I am running a Supermicro server with Ubuntu 14.04.4 LTS. We have an app that is maxing out the sy cpu time, while barely touching user space. I run the same app on older hardware, with Ubuntu 12.04.4 LTS and the cpu is much more balanced between user and system. I've used strace to get some performance info about the process, and I see epoll_wait taking up 98% of the system cpu time. I can't find to much information about this call, and what I do find I'm not understanding. Can anyone shed some light on this please?
You can see where CPU is being consumed in even more detail using "perf":
sudo apt-get install linux-tools-$(uname -r)
Next, start the program that consumes all the CPU resources
Next, run perf and capture all scheduling events for say 60 seconds:
sudo perf record -a sleep 60
Once that has completed, you can get a log of all the perf events using:
sudo perf script > perf.log
and you can look at that, or better, one can interactively look at the CPU hotspots using:
sudo perf report
If you can capture the output from tracing the program using the -e epoll_wait option just to capture the epoll syscall then add that to this question we can then figure out what is happening.
The epoll_wait system call is basically waiting for some epoll events and a high CPU consumption on epoll_wait system calls could mean either that a provided timeout is too small and this is causing many tens of thousands of epoll_wait calls if in a loop, or that there are indeed a lot of events occurring which the epoll_wait is waiting on and the code is handling. Alternatively, it could be a bug in the program and it is spinning on some kind of error. Typical error conditions would be EINVAL, where an invalid parameter is being passed to the system call, or EBADF where an invalid file descriptor is being used (possibly from a failed open)
So, strace the program again:
strace -f -e epoll_wait program-name >& strace.log
and see what kind of epoll_wait failures are occurring (return of -1). If no failures are occurring, then check if a timeout occurs (0) or a file descriptor becomes ready for I/O (return > 0).