1

I am running a Supermicro server with Ubuntu 14.04.4 LTS. We have an app that is maxing out the sy cpu time, while barely touching user space. I run the same app on older hardware, with Ubuntu 12.04.4 LTS and the cpu is much more balanced between user and system. I've used strace to get some performance info about the process, and I see epoll_wait taking up 98% of the system cpu time. I can't find to much information about this call, and what I do find I'm not understanding. Can anyone shed some light on this please?

2

You can see where CPU is being consumed in even more detail using "perf":

Install perf:

sudo apt-get install linux-tools-$(uname -r)

Next, start the program that consumes all the CPU resources

Next, run perf and capture all scheduling events for say 60 seconds:

sudo perf record -a sleep 60

Once that has completed, you can get a log of all the perf events using:

sudo perf script > perf.log

and you can look at that, or better, one can interactively look at the CPU hotspots using:

sudo perf report
  • Thanks! I'll run perf tomorrow. That app consumes market data, and market data is only available certain times of the day. I'll post an update tomorrow. – scott Feb 24 '16 at 23:09
  • What's the difference between the "good server" and "bad server" in terms of configuration, kernel version, etc.. – Colin Ian King Feb 25 '16 at 14:38
  • @ColinIanKing +1. Thank you. if I could upvote this 100 times I would. This has proven quite helpful as a first step in diagnosting another high cpuissue. – Joshua McKinnon Aug 25 '17 at 21:18
0

If you can capture the output from tracing the program using the -e epoll_wait option just to capture the epoll syscall then add that to this question we can then figure out what is happening.

The epoll_wait system call is basically waiting for some epoll events and a high CPU consumption on epoll_wait system calls could mean either that a provided timeout is too small and this is causing many tens of thousands of epoll_wait calls if in a loop, or that there are indeed a lot of events occurring which the epoll_wait is waiting on and the code is handling. Alternatively, it could be a bug in the program and it is spinning on some kind of error. Typical error conditions would be EINVAL, where an invalid parameter is being passed to the system call, or EBADF where an invalid file descriptor is being used (possibly from a failed open)

So, strace the program again:

strace -f -e epoll_wait program-name >& strace.log

and see what kind of epoll_wait failures are occurring (return of -1). If no failures are occurring, then check if a timeout occurs (0) or a file descriptor becomes ready for I/O (return > 0).

  • Running strace -e epoll_wait -p 1921 returns epoll_wait(19, {{EPOLLIN, {u32=137439904, u64=137439904}}}, 1024, 60000) = 1 – scott Feb 24 '16 at 20:42
  • So the call is returning 1, so there is data available on epoll fd 19. The u32=137439904 is a bit field showing the bitmap of events that are ready, so quite a few. The 1024 refers to 1024 events to wait on, and the 60000 is the 60 x 1000ms (or 1 minute) as a timeout. So it looks like if this is a typical epoll_wait call then the code is getting a lot of input events to handle; hence why the CPU is busy – Colin Ian King Feb 24 '16 at 22:49
  • If you want to see where in the kernel the CPU is being consumed you can use "perf" (see next answer below) – Colin Ian King Feb 24 '16 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.