How can I find out if a process is bound to CPU, Memory or Disk?
-
If you're using a gnome GUI, you can add the System Monitor applet to your panel. In the settings you can set cycles lost to IO-Wait to show up as white, and set the Memory, CPU and Disk Read / Write graphs to show. I usually change the default graph colours to make it easier to distinguish user/OS memory and disk read/write. It's not a real solution, but it's handy to have alongside all the tools mentioned in the answers.– JeremyNov 17, 2010 at 1:47
-
1@luis-alvarado I would disagree that this question isn't useful, though I'd argue this might fit in better on StackOverflow or the like. I came here looking for a solution.– skeggseMay 12, 2014 at 18:42
-
The problem with the question as-asked is that it does not take into account the environment of execution. Which method is most effective at discovering the host component bottleneck of a job/process is intimately tied to the runtime, execution environment, host OS, hardware, etc. of the job and what tooling is available for each component to inspect it. Some runtimes (the Erlang runtime or IBM's JVM, for example) have in-depth tools for this than span the entire environment, others totally lack tooling and leave you making educated guesses based on whatever htop/iotop/etc. can show you.– zxq9Jan 22, 2018 at 4:10
-
Related: stackoverflow.com/questions/3156334/…– Ciro Santilli OurBigBook.comSep 10, 2019 at 15:04
6 Answers
That requires some expert skills. It depends. Example:
If there's enough of memory and disks don't seem too busy, it may be CPU-bound. Look at CPU usage and if its bordering at 100% it's CPU bound. If it's not there's an artificial bottleneck in the implementation. E.g. on a dual-core CPU a single threaded process will not go above 50% CPU usage.
If CPU and memory are available, but disks are very busy, or IO latency seems high, its likely that its IO bound. See if adding more disks (RAID?) helps.
None of the above? Check memory available.
Enough memory? There may be an artificial bottleneck in the process itself i.e. maybe someone forgot to remove a sleep(1)? Naah its not that easy usually. ;)
There's a reason why we have a whole lab for performance engineers in most companies dealing with performance sensitive products!
Use tools like sar, vmstat, iostat, oprofile, lockstat, dtrace, product specific perf monitoring tools etc, to debug perf problems.
-
2
-
usually no, but if dtrace is available on your system (Solaris/FreeBSD/Mac OS X?) you may get far just by using that. Aug 7, 2010 at 6:21
-
I just remembered there's Systemtap on linux (similar to dtrace), except I am not sure about the maturity of the tool. Aug 7, 2010 at 6:34
-
1
check out iotop
, can be useful
-
-
1Yes iotop is usually not installed by default, inc on Ubuntu
sudo apt install iotop
for Ubuntu/Debian based systems (inc Mint), orsudo yum install iotop
for RedHat based systems. Oct 27, 2020 at 2:55
A tool that can be useful for real-time checking a number of process statistics (memory, cpu-usage, I/O, etc.) is htop. It doesn't replace the more specialised tools named by Sudhanshu, but might be a good start.
As well as the other tools mentioned, run ps l PID
, inserting the relevant process id, or look at the STATE and WCHAN columns in top or htop.
If it's in D (for disk) state, then it's doing file IO. This could be because it's either reading a lot of files, or because it's using a lot of memory and swapping. The WCHAN column will tell you what kernel function it's inside; googling for them or asking here may give you some indication what they mean.
If it's in R (run) state, it's using the CPU in user space, in other words it's CPU bound at that moment.
If it's in S (sleep) state, it's inside an interruptible system call, which may mean it's either actually sleeping, or it's doing something like waiting for network traffic or a lock. Again, looking at the specific wchan will tell you more.
Run top
and look at the cpu usage line. A high user % indicates that it is cpu bound. A high wait % indicates that it is IO bound.
I usually go:
sudo apt install iotop htop
# Ubuntu/debian based systems
or
sudo yum install iotop htop
# Redhat/RPM based systems
Then have a quick look at htop
- is the system swapping (fully filled memory bar)? If so, you want to fix that first. Swapping is super expensive in terms of performance.
If the system is not swapping, check out the cpu bars. Are they near 100% most of the time? Well, then your CPU is busy :)
Next, quit htop
and start sudo iotop
and check the top right numbers (total and actual disk write). How high are they?
If they about match the maximum reasonable throughput of your disk which is being read from / written too (and usually write operations are the bottleneck as disk write speeds are almost always lower then read speeds), then your IO is busy :)
Note I am referring to the actual reasonable throughput on your disk (what you were to approximately get if you copied a set of files manually when the server was otherwise idle), NOT the "label" performance, which is often impossible to achieve to start with.
Thus, doing a few simple check, you worked out if your system is memory, cpu or disk bound. If you want more detailed numbers or analysis, checkout some of the other tools, but this gives you a basic idea, which is often sufficient to help you scale etc.