How can I see what process spiked CPU usage and froze the system a few seconds ago?

Question

My Ubuntu 18.04 randomly freezes for a few seconds. I can move the mouse cursor (sometimes), but otherwise the OS is unresponsive and I can't switch to any other application.

When it resumes, I can go to the System Monitor and see a spike in CPU usage a few seconds ago (only to 70% though), but that doesn't tell me WHAT used the CPU then.

How can you find out what processed chewed up the CPU very recently? (I don't think this is due to disk I/O).

UPDATE: In the meantime I've determined that the culprit is WebStorm, a Java-based IDE, by isolating suspected apps. While using VS Code, there were no freezes.

Below is some additional diagnostic info asked for in the comments.

$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        8.6G        2.3G        2.0G        4.5G        4.4G
Swap:           15G        487M         15G
$ sysctl vm.swappiness
vm.swappiness = 10

When WebStorm freezes, the load in System Monitor does show a spike, but nowhere near 100%.

Answer 1

To get past history of CPU utilization is not possible unless one is monitoring the system with a tool that captures CPU activity. One method is to run cpustat and capture the output and see where the CPU utilization is occurring, for example:

sudo apt-get install cpustat
cpustat -xS | tee cpu.log

and when you get a slowdown one can view cpu.log and see what was busy running.

Answer 2

Try atop

A more graphical way to capture past activity by system-state snapshoting continuously, is to use atop. atop is similar to programs like top or htop, with the notable difference that it runs a periodic cron job to generate & preserve full process & system activity data. This allows you to go back in time later to investigate issues. atop also provides a utility atopsar which is similar to traditional Unix sar. Both utilities share the same system-data snapshot database.

Here's an atop screenshot showing a system during disk-utilization stress. Note the 100% disk utilization on sda and LVM which are highlighted in red color. Credit: atop author, Gerlof Langeveld, atoptool.nl.

To install:

    sudo apt-get install atop

Now you will need to wait for ~10 minutes for the 1st accounting snapshot to be performed. snapshots use per metric point per entity. Entities tracked are:

• Processes (by executable)
• Per-core CPU utilization, frequency & scaling, system vs user
• Memory & swap usage
• Disk-partitions: reads, writes, %utilization
• Network-interfaces: packets in/out (both UDP and TCP), errors, packet-retransmits and more

All metrics are cumulative totals for the watched snapshot.

To view past activity

This effectively gives you a little "time-machine". You can move backward and forward in time to see what happened in every time-slice in the past watched day(s).

   atop -r [/var/log/atop/...]

Without the snapshot-file argument, atop will show a view of the past day (pick any existing snapshot-file to show a different day), starting from midnight. The most important keys to remember are:

t         move forward in time (to the next time-slice)

T         move backward in time (to the previous time-slice)

h         help

q         quit

The snapshot deltas are correctly implemented by using process accounting at each process exit() so even if you have many short running processes, their sum of parts will be added together and attributed correctly to both the appropriate executable and the appropriate time-slice.

Not only processes are captured. The full system state is captured. The top half of the screen shows all the important system metrics, CPU, memory, disk and network utilization for every entity. The data includes CPU frequencies and scaling factors, network errors and much more. For more helpfulness, abnormal values are highlighted in color, for example any time-slice disk utilization of 100% will show in bright red, near maxed-out values will show in different color, so any stressed-out entity is hard to miss.

If you're more of a batch style person, you may prefer to use atopsar over atop. For example, to dump a full time-range batch-style, you could use:

   atopsar -D -b 14:05 -e 14:45

Which will show the top 3 processes by (-D) %percent disk utilization between (-b: begin) 14:05 and (-e: end) 14:45 today. man atopsar for more detailed usage.

If you want to focus on certain sub-areas, you may use these atopsar options (atop uses the same letters interactively):

    -C  sort processes in order of cpu-consumption (default)
    -M  sort processes in order of memory-consumption
    -D  sort processes in order of disk-activity
    -N  sort processes in order of network-activity
    -A  sort processes in order of most active resource (auto mode)

There's much more you can do with atop and atopsar. Use man atop and man atopsar for full details. The above was the gist of it.

Answer 3

In my experience, it's uncommon for Linux to become unresponsive due to just CPU usage. Excessive CPU usage tends to just make everything a bit sluggish.

I/O issues (writing many/large files, swapping, faulty disk, etc) on the other hand can easily cause unresponsiveness where everything seems to grind to a halt, then maybe continue for a bit, only to halt again. The fact that sometimes even your mouse stops moving leads me to believe that your problems fall in this category.

A reasonably simple and effective way to determine if I/O is the culprit is using the standard tool vmstat. You can run vmstat -w 5 somewhere (in screen, or just in a terminal); this will print a line of statistics every 5 seconds. Then you can go back and inspect the numbers (and/or post them on AskUbuntu ;) after you experienced a freeze.

The output looks like this:

procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
 r  b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st
 3  0       865332       328876     18014392      8262980    0    0   108    89    7    7  21   6  73   0   0
 0  0       865332       330016     18006044      8267348    0    1     0   332 2169 8117  25   6  69   0   0

Interesting columns (for this purpose) include:

• CPU: wa indicates what percentage of CPU is blocked waiting for I/O to finish. High numbers here suggest I/O is the problem, not CPU usage. Also useful for determining bottlenecks.
• Swap: si and so display the number of KiB/s swapped in and out respectively. Should be pretty much 0 if you have plenty of RAM. High numbers suggest that your memory requirements exceed your memory size.
• I/O: bi and bo display the number of KiB/s read/written from disks (the swap activity is included in this). Unexpected high write numbers might warrant a search for what process is doing that writing (using e.g. iotop). Freezes with low/medium numbers suggest your disk is slow.