A more graphical way to capture past activity by system-state snapshoting continuously, is to use
atop is similar to programs like
htop, with the notable difference that it runs a periodic cron job to generate & preserve full process & system activity data. This allows you to go back in time later to investigate issues.
atop also provides a utility
atopsar which is similar to traditional Unix
sar. Both utilities share the same system-data snapshot database.
atop screenshot showing a system during disk-utilization stress. Note the 100% disk utilization on
sda and LVM which are highlighted in red color. Credit:
atop author, Gerlof Langeveld, atoptool.nl.
sudo apt-get install atop
Now you will need to wait for ~10 minutes for the 1st accounting snapshot to be performed. snapshots use per metric point per entity. Entities tracked are:
- Processes (by executable)
- Per-core CPU utilization, frequency & scaling, system vs user
- Memory & swap usage
- Disk-partitions: reads, writes, %utilization
- Network-interfaces: packets in/out (both UDP and TCP), errors, packet-retransmits and more
All metrics are cumulative totals for the watched snapshot.
To view past activity
This effectively gives you a little "time-machine". You can move backward and forward in time to see what happened in every time-slice in the past watched day(s).
atop -r [/var/log/atop/...]
Without the snapshot-file argument,
atop will show a view of the past day (pick any existing snapshot-file to show a different day), starting from midnight. The most important keys to remember are:
t move forward in time (to the next time-slice)
T move backward in time (to the previous time-slice)
The snapshot deltas are correctly implemented by using process accounting at each process
exit() so even if you have many short running processes, their sum of parts will be added together and attributed correctly to both the appropriate executable and the appropriate time-slice.
Not only processes are captured. The full system state is captured. The top half of the screen shows all the important system metrics, CPU, memory, disk and network utilization for every entity. The data includes CPU frequencies and scaling factors, network errors and much more. For more helpfulness, abnormal values are highlighted in color, for example any time-slice disk utilization of 100% will show in bright red, near maxed-out values will show in different color, so any stressed-out entity is hard to miss.
If you're more of a batch style person, you may prefer to use
atop. For example, to dump a full time-range batch-style, you could use:
atopsar -D -b 14:05 -e 14:45
Which will show the top 3 processes by (
-D) %percent disk utilization between (
-b: begin) 14:05 and (
-e: end) 14:45 today.
man atopsar for more detailed usage.
If you want to focus on certain sub-areas, you may use these
atopsar options (
atop uses the same letters interactively):
-C sort processes in order of cpu-consumption (default)
-M sort processes in order of memory-consumption
-D sort processes in order of disk-activity
-N sort processes in order of network-activity
-A sort processes in order of most active resource (auto mode)
There's much more you can do with
man atop and
man atopsar for full details. The above was the gist of it.