3

Ubuntu 18.04 lsb, I'm trying to figure out the best way to diagnose whats happening when the Amazon Ec2 instance (free tier) hangs.

There are experimental services running and there may be / is a memory leak.

For quality of life I'm using a utility called lnav to help me browse system logs. Also I've installed a utility called monitorix to visiualise whats happening.

Can I / how do I to identify the specific process causing the problem from system logs ? Which log might help me? (/var/log/syslog does not help)

These charts show high CPU load associated with system swap space being consumed until catastrophic failure occurs.

High CPU load

Swap consumption

But this does not tell me the specific process. How can I do this through the terminal?

Is there some other process monitoring I could to configure ?

Any help appriciated...

Edit: Thanks to the hint from @Rinzwind sar is now installed and cron is running it every 2 mins... but it doesn't give process level info. So with help from this other answer:

pidstat 5 > pidhist.log pipes out to a text file, and running it in persistant session will aid diagnosis when the event happens again.

@heynnema suggested iotop

Running iotop -P -a which is top for file I/O as a totaliser. It indicated that the experimental process (a mono service) was the one consuming the most swap with SWAPIN enter image description here
enter image description here ****

It's more visible in top enter image description here enter image description here

We see can see the same pattern of consumption, and then after restarting the process return to normal ~20% from monitorix.

enter image description here

The system is stable for weeks on end between these random events. The evidence from iotop proves the underlying issue is within the experimental process!

Yet, this is still a run time diagnosis. Is there a way to determine from existing logs which process was at fault after the fact? to do that without without preemptive monitoring and logging.

That proof of what went wrong is the critical issue to be resolved. how can we do that without waiting for it to reoccur if no logging is enabled? kernel logs???

Thanks for any help.

5
  • 2
    sar is the command for a system activity report. Pipe it to a text file and the end of the text file might explain what happens
    – Rinzwind
    Mar 18, 2019 at 16:40
  • @Rinzwind thanks, sar looks to be ideal. But I'm stumped on how to get the report from pidstat included in its history. Is it something easy im missing?
    – Kickaha
    Mar 18, 2019 at 17:31
  • 1
    Edit your question to include free -h and sysctl vm.swappiness and cat /etc/fstab. Install iotop. Report back to @heynnema
    – heynnema
    Mar 19, 2019 at 1:31
  • @heynnema, thanks, can you create an anwser now ?
    – Kickaha
    Mar 19, 2019 at 14:32
  • @heynnema Sorry dude, you missed the mark. Not so obvious it seems.
    – Kickaha
    Mar 19, 2019 at 17:44

2 Answers 2

1

From the comments...

We looked at the output of free -h and sysctl vm.swappiness and cat /etc/fstab, and installed iotop to determine why swap if used so much.

There are a few reasons why the system is thrashing.

  1. you don't have enough RAM

  2. you don't have enough swap

  3. vm.swappiness has been modified incorrectly

The fix...

  1. add more RAM

  2. increase /swapfile space

  3. set vm.swappiness to 60-90 (60 is default)

11
  • Sorry, your reasons are wrong. You don't solve a memory leak by adding resources. Also the swappiness setting of 10 is appropriate for a server, the default of 60 is for desktops. The question was "how to diagnose a process leaking memory from logs after catastrophic failure". It wasn't answered exactly, instead monitoring methods to identify reoccurance were suggested. This is not an acceptable answer. If you can change it by deleting your reasons and fix I'll accept it. Thanks again.
    – Kickaha
    Mar 19, 2019 at 16:15
  • @Kickaha When you only have ~1G RAM, and vm.swappiness is set to be more RAM intensive (vs swap intensive) the poor system can't even get out of its own way. The "catastrophic failure" is the system thrashing. Adding resources is exactly what needs to happen. You could try doubling your swap, and setting vm.swappiness to 90, and see if things improve.
    – heynnema
    Mar 19, 2019 at 16:33
  • you're looking at this from a different perspective. The purpose of this experiment is to run the process within the constraints. The memory leak behaviour is not a feature to be accommodated by adding resources. Adding resources is the opposite of what needs to happen. The swap is configured with the low value of 10 because using swap is a last resort, the default is to have no swap at all. The server response time is optimal when RAM is where processes reside. Yes it's nicer to have more resources, it's just not appropriate here.
    – Kickaha
    Mar 19, 2019 at 16:50
  • @Kickaha yes, I am looking at this from a different perspective. You're looking at this as if you have plenty of RAM resources, and don't want to swap at all. Back-ass-words. Try my way... as I mention in my last comment... and it won't cost you anything, and although you'll swap more, that's better than thrashing, which is what's happening now. Listen to someone else who may have a little more experience in this area :-)
    – heynnema
    Mar 19, 2019 at 16:58
  • There are plenty of RAM resources, normal operation runs around 20%. Swap usage is a symptom of a memory leak in the logic of the process. My question was originally how to diagnose which process caused the problem after the fact from logs. It's not about how to configure swap. Can you help answer that?
    – Kickaha
    Mar 19, 2019 at 17:25
0

We don't add RAM to solve this issue.

Identifying a process causing a memory leak has nothing to do with system configuration.

iotop -P -a helped identify the process consuming swap during reoccurance of the event.

Steps for digital forensic log investigation would be a better solution.

1
  • After patching a firewall misconfiguration, a denial of service vulnerability is the likely root cause here.
    – Kickaha
    Apr 1, 2019 at 14:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .