Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have an Ubuntu 12.04 server that just crashed because of a very obvious cause: 30+ of apt-check processes consuming all memory, the OOM killer kicking in, killing vital services. I'm not sure where the apt-check processes come from, but I guess my Nagios/Icinga plugins check_apt might use it, as well as the byobu status line may want to display its output. I guess something locked up and all of the processes were just waiting, yet holding memory.

How can I prevent to have so many instances of apt-check on the system? It doesn't make sense to me and it should just quit as soon as it can't get a read lock on the dpkg database.

It seems that I'm not the only one running into trouble here. All suggestions for apt-check are pretty negative:

enter image description here

(clean browser, not logged in, no personalised search)

share|improve this question

2 Answers 2

up vote 5 down vote accepted

Some dive into apt-check gave me these clues for being it a very blunt script that needs fixing. With all due respect to the authors of it, it is failing on my servers. Here are my thoughts:

  • apt-check == /usr/lib/update-notifier/apt_check.py
  • forces nicelevel 19 for itself
  • no timeouts set on actions

The combination of the last two allows it to pile up endlessly in a spiral downwards. If the system is used for some other purposes with higher priority, the amount of processes will just increase and there's no end to it, as apt-check will never get any priority over it. Trouble will only get worse once the OOM killer decides to kill your vital system processes.

If either of these two aspects in behaviour was different, it would not allow the system to end up in such a broken state is my assumption.

While strings is right about the parent processes being responsible in this too, I believe below points are flaws in apt-check and has to be reported as a bug to get addressed properly:

  • it should hint the OOM killer to have itself killed first
  • it should not set the nicelevel hardcoded
  • it should exit if it takes an unreasonable amount of time to get pieces of information

Actually, it seems that the Linux OOM killer is doing some heuristic on this. Niced processes will get an increased score, and long-running processes are decreased. (source - thanks to Ulrich Dangel for pointing it out)

Possible solution I may propose:

  • cache results after processing
  • output cache if less than N amount of seconds without loading all Python-APT libraries for every simple (even --help) invocation.
  • make the nicelevel configurable - Allow me to change/disable this, please! I believe that setting it to 0 will actually help
  • have it increase the OOM killer score
share|improve this answer
    
I think you've missed the biggest thing it should do: use a lockfile and not allow more than one instance to run at once. –  derobert Jan 10 '13 at 20:55
    
@derobert It's a script that any user can run, without specific privileges on Ubuntu, not a daemon. Or can I use /var/run / /tmp for this world-readable/writable lock file safely? Big hole there: add lock file and admin won't be notified of system updates! –  gertvdijk Jan 10 '13 at 21:27
    
Whatever is launching it automatically (leading to 30+ copies running) needs to do the locking. Or it could do it itself, per-user. Either way, that's a bug that needs to be solved. –  derobert Jan 10 '13 at 23:33
    
Nagios/Icinga seems to avoid the pitfall. At least, it had some 10s timeout and warned if it was exceeded. (Though I can't find how to configure the timeout - I'd prefer longer). Byobu on Debian is what caused the problem for me; on Ubuntu it's supposed to be fixed. –  sourcejedi Feb 14 '13 at 10:50

You need to find out what process is spawning apt-check. you can use something like ps to get the process tree.

ps -A --forest

If apt-check has no parents, then it might be a issue with apt-check its self and not one particular program. if that is the case I would try to debug apt-check.

share|improve this answer
    
Thanks. Gave me some ideas to look further into. However, this lead me to believe it's a problem of apt-check really - see my own answer. –  gertvdijk Jan 10 '13 at 12:39
    
If it's consuming memory and CPU time, it's not a zombie. –  Gilles Jan 10 '13 at 19:24
    
@Gilles good point. –  strings Jan 10 '13 at 19:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.