36

I use my computer for scientific programming. It has a healthy 8GB of RAM and 12GB of swap space. Often, as my problems have gotten larger, I exceed all of the available RAM. Rather than crashing (which would be preferred), it seems Ubuntu starts loading everything into swap, including Unity and any open terminals. If I don't catch a run-away program in time, there is nothing I can do but wait - it takes 4-5 minutes to switch to a command prompt eg. Ctrl-Alt-F2 where I can kill the offending process.

Since my own stupidity is out of scope of this forum, how can I prevent Ubuntu from crashing via thrashing when I use up all of the available memory from a single offending program?

At-home experiment*!

Open a terminal, launch python and if you have numpy installed try this:

>>> import numpy
>>> [numpy.zeros((10**4, 10**4)) for _ in xrange(50)]

* Warning: may have adverse effects, monitor the process via iotop or top to kill it in time. If not, I'll see you after your reboot.

2 Answers 2

23

The shell built-in ulimit allows you to restrict resources. For your case, to limit memory use in the shell (and its children), use ulimit -v.

Demonstration setting a memory limit of 100 MB (100000 KB):

$ ulimit -v
unlimited
$ python -c '[ "x" * 100000000 ]'
$ ulimit -v 100000
$ python -c '[ "x" * 100000000 ]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
MemoryError

It's observed using ps uww -C script-name-here that python requires at least 29MB of memory (VSZ column). The RSS limit grows as your python script needs more memory so adapt that column.

5
  • 1
    Too bad we don't know what the actual app looks like. The ulimit as demonstrated impacts all processes who are children of that shell. We don't even know if it runs as a daemon, or if it has lots of children which would further complicate things. See coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/…
    – ppetraki
    Apr 11, 2012 at 18:09
  • 1
    Is there no user-friendly front-end for ulimit and cgroups? In Windows, when the OS gets close to running out of RAM, a warning pop-up has always enabled me to avoid system freezes. In Ubuntu, do I have to keep an eye on the memory usage all the time? Dec 30, 2012 at 6:19
  • 1
    @DanDascalescu If you have a swap file configured, then you will suffer longer delays but still be able to work. Once you notice that swapping starts, you can check the system monitor for RAM usage. Should your system really run out of memory, the out-of-memory (OOM) killer is invoked which kills a process with the highest "badness" score (usually the process that consumes most memory).
    – Lekensteyn
    Jan 14, 2015 at 8:05
  • Is there a way to set ulimit for specific resource/process, for example, java?
    – Kulfy
    Feb 29, 2020 at 19:08
  • @Kulfy Yes and no. Yes, if you wrap the Java execution in a script (e.g. sh -c "ulimit ...; java ..."), then you would control ulimit for Java and its child processes. No, you can usually not run a command and apply the ulimit for any Java process. To limit memory usage in Java, have a look at the java -Xmx option. E.g. java -Xmx1024M restricts the size of the application-accessible memory pool to 1G. Do note that the JVM consumes additional memory.
    – Lekensteyn
    Mar 1, 2020 at 13:22
12

Cgroups should let you limit your memory usage on a per process basis.

https://en.wikipedia.org/wiki/Cgroups

http://www.mjmwired.net/kernel/Documentation/cgroups/memory.txt

Scientific computing is notoriously memory intensive, by sandboxing your app in a cgroup, the rest of the processes should not become victims as memory pressure will be alleviated.

Alternatively, a VM could be used as a sort of hard limit as the app can only use the memory delegated to the virtual machine, at the expense of performance of course. However a VM is much easier to configure for the uninitiated when compared to setting up and maintaining a cgroup.

Decisions decisions :) Good luck!

4
  • 1
    Is there no user-friendly front-end for ulimit and cgroups? In Windows, when the OS gets close to running out of RAM, a warning pop-up has always enabled me to avoid system freezes. In Ubuntu, do I have to keep an eye on the memory usage all the time? Dec 30, 2012 at 6:19
  • Nope. Also your OS shouldn't be freezing either, that's a separate problem. The windows pop up is just a bandaid to begin with. If you're using tons of memory maybe you should investigate why? For example, modern browsers with their sanboxing can easily consume 70-150M PER TAB. Turning that off or tuning it can free substantial memory used on a regular basis by an end user. Schedulers are supposed to swap out seldom used programs to save space but if they wake up more than they should then you need to fix them, or if your primary task is memory intensive (web) then more ram is needed.
    – ppetraki
    Dec 31, 2012 at 15:55
  • 8
    I know what consumes the memory - Chrome. But the OS should protect its stability against applications that run amok nevertheless. Jan 3, 2013 at 11:30
  • 1
    I understand where you're coming from as a user. What you're asking for however means to turn a general computing platform into an appliance. As the least complex (error prone) method an operating system engineer must implement to provide those safeguards is to make dramatically opinionated decisions on how the entire computer is run. That usually comes at the expense of end user flexibility. You see examples of this in tablet computing. In desktops however the flexibility is preserved.
    – ppetraki
    Jun 4, 2013 at 14:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .