Running script was killed

Question

I was running a script on 30k images and suddenly it was killed. What could have caused this?

mona@pascal:~/computer_vision/deep_learning/darknet$ ./darknet coco test cfg/yolo-coco.cfg yolo-coco.weights images
0: Convolutional Layer: 448 x 448 x 3 image, 64 filters -> 224 x 224 x 64 image
1: Maxpool Layer: 224 x 224 x 64 image, 2 size, 2 stride
2: Convolutional Layer: 112 x 112 x 64 image, 192 filters -> 112 x 112 x 192 image
3: Maxpool Layer: 112 x 112 x 192 image, 2 size, 2 stride
4: Convolutional Layer: 56 x 56 x 192 image, 128 filters -> 56 x 56 x 128 image
5: Convolutional Layer: 56 x 56 x 128 image, 256 filters -> 56 x 56 x 256 image
6: Convolutional Layer: 56 x 56 x 256 image, 256 filters -> 56 x 56 x 256 image
7: Convolutional Layer: 56 x 56 x 256 image, 512 filters -> 56 x 56 x 512 image
8: Maxpool Layer: 56 x 56 x 512 image, 2 size, 2 stride
9: Convolutional Layer: 28 x 28 x 512 image, 256 filters -> 28 x 28 x 256 image
10: Convolutional Layer: 28 x 28 x 256 image, 512 filters -> 28 x 28 x 512 image
11: Convolutional Layer: 28 x 28 x 512 image, 256 filters -> 28 x 28 x 256 image
12: Convolutional Layer: 28 x 28 x 256 image, 512 filters -> 28 x 28 x 512 image
13: Convolutional Layer: 28 x 28 x 512 image, 256 filters -> 28 x 28 x 256 image
14: Convolutional Layer: 28 x 28 x 256 image, 512 filters -> 28 x 28 x 512 image
15: Convolutional Layer: 28 x 28 x 512 image, 256 filters -> 28 x 28 x 256 image
16: Convolutional Layer: 28 x 28 x 256 image, 512 filters -> 28 x 28 x 512 image
17: Convolutional Layer: 28 x 28 x 512 image, 512 filters -> 28 x 28 x 512 image
18: Convolutional Layer: 28 x 28 x 512 image, 1024 filters -> 28 x 28 x 1024 image
19: Maxpool Layer: 28 x 28 x 1024 image, 2 size, 2 stride
20: Convolutional Layer: 14 x 14 x 1024 image, 512 filters -> 14 x 14 x 512 image
21: Convolutional Layer: 14 x 14 x 512 image, 1024 filters -> 14 x 14 x 1024 image
22: Convolutional Layer: 14 x 14 x 1024 image, 512 filters -> 14 x 14 x 512 image
23: Convolutional Layer: 14 x 14 x 512 image, 1024 filters -> 14 x 14 x 1024 image
24: Convolutional Layer: 14 x 14 x 1024 image, 1024 filters -> 14 x 14 x 1024 image
25: Convolutional Layer: 14 x 14 x 1024 image, 1024 filters -> 7 x 7 x 1024 image
26: Convolutional Layer: 7 x 7 x 1024 image, 1024 filters -> 7 x 7 x 1024 image
27: Convolutional Layer: 7 x 7 x 1024 image, 1024 filters -> 7 x 7 x 1024 image
28: Local Layer: 7 x 7 x 1024 image, 256 filters -> 7 x 7 x 256 image
29: Connected Layer: 12544 inputs, 4655 outputs
30: Detection Layer
forced: Using default '0'
Loading weights from yolo-coco.weights...Done!

Killed

mona@pascal:~/computer_vision/deep_learning/darknet/src$ dmesg | tail -5
[2265064.961124] [28256]  1007 28256    27449       11      55      271             0 sshd
[2265064.961126] [28257]  1007 28257     6906       11      19      888             0 bash
[2265064.961128] [32519]  1007 32519 57295584 16122050   62725 15112867             0 darknet
[2265064.961130] Out of memory: Kill process 32519 (darknet) score 941 or sacrifice child
[2265064.961385] Killed process 32519 (darknet) total-vm:229182336kB, anon-rss:64415788kB, file-rss:72412kB

and

[2265064.961128] [32519]  1007 32519 57295584 16122050   62725 15112867             0 darknet
[2265064.961130] Out of memory: Kill process 32519 (darknet) score 941 or sacrifice child
[2265064.961385] Killed process 32519 (darknet) total-vm:229182336kB, anon-rss:64415788kB, file-rss:72412kB

After the process is killed I have:

$ top | grep -i mem
KiB Mem:  65942576 total,  8932112 used, 57010464 free,    50440 buffers
KiB Swap: 67071996 total,  6666296 used, 60405700 free.  7794708 cached Mem
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                              
KiB Mem:  65942576 total,  8932484 used, 57010092 free,    50440 buffers
KiB Swap: 67071996 total,  6666296 used, 60405700 free.  7794736 cached Mem
KiB Mem:  65942576 total,  8932608 used, 57009968 free,    50448 buffers
KiB Mem:  65942576 total,  8932480 used, 57010096 free,    50448 buffers

My vmstat is:

$ vmstat -s -SM
        64397 M total memory
         8722 M used memory
          305 M active memory
         7566 M inactive memory
        55674 M free memory
           49 M buffer memory
         7612 M swap cache
        65499 M total swap
         6510 M used swap
        58989 M free swap
    930702519 non-nice user cpu ticks
        33069 nice user cpu ticks
    121205290 system cpu ticks
   4327558564 idle cpu ticks
      4518820 IO-wait cpu ticks
          148 IRQ cpu ticks
       260645 softirq cpu ticks
            0 stolen cpu ticks
    315976129 pages paged in
    829418865 pages paged out
     38599842 pages swapped in
     46593418 pages swapped out
   2984775555 interrupts
   3388511507 CPU context switches
   1475266463 boot time
       162071 forks

The other time I ran this script with only 3000 images instead of 30k I got this error:

28: Local Layer: 7 x 7 x 1024 image, 256 filters -> 7 x 7 x 256 image
29: Connected Layer: 12544 inputs, 4655 outputs
30: Detection Layer
forced: Using default '0'
Loading weights from yolo-coco.weights...Done!
OpenCV Error: Insufficient memory (Failed to allocate 23970816 bytes) in OutOfMemoryError, file /build/buildd/opencv-2.4.8+dfsg1/modules/core/src/alloc.cpp, line 52
terminate called after throwing an instance of 'cv::Exception'
  what():  /build/buildd/opencv-2.4.8+dfsg1/modules/core/src/alloc.cpp:52: error: (-4) Failed to allocate 23970816 bytes in function OutOfMemoryError

Aborted (core dumped)

It used 61G of my 64G RES memory as shown in htop.

Answer 1

It's the OOM (Out Of Memory) killer of Linux kernel killing the process.

Linux kernel allows processes to overcommit memory i.e. a process can map (e.g. mmap(2)) more memory than actually available. This is defined by the value of the file /proc/sys/vm/overcommit_memory. Possible values:

• 0 : Heuristics based overcommit (default)
• 1 : Always overcommit
• 2 : Never overcommit

Overcommitting is enabled by default because it is considered that a process would not use all the memory it maps, well, at least not at the same time.

The problem begins, when a process is asking to allocate the memory (e.g. malloc(2)) but there is not enough memory available. The kernel will then trigger the OOM killer, and which will then kill process(es) based on their OOM score(s), defined in the file /proc/PID/oom_score with values ranging from 0 to 1000, the higher the value the bigger chance that OOM killer will kill the process in case of an OOM situation.

The OOM score is calculated by a complex algorithm considering factors such as who owns the process, how long it's been running, how many children it has, how much memory it is using, and so on. Note that, the root owned process always get 30 deducted (when >=30) from their real OOM score.

You can influence the OOM score by providing an adjustment score yourself in /proc/PID/oom_score_adj file, allowed values range from -1000 to +1000, with a negative to keep the process, and positive to influence killing. So you can check the oom_score of the process in question, and make necessary adjustments so that OOM killer will not have it in it's priority list when start killing. Although note that, this is not recommended when the process you are trying to keep is actually hogging the memory (like in your case).

The alternate solutions would include installing more memory obviously, better check if anything can be done within the program like changing it's algorithm for example, or impose a resource limit via e.g. cgroups, which could result in the same situation i'm afraid.

Answer 2

While it's a relatively sane decision to over-commit memory kill processes once the system runs out of memory, disabling over-commitment will just make the memory-hoggig program terminate abnormally or crash because it can't allocate any more memory to perform its task.

The only ways around that are

• to not use that program with that particular task on that particular data set,

• tweak the program and its mode of operation to use less memory – many data processing algorithms have options for that though the specifics of this program are something for a different question –,

• terminate other non-vital, memory hogging processes running at the same time,

• add more physical main memory (RAM) to the system,

• add more virtual main memory (swap space) to the system – this will obviously slow down the system once it runs out of physical memory but at least your process will finish its work eventually and it's also something for a different question.