Comparing Virtual Memory Size with Resident Size

Question

I came across this stackoverflow post:

https://stackoverflow.com/questions/10400751/how-do-vmrss-and-resident-set-size-match

The answer voted correctly states the following:

"So, VSS should be greater than RSS. If they are close to equal, that means your process is sitting comfortably in memory. If VSS is much larger, that means there isn't enough memory and parts of it have to be swapped out to disk (i.e., because of a competing process, etc.)."

That statement confuses me a lot because when I inspect my system, I notice the following.

First, I notice I have a lot of free memory:

$ cat /proc/meminfo
MemTotal:        6113156 kB
MemFree:         3668992 kB

That means I have 3.5 gigabytes of pure memory (no swap, no disk, etc)

However, when I look at my spawned apache2 child processes, I come to a surprise:

$ ps aux | grep apache2
USER       PID  %CPU %MEM  VSZ  RSS     TTY   STAT START    TIME    COMMAND
root      1130  0.0  0.1 149080 10600 ?        Ss   Jul11   0:03 /usr/sbin/apache2 -k start
www-data 23211  0.0  0.3 163408 23784 ?        S    10:34   0:03 /usr/sbin/apache2 -k start
www-data 23215  0.0  0.4 164436 24832 ?        S    10:34   0:02 /usr/sbin/apache2 -k start
www-data 23287  0.0  0.3 163608 23992 ?        S    10:36   0:02 /usr/sbin/apache2 -k start
www-data 23351  0.0  0.3 163660 24064 ?        S    10:40   0:01 /usr/sbin/apache2 -k start
www-data 23440  0.0  0.3 161580 23588 ?        S    10:46   0:00 /usr/sbin/apache2 -k start
www-data 24393  0.0  0.3 163620 23496 ?        S    11:32   0:00 /usr/sbin/apache2 -k start
www-data 25377  0.0  0.2 150656 12316 ?        S    12:20   0:00 /usr/sbin/apache2 -k start
www-data 25378  0.0  0.3 158224 18400 ?        S    12:20   0:00 /usr/sbin/apache2 -k start
www-data 27038  0.0  0.1 149360  7816 ?        S    13:01   0:00 /usr/sbin/apache2 -k start
www-data 27041  0.0  0.1 149368  7660 ?        S    13:01   0:00 /usr/sbin/apache2 -k start
1000     27124  0.0  0.0   8112   900 pts/0    S+   13:04   0:00 grep apache2

(Note that grep removes column headers, so I artificially add them back)

Look how much larger virtual memory is compared to resident memory. I mean, for example, for the apache parent process (the parent process is 1130):

$ ps xao pid,ppid,pgid,sid,comm | grep apache2 
 1130     1  1130  1130 apache2
23211  1130  1130  1130 apache2
23440  1130  1130  1130 apache2
27038  1130  1130  1130 apache2
27041  1130  1130  1130 apache2
27183  1130  1130  1130 apache2
27242  1130  1130  1130 apache2
27349  1130  1130  1130 apache2
27405  1130  1130  1130 apache2
27456  1130  1130  1130 apache2
27457  1130  1130  1130 apache2

That parent process is taking up 146 megabytes of virtual memory compared to 10 megabytes of resident memory. That is a difference of 136 megabytes of swap space being used!

So this doesn't make sense to me. I have so much free memory, but it's using so much more swap space?? According to the post on stackoverflow, he says "means there isn't enough memory". Well that's not true. I have plenty of memory.

Answer 1

The "swapped out" conclusion the SO post makes is wrong. For example, here is a trivial program:

#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

int main() {
    printf("Started - sleeping 10s; pid = %i\n", (int)getpid());
    sleep(10);

    int fd = open("10469068800-byte-file", O_RDONLY);
    void *map = mmap(NULL, 10469068800, PROT_READ, MAP_SHARED, fd, 0);

    printf("Mapped - sleeping 10s; fd %i to %p\n", fd, map);
    sleep(10);

    return 0;
}

When I check ps after it prints out the started message (before the mapped message):

anthony@Zia:~$ ps u 13420
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
anthony  13420  0.0  0.0   4080   348 pts/13   S+   16:10   0:00 ./test

and after:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
anthony  13420  0.0  0.0 10227780 348 pts/13   S+   16:10   0:00 ./test

Part of the program's address space is currently on disk, but that's because its a memory-mapped file which has not yet been read (or may never be). A similar thing happens with this program:

#include <sys/types.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

int main() {
    printf("Started - sleeping 10s; pid = %i\n", (int)getpid());
    sleep(10);

    void *mem = malloc(1024*1024*1024);

    printf("Allocated - sleeping 10s; mem at %p\n", mem);
    sleep(10);

    return 0;
}

Before:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
anthony  15150  0.0  0.0   4080   352 pts/13   S+   16:18   0:00 ./test2

After:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
anthony  15150  0.0  0.0 1052660  352 pts/13   S+   16:18   0:00 ./test2

In this case, the memory has been allocated but, as an optimization, the kernel doesn't actually put actual pages of memory behind those addresses until the program uses them. So, again, you see a much higher VSZ than RSS.

You're probably seeing the above two things (and maybe a few more) in Apache. You can use pmap -x to tell. Here is what the second program (the malloc one) looks like:

anthony@Zia:~$ pmap -x 15997
15997:   ./test2
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000       4       4       0 r-x--  test2
0000000000600000       4       4       4 rw---  test2
00007fba82f94000 1048580       4       4 rw---    [ anon ]      <--- HERE
00007fbac2f95000    1672     300       0 r-x--  libc-2.17.so
00007fbac3137000    2048       0       0 -----  libc-2.17.so
00007fbac3337000      16      16      16 r----  libc-2.17.so
00007fbac333b000       8       8       8 rw---  libc-2.17.so
00007fbac333d000      16      12      12 rw---    [ anon ]
00007fbac3341000     132     104       0 r-x--  ld-2.17.so
00007fbac3533000      12      12      12 rw---    [ anon ]
00007fbac355f000      12      12      12 rw---    [ anon ]
00007fbac3562000       4       4       4 r----  ld-2.17.so
00007fbac3563000       8       8       8 rw---  ld-2.17.so
00007fffb7163000     132      12      12 rw---    [ stack ]
00007fffb71fe000       8       4       0 r-x--    [ anon ]
ffffffffff600000       4       0       0 r-x--    [ anon ]
----------------  ------  ------  ------
total kB         1052660     504      92

Note you can see the anonymous mapping which is huge but has almost nothing resident. For the program with the mmap, you get:

Address           Kbytes     RSS   Dirty Mode   Mapping
⋮
00007f8e50cf2000 10223700       0       0 r--s-  10469068800-byte-file
⋮

That shows you that the memory mapped file is there, but none of it is resident.