Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

The memory usage on my Ubuntu 13.04 system doesn't add up over time. I've tried the 3.8.0-xx kernels and the latest 3.9 from /~kernel-ppa/mainline.

When I first boot, it all adds up correctly but after a couple of days, this is what it looks like.

$ free -k
             total       used       free     shared    buffers     cached
Mem:       7978976    7319920     659056          0      97248    1344356
-/+ buffers/cache:    5878316    2100660
Swap:      2438140          0    2438140

Free is reporting that about 5.8G of resident memory is being used excluding cache and buffers.

When I sum the rss reported by the ps command, the totals differ dramatically.

sudo ps -eo pid,rss,args  > mem.txt

  PID      RSS  COMMAND
6377    848,268  /usr/lib/firefox/firefox
6095    243,664  /usr/bin/gnome-shell
1752    72,300  /usr/sbin/mysqld
6167    71,460  /var/lib/dropbox/.dropbox-dist/dropbox
7608    66,952  /usr/bin/python /usr/bin/deluge-gtk
23320   57,892  /usr/bin/X :3 -background none -verbose -auth /run/gdm/auth-for-gdm-9zmczs/database -nolisten tcp vt7
6445    45,476  /usr/lib/firefox/plugin-container /usr/lib/adobe-flashplugin/libflashplayer.so -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 6377 true plugin
6159    39,784  nautilus -n
6153    37,276  /usr/bin/python /usr/bin/autokey-gtk
9306    29,432  gnome-terminal
6152    27,152  nm-applet
6066    24,796  /usr/lib/gnome-settings-daemon/gnome-settings-daemon
6434    20,120  /usr/lib/firefox/plugin-container /opt/google/talkplugin/libnpgoogletalk.so -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 6377 true plugin
9505    17,700  /usr/sbin/apache2 -k start
6154    16,096  update-notifier
6437    13,516  /opt/google/talkplugin/GoogleTalkPlugin
6184    13,192  /usr/lib/evolution/evolution-calendar-factory
5983    12,372  gnome-session --session=gnome
3746    9,468   /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
9510    8,476   /usr/sbin/apache2 -k start
9511    8,476   /usr/sbin/apache2 -k start
9512    8,476   /usr/sbin/apache2 -k start
9513    8,476   /usr/sbin/apache2 -k start
9514    8,476   /usr/sbin/apache2 -k start
6112    7,732   /usr/lib/evolution/evolution-source-registry
1890    7,212   /usr/bin/atop -a -w /var/log/atop/atop_20130425 600
6106    6,912   /usr/lib/gnome-shell/gnome-shell-calendar-server
1646    6,868   NetworkManager
6073    6,404   /usr/bin/pulseaudio --start --log-target=syslog
6125    6,080   /usr/lib/telepathy/mission-control-5
5588    5,592   /usr/lib/udisks2/udisksd --no-debug
1229    5,568   /usr/lib/colord/colord
6129    5,560   /usr/lib/gvfs/gvfs-udisks2-volume-monitor
1205    5,532   smbd -F
5954    5,220   gdm-session-worker [pam/gdm-password]
1656    5,164   /usr/lib/policykit-1/polkitd --no-debug
9312    5,112   -bash
5965    5,040   /usr/bin/gnome-keyring-daemon --daemonize --login
9531    4,952   -bash
1759    4,900   whoopsie
1606    4,412   /usr/sbin/winbindd -F
3149    4,300   /usr/lib/upower/upowerd
2992    4,068   /usr/sbin/console-kit-daemon --no-daemon
23316   3,952   /usr/lib/gdm/gdm-simple-slave --display-id /org/gnome/DisplayManager/Displays/_3
6299    3,800   /usr/lib/gvfs/gvfsd-metadata
1090    3,748   /usr/sbin/cupsd -F
1721    3,708   /sbin/dhclient -d -sf /usr/lib/NetworkManager/nm-dhcp-client.action -pf /run/sendsigs.omit.d/network-manager.dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-681b428f-beaf-8932-dce4-687ed5bae28e-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
2988    3,672   /usr/lib/accountsservice/accounts-daemon
6228    3,540   /usr/lib/gvfs/gvfsd-trash --spawner :1.4 /org/gtk/gvfs/exec_spaw/0
6043    3,296   /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session
1728    3,288   gdm
6136    3,220   /usr/lib/gvfs/gvfs-gphoto2-volume-monitor
6144    3,212   /usr/lib/gvfs/gvfs-afc-volume-monitor
6036    3,204   /usr/lib/at-spi2-core/at-spi-bus-launcher
1607    3,196   /usr/sbin/modem-manager
1   3,096   /sbin/init
6087    3,044   /usr/lib/pulseaudio/pulse/gconf-helper
6047    2,992   /usr/lib/gvfs/gvfsd
6089    2,900   /usr/lib/x86_64-linux-gnu/gconf/gconfd-2
6274    2,832   /usr/lib/dconf/dconf-service
1248    2,772   /usr/sbin/cups-browsed
1571    2,696   /usr/sbin/sshd -D
6140    2,648   /usr/lib/gvfs/gvfs-mtp-volume-monitor
1484    2,608   dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
9662    2,600   sudo ps -eo pid,rss,args
3614    2,428   /usr/sbin/afpd -U uams_dhx2.so,uams_clrtxt.so -g nobody -c 50 -n cheetah
771 2,340   dbus-daemon --system --fork
5376    2,124   /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 117:126
6032    1,908   //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
1642    1,780   /usr/sbin/winbindd -F
3460    1,760   qmgr -l -t fifo -u
928 1,728   avahi-daemon: running [cheetah.local]
866 1,704   /usr/sbin/bluetoothd
719 1,700   /sbin/udevd --daemon
9637    1,696   man ps
1627    1,684   nmbd -D
762 1,652   rsyslogd -c5
2850    1,636   /usr/lib/postfix/master
1239    1,608   smbd -F
7599    1,608   pickup -l -t fifo -u -c
6040    1,488   /bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3
1760    1,472   /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/NetworkManager/dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
5399    1,368   /sbin/udevd --daemon
5400    1,368   /sbin/udevd --daemon
1233    1,356   rpc.statd -L
3516    1,308   /usr/lib/rtkit/rtkit-daemon
9648    1,188   pager -s
1719    1,044   cron
1675    976 /sbin/getty -8 38400 tty5
1691    976 /sbin/getty -8 38400 tty2
1692    972 /sbin/getty -8 38400 tty3
1695    972 /sbin/getty -8 38400 tty6
3971    972 /sbin/getty -8 38400 tty1
1668    968 /sbin/getty -8 38400 tty4
1216    964 rpcbind -w
716 900 upstart-udev-bridge --daemon
2893    880 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new
9663    868 ps -eo pid,rss,args
9311    824 gnome-pty-helper
1703    816 acpid -c /etc/acpi/events -s /var/run/acpid.socket
1737    776 /usr/sbin/irqbalance
3606    760 /usr/sbin/cnid_metad -l log_note
1235    732 upstart-socket-bridge --daemon
763 680 rpc.idmapd
6031    592 /usr/bin/dbus-launch --exit-with-session gnome-session --session=gnome
933 472 avahi-daemon: chroot helper
457 420 upstart-file-bridge --daemon
6028    316 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session gnome-session --session=gnome
1717    168 atd
2   0   [kthreadd]
3   0   [ksoftirqd/0]
5   0   [kworker/0:0H]
7   0   [kworker/u:0H]
8   0   [migration/0]
9   0   [rcu_bh]
10  0   [rcu_sched]
11  0   [watchdog/0]
12  0   [watchdog/1]
13  0   [migration/1]
14  0   [ksoftirqd/1]
16  0   [kworker/1:0H]
17  0   [watchdog/2]
18  0   [migration/2]
19  0   [ksoftirqd/2]
21  0   [kworker/2:0H]
22  0   [watchdog/3]
23  0   [migration/3]
24  0   [ksoftirqd/3]
26  0   [kworker/3:0H]
27  0   [watchdog/4]
28  0   [migration/4]
29  0   [ksoftirqd/4]
31  0   [kworker/4:0H]
32  0   [watchdog/5]
33  0   [migration/5]
34  0   [ksoftirqd/5]
36  0   [kworker/5:0H]
37  0   [khelper]
38  0   [kdevtmpfs]
39  0   [netns]
40  0   [bdi-default]
41  0   [kintegrityd]
42  0   [kblockd]
43  0   [ata_sff]
44  0   [khubd]
45  0   [md]
46  0   [devfreq_wq]
48  0   [irq/72-AMD-Vi]
49  0   [khungtaskd]
50  0   [kswapd0]
51  0   [ksmd]
52  0   [khugepaged]
53  0   [fsnotify_mark]
54  0   [ecryptfs-kthrea]
55  0   [crypto]
66  0   [kthrotld]
74  0   [binder]
93  0   [deferwq]
94  0   [charger_manager]
198 0   [scsi_eh_0]
204 0   [scsi_eh_1]
205 0   [firewire]
219 0   [kworker/u:3]
220 0   [scsi_eh_2]
239 0   [scsi_eh_3]
251 0   [scsi_eh_4]
255 0   [scsi_eh_5]
259 0   [scsi_eh_6]
260 0   [scsi_eh_7]
269 0   [kworker/u:6]
283 0   [scsi_eh_8]
284 0   [scsi_eh_9]
360 0   [kworker/5:1H]
361 0   [jbd2/sda1-8]
362 0   [ext4-dio-unwrit]
643 0   [xfsalloc]
644 0   [xfs_mru_cache]
645 0   [xfslogd]
646 0   [xfs-data/sdb1]
647 0   [xfs-conv/sdb1]
648 0   [xfs-cil/sdb1]
650 0   [xfsaild/sdb1]
695 0   [xfs-data/sdc1]
696 0   [xfs-conv/sdc1]
697 0   [xfs-cil/sdc1]
698 0   [xfsaild/sdc1]
730 0   [rpciod]
743 0   [nfsiod]
922 0   [krfcommd]
944 0   [kworker/4:1H]
1017    0   [edac-poller]
1029    0   [hd-audio0]
1031    0   [kpsmoused]
1060    0   [kvm-irqfd-clean]
1107    0   [hd-audio1]
1157    0   [kworker/3:1H]
1190    0   [kworker/2:1H]
1198    0   [kworker/1:1H]
1793    0   [kauditd]
1845    0   [flush-8:0]
3569    0   [iprt]
4867    0   [kworker/4:1]
5403    0   [nfsv4.0-svc]
5848    0   [kworker/0:1H]
5935    0   [kworker/2:3]
9274    0   [kworker/5:2]
9527    0   [kworker/5:0]
9630    0   [flush-8:32]
10694   0   [kworker/3:3]
13358   0   [kworker/1:0]
18154   0   [kworker/0:0]
19625   0   [kworker/4:0]
23502   0   [kworker/3:2]
31403   0   [kworker/2:1]
32141   0   [kworker/0:2]
32655   0   [kworker/1:1]
    1,939,900   

ps reports that the sum off all the resident memory used by all my processes is about 1.9G.

What processes are using the remaining 3.9G?

EDIT: More information on kernel memory usage. slabtop reports the following.

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
2233988 2233988 100%    0.88K  62057       36   1985824K xfs_inode              
735948 714094  97%    0.22K  20443       36    163544K xfs_ili                
129280 117946  91%    1.00K   4040       32    129280K nfs_inode_cache        
 92344  92344 100%    0.93K   2716       34     86912K ext4_inode_cache       
292866 292866 100%    0.19K  13946       21     55784K dentry                 
 75096  71346  95%    0.55K   2682       28     42912K radix_tree_node     

Does this mean that the kernel is using almost 2GB to cache xfs_inodes, cache that's not reported in the cached column of the free command?

share|improve this question
1  
ps is not an accurate way to measure system memory usage, for various reasons. I've seen the following command, similar to your command but includes a total: sudo ps -e -orss=,args= | sort -b -k1,1n | awk '{total = total + $1}END{print total}' –  Bert Apr 25 '13 at 15:05
    
Thanks, much easier than loading in a spreadsheet. The total that it reports is the same though at 1896504 bytes. I still have almost 4G missing. Anyway to determine what's using it? –  Richard Ayotte Apr 25 '13 at 16:12
    
Have you tried the command top? –  somoso Apr 30 '13 at 11:56
    
@somoso yes, top, htop, system monitor, they all report the same thing. –  Richard Ayotte Apr 30 '13 at 13:32
add comment

2 Answers

up vote 5 down vote accepted

The memory has been found. It was in the xfs_inode cache which wasn't reported by ps or free. slabtop did however show me what I needed. The reason for such a large xfs_inode cache is because of a nightly backup. The memory can be released with the following command.

echo 3 > /proc/sys/vm/drop_caches

From http://www.linuxinsight.com/proc_sys_vm_drop_caches.html

To free pagecache:

echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

echo 3 > /proc/sys/vm/drop_caches

More information can be found on this blog.

http://serenadetoacuckooo.blogspot.ca/2012/03/linux-memory-tweaks-for-improved.html

I have to decide now on whether I want to drop the cache after backups, set vfs_cache_pressure back to 100 or just buy more memory.

share|improve this answer
add comment

The numbers in ps do not eliminate redundancy among processes which map the same objects into their virtual memory address spaces, such as executables and shared libraries. Say that three instances of some program are running, and the libraries and executable add up to 20 megs, and are 100% resident in memory. That memory will simply appear three times in the ps listing so it looks like 60 megs (and some of the libraries like libc will also count against other processes).

If you want to tabulate virtual memory usage accurately, you have to iterate over all the running processes, scan the /proc/<PID>/maps data for all of them and merge the information. The problem is that the map info won't tell you how much of each virtual memory mapping is actually resident, so it won't be easy to correlate to page level info given in free.

A whole other problem is that not all the memory you see in free is accounted by processes. It's based on how many pages are free in the system as a whole. Some pages are assigned to resident memory mappings, as well as anonymous mappings (e.g. malloc data private to a process, not backed by a file). But some pages are allocated by the kernel. If you insmod a driver, space is allocated for it. When network packets are received, space is allocated, etc. The kernel has numerous data structures that are allocated from the pool of pages, and that shows up as used memory in free (which is based on /proc/meminfo). The kernel has a virtual memory allocation area of its own managed by the vmalloc function, and that we can be informed about via /proc/vmallocinfo, but that doesn't report on things allocated via kmalloc or directly via get_free_page.

share|improve this answer
    
wrt ps, you're saying that the memory usage problem is worse than what I've shown, which is probably true. You're also saying that the kernel is using the memory that's not being reported which is what I suspected. –  Richard Ayotte May 3 '13 at 12:32
    
Thank you for the hints on kernel memory usage; it put me on the right path. –  Richard Ayotte May 3 '13 at 13:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.