60

I use an application that consumes inotify watches. I've already set

fs.inotify.max_user_watches=32768

in /etc/sysctl.conf but last night the application stopped indexing unless I ran it manually, which leads me to suspect I am out of watches.

Since I don't know what the trade off is when I increase this number (does it consume more RAM?), I don't know if I should just increase this number, so I'd like to know if there's a way I can tell if it's using all these watches and what the tradeoffs might be for increasing it.

2
  • This question has become quite relevant again as it turns out in 18.04 unlocking the screen is not possible if one has run out of inode watches.
    – kasperd
    Jan 21, 2019 at 21:33
  • It would be possible to see the number of inotify watches in use if the kernel patch I linked here got merged: github.com/prometheus/node_exporter/issues/… May 24, 2022 at 20:06

2 Answers 2

80

How do you know if you are out of watches? tail will tell!

  • Start tail with the -f (follow) option on any old file, e.g. tail -f /var/log/dmesg:
    • If all is well, it will show the last 10 lines and pause; abort with Ctrl-C
    • If you are out of watches, it will fail with this somewhat cryptic error:
      tail: cannot watch '/var/log/dmsg': No space left on device

For the curious: Why is tail a "telltail"?

  • Actually, any well-written app should have the courtesy of telling you, since the inotify API/calls clearly tells them what the deal is.
  • Try strace tail -f ... instead, and when it succeeds, it ends with:
    inotify_add_watch(4, "/var/log/dmesg", IN_MODIFY...) = 1
    
  • but if it fails, i.e. you are out of watches, it'll say:
    inotify_add_watch(4, "/var/log/dmesg", IN_MODIFY..)
    = -1 ENOSPC (No space left on device)
    

Can you increase the watches? By how much? Any tradeoffs?

Short answer: Sure, no sweat. Go to straight to a half-million (524288) if you want...the additional memory used should be negligible on a modern system with 4GB+ of memory.

  • Each used inotify watch takes up 540 bytes (32-bit system), or 1 kB (double - on 64-bit) [sources: 1, 2]
  • This comes out of kernel memory, which is unswappable.
  • So, assuming you set the max at 524288, and all were used (improbable), you'd be using approx. 256MB/512MB of 32-bit/64-bit kernel memory

    • Note that your application will also use additional memory to keep track of the inotify handles, file/directory paths, etc. -- how much depends on its design.
  • What's the max value? I guess none, in theory, as long as you have enough RAM. In practice, 524288 has been officially recommended by apps, and people have been setting it to 2 million, with the accompanying memory usage, of course.

3
  • 8
    If you also want to know the actual current number of inotify watches (thus more then yes/no), go for sudo lsof | grep -i inotify | wc -l
    – Frank N
    Sep 14, 2017 at 12:14
  • 2
    "well-written-app" link broken. Apr 9, 2018 at 7:15
  • @FrankNocke You might want to be a little more careful with your grep. That can catch many things that aren't an inotify watch. For instance, if you have a file open named "inotify.txt", it'll count that. In fact, on my system, this seems to catch two distinct types of lines: ones whose NAME (last column) is inotify, and ones whose TASKCMD (4th column) is inotify_reader. I'm not sure which of those two is an actual inotify watch.
    – Nick S
    Aug 24, 2022 at 21:05
15

I don't know if I should just increase this number

The easy way of checking if you reached your max_user_watches value is, with your user, to use inotifywatch, from the package inotify-tools, and check if you can still collect information from a file.

For example inotifywatch -v /home/bruno/.profile for me returns:

Establishing watches...
Total of 1 watches.
Finished establishing watches, now collecting statistics.

So inotify has no issues creating a new watch, no issues here.

If you have reached your maximum limit in inotify watches it will return something like

Failed to watch /home/bruno/.profile; upper limit on inotify watches reached!

If you see something like this then you have reached the limit and will need to increase the allowed watches limit.

Does it consume more RAM?

Yes, it does. But according to this old article the amount it consumes is minimal compared with other aspects of a running desktop.

--MEMORY USAGE--

The inotify data structures are light weight:

inotify watch is 40 bytes inotify device is 68 bytes inotify event is 272 bytes

So assuming a device has 8192 watches, the structures are only going to consume 320KB of memory. With a maximum number of 8 devices allowed to exist at a time, this is still only 2.5 MB

Each device can also have 256 events queued at a time, which sums to 68KB per device. And only .5 MB if all devices are opened and have a full event queue.

So approximately 3 MB of memory are used in the rare case of everything open and full.

Each inotify watch pins the inode of a directory/file in memory, the size of an inode is different per file system but lets assume that it is 512 byes.

So assuming the maximum number of global watches are active, this would pin down 32 MB of inodes in the inode cache. Again not a problem on a modern system.

I am of course assuming things did not change a lot since the article was written but looking at the numbers I would not worry and increasing the limit will not increase RAM consumption by much.


Related posts about inotify

1
  • 1
    Sorry man, I had a draft open yesterday for a few hours and didn't see your answer before I posted mine. I guess it's OK since they have two different approaches :-) Could you just clarify that in recent kernels, inotify uses 0.5KB (on 32-bit) or 1KB (64-bit) of kernel memory per watch, since the old info from 2005 doesn't seem to be true any longer?
    – ish
    Jun 25, 2012 at 7:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .