how to increase open file limits (nofile and epoll) in 10.04?

Question

I'm running into a fairly common problem in configuring a hadoop cluster (actually, its using Cloudera's pseudo distributed cluster on a single machine), where the number of files that hadoop is opening is exceeding the file system limits. Cloudera recommends adding the following lines to /etc/security/limits.conf:

hdfs hard nofile 16384 # hdfs is my 'hadoop' user equivalent

and, since I'm running kernel 2.6.32, also editing /etc/sysctl.conf:

fs.epoll.max_user_instances = 4096

After making these changes and restarting my server, I am still getting the same error! It still appears that hdfs's open file limits have not been increased beyond the 1024 default:

[bash]$ sudo lsof | awk '{print $3}' | sort | uniq -c | sort -nr
   2145 root
   1495 hdfs
    610 mapred
    359 www-data
    305 rdm
    116 mysql
     83 rabbitmq
     32 messagebus
     30 snmp
     25 ntp
     23 syslog
     16 daemon
      1 USER

As I've done more research its becoming clear that increasing the file size limits is highly system dependent (even within Ubuntu; here, here, and here), so I wanted to see what the Ubuntu method is. Does anyone know how to increase these limits in Ubuntu 10.04?

I definitely prefer solutions that do not increase the limits for all users, but at this point I would be willing to try anything. Thanks for your help!

Answer 1

In order to set these limits, I did a combination of things found here and here. Since I want to restrict these file limits to the hdfs and mapred users, I added each of these users to the hadoop group on my system and edited /etc/security/limits.conf to include the line:

@hadoop hard nofile 16384

which allows both users to open as many as 16384 files at once, which is apparently important in pseudo-distributed mode. I also had to add the following line to /etc/pam.d/common-session:

session required pam_limits.so

which makes these file limits persist across daemon processes like hdfs and mapred. After restarting the server, everything appears to be working perfectly as hdfs currently has more than the default number (1024) files open:

[dsftar01 ~]$ sudo lsof | awk '{if(NR>1) print $3}' | sort | uniq -c | sort -nr
   1972 root
   1530 hdfs
    608 mapred
    360 www-data
    166 rdm
     97 mysql
     83 rabbitmq
     41 nobody
     35 syslog
     31 messagebus
     30 snmp
     25 ntp
     16 daemon

Answer 2

add ulimit -n 16384 to hdfs users .bashrc (or /etc/profile this will set the value for all users]

If the user does not have a home directory append ulimit -n 16384 to /etc/profile

Then you have to restart the computer.