I'm running into a fairly common problem in configuring a hadoop cluster (actually, its using Cloudera's pseudo distributed cluster on a single machine), where the number of files that hadoop is opening is exceeding the file system limits. Cloudera recommends adding the following lines to /etc/security/limits.conf
:
hdfs hard nofile 16384 # hdfs is my 'hadoop' user equivalent
and, since I'm running kernel 2.6.32, also editing /etc/sysctl.conf
:
fs.epoll.max_user_instances = 4096
After making these changes and restarting my server, I am still getting the same error! It still appears that hdfs's open file limits have not been increased beyond the 1024 default:
[bash]$ sudo lsof | awk '{print $3}' | sort | uniq -c | sort -nr
2145 root
1495 hdfs
610 mapred
359 www-data
305 rdm
116 mysql
83 rabbitmq
32 messagebus
30 snmp
25 ntp
23 syslog
16 daemon
1 USER
As I've done more research its becoming clear that increasing the file size limits is highly system dependent (even within Ubuntu; here, here, and here), so I wanted to see what the Ubuntu method is. Does anyone know how to increase these limits in Ubuntu 10.04?
I definitely prefer solutions that do not increase the limits for all users, but at this point I would be willing to try anything. Thanks for your help!