Ubuntu: 12.04 LTS (Linux mysql02 3.2.0-40-generic #64-Ubuntu SMP Mon Mar 25 21:22:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux)
MySQL: Ubuntu distro 5.5.31
Apparmor: REMOVED!
Server has been running rock solid for over a year. Then this Monday MySQL began failing. An update has caused the problem and we cannot figure it out what it is. We have even tried to roll back to MySQL 5.5.30 but with no luck. We returned at 5.5.31.
MySQL error log entries:
130430 7:55:46 [ERROR] Error in accept: Too many open files
130430 7:55:46 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/fclvod.frm' (errno: 24)
130430 7:55:46 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/fcnote.frm' (errno: 24)
130430 7:55:47 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/ffcont.frm' (errno: 24)
130430 7:55:47 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/ffcontv.frm' (errno: 24)
130430 7:55:47 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/ffnote.frm' (errno: 24)
130430 7:55:47 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/frcfcl.frm' (errno: 24)
It appears we are running into ulimit problem. We have removed APPARMOR completely. We have increased the /etc/security/limits.conf and still no luck:
# Out of desperation....
* soft nofile 49152
* hard nofile 65536
# No effect!?!!?
#mysql soft nofile 49152
#mysql hard nofile 65536
And to show the limits.conf is working:
root@mysql02:/etc/security# ulimit -Sa | grep "open files"
open files (-n) 49152
root@mysql02:/etc/security# ulimit -Ha | grep "open files"
open files (-n) 65536
And here are the important entries in my.cnf
[mysqld_safe]
open_files_limit = 16384
[mysqld]
open_files_limit = 16384
However:
root@mysql02:/etc/mysql# mysqladmin -u root -pThePassword variables| grep open_files_limit
open_files_limit | 1024
We are totally stumped and down. Any assistance would be greatly appreciated.