A couple of weeks ago, my OSSEC daemon started sending me syslogs about a segfault in my php5-fpm
. Here's the portion of the logs:
Dec 19 15:09:38 mydropplet kernel: [165740.998932] php5-fpm[26936]: segfault at 7fff621ad001 ip 000000000079fe74 sp 00007fff621a5040 error 6 in php5-fpm[400000+800000]
So I peeked into my error logs to see if I had some PHP errors somewhere that were causing the segfault (since I am not sure what that even means, but sounds like it's something in the PHP core) My error logs had this in it (identifying information removed):
2015/12/19 01:36:49 [error] 6177#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 01:36:54 [error] 6177#0: gv.symcd.com could not be resolved (110: Operation timed out) while requesting certificate status, responder: gv.symcd.com
2015/12/19 02:00:06 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:11 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:16 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:21 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:26 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:31 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
So I took a closer look at my syslogs it had this in it. This is the same portion of the logs I was seeing in the emails from OSSEC:
Dec 19 09:44:06 mydomain kernel: [146209.162601] php5-fpm[13787]: segfault at 7fff621ad001 ip 000000000079fe74 sp 00007fff621a5040 error 6 in php5-fpm[400000+800000]
Here is the rest of the logging:
Dec 19 09:44:59 mydomain postfix/anvil[26420]: statistics: max connection rate 1/60s for (smtp:195.22.126.137) at Dec 19 09:41:39
Dec 19 09:44:59 mydomain postfix/anvil[26420]: statistics: max connection count 1 for (smtp:195.22.126.137) at Dec 19 09:41:39
Dec 19 09:44:59 mydomain postfix/anvil[26420]: statistics: max cache size 1 at Dec 19 09:41:39
[...]
Dec 19 09:41:39 mydomain postfix/smtpd[26418]: connect from unknown[195.22.126.137]
Dec 19 09:41:39 mydomain postfix/smtpd[26418]: disconnect from unknown[195.22.126.137]
I'm pretty new to self-managed servers, so I am not sure how to debug this one. I manage my personal website on this server and the only thing I did a couple weeks ago is installed a new plugin from the Wordpress repository, due to lack of time to research a proper configuration I hastily activated it and configured it, and it bricked the front end of my site. So I deactivated it, and decided to give it a go later on. After that the kernel errors started happening.
So I uninstalled the plugin completely thinking it might be the cause, but it didn't solve anything. The plugin would have only had write access to my web directory and would not have been able to change any core configuration settings. I'm not sure if there's a connection there, but I thought I would mention it.
I have also been running this PHP configuration since October with no problems. I haven't added any new/weird modules in PHP. It's basically out of the box, except for a few security precautions. Before seeing the error I executed a dist-upgrade as a part of my regular maintenance:
sudo apt-get dist-upgrade
sudo apt-get update
sudo shutdown -r now
This was around the same time as the plugin installation. In an attempt to fix the errors, I executed the commands again, thinking if there were a bug surely a patch would have been pushed. This, however, didn't help. So I'm at a loss of what I can do to beat this.
Update: I rolled back to a previous snapshot, and everything worked for about a week, then I started getting emails again about the same seg fault issue shortly after Christmas. So I am back to square one.
Any advice would be great! I'm running LEMP Ubuntu 14.04 LTS server. How do I debug an issue like this? Thank you for any help ahead of time.