How do you debug a php5-fpm kernel segfault error?

Question

A couple of weeks ago, my OSSEC daemon started sending me syslogs about a segfault in my php5-fpm. Here's the portion of the logs:

Dec 19 15:09:38 mydropplet kernel: [165740.998932] php5-fpm[26936]: segfault at 7fff621ad001 ip 000000000079fe74 sp 00007fff621a5040 error 6 in php5-fpm[400000+800000]

So I peeked into my error logs to see if I had some PHP errors somewhere that were causing the segfault (since I am not sure what that even means, but sounds like it's something in the PHP core) My error logs had this in it (identifying information removed):

2015/12/19 01:36:49 [error] 6177#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 01:36:54 [error] 6177#0: gv.symcd.com could not be resolved (110: Operation timed out) while requesting certificate status, responder: gv.symcd.com
2015/12/19 02:00:06 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:11 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:16 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:21 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:26 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:31 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx

So I took a closer look at my syslogs it had this in it. This is the same portion of the logs I was seeing in the emails from OSSEC:

Dec 19 09:44:06 mydomain kernel: [146209.162601] php5-fpm[13787]: segfault at 7fff621ad001 ip 000000000079fe74 sp 00007fff621a5040 error 6 in php5-fpm[400000+800000]

Here is the rest of the logging:

Dec 19 09:44:59 mydomain postfix/anvil[26420]: statistics: max connection rate 1/60s for (smtp:195.22.126.137) at Dec 19 09:41:39
Dec 19 09:44:59 mydomain postfix/anvil[26420]: statistics: max connection count 1 for (smtp:195.22.126.137) at Dec 19 09:41:39
Dec 19 09:44:59 mydomain  postfix/anvil[26420]: statistics: max cache size 1 at Dec 19 09:41:39
[...]
Dec 19 09:41:39 mydomain postfix/smtpd[26418]: connect from unknown[195.22.126.137]
Dec 19 09:41:39 mydomain postfix/smtpd[26418]: disconnect from unknown[195.22.126.137]

I'm pretty new to self-managed servers, so I am not sure how to debug this one. I manage my personal website on this server and the only thing I did a couple weeks ago is installed a new plugin from the Wordpress repository, due to lack of time to research a proper configuration I hastily activated it and configured it, and it bricked the front end of my site. So I deactivated it, and decided to give it a go later on. After that the kernel errors started happening.

So I uninstalled the plugin completely thinking it might be the cause, but it didn't solve anything. The plugin would have only had write access to my web directory and would not have been able to change any core configuration settings. I'm not sure if there's a connection there, but I thought I would mention it.

I have also been running this PHP configuration since October with no problems. I haven't added any new/weird modules in PHP. It's basically out of the box, except for a few security precautions. Before seeing the error I executed a dist-upgrade as a part of my regular maintenance:

sudo apt-get dist-upgrade
sudo apt-get update
sudo shutdown -r now

This was around the same time as the plugin installation. In an attempt to fix the errors, I executed the commands again, thinking if there were a bug surely a patch would have been pushed. This, however, didn't help. So I'm at a loss of what I can do to beat this.

Update: I rolled back to a previous snapshot, and everything worked for about a week, then I started getting emails again about the same seg fault issue shortly after Christmas. So I am back to square one.

Any advice would be great! I'm running LEMP Ubuntu 14.04 LTS server. How do I debug an issue like this? Thank you for any help ahead of time.

Answer 1

If you think the error is related to Nginx, you should post your related Nginx logs. There are references to resolver in your logs posted, which might relate to Nginx.

If the Nginx resolver is at fault, you might be able to workaround the issue by using raw IPs in your Nginx configuration instead of domain names.

Also, check your current version of php5-fpm vs the latest version available in their Git repository. Then check the Changelog for the project for changes that are newer than the version you are using. If there is a newer version available, see about upgrading.

Segfaults imply a software bug, not a configuration problem. Even if the face of a bad configuration, software shouldn't segfault. Since it appears php5-fpm was the process that segfaulted, consider reporting the bug on their mailing list or issue tracker if it's not fixed in the latest version.

It's also possible the software running via php5-fpm is what segfault'ed and php5-fpm is just the messenger delivering the notice of a segfault. Again, the php5-fpm project will be the authoritative reference for figuring how out segfaults with that software.