I have a variety of Ubuntu machines running in production, with about 30 that were upgraded from 15.04 to 15.10. With most of the machines, the upgrade has been a breeze and everything has functioned perfectly.
However, 10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs. The most recent log entries I see on one machine show:
/var/log/syslog:Dec 18 00:28:58 xxx-web-4a dhclient: DHCPREQUEST of 10.xxx.xxx.104 on eth0 to 10.xxx.xxx.1 port 67 (xid=0x616a091d)
/var/log/syslog:Dec 18 00:28:58 xxx-web-4a dhclient: DHCPACK of 10.xxx.xxx.104 from 10.xxx.xxx.1
/var/log/syslog:Dec 18 00:28:58 xxx-web-4a dhclient: bound to 10.xxx.xxx.104 -- renewal in 1640 seconds.
But my Instance Status Checks didn't begin failing until 00:32:00 (when the first of several checks failed to respond). There is absolutely nothing in the logs during the period following the entries above.
Now, like I said, my ~20 other 15.10 instances have never crashed in the over 6 weeks since their upgrade, only this set of webservers, and they're all crashing. So, what's different about these machines? Only two things, really.
- They're my highest-traffic 15.10 instances, sending and receiving about 5-10Mb/sec on average, peaking to a bit over 30-40 on occasion.
- They're my only instances of type
c4.xlargeorm4.xlarge. Originally, they were allc4.xlarge, but I replaced them withm4.xlargeto try to isolate the problem. It seems to be less frequent with them4.xlarge, but I've still seen 3 or 4 or so crashes a day between the 10 webservers. Generally, I'm seeing each instance crash at least once a day, at seemingly random times.
These instances are running Apache 2.4.x, mod_php 5.6.11, and memcached 1.4.24, but I have other machines receiving less traffic on a smaller instance type that are perfectly stable.
Not sure if related, but all of these machines are seeing periodic ifquery segfaults, for example:
/var/log/syslog:Dec 17 14:02:05 xxx-web-3a kernel: [ 22.592488] ifquery[476]: segfault at 1 ip 0000000000403187 sp 00007ffde8596050 error 4 in ifup[400000+d000]
/var/log/syslog:Dec 17 14:02:05 xxx-web-3a kernel: [ 23.593774] ifquery[510]: segfault at 1 ip 0000000000403187 sp 00007ffde6087b90 error 4 in ifup[400000+d000]
/var/log/syslog:Dec 17 14:02:05 xxx-web-3a kernel: [ 24.594994] ifquery[531]: segfault at 1 ip 0000000000403187 sp 00007ffe70747a50 error 4 in ifup[400000+d000]
/var/log/syslog:Dec 17 14:04:12 xxx-web-3a kernel: [ 2.623024] ifquery[367]: segfault at 1 ip 0000000000403187 sp 00007ffefc980f60 error 4 in ifup[400000+d000]
One system, prior to the c4.xlarge --> m4.xlarge migration, saw a General Protection Fault logged a single time in the system console log, but I have not seen this again.
I'm not seeing these segfaults on my other 15.10 machines which are not crashing.
These are all Enhanced Networking instances with Intel 82599 10G Ethernet, which I slightly suspect may contribute to the issue, but, I have other (much-lower-traffic) machines with the same adapter running 15.10 without ever crashing.
Is anyone seeing similar problems, or have any ideas for debugging or fixing this? Thanks!
Edit
Looking at the Console Log, one of my frequently-crashing systems reported a General Protection Fault right before rebooting:
[171009.844097] general protection fault: 0000 [#1] [ 0.000000] Initializing cgroup subsys cpuset
Edit 2
We got a trace now with netconsole! Reporting a bug.
[21410.260077] general protection fault: 0000 [#1] SMP
[21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev intel_rapl iosf_mbi xen_fbfront fb_sys_fops input_leds serio_raw i2c_piix4 parport_pc 8250_fintek parport mac_hid netconsole configfs autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea sysfillrect sysimgblt aesni_intel ttm aes_x86_64 drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd psmouse drm ixgbevf pata_acpi floppy
[21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu
[21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015
[21410.264054] task: ffff8803f9809b80 ti: ffff8803f999c000 task.ti: ffff8803f999c000
[21410.264054] RIP: 0010:[<ffffffff810e5c36>] [<ffffffff810e5c36>] run_timer_softirq+0x116/0x2d0
[21410.264054] RSP: 0000:ffff8803ff203e98 EFLAGS: 00010086
[21410.264054] RAX: dead000000200200 RBX: ffff8803ff20e9c0 RCX: ffff8803ff203ec8
[21410.264054] RDX: ffff8803ff203ec8 RSI: 0000000000011fc0 RDI: ffff8803ff20e9c0
[21410.264054] RBP: ffff8803ff203f08 R08: 000000000000a77a R09: 0000000000000000
[21410.264054] R10: 0000000000000020 R11: 0000000000000004 R12: 000000000000007c
[21410.264054] R13: ffffffff8172aaf0 R14: 0000000000000000 R15: ffff8803af955be0
[21410.264054] FS: 00007fb0ce6e8780(0000) GS:ffff8803ff200000(0000) knlGS:0000000000000000
[21410.264054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[21410.264054] CR2: 00007fb0ce51e130 CR3: 00000003fb233000 CR4: 00000000001406f0
[21410.264054] Stack:
[21410.264054] ffff8803ff203eb8 ffff8803ff20f5f8 ffff8803ff20f3f8 ffff8803ff20f1f8
[21410.264054] ffff8803ff20e9f8 ffff8803af955b58 dead000000200200 00000000f60fabc0
[21410.264054] 0000000000011fc0 0000000000000001 ffffffff81c0b0c8 0000000000000001
[21410.264054] Call Trace:
[21410.264054] <IRQ>
[21410.264054] [<ffffffff8107f846>] __do_softirq+0xf6/0x250
[21410.264054] [<ffffffff8107fb13>] irq_exit+0xa3/0xb0
[21410.264054] [<ffffffff814a4499>] xen_evtchn_do_upcall+0x39/0x50
[21410.264054] [<ffffffff817f1f6b>] xen_hvm_callback_vector+0x6b/0x70
[21410.264054] <EOI>
[21410.264054] Code: 81 e6 00 00 20 00 48 85 d2 48 89 45 b8 0f 85 30 01 00 00 4c 89 7b 08 0f 1f 44 00 00 49 8b 07 49 8b 57 08 48 85 c0 48 89 02 74 04 <48> 89 50 08 41 f6 47 2a 10 48 b8 00 02 20 00 00 00 ad de 49 c7
[21410.264054] RIP [<ffffffff810e5c36>] run_timer_softirq+0x116/0x2d0
[21410.264054] RSP <ffff8803ff203e98>
Edit 3
Here's the Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345
kdumpper help.ubuntu.com/stable/serverguide/kernel-crash-dump.html on 4/10 machines, so hopefully I can get a meaningful crash report today.