Various segfaults in syslog; firefox, apt-get and other apps broken

Question

I am working on an Ubuntu 12.04 installation that has recently started misbehaving. It is used by multiple people at an office, so it's possible someone messed things up (although the only user who is an administrator is pretty smart and keeps things updated). In any case I don't know how it ended up in this state.

The problem is that Firefox, Update Manager, and possibly other programs no longer work. It boots up to a normal-looking desktop. Thunderbird and LibreOffice seem to work fine and I have internet connectivity, so the system isn't completely broken.

I looked at /var/log/syslog and see a bunch of segmentation faults. I'm puzzled by the pattern of broken libraries... they seem unrelated, and I am at a loss to explain what happened. I'll continue to investigate but if anyone has any ideas, I'd love to hear them!

Note that libapt-pkg.so is broken, so I can't seem to get the system to clean itself up. I removed *.bin from /var/cache/apt as mentioned elsewhere, but that didn't help.

Here is /var/log/syslog:

Mar 17 15:52:53 hostname rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="493" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Mar 17 15:52:54 hostname rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="493" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Mar 17 15:53:18 hostname anacron[895]: Job `cron.daily' terminated
Mar 17 15:53:18 hostname anacron[895]: Normal exit (1 job run)
Mar 17 15:53:55 hostname kernel: [  895.453234] apt-get[2737]: segfault at bfec00f7 ip 0061b1c4 sp bfa05710 error 4 in libapt-pkg.so.4.12.0[5bf000+124000]
Mar 17 15:54:28 hostname kernel: [  928.805036] apt-get[2973]: segfault at bffdbdf7 ip 00c9d1c4 sp bfeda2c0 error 4 in libapt-pkg.so.4.12.0[c41000+124000]
Mar 17 15:54:30 hostname kernel: [  931.007931] apt-get[2978]: segfault at bfee85f7 ip 00e201c4 sp bf823c40 error 4 in libapt-pkg.so.4.12.0[dc4000+124000]
Mar 17 15:54:39 hostname kernel: [  940.135040] apt-get[2991]: segfault at bfd3e5f7 ip 00c941c4 sp bfd1dc80 error 4 in libapt-pkg.so.4.12.0[c38000+124000]
Mar 17 15:54:43 hostname kernel: [  943.368954] apt-get[2996]: segfault at bfff30f7 ip 0058e1c4 sp bfbf2520 error 4 in libapt-pkg.so.4.12.0[532000+124000]
Mar 17 15:54:47 hostname kernel: [  948.090732] apt-get[3002]: segfault at bfdfe0f7 ip 008701c4 sp bfde9720 error 4 in libapt-pkg.so.4.12.0[814000+124000]
Mar 17 15:55:34 hostname kernel: [  994.745018] firefox[3034] general protection ip:284718 sp:bfdf9999 error:0 in libstdc++.so.6.0.16[25a000+d8000]
Mar 17 15:56:05 hostname kernel: [ 1026.148231] apt-get[3038]: segfault at ac39d81f ip 0016c20c sp bfe3e220 error 4 in libapt-pkg.so.4.12.0[110000+124000]
Mar 17 15:58:34 hostname kernel: [ 1175.049420] firefox[3057]: segfault at 0 ip 0028472f sp bf91badc error 6 in libc-2.15.so[12e000+1a3000]
Mar 17 15:59:00 hostname dbus[467]: [system] Activating service name='org.debian.apt' (using servicehelper)
Mar 17 15:59:00 hostname AptDaemon: INFO: Initializing daemon
Mar 17 15:59:00 hostname dbus[467]: [system] Successfully activated service 'org.debian.apt'
Mar 17 15:59:01 hostname AptDaemon.PackageKit: INFO: Initializing PackageKit compat layer
Mar 17 15:59:01 hostname kernel: [ 1201.659743] software-center[3060]: segfault at bfb5f8f7 ip 097ab1c4 sp bfb11d60 error 4 in libapt-pkg.so.4.12.0[974f000+124000]
Mar 17 16:00:45 hostname kernel: [ 1305.806837] firefox[3092]: segfault at 459c84eb ip 01ee43f1 sp bfb6ebbc error 4 in libxul.so[fa8000+1d11000]
Mar 17 16:01:26 hostname kernel: [ 1346.971439] firefox[3122] general protection ip:284706 sp:bfc543ec error:0 in libc-2.15.so[1f5000+1a3000]
Mar 17 16:01:30 hostname kernel: [ 1351.221917] firefox[3127]: segfault at 9fffe7ad ip 00284716 sp bf86b0bc error 6 in libpangoft2-1.0.so.0.3000.0[283000+2a000]
Mar 17 16:04:01 hostname AptDaemon: INFO: Quitting due to inactivity
Mar 17 16:04:01 hostname AptDaemon: INFO: Quitting was requested

I'm hoping I don't need to reinstall from scratch.

Answer 1

When a machine starts to misbehave like this, it can be a RAM chip gone bad, causing a few bits to flip. Given the symptoms, it's possible that some library files were corrupted when they were installed.

Reboot and run a memory test. Let it run for at least one full pass (keep it running over lunch or overnight). If any of the RAM chips is faulty, change it immediately.

If you can compare the library files on your system with a clean installation, single-bit errors always at the same bit position are a sure sign of faulty RAM (but the error can follow other patterns as well).

If you've identified bad RAM, verify every file on your system against good versions, including all your data. Beware that backups may themselves have become corrupted. Don't attempt restoring anything until you know that the RAM is good.

Answer 2

You can try to reinstall manually some of the packages that provide those files (libc6 and libstdc++6), but I doubt it would help. Those segfaults should have other origin.

Download the debs from here and here. Then install each using dpkg:

sudo dpkg -i /path/to/each/file.deb

reboot and check if you get less errors. If so you can try to reproduce the same procedure to each file (use apt-file to find which package provide each file).

Also if you know when the system started to misbehave you can try to check /var/log/apt/history.log and see if there was any update/package installation that could be causing this.

Hope it helps.