50

php-fpm crashed on us and dumped a file in

/var/crash/_usr_sbin_php5-fpm.1002.crash

There is some info in that file but what I'm after is in the section called CoreDump in a base64 encoded format. How can I read what was running at the time of the crash?

3 Answers 3

59

In case if you don't want to install a bunch of sub-dependencies for apport-retrace tool, you can unpack apport format into separate files and to use only CoreDump dump with gdb as usual.

  1. apport-unpack systemGeneratedCrashReportPath.crash yourNewUnpackDirectoryHere

  2. cd yourNewUnpackDirectoryHere/

  3. gdb `cat ExecutablePath` CoreDump (pay attention to backticks here!)

  4. bt (output actual back-trace)

    Note: apport-unpack will sometimes crash itself on unpack operation (apport seems broken all around... xD), but your CoreDump and other files will be there, just ignore it and delete all .crash files in /var/crash after you move them elsewhere in order to allow system to output new crash reports from same apps there.

6
  • 1
    Worked like a charm, without requiring to install apport-retrace, thanks!
    – greuze
    Oct 31, 2018 at 9:16
  • 1
    @digital_infinity no?! See it for yourself. There's a large difference between echo and cat...
    – stamster
    Feb 27, 2019 at 16:19
  • 2
    @stamster You are right. Sorry I missed that there is a file with name ExecutablePath. I thought the reader must fill the executable path there. Feb 28, 2019 at 8:27
  • 3
    Great answer. Just a slight correction (for other silly people who get hung up on details) that step 3 contains no tildes. I assume that you mean the backticks. Sep 27, 2021 at 1:02
  • 1
    I order to get debug symbols (for use in gdb), you can follow information in wiki.ubuntu.com/Debug%20Symbol%20Packages
    – iuridiniz
    Jun 25, 2022 at 13:31
17

There is a tool called apport-retrace that reads the .crash files and allows you to either fill it with a fully-symbolic stack trace or run a gdb session using the core dump. To start a gdb session, run apport-retrace -g CRASHFILE.crash. Note that you need to have the -dbg packages installed to get a good stack trace.

That being said (I'm not an expert on PHP), it might actually be something that you wrote in one of your files that is causing the crash.

4
  • 6
    Doesnt seem to work: # apport-retrace -g _usr_sbin_php5-fpm.1002.crash ERROR: report file does not contain one of the required fields: CoreDump DistroRelease Package ExecutablePath` # grep CoreDump _usr_sbin_php5-fpm.1002.crash CoreDump: base64
    – user76369
    Mar 17, 2014 at 3:26
  • 13
    Edit the crash file and add the field "Package: 0" under ExecutableTimestamp.
    – DarkNeuron
    Nov 9, 2016 at 9:28
  • See wiki.ubuntu.com/DebuggingProgramCrash for documentation Mar 18, 2020 at 7:04
  • Instead of manually editing the crash file, the --rebuild-package-info should also work as long as you are on the same system the crash happened, I think? I get errors because of missing deb-src URIs and can't be bothered to get it to work, I'll try the apport-unpack version.
    – mxmlnkn
    Dec 5, 2020 at 23:29
0

How to also obtain source code on GDB

Tested on Ubuntu 23.10, I managed to get GDB working with stamster's answer: https://askubuntu.com/a/947532/52975 however there was no source code visible for the packages. Since I had a recent crash I wanted to debug, I'll take this chance to document how to do it.

First set things up so that you are able to do a package debugging with source hello world and properly debug gdb ls with source as explained at: How to install debug symbols for installed packages?

Next, after doing:

gdb `cat ExecutablePath` CoreDump

this is what I saw:

>>> bt
#0  __pthread_kill_implementation (no_tid=0, signo=5, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=5, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=5) at ./nptl/pthread_kill.c:89
#3  0x00007f62b5842866 in __GI_raise (sig=5) at ../sysdeps/posix/raise.c:26
#4  0x0000562470b03bda in ??? ()
#5  0x00007f62b5842910 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007f62b62e5867 in g_log_structured_array () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#7  0x00007f62b62e5b52 in g_log_default_handler () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#8  0x00007f62b62e69e2 in g_logv () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#9  0x00007f62b62e6ca3 in g_log () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#10 0x00007f62b68ab59e in ??? () at /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0
#11 0x00007f62b5704a5b in _XError () at /lib/x86_64-linux-gnu/libX11.so.6
#12 0x00007f62b1f63adb in ??? () at /lib/x86_64-linux-gnu/libGLX.so.0
#13 0x00007f62b5f0962f in ??? () at /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-cogl-13.so.0

but no frame had source.

From the function names, the first possibly interesting frame was #10 0x00007f62b68ab59e in ??? () at /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0 so let's try to get its source code.

First I identify the package:

apt-file search /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0

which gives:

libmutter-13-0: /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0
libmutter-13-0: /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0.0.0

and so we obtain the debug symbols and source code with:

sudo apt install libmutter-13-0-dbgsym

but TODO hmmm that fails with:

The following packages have unmet dependencies:
 libmutter-13-0-dbgsym : Depends: libmutter-13-0 (= 45.2-0ubuntu3) but 45.2-0ubuntu2~really45.0 is to be installed
E: Unable to correct problems, you have held broken packages.

Edit: it later worked when I tried again some time later. Go figure.

OK let's try another interesting frame at #11 0x00007f62b5704a5b in _XError () at /lib/x86_64-linux-gnu/libX11.so.6:

apt-file search /lib/x86_64-linux-gnu/libX11.so.6

which gives:

libx11-6: /usr/lib/x86_64-linux-gnu/libX11.so.6
libx11-6: /usr/lib/x86_64-linux-gnu/libX11.so.6.4.0

and so we get the debug symbols with:

sudo apt install libx11-6-dbgsym

and the sources with:

cd /usr/src
sudo apt source libx11-6

Then on GDB:

frame 11

shows:

#11 0x00007f62b5704a5b in _XError (dpy=0x562471d040d0, rep=<optimized out>) at ../../src/XlibInt.c:1503
1503    ../../src/XlibInt.c: No such file or directory.

so we do:

set substitute-path ../.. /usr/src/libx11-1.8.6

and then:

list

gives us source:

1498            dpy->error_threads = &thread_info;
1499            if (dpy->lock)
1500                (*dpy->lock->user_lock_display)(dpy);
1501            UnlockDisplay(dpy);
1502    #endif
1503            rtn_val = (*_XErrorFunction)(dpy, (XErrorEvent *)&event); /* upcall */
1504    #ifdef XTHREADS
1505            LockDisplay(dpy);
1506            if (dpy->lock)
1507                (*dpy->lock->user_unlock_display)(dpy);

OK let's try another one on #12 0x00007f62b1f63adb in ??? () at /lib/x86_64-linux-gnu/libGLX.so.0:

apt-file search /lib/x86_64-linux-gnu/libGLX.so.0

gives:

libglx0: /usr/lib/x86_64-linux-gnu/libGLX.so.0
libglx0: /usr/lib/x86_64-linux-gnu/libGLX.so.0.0.0

so:

sudo apt install libglx0-dbgsym
cd /usr/src
sudo apt source libglx0

then in GDB:

frame 12

gives:

#12 0x00007f62b1f63adb in __glXSendError (dpy=0x562471d040d0, errorCode=<optimized out>, resourceID=0, minorCode=<optimized out>, coreX11error=0) at ../src/GLX/libglx.c:805
805     ../src/GLX/libglx.c: No such file or directory.

so:

set substitute-path .. /usr/src/libglvnd-1.6.0

and now list gives:

>>> list
800         error.majorCode = dpyInfo->glxMajorOpcode;
801         if (!coreX11error) {
802             error.errorCode += dpyInfo->glxFirstError;
803         }
804
805         _XError(dpy, &error);
806
807         UnlockDisplay(dpy);
808     }
809

OK, it worked for two packages, we clearly see that library calling the _XError from before!

This makes me wonder: TODO how to use a different substitute-path for each separate library? Because there will be conflicts.

Related:

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .