How to get coredump from process crashed inside a docker container using apport?

Question

We have a bunch of microservices, published as docker images, and run them on-premise for local testing and in AKS k8s cluster. We use ubuntu 16.04, because currently AKS uses it.

We would like to automatically collect coredump when our microservice crashes inside the docker container. We have successfully achieve it using instructions on local environment, but we would like to use ubuntu apport interceptor to collect CoreDump.

But we unable to collect dump because of the following error:

ERROR: apport (pid 26994) Thu Apr 9 16:17:34 2020: host pid 26979 crashed in a container without apport support

docker run -d --privileged -v /var/crash:/var/crash ubuntu:16.04 /bin/sleep 50000
docker kill <CreatedContainerId>

After looking at what apport is doing it seems that we have to install apport inside the container. Not a problem, create a Dockerfile with apport installation:

FROM ubuntu:16.04

RUN apt-get update && apt-get install -y --no-install-recommends systemd apport && rm -rf /var/lib/apt/lists/*
RUN sed -i "s/enabled=0/enabled=1/g" /etc/default/apport
RUN sed -i "s/'problem_types'/#'problem_types'/g" /etc/apport/crashdb.conf
COPY ./super.sh /opt/super.sh
RUN ["chmod", "+x", "/opt/super.sh"]
WORKDIR /opt
ENTRYPOINT [ "/opt/super.sh" ]

where super.sh is:

#!/bin/bash
service apport start
sleep 50000

But these also leads to the same result. There must be an apport.socket entry under host /proc/<host_pid>/root/run/ directory, which probably tells apport that another instance of it is up and running inside the container. apport.socket is the activation socket, which will require systemd to be running inside the container.

This is not so good

Questions

1. Is there a way to enable core dump forwarding for application inside container to host machine using apport?
2. What should be done inside container to enable apport to correctly pass coredump collection?
3. Do we really require systemd to be installed inside the docker in order for this to work?

P.S.

We have achieved behavior we wanted when comment out some code blocks in /usr/share/apport/apport, introduced by

869366238 data/apport (Brian Murray 2017-11-20 08:46:52 -0800)

but this is not the solution we would like to accept.

Answer 1

For those who really interested, here is our findings.

In order for apport.socket to be listened, you have to install systemd first. In order for systemd works properly, it must be the first process in the container, so you have to change your ENTRYPOINT command to be /sbin/init. In order for apport to work properly inside the container, you have to install python3-systemd module which enables systemd activation and parameters propogation. In order to run your process, you have to write simple unit file which describes how to launch your application with systemd. For ubuntu 18.04 you also have to install init package to allow /sbin/init to be run.

If you use some other kind of other virtualization, like vmware in my case (which runs host ubuntu, on which docker containers are running), you also have to remove ConditionVirtualization=container from apport-forward.socket file, because otherwise systemd will not listen activation socket mentioned earlier.

Combining altogether, here what you can do in order to enable apport inside container to work properly.

Dockerfile

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-bionic
COPY dotnet.service /lib/systemd/system/
RUN apt-get update && apt-get install -y --no-install-recommends \
    systemd \
    init \
    apport \
    python3-systemd && rm -rf /var/lib/apt/lists/*
RUN sed -i "s/enabled=0/enabled=1/g" /etc/default/apport && \
    sed -i "s/'problem_types'/#'problem_types'/g" /etc/apport/crashdb.conf && \
    sed -i "s/ConditionVirtualization=container//g" /lib/systemd/system/apport-forward.socket
RUN systemctl enable apport-forward.socket && \
    systemctl enable dotnet.service

# our app
COPY . /opt/app
WORKDIR /opt/app
EXPOSE 5000
ENTRYPOINT [ "/sbin/init" ]

dotnet.service

[Unit]
Description=My service

[Service]
Type=oneshot
ExecStart=/opt/app/MyServiceElf

[Install]
WantedBy=multi-user.target

all this adds additional ~65MB of data, which is looks crazy, so we forward this question to apport repository