48

Some minutes ago my computer crashed because apparently my hard drive was full. After booting in recovery mode I then found out that my syslog file in /var/log was 64GB large. I saved the end of the file on another partition and then deleted it. Apparently docker seems to have been the problem because I found a lot of this at the end of the file and there was a docker process running at 200% CPU the whole time. After clearing the log and killing docker everything seems to be normal.

Nov 15 01:44:08 Elemental docker.dockerd[1120]: 
time="2019-11-15T01:44:08.727060251Z" level=error 
msg="failed to get event" error="rpc error: code = 
Unavailable desc = all SubConns are in TransientFailure, latest connection 
error: connection error: desc = \"transport: Error 
while dialing dial unix /run/containerd/containerd.sock: 
connect: permission denied\"" module=libcontainerd namespace=plugins.moby
Nov 15 01:44:08 Elemental docker.dockerd[1120]:     time="2019-11-15T01:44:08.727116701Z" 
...

And so on. I hope this problem won't appear again but still I would like to know what might have happened here.

  • I was getting disk full warnings and system was not logging in due to huge syslog file. I deleted the docker from snap to solve it. – ahmadalibaloch Nov 18 at 10:36
48

I had docker package installed through apt and snap at the same time. So the issue is fixed by removing docker through the snap packaging system.

# apt list --installed | grep docker   
docker/bionic,now 1.5-1build1 amd64 [installed]
docker-ce/bionic,now 5:19.03.5~3-0~ubuntu-bionic amd64 [installed]
docker-ce-cli/bionic,now 5:19.03.5~3-0~ubuntu-bionic amd64 [installed,automatic]

# snap list | grep docker
docker                18.09.9                     418   stable    canonical✓  -

# snap remove docker
docker removed
  • 8
    Likewise, I didn't realise I had docker installed via snap and had issues starting at 04:00Z today. I needed to run sudo snap remove docker --purge to fix it. Thanks for the response. – Belly Buster Nov 15 at 9:43
  • 2
    Odd that this happened to me at just about the same time as you! I wonder if some update got pushed - in any case, sudo snap remove docker --purge fixed my high CPU issues! – Try431 Nov 15 at 18:30
  • 2
    This hit me as well. docker auto-installed itself via snap earlier today, which generated 800GB of logs and filled up our root disk. WTF? – Chris Heald Nov 15 at 22:42
16

Someone at Cannonical screw things up few hours ago - it's tracked on launchpad - https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1852720

My system was installed yesterday with docker only and same as rest of folks here - syslog killed entire free space on disk, purging docker from snap helped.

  • 6
    What happened that allowed Cannonical the ability to remotely install packages. This is actually ridiculous and brought down quite a few of my own servers – Jaymes Bearden Nov 15 at 18:48
9

You have multiple docker services running. Looks like dockerd is trying to access the different installation of containerd, spamming syslog.

In my case, the teamcity agent and docker installation was conflicted.

sudo apt-get purge docker-ce
sudo apt autoremove
sudo rm -rf /var/lib/docker
sudo truncate -s 0 /var/log/syslog

Ran those commands above and restarted. everything went okay.

6

Happened to me too. I wonder if it was a snap updated that made it happen:

ID   Status  Spawn               Ready               Summary
17   Done    today at 00:34 UTC  today at 00:34 UTC  Auto-refresh snap "docker"

Uninstall+reinstall of Ubuntu's docker.io package brings back this problem, but for some reason I also had a snap of docker installed, so that's what triggered the conflict.

Name    Version    Rev   Tracking  Publisher   Notes
core    16-2.42.1  8039  stable    canonical✓  core
docker  18.09.9    418   stable    canonical✓  -

However, I don't remember that I ever installed the docker snap package... but this is a test-system after all, someone else might have thought that it was a good idea...

  • 1
    Thank you, my machine was freaking out this morning as a result of this. – Duncanmoo Nov 15 at 7:47
4

I ran into this problem too, so just in case other people wander in here I can describe the solution that worked for me. I first tried Chris Sung's suggestion but that did not work.

The symptoms were:

  1. dockerb running at 200+% CPU, and would constantly restart with a kill command

  2. Giant files being filled in syslog as Philip Z. saw. When I came in this morning it had filled up my entire hard drive with a 700GB file.

Frist remove the huge file so you can actually do stuff. It will start filling up again, but you should have some time.

sudo truncate -s 0 /var/log/syslog

Then remove the snap docker installation. This was the problem for me, not docker-ce

sudo snap stop docker
sudo snap remove docker

I'm not sure it's necessary, but I went ahead and got rid of snap altogether too

sudo apt purge snap

You should no longer see dockerb in top. You can then rerun the log truncator to remove any junk that was written while you were doing the above commands. If you screw up as I did and delete the syslog altogether, make sure that you give the new syslog the correct permissions.

sudo cd /var/log
sudo touch syslog
sudo chown syslog:adm syslog
sudo service rsyslog restart
2

Same problem here too since this night.

Docker containers (mosquitto/influxdb/grafana) are not accessible on their respective ports.

Unable to stop containers, until using --force.

Forced stop all containers but message on restarting each one is always the same :

"docker: Error response from daemon: all SubConns are in TransientFailure"

I did Silvio's package removal. Then, to restart docker daemon, I ran these commands as root:

systemctl unmask docker.service
systemctl unmask docker.socket
systemctl start docker.service
service docker start

Seems to be good now. Thanks to @Silvio.

1

I got low space warning on ubuntu.

The space is not enough for me to analyze which repeated log mess up/var/log

But the most recent logs are exact same as you.

I tried check which process write most to disk, but found docker's cpu % is over 250.

snap changes | grep docker

it seems docker snap installed by itself

54 Done yesterday at 23:31 EST yesterday at 23:31 EST Auto-refresh snap "docker

sudo snap remove docker --purge

should remove the docker snap and immediately release some space

Not the answer you're looking for? Browse other questions tagged or ask your own question.