13

When unattended-upgrades is installed, 9 out of 10 shutdowns/reboots hang while "starting unattended upgrades shutdown". This hang stalls the shutdown process for 5-10 mins.

If I disable unnattended-upgrades via the /etc/apt/apt.conf.d/20auto-upgrades and/or 50unattended-upgrades, the problems occurs.

If I terminate the service before shutdown/reboot (sudo service unattended-upgrades stop) the problem still occurs.

If I remove the package (sudo apt remove unattended-upgrades) the problem no longer occurs.

This occurs on a freshly installed version of Ubuntu Server 16.04.1 (both unattended-upgrades installed via install GUI or manual install of unattended-upgrades)

Both Kern.log & syslog do not show the shutdown process (I believe because the filesystems have already unmounted)

Has anyone else seen or fixed this issue? Going crazy trying to troubleshoot it.

  • Unable to reproduce in a 16.04.1 test VM. Shutdown is not delayed here. – user535733 Feb 1 '17 at 12:36
  • Could it be hardware based? I'm not entirely sure what happens with unattended-upgrades during shutdown. – garullon245136 Feb 1 '17 at 12:38
  • I'm wondering why u-u is always running at the time of shutdown: u-u is not a daemon; it's merely a script that runs briefly once each day and then terminates. – user535733 Feb 1 '17 at 15:55
  • It seems as though the shutdown process tries to run u-u during the stage where all of the filesystems are being dismounted.This doesn't seem to be controlled by the /etc/rc6.d/ or /etc/rc0.d/ links as I have removed all of the links and the process still runs during shutdown. – garullon245136 Feb 3 '17 at 3:41
  • 1
    Look in /etc/apt/apt/conf.d/50unattended-upgrades for the 'run u-u at shutdown' option (around line 25). Ensure it is 'false' or commented out. – user535733 Feb 3 '17 at 13:21
14

Looking around to get closer to the root cause

The problem seems to be the script running at shutdown.

I identified the corresponding file with:

find /etc/systemd -name *unattended*

which gaves me the related systemd script:

/etc/systemd/system/shutdown.target.wants/unattended-upgrades.service

which then told me the script executed on shutdown:

/usr/share/unattended-upgrades/unattended-upgrade-shutdown

Investigating deeper to find the root cause

within this script there is a section in line 120 related to the section in /etc/apt/apt.conf.d/50unattended-upgrades -> Unattended-Upgrade::InstallOnShutdown

Line 120 of /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):

The problem: it expects the keyword "False" while in the apt conf we should add "false" (exact string comparison)!

Solution

I was able to fix/workaround the stalling shutdown in 3 different ways:

Workaround A

  • write "False" instead of "false" in /etc/apt/apt.conf.d/50unattended-upgrades

This setting is upgrade safe until a real fix is provided because the file we change here gets not overwritten by an update of unattended-upgrades. Problem: When the root cause gets fixed this will result in a stalling shutdown again so I suggest to combine this with Workaround B.

OR: Workaround B

  • decrease the wait time in /etc/systemd/system/shutdown.target.wants/unattended-upgrades.service from default to 15 seconds:

vim /etc/systemd/system/shutdown.target.wants/unattended-upgrades.service

[Service]
Type=oneshot
ExecStart=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStartSec=15

This setting is NOT upgrade safe because the file we change here may get overwritten by an update of unattended-upgrades. Besides this it is really far away from fixing something but it will ensure that your system will not wait several minutes when shutting down. Keep in mind that after an upgrade of unattended-upgrades you may have to set this again!

OR: Fix C (have to be reported upstream)

  • fix /usr/share/unattended-upgrades/unattended-upgrades-shutdown to expect "false" instead of "False"

patching /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

--- /tmp/unattended-upgrade-shutdown    2017-02-03 14:53:03.238103238 +0100
+++ /tmp/unattended-upgrade-shutdown_fix    2017-02-03 14:53:17.685589001 +0100
@@ -117,7 +117,7 @@
     # run it
     p = None
     apt_pkg.init_config()
-    if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):
+    if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", false):
         env = copy.copy(os.environ)
         env["UNATTENDED_UPGRADES_FORCE_INSTALL_ON_SHUTDOWN"] = "1"
         logging.debug("starting unattended-upgrades in shutdown mode")

Conclusion

tbh only the last one is a real fix. the both other options are just workarounds until the real fix would be implemented.

This has to be done upstream and as this affects both Debian (tested on Debian Stretch) and Ubuntu (tested on Ubuntu 16.04.1) for both distributions.

I have opened a bug report here: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611

  • 2
    apt_pkg.config.find_b() returns a boolean not a string. "find_b(key[, default=False]) → bool Return the boolean value stored at key, or the value given by the bool object default if the requested option is not set." apt.alioth.debian.org/python-apt-doc/library/… So it's not a bug in unattended-upgrade-shutdown as checking for False is correct. – Brian Murray Feb 7 '17 at 21:51
  • As anyone who follows the above Launchpad bug report link will see, this issue was not in fact caused by a failing string comparison, but instead was due to a systemd sequencing bug that has reportedly now been fixed. – sampablokuper Dec 21 '17 at 22:49
1

As an workaround I use this script to fix it:

#!/usr/bin/env bash

sed -i '/if apt_pkg.config.find_b/s/False/false/' /usr/share/unattended-upgrades/unattended-upgrade-shutdown

exit

Hopefully it will be soon upstreamed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.