25

I have configured unattended-upgrades to install security packages and notify by mail when it does so.

I have noticed that the installation happens at very random times. I know that the latest versions added a random delay up to 30 minutes starting from the cron.daily execution time.

However the delays I am experiencing are much bigger than that. I see unattended-upgrades executing at 9am, 3pm, 12am... The logs show the same, so it is not just the email delivery that takes longer.

The unattended upgrades task is the first one in cron.daily, meaning that there is no previous task with huge execution times.

Anybody experienced a similar thing?

2
  • The random behavior is deliberate - to smooth out demand instead of millions of systems hammering a few mirrors at the same times each day. Ordinary desktop users should not notice the behavior at all. Some enterprise users way wish to change the behavior to something a bit more predictable, and are certainly welcome to do so.
    – user535733
    Commented Feb 5, 2019 at 13:50
  • Yes, the reason behind this choice is clear. It is just that this behavior is unacceptable for production systems. At the time i asked this question, this behavior (and the fix) was not documented anywhere
    – daniel f.
    Commented Feb 5, 2019 at 14:06

5 Answers 5

35

After debugging this I found the solution.

The root cause of this issue resides in the fact that under Ubuntu 16.04 and newer, unattended-upgrades uses systemd - not cron - to schedule the updates with a huge randomized delay:

/lib/systemd/system/apt-daily.timer is configured with

OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h

This means that it will run twice a day, at 6:00 and 18:00, with a random delay of up to 12 hours. As this is not always acceptable for production environments, I had to override these settings.

In order to keep the package config files untouched, I defined my override in /etc/systemd/system/apt-daily.timer.d/override.conf (Update: please read the edit at the bottom of this answer for further information on filename and location, as it seems to be slightly subject to change).

There I set

[Timer]
OnCalendar=
OnCalendar=06:00
RandomizedDelaySec=1h

to have unattended-upgrades run at 6:00 plus a random delay of up to an hour.

Then I simply restarted the timer with systemctl restart apt-daily.timer (eventually need to reload the daemon).

Unattended-updates now runs at predictable times again!

Edit: It would seem as if for Ubuntu 18.04 things changed a bit. The override should now be stored in /etc/systemd/system/apt-daily-upgrade.timer.d/override.conf and look like this:

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=1h

@PerlDuck has mentioned a way of creating an override file with the right name and location in a comment below. Instead of manually creating a file, please consider running sudo systemctl edit apt-daily.timer

8
  • 1
    Why do you clear OnCalendar first?
    – jarno
    Commented Dec 14, 2016 at 10:15
  • 3
    Because otherwise it would only ADD a new timer at 6am, leaving the existing one as well. Since I want unattended-upgrades to ONLY run at 6, I need to clear the schedule first.
    – daniel f.
    Commented Dec 14, 2016 at 10:19
  • I added "OnBootSec=5min" to enable running after boot as well, but it did not work. (also added OnUnitActiveSec=12h so that it is not run too frequently.)
    – jarno
    Commented Dec 14, 2016 at 15:09
  • systemd, or rather, the systemd mindset, strikes again. I may have to rethink upgrading to Xenial Xerus in production after this gem bit me.
    – Joe
    Commented Jan 11, 2017 at 11:30
  • 2
    @daniel f. I have an apt-daily.timer file in /lib/systemd/system/ as well as one in /etc/systemd/system/timers.target.wants/ However I don't have any in /etc/systemd/system/ itself like you. Would you know if I should be creating the apt-daily.timer.d directory and the override.conf under /lib/systemd/system instead? All advice gratefully recd with thx.
    – Purvez
    Commented Apr 27, 2017 at 10:24
13

The best way to update the unattended upgrade time, as I compiled it from various sources and tested on our system, is to exclusively use the systemctl commands and avoid trying to find the proper files to edit.

The only thing you have to know for sure is the service name, which in our case, is apt-daily-upgrade (if unsure, search for it via $ systemctl | grep apt). When a systemd service has a timer defined, it is referenced as #{service_name}.timer, thus it’s apt-daily-upgrade.timer for us.

As the system configuration should not be edited, we’ll have to override the default timer config in systemd. For this you need to copy-edit some parts of the original config, so let’s show it first:

$ systemctl cat apt-daily-upgrade.timer 
# /lib/systemd/system/apt-daily-upgrade.timer
[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=12h
Persistent=true

[Install]
WantedBy=timers.target

We’ll need to update the OnCalendar and RandomizedDelaySec values in the [Timer] section. Let’s create the override config file via the following command:

$ systemctl edit apt-daily-upgrade.timer

This should open an editor with a blank file and we need to put there the amended [Timer] section, at least:

[Timer]
# Reset the system calendar config first
OnCalendar=
# Set a new calendar timer with a 60 minute threshold
OnCalendar=*-*-* 21:00
RandomizedDelaySec=60m

As you can see, we’ve updated the OnCalendar value to trigger the automatic updates in the evenings, instead of mornings. The blank OnCalendar line above it must be present as this config value is additive, i.e. it may be specified more than once and only setting it to a blank value resets all previous OnCalendar values (the ones from the system config).

Once we save the file, we can verify that systemd knows about it (there’s no need to run systemctl daemon-reload, the edit command does that for us upon leaving the editor) by running systemctl again as above:

$ systemctl cat apt-daily-upgrade.timer

# /lib/systemd/system/apt-daily-upgrade.timer
[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=12h
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/apt-daily-upgrade.timer.d/override.conf
[Timer]
# Reset the system calendar config first
OnCalendar=
OnCalendar=*-*-* 21:00
RandomizedDelaySec=60m

Now it shows two configurations, with our custom one overriding the default one. Good!

The final check that it all works as expected can be done via the list-timers command of systemctl:

$ systemctl list-timers
NEXT                        LEFT          LAST                        PASSED       UNIT                         ACTIVATES                     
...
Thu 2020-08-06 21:51:36 UTC 12h left      Thu 2020-08-06 07:10:20 UTC 2h 20min ago apt-daily-upgrade.timer      apt-daily-upgrade.service               
...             

Find the proper line in the output and look at the NEXT column - the value in there should reflect the time of your newly configured unattended upgrade.

1
  • nice answer and explanation ! Thank you ! Any idea what if we should do this also for apt-daily.service ? Commented Nov 28, 2021 at 20:35
10

The official debian documentation on https://wiki.debian.org/UnattendedUpgrades currently has a mistake in it that is misleading a lot of people. It claims that you can override the upgrade time by creating a file called

/etc/systemd/system/apt-daily-upgrade.d/override.conf

However the correct path is

/etc/systemd/system/apt-daily-upgrade.timer.d/override.conf
6
  • 3
    Nice finding. IMHO the safest thing is to use sudo systemctl edit apt-daily.timer. This will open an editor with the correct drop-in file.
    – PerlDuck
    Commented Feb 5, 2019 at 10:39
  • 2
    Thank you PerlDuck, I edited the Debian Wiki page with your suggestion Commented Feb 5, 2019 at 12:45
  • Upvoted this answer for being useful – and for updating the Debian wiki page! Commented Feb 27, 2020 at 21:16
  • @PerlDuck - Under 18.04LTS, I just ran sudo systemctl edit apt-daily.timer. It created /etc/systemd/system/apt-daily.timer.d/override.conf... Why would it be so? I will check if it works properly. Commented Apr 13, 2020 at 11:04
  • @sancho.sReinstateMonicaCellio Maybe you upgraded your 18.04 from a previous version instead of doing a fresh install. I reckon they didn't change the location of the unit files during an update but only use the new location (/lib/… instead of /etc/…) in a fresh install. Just a guess, though. Still: if systemctl edit … thinks /etc/… is the correct path, then you can trust it. I wouldn't bother. You can also try systemctl cat … to see the current path (and content) of that unit. (BTW: I'm not the author of this answer, I just fixed some formatting.)
    – PerlDuck
    Commented Apr 14, 2020 at 11:22
8

I tried Daniel's solution but the upgrade still ran at incorrect times. Turned out there are two systemd overrides needed:

Used for downloads

/lib/systemd/system/apt-daily.timer

...that is overridden with:

/etc/systemd/system/apt-daily.timer.d/override.conf

Used for upgrading

/lib/systemd/system/apt-daily-upgrade.timer

...that is overridden with:

/etc/systemd/system/apt-daily-upgrade.timer.d/override.conf
2
  • 1
    Which version are you on? So you had to override both mentioned timers at the same time?
    – daniel f.
    Commented Jan 23, 2019 at 7:48
  • Ubuntu 16.04.4 x64
    – Niels Rask
    Commented Oct 19, 2019 at 15:11
0

The apt-daily.timer is set to run twice a day, and will not interrupt a production environment, as it simply updates and downloads the packages that need to be updated. You should only need to override apt-daily-upgrade.timer to a time that would be less likely to impact your production environment as this is when the updates are actually installed. Further, there is a setting to reboot for things that require it in 50unattended-upgrades that you should consider setting so they actually get applied and that also has a timer. Also, as of 22.04 the upgrade timer has been changed to default 6am + 1hr, so most will not need to change this.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .