I recently noticed a server performing apt-get update && apt-get dist-upgrade

Amongst other things, docker was updated and restarted.

However, I noticed some changes I made to the service file previously were now gone. The file at /lib/systemd/system/docker.service seems to have been reset.

Is this possible? I don't recall if I ran daemon-reload after these changes.

EDIT: To be clear: I'm wondering is dist-upgrade basically removed the service file and replaced it with a default one?

And does daemon-reload prevent this in the future?

EDIT2: Ok, looks like sudo systemctl edit docker.service is the way to go and performs a daemon-reload on save.

  • 6
    Does this answer your question? How do I override or configure systemd services?
    – muru
    Jun 25 at 11:13
  • @muru No, my question is more "did dist-upgrade forcibly replace the entire service file with a new one, ignoring the existing one?"
    – KdgDev
    Jun 25 at 12:39
  • 2
    @muru Ah, I read a bit further and it does detail not to edit /lib/systemd/ files. Yeah this is going to be it.
    – KdgDev
    Jun 25 at 12:43

Don't edit files in /lib/systemd/ or /usr/share/systemd as they will get overwritten on updates.

Instead, copy the file to /etc/systemd/ and make the changes there.

The /etc/ directory (at least for systemd) is considered the place to store local config files. All other /**/systemd/ directories are considered sources of default and sample config files that should be replaced on any upgrade.

Another reason to not edit these package supplied config files is that if you copy it to /etc/systemd/... and edit it and make a mistake, you can always compare with the original file.

systemctl daemon-reload doesn't prevent anything. It just tells systemd to re-examine all of its config and use whatever has changed.

  • think /usr/local/lib/systemd/system/ is better than /etc/systemd... 1
    – bac0n
    Jun 25 at 13:11
  • @bac0n the systemd-sanctioned location is /etc/systemd. That’s what systemctl edit uses. Jun 25 at 20:51
  • I wouldn't edit the one in /usr/local/lib/systemd either, I'd copy that to /etc/systemd as well before editing.
    – user10489
    Jun 25 at 22:20
  • @StephenKitt, what you suggest is something completely different and deserves its own answer and even be the preferred one, still, I think, copying files to /etc/systemd/ is not recommended, e.g., you lose the ability to mask units.
    – bac0n
    Jun 26 at 6:20
  • 3
    @bac0n it’s already covered in muru’s answer here, no point in duplicating it IMO ;-). Jun 26 at 7:38

There are some drawbacks of storing regular files in /etc/systemd/system, not because of the systemd itself, but because systemctl resides in this location. Placing regular files in this directory will break some of the functionality of systemctl, in this case, the ability to mask your .service, and there is no reason to believe other applications will handle this any differently. Now, systemd has a predefined set of unit search paths, most of which are preoccupied with the distribution, this makes locations where you can place your .service basically limited to (or at least until this is resolved):


This work exceptionally well and without loss in functionality:

# cp -a hello-world.service /usr/local/lib/systemd/system
'hello-world.service' -> '/usr/local/lib/systemd/system/hello-world.service'

# systemctl daemon-reload
# dpkg -i hello-world_1.0-1_all.deb 
Selecting previously unselected package hello-world.
(Reading database ... 396452 files and directories currently installed.)
Preparing to unpack hello-world_1.0-1_all.deb ...
Unpacking hello-world (1.0) ...
Setting up hello-world (1.0) ...
Created symlink /etc/systemd/system/multi-user.target.wants/hello-world.service → /usr/local/lib/systemd/system/hello-world.service.

# systemctl mask hello-world
Created symlink /etc/systemd/system/hello-world.service → /dev/null.

the same chronological order applies to drop-ins as well, where /etc take precedence over /run which in turn take precedence over /lib ... and so on, drop-ins with different names will be applied in lexicographic order regardless of location. If you have overlapping directives, then the last will take precedence:

: systemctl cat hello-world
# /lib/systemd/system/hello-world.service
Description=Hello world (lib).

ExecStart=/opt/bin/hello.sh lib


# /usr/local/lib/systemd/system/hello-world.service.d/10-local.conf
Description=Hello world (local).

ExecStart=/opt/bin/hello.sh local

# /etc/systemd/system/hello-world.service.d/override.conf
ExecStart=/opt/bin/hello.sh etc

: systemctl start hello-world
jun 28 15:20:24 betazoid systemd[1]: Starting Hello world (local)....
jun 28 15:20:24 betazoid hello[402381]: hello etc
jun 28 15:20:24 betazoid systemd[1]: hello-world.service: Succeeded.
jun 28 15:20:24 betazoid systemd[1]: Finished Hello world (local)..
  • According to 0pointer.de/blog/projects/three-levels-of-off systemd supports "three levels of off": service stopped, service disabled, service masked. Masked is for when you never want to start the service. It makes no sense to have a customized file for a service you never want to start. Your solution is just trying to have its cake and eat it too.
    – user10489
    Jun 30 at 1:15
  • The /usr/local/lib/systemd/system/ directory is the target for hand installed programs to install their services. While a normal system upgrade won't overwrite this, rerunning make install for programs that have service will overwrite that. The correct directory for custom service files for the administrator to install is /etc/system/systemd
    – user10489
    Jun 30 at 1:16
  • @user10489 create a service and put it in /etc/systemd/system and systemctl mask your.service what happens?
    – bac0n
    Jun 30 at 2:03
  • What happens if you type systemctl edit --full your.service ? This is what systemd intends.
    – user10489
    Jun 30 at 4:41
  • Yes, this will give roughly the same headache, I can understand that the file has to start somewhere, so I forgive them this time ;-)
    – bac0n
    Jun 30 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.