prevent systemd service starting in recovery-mode

Question

I have a systemd service that has:

[Unit]

After=network-online.target krb5-kdc.service krb5-admin-server.service

Wants=krb5-kdc.service network-online.target krb5-admin-server.service

The service should not start unless the networking stack is fully ready / fully available, and the Kerberos services have started. (That is what I want to achieve, and when both of these are not true then my service should not try to start).

My unit file seems to work fine for regular use.

On reboot, from grub2 menu , if I choose "*Advanced options for Ubuntu" and then choose "recovery mode" for my kernel, and then open a root shell, and do not run 'Enable Networking', I can see that my service tries to start (and it subsequently fails because it needs the fully operating network, along with the kerberos services).

What must I change so that the service will not start unless the networking stack is fully available and the kerberos services are started?

In recovery-mode root shell (without "Enable networking"):

systemctl list-units --type target

UNIT                     LOAD   ACTIVE SUB    DESCRIPTION
basic.target             loaded active active Basic System
cryptsetup.target        loaded active active Encrypted Volumes
friendly-recovery.target loaded active active Friendly Recovery Mode
local-fs-pre.target      loaded active active Local File Systems (Pre)
local-fs.target          loaded active active Local File Systems
network-online.target    loaded active active Network is Online
network-pre.target       loaded active active Network (Pre)
network.target           loaded active active Network
paths.target             loaded active active Paths
slices.target            loaded active active Slices
sockets.target           loaded active active Sockets
sound.target             loaded active active Sound Card
swap.target              loaded active active Swap
sysinit.target           loaded active active System Initialization
time-sync.target         loaded active active System Time Synchronized
timers.target            loaded active active Timers

In the above list, I do not see any kerberos services, and I don't know why the target network.online is active, it must mean something different from my assumptions.

I also see on the console during boot into recovery mode:

[ DEPEND ] Dependency failed for  Network Manager wait online
...
Reached Target Network is online

However, the network is not operating in the normal way: I cannot ping the static IP4 address of this hostname (result: Network is unreachable). I cannot reference any other computer on my local network, although I can ping 127.0.0.1 . So the networking is not fully operating and that is what I want to avoid, i.e. I want my service to only start when the network is fully operating.

Seems I have not understood what the target network-online really means?

When my service tries to start, it will timeout because the relevant services are not available, so the systemctl status myservice.service shows

● myservice.service - description
   Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: timeout) since Fri 2020-07-24 16:40:45 BST; 7s ago
  Process: 1396 ExecStartPre=/usr/local/bin/get_kerberos_tickets.ksh (code=killed, signal=HUP)
    Tasks: 0
   Memory: 3.4M
      CPU: 529ms

Jul 24 16:40:45 t7810 systemd[1]: myservice.service: Unit entered failed state.
Jul 24 16:40:45 t7810 systemd[1]: myservice.service: Failed with result 'timeout'.

If I modify my unit file to add multi-user.target to the WANTS and AFTER lines, then my service also tries to start, even though the root shell then shows systemctl list-units -target has multi-user.target "loaded inactive dead".

Answer 1

The recovery mode on your systems loads network-online.target.

Run the command

systemctl list-units --type target

to list the targets loaded in recovery mode.

The line

network-online.target    loaded active active Network is Online

shows that networks' services are loaded. Hence your system behaves as expected.

An idea is to use Conflicts, from systemd.units(7)

Conflicts=
       A space-separated list of unit names. Configures negative requirement dependencies. If a unit has a
       Conflicts= setting on another unit, starting the former will stop the latter and vice versa.

       Note that this setting does not imply an ordering dependency, similarly to the Wants= and Requires=
       dependencies described above. This means that to ensure that the conflicting unit is stopped before the
       other unit is started, an After= or Before= dependency must be declared. It doesn't matter which of the
       two ordering dependencies is used, because stop jobs are always ordered before start jobs, see the
       discussion in Before=/After= below.

       If unit A that conflicts with unit B is scheduled to be started at the same time as B, the transaction
       will either fail (in case both are required parts of the transaction) or be modified to be fixed (in case
       one or both jobs are not a required part of the transaction). In the latter case, the job that is not
       required will be removed, or in case both are not required, the unit that conflicts will be started and
       the unit that is conflicted is stopped.

When entering the recovery mode in grub, systemd starts to reach the friendly-recovery.target. Hence, try to set this statement in your service unit:

Conflicts=friendly-recovery.target

which should prevent to start your service while in recovery mode.