0

I did apt-get update; apt upgrade on a remote server over SSH. netplan.io and some systemd packages were upgraded. I switched to a different screen window, but after just a few seconds the SSH connection stopped. All network connectivity is now lost.

This happened once before during an update of netplan.io. Then, a simple sudo ifdown (WAN-if); sudo ifup (WAN-if) on the physical console fixed the problem.

My server runs 18.04.1 LTS with blank configuration for netplan, blank configuration for systemd-networkd (which is the backend), with all interfaces statically configured in /etc/network/interfaces, and "ifupdown" installed. Reboots work just fine.

Could it be that netplan apply is only triggered on apt upgrade of netplan.io?

EDIT: These are the packages that were updated that killed the WAN interface, as copied from /var/log/apt/history.log:

Start-Date: 2019-01-15  12:43:10
Commandline: apt upgrade
Requested-By: user (1234)
Upgrade: libkrb5-3:amd64 (1.16-2build1, 1.16-2ubuntu0.1), libgssapi-krb5-2:amd64 (1.16-2build1, 1.16-2ubuntu0.1), netplan.io:amd64 (0.40.1~18.04.3, 0.40.1~18.04.4), libcom-err2:amd64 (1.44.1-1, 1.44.1-1ubuntu1), gnupg-utils:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), gpg-wks-client:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), python3-software-properties:amd64 (0.96.24.32.6, 0.96.24.32.7), gnupg-l10n:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), libibverbs1:amd64 (17.1-1, 17.1-1ubuntu0.1), libsystemd0:amd64 (237-3ubuntu10.9, 237-3ubuntu10.11), e2fsprogs:amd64 (1.44.1-1, 1.44.1-1ubuntu1), librbd1:amd64 (12.2.7-0ubuntu0.18.04.1, 12.2.8-0ubuntu0.18.04.1), ibverbs-providers:amd64 (17.1-1, 17.1-1ubuntu0.1), gpg-wks-server:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), gpg:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), libk5crypto3:amd64 (1.16-2build1, 1.16-2ubuntu0.1), udev:amd64 (237-3ubuntu10.9, 237-3ubuntu10.11), librdmacm1:amd64 (17.1-1, 17.1-1ubuntu0.1), initramfs-tools-bin:amd64 (0.130ubuntu3.5, 0.130ubuntu3.6), libudev1:amd64 (237-3ubuntu10.9, 237-3ubuntu10.11), krb5-locales:amd64 (1.16-2build1, 1.16-2ubuntu0.1), nplan:amd64 (0.40.1~18.04.3, 0.40.1~18.04.4), dirmngr:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), libss2:amd64 (1.44.1-1, 1.44.1-1ubuntu1), libext2fs2:amd64 (1.44.1-1, 1.44.1-1ubuntu1), psmisc:amd64 (23.1-1, 23.1-1ubuntu0.1), libkrb5support0:amd64 (1.16-2build1, 1.16-2ubuntu0.1), systemd-sysv:amd64 (237-3ubuntu10.9, 237-3ubuntu10.11), gpgv:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), libpam-systemd:amd64 (237-3ubuntu10.9, 237-3ubuntu10.11), systemd:amd64 (237-3ubuntu10.9, 237-3ubuntu10.11), libnss-systemd:amd64 (237-3ubuntu10.9, 237-3ubuntu10.11), libnss3:amd64 (2:3.35-2ubuntu2, 2:3.35-2ubuntu2.1), linux-firmware:amd64 (1.173.2, 1.173.3), gnupg:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), gpg-agent:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), librados2:amd64 (12.2.7-0ubuntu0.18.04.1, 12.2.8-0ubuntu0.18.04.1), initramfs-tools-core:amd64 (0.130ubuntu3.5, 0.130ubuntu3.6), gpgconf:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), initramfs-tools:amd64 (0.130ubuntu3.5, 0.130ubuntu3.6), gpgsm:amd64 (2.2.4-1ubuntu1.1, 2.2.4-1ubuntu1.2), tzdata:amd64 (2018g-0ubuntu0.18.04, 2018i-0ubuntu0.18.04), software-properties-common:amd64 (0.96.24.32.6, 0.96.24.32.7)
End-Date: 2019-01-15  12:45:04

The WAN interface configuration:

auto eno1
#iface eno1 inet dhcp
iface eno1 inet static
 address a.b.c.X
 netmask 255.255.255.NNN
 gateway a.b.c.Y
 dns-nameservers e.f.g.h i.j.k.l
 up ip addr add a.b.c.Z/BB dev eno1

The funny thing is that the LAN interface used for trunking, was unaffected (still up and running):

auto eno2
iface eno2 inet manual
 up ifconfig $IFACE 0.0.0.0 up
 down ifconfig $IFACE down

auto vlan3
iface vlan3 inet manual
 vlan-raw-device eno2
 up ifconfig $IFACE 0.0.0.0 up
 down ifconfig $IFACE down

auto br3
iface br3 inet static
 address a.b.m.n
 netmask 255.255.255.128
 up ip addr add a.b.m.n-1/25 dev br3
 down ip addr del a.b.m.n-1/25 dev br3
 bridge_ports vlan3
 bridge_stp off
Improve this question
6
  • It sounds like while it was running the upgrades it performed an ifupdown which caused the network to disconnect. The other issue with this is that it can end the upgrade process once your system disconnected so it may not bring the network backup automatically. Actual server hardware gets around this with remote serial consoles like BMC, iLO or LOM that are not affected by network updates. You will probably need to visit the remote system or have someone close to it check it or reboot it, etc. – Terrance Jan 16 '19 at 16:39
  • ifupdown could be a possible explanation as well. The first time this happened, it happened exactly as netplan.io was upgraded and configured, because that's where the output stopped. – perry Jan 17 '19 at 8:20
  • A comment in /etc/network/interfaces caused raise network interfaces to fail: 'iface br4 inet static # COMMENT'. After removing, raise network interfaces doesn't fail. The command was 'systemctl restart networking', and the output in journalctl -xe was 'ifup[30436]: /etc/network/interfaces:98: extra parameter for the iface line not understood and ignored: #' and 'systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE' and 'systemd[1]: networking.service: Failed with result 'exit-code'.' and 'systemd[1]: Failed to start Raise network interfaces.' – perry Jan 19 '19 at 8:46
  • It seems a little over the top that a comment on one bridge should cause a totally different interface to lose its configuration, but as I found dozens and dozens of log errors about interfaces:98 for just one run of restart networking, and the end result was FAILED, I just assume that the process stopped half-way through and that the WAN interface inconsistent state happened because it was the first interface to be reconfigured. – perry Jan 19 '19 at 8:56
  • Sounds like that is your answer. You should write that up separately and select it as the accepted answer. When running the updates remotely like that, when you lose the connection because of the cycle of the interface it will stop the whole upgrade process at that time as the session will end. Many times in my first dealings with remote server administration, myself and the other guys I had worked with had to run back to the office and bring it back up and start over. – Terrance Jan 19 '19 at 15:11
1

Conclusion: "systemctl restart networking" (networking.service) is not as clever as the boot process. Avoid mixing comments with configuration in /etc/network/interfaces, put them on separate lines in stead.

Canonical: Please remove "restart networking" from apt upgrades (netplan.io, nplan, etc). Upgrading packages should always be networking safe, not something you delay for a fear of being cut off from a remote server.

Explanation: I got the remote server up and running by logging in to the physical console and doing a simple "ifdown (WAN-interface)" and "ifup (WAN-interface)". The problem seemed to be that /etc/network/interfaces contained one interface with a comment at the end of a line, like this:

iface br4 inet static # COMMENT

This caused the raise network interfaces job to fail during the apt upgrade, as shown in the logs (dozens and dozens of errors like the first one for one run of 'systemctl restart networking'):

ifup[30436]: /etc/network/interfaces:98: extra parameter for the iface line not understood and ignored: #
systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: networking.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Raise network interfaces.

I assume this means that restart networking reads /etc/network/interfaces multiple times, at least once for every interface, and that the process only got as far as shutting down the first interface before it failed. This interface was not the one with a comment, so the order of events of the failure is a bit mysterious.

Generally speaking, I don't understand why 18.04 runs restart networking during an apt upgrade. This does not seem networking safe for remote management. (And it marks the first time in more than 12 years that an Ubuntu server caused a two day travel to login physically after an upgrade.)

I also don't understand why the server would boot just fine with that comment in the interfaces file, but not be able to upgrade properly. One should think that it was the same processes doing the same things at boot and during upgrades.

Technically speaking, I didn't have the time on site to verify that upgrading netplan works after removing the comment from the interfaces file. But I did manage to reproduce the errors that the upgrade process produced, and to eliminate them when running restart networking on the console. In other words, restart networking now works.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.