3

I ran apt-get upgrade on our Ubuntu systems this week. Now network mounts are breaking on our 16.04 systems every 5-10 minutes. We get errors like this:

ls: cannot access '/mnt/server_a/dir_a': Host is down

The host is definitely not down, I updated a bunch of 14.04 systems at the same time and they're not having any issues. It looks like a reboot or umount && mount of the broken mount fixes it for a few minutes, then it breaks again (even when the system is completely idle). The lines in /etc/fstab are:

//server_a/dir_a /mnt/server_a/dir_a cifs uid=my_user,soft,rw,exec,credentials=/root/creds/mnt_server_a,file_mode=0777,dir_mode=0777,iocharset=utf8,sec=ntlm 0 0
//server_b/dir_b /mnt/server_b/dir_b cifs uid=my_user,soft,rw,exec,credentials=/root/creds/mnt_server_b,file_mode=0777,dir_mode=0777,iocharset=utf8,sec=ntlm 0 0

The cred file for server_a is a local user, (two lines "username=foo" and "password=bar"). The server_b cred file is for an domain user, which is also specified in the cred file.

It looks like the update was one of these (from /var/apt/install/history.log):

update-manager-core:amd64 (1:16.04.5, 1:16.04.6)
libapt-inst2.0:amd64 (1.2.19, 1.2.20)
update-notifier-common:amd64 (3.168.3, 3.168.4)
libgtk-3-common:amd64 (3.18.9-1ubuntu3.2, 3.18.9-1ubuntu3.3)
apt:amd64 (1.2.19, 1.2.20)
libgtk-3-0:amd64 (3.18.9-1ubuntu3.2, 3.18.9-1ubuntu3.3)
snapd:amd64 (2.22.6, 2.24.1)
snap-confine:amd64 (2.22.6, 2.24.1)
dnsmasq-base:amd64 (2.75-1ubuntu0.16.04.1, 2.75-1ubuntu0.16.04.2)
grub-legacy-ec2:amd64 (0.7.9-48-g1c795b9-0ubuntu1~16.04.1, 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1)
libapt-pkg5.0:amd64 (1.2.19, 1.2.20)
cifs-utils:amd64 (2:6.4-1ubuntu1, 2:6.4-1ubuntu1.1)
ntp:amd64 (1:4.2.8p4+dfsg-3ubuntu5.3, 1:4.2.8p4+dfsg-3ubuntu5.4)
libgtk-3-bin:amd64 (3.18.9-1ubuntu3.2, 3.18.9-1ubuntu3.3)
python3-update-manager:amd64 (1:16.04.5, 1:16.04.6)
ubuntu-core-launcher:amd64 (2.22.6, 2.24.1)
apt-utils:amd64 (1.2.19, 1.2.20)
pciutils:amd64 (1:3.3.1-1.1ubuntu1, 1:3.3.1-1.1ubuntu1.1)
apt-transport-https:amd64 (1.2.19, 1.2.20)
libpci3:amd64 (1:3.3.1-1.1ubuntu1, 1:3.3.1-1.1ubuntu1.1)

I tried to revert, but apt-get would only let me downgrade those listed below, and none of them fixed the issue (although I only really suspected cifs or dnsmasq might be to blame):

  • cifs-utils:amd64=2:6.4-1ubuntu1
  • dnsmasq-base:amd64=2.75-1ubuntu0.16.04.1
  • ntp:amd64=1:4.2.8p4+dfsg-3ubuntu5.3
  • pciutils:amd64=1:3.3.1-1.1ubuntu1
  • libpci3:amd64=1:3.3.1-1.1ubuntu1

Does anyone have any ideas how to get my mounts working again? I'm seriously desperate, this is show-stopper for us, if I can't get it working in the next few days we'll have to switch our whole infrastructure back to Ubuntu 14.04.

  • The cifs filesytem is implemented in the kernel. The two relevant packages should be your kernel and cifs-utils. I don't see any other packages in that list that could break this other than cifs-utils. Is there anything relevant in your kernel logs? – jelmer Apr 30 '17 at 11:42
  • @jelmer, good point! Looking close install history I see a day earlier it ran some updates automatically? Start-Date: 2017-04-25 00:49:34 Commandline: /usr/bin/unattended-upgrade Install: linux-image-4.4.0-75-generic:amd64 (4.4.0-75.96, automatic), linux-image-extra-4.4.0-75-generic:amd64 (4.4.0-75.96, automatic), linux-headers-4.4.0-75-generic:amd64 (4.4.0-75.96, automatic), linux-headers-4.4.0-75:amd64 (4.4.0-75.96, automatic) Upgrade: ... – EJP May 1 '17 at 17:11
  • As I noted below, adding a line to my crontab */1 * * * * ls /mnt/ seems to have "fixed" the problem (in the loosest definition of the word), and I left them in that state over the weekend. Looking at dmesg today I don't see any messages that appear to be relevant. I'm not satisfied with this as a long term solution, but at least everything appears to be working this morning. I'm going to turn off the crontab on one of the systems and see monitor the kernel logs, see if anything useful pops out. – EJP May 1 '17 at 17:22
  • Nothing pops out in kern.log or dmesg – EJP May 1 '17 at 22:12
1

I had the same problem, by me it helps to remove the latest kernel. I did it like this:

  1. check that there is a second older kernel:

    dpkg --list | grep linux-image

  2. if there is an older one, remove the newest:

    apt remove --purge 4.4.0-75-*

  3. update grub:

    update-grub

Now it needs a reboot and after it should be working.

When you want later a new kernel, you have to install them with: apt install linux-generic

Another solution is to add vers=3.0 to the fstab mount statement.

  • Interesting! It looks like kernel updates are happening automatically, this issue probably would have hit me next time I rebooted these systems. When I was poking this on Friday, trying to determine the frequency with which the connections failed I noticed that actively using the mounts seemed to prevent the failure state. I left these systems running over the weekend with the following line in their crontabs, and they seem to be working happily! */1 * * * * ls /mnt/* My gut tells me it would be preferable to leave this work around in place so I get the latest security updates. Any thoughts? – EJP May 1 '17 at 17:18
  • Kernel updates gets only installed by apt(-get) dist-upgrade. Have you also the latest kernel (4.4.0-75) installed? For cifs-utils they only add pam_cifscreds, so no security relevant fixes, and the kernel update I think is also no so important, but you can take a look to the changelog. I open up a bug ticket, here you can see when they fix it: bugtracker – jb_alvarado May 1 '17 at 17:48
  • Ah I see you use unattended-upgrade. I was wondering how you get kernel updates automatically installed :). – jb_alvarado May 1 '17 at 17:59
  • Looking at history.log I see unattended-upgrade ran, and it installed a new kernel. It bumped me from 4.4.0.72.78 to 4.4.0.75.81 on April 25th. I've disabled the cron work-around on one of our systems and am monitoring it, I wanted to see if we get anything interesting in the kernel logs (but no failures yet this morning). Once it starts failing and I get the logs, I'll try the hold packages work around. – EJP May 1 '17 at 18:03
  • Yeah, I'm not sure it was wise to have unattended-upgrades enabled on these systems, I'm not sure why that was done. I think we'll be turning that off after this experience! – EJP May 1 '17 at 18:04
0

I had exactly the same problem -- for the last 3 or 4 days, my Plex media server as a ESXi VM would drop its permanent SMB mount (defined in fstab) from a bare-metal Freenas server, with the "Host is down" error message; even umount wouldn't work, indicating the target is busy.

Reverting to 4.4.0-72-generic just did the trick.

0

Looks like this is a bug in the 4.4.0 kernel. Some sort of race condition every 15 minutes causes a flood and disconnect.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=856843

I upgraded to 4.9.30 and it appears to have resolved the issues. The steps I followed are:

Download all the Kernel deb's

wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9/linux-headers-4.9.0-040900_4.9.0-040900.201612111631_all.deb

wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9.30/linux-headers-4.9.30-040930_4.9.30-040930.201705251131_all.deb

wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9/linux-headers-4.9.0-040900-generic_4.9.0-040900.201612111631_amd64.deb

Then install with:

sudo dpkg -i *.deb

Then reboot into the new kernel. Confirm with:

uname -r

Obviously be careful if you have hardware or services deeply dependent on a certain kernel. My server only runs Plex and I was able to update with no ill effects.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.