5

Summary: Is there a way to make sure that the NFS server isn't started by systemd until it can properly resolve client machine names specified in /etc/exports?

Problem description: I have found that NFS shares are not made properly available after the server (running 16.10) reboots. Clients get "access denied by server" errors, until exportfs -ra or service nfs-server restart is manually run on the server. After that, everything works as expected.

The server's /etc/exports contains only:

/mnt/raidarray clientmachine(rw)

where clientmachine is the hostname of the NFS client machine on the local network.

Problem identification: The output of systemctl status nfs-server (below) makes the problem clear: the name of the client can't be resolved at the time that the NFS server was started.

● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled
Active: active (exited) since Tue 2017-01-17 16:47:38 CST; 26min ago
Main PID: 1520 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/nfs-server.service
Jan 17 16:47:38 servermachine exportfs[1511]: exportfs: Failed to resolve clientmachine
Jan 17 16:47:38 servermachine systemd[1]: Started NFS server and services.

NetworkManager-wait-online.service is enabled, which I had understood serves to make sure that network.target (on which nfs-server depends) is not satisfied until network-online.target is satisfied.

Things that did not help:

  1. In case I misunderstood what NetworkManager-wait-online.service does, I tried adding an explicit After=network-online.target and Wants=network-online.target to nfs-server.service. This doesn't fix anything. I see the new dependency show up in systemctl list-dependencies, but name resolution still fails at boot. So it seems that network-online.target doesn't guarantee that hostname resolution can happen.

  2. Some googling suggested that requiring nss-lookup.target would ensure that network resolution is available, but adding it as a Wants and/or After dependency to nfs-server.service doesn't fix things either!

  3. Adding a Wants and/or After dependency on systemd-resolved.service instead of nss-lookup.target doesn't fix things either.

After adding all of these dependencies, nfs-server starts up very very late in the boot process (just before the desktop login things), but it still can't resolve hosts. Based on systemd-analyze plot, it appears that nmbd is wedged around this time, but I don't know if that's related.

Configuration information: This is on a desktop version of kubuntu 16.10 that is basically a fresh install.

NetworkManager.service is enabled and systemd-networkd.service is disabled -- I didn't change that from the default.

Here is the NetworkManager-wait-online service definition:

[Unit]
Description=Network Manager Wait Online
Documentation=man:nm-online(1)
Requisite=NetworkManager.service
After=NetworkManager.service
Before=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/bin/nm-online -s -q --timeout=30
RemainAfterExit=yes
[Install]
WantedBy=network-online.target

As a workaround I could hard-code IP addresses instead of hostnames into /etc/exports or /etc/hosts, but this seems somewhat brittle. Is there a better answer?

EDIT - Update: following @muru's advice below, I tried making a script to wait to resolve hostnames -- just to see how long it takes. My script has to wait tens of seconds after systemd-resolved starts before it can actually resolve a host. This is very bizarre. I wonder if it's actually a local networking issue? Sounds like maybe a hard-coded (and auto-updating, perhaps, as suggested by @mark-stosberg) /etc/exports or /etc/hosts file is warranted.

6
  • Hack: Create a script which waits for successful DNS resolution, then use that as an ExecStartPre for the nfs-server service
    – muru
    Jan 18, 2017 at 6:35
  • Welcome to Ask Ubuntu! ;-) Could you do a systemd-analyze plot > /tmp/systemd.svg, post a link to systemd.svg and post the contents of /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service as well? then drop me a note @fabby
    – Fabby
    Jan 18, 2017 at 7:29
  • 1
    Hrmpf! What you already did was what I was going to propose... BTW, you should edit your question and provide the above critical info there so the next user looking at your question doesn't have to dig through the entire comment section.
    – Fabby
    Jan 18, 2017 at 10:43
  • @muru IMNSHO, your suggestion is the best one. Could you provide an answer for that??? If you ping me, I'll come back and upvote! ;-)
    – Fabby
    Jan 18, 2017 at 18:24
  • Why hard-code and auto-update /etc/hosts instead of /etc/exports itself?
    – muru
    Jan 19, 2017 at 1:13

5 Answers 5

0

In terms of what you are trying to do with systemd, you've already tried to do the best thing, which is to set your service to start after network-online. I read a lot of a systemd questions, and others have reported problems with the network not being fully online when they do this.

I recommend the idea of putting IP addresses in /etc/exports or /etc/hosts as you recommend. This simplifies an important disk mounting task during boot-up.

To make the system more robust, you could use a cron job or systemd timer that checks periodically to see if the DNS for the interesting hosts have changed. If so, automatically update /etc/hosts with the new values.

9
  • Thanks! I'm not sure which seems like a more robust / less kludgey solution: the suggestion from @muru to make a script that waits for successful name resolution as an ExecStartPre component of nfs-server.service, or make a cron script to edit /etc/hosts with the latest DNS information for hosts of interest. Thoughts anyone?
    – zachrahan
    Jan 18, 2017 at 15:23
  • If you wait for successful name resolution in ExecStartPre, it's possibly it will take very long or maybe never happen. If you update /etc/hosts with valid DNS entries only when valid DNS queries can be completed, the file will be maximally up to date. Jan 18, 2017 at 15:40
  • Copy-paste of comments as an answer. -1
    – Fabby
    Jan 18, 2017 at 19:30
  • @Fabby I'm not sure what you are referring to- I didn't copy/paste anything. Jan 18, 2017 at 19:47
  • 1
    @Fabby This answer was useful to me, at least, so it couldn't have been a total copy-paste of what I said. (And nobody other than I said anything about /etc/hosts...)The idea of automatically updating /etc/hosts was a good one, too, and hadn't occurred to me. Plus the context of this being a common systemd bugaboo is useful.
    – zachrahan
    Jan 18, 2017 at 20:50
0

I had the same problem on an ubuntu 16.04.03 LTS installation.
The exportfs did not run properly after a reboot and thus the nfs-kernel-server did not start properly.
I tried to add a sleep to the start case in /etc/init.d/nfs-kernel-server but that did not help.
I also tried some hints found https://discourse.osmc.tv/t/nfs-kernel-server-wont-start-on-boot/5936/7
I finally solved the problem by adding the following line to /etc/rc.local

systemctl restart nfs-kernel-server
0

Based on @muru's comment, I made a python script to wait until DNS resolution:

import socket
import time
import itertools

TIMEOUT = 30
HOSTS = ['stanford.edu', 'google.com', 'example.com']

def main():
    hosts = itertools.cycle(HOSTS)
    t0 = time.time()
    for host in hosts:
        t = time.time()
        elapsed = t - t0
        if elapsed > TIMEOUT:
            break
        try:
            socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP)
            print('Resolved {} at t = {}'.format(host, elapsed))
            break
        except socket.gaierror:
            print('Could not resolve {} at t = {}'.format(host, elapsed))
        time.sleep(0.25)
        t = time.time()

if __name__ == '__main__':
    main()

I saved that script as /etc/systemd/system/nfs-server.service.d/wait_for_dns.py (one has to make the parent directory first), and then ran sudo systemctl edit --full nfs-server, adding the following before other ExecStartPre lines:

ExecStartPre=/usr/bin/python3 /etc/systemd/system/nfs-server.service.d/wait_for_dns.py

This worked fine, though it certainly feels a bit hackish and could be improved in many ways. (I eventually just gave up on NFS in this context; it was more of a pain than it was worth.)

0

this was bugging me for weeks. At first I didn't figure it out why does KODI on my TV suddenly don't see NFS shares. And then I couldn't figure it out how to solve that because all solutions I found didn't work on 19.04.

But adding a cron job with

systemctl restart nfs-kernel-server 

as command after every reboot solved it for me.

0

Had the same issue today (I am on AlmaLinux 8.6), and fixed with systemctl edit nfs-server:

[Unit]  
Requires=network.target network-online.target  
After=network.target network-online.target  

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.