mount.nfs: access denied by server while mounting

Question

I have a problem I have been googling for at least 6 hours 28 hours with no luck. Whenever I try to mount a directory I get the following error (verbose version at end):

mount.nfs: access denied by server while mounting :

Initally I was using computer1 as the nfs server, but I have moved it over to computer2 and I am trying to mount on computers3-8. As a further note, computer2 is now acting as the dhcp router for all other computers (even computer1), but I am almost certain that the problem has to do with computer2's nfs settings as I have tried the following:

1. Try mounting a folder from the original computer1 server to one of the computers3-8 clients -- SUCCESS
2. Try mounting a folder from the original computer1 server to the new computer2 server -- SUCCESS
3. Try mounting a folder from the new computer2 server to one of the computers3-8 clients -- FAILURE
4. Try mounting a folder from the new computer2 server to the old computer1 server -- FAILURE
5. Try exporting then mounting any other folder on computer2 -- FAILURE
6. ssh to and from computer2 -- SUCCESS

In addition to this, I have also checked to make sure that the relevant parts of the /etc/exports file do not have any mistakes. Therefore, I am almost absolutely certain this is not a permission issue.

Question: Given that this is not a network error, can anyone advise me as to how I can start debugging this problem.

Verbose output:

mount: proc already mounted on /proc
mount: /dev/mmcblk0p5 already mounted on /boot
mount: tmpfs already mounted on /var/tmp
mount: tmpfs already mounted on /var/log
mount.nfs: timeout set for Fri Nov 15 07:27:02 2013
mount.nfs: trying text-based options 'vers=3,rsize=8192,wsize=8192,nolock,addr=192.168.7.1'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.7.1 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.7.1 prog 100005 vers 3 prot UDP port 37300
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 192.168.7.1:/home/raspi/Documents/raspi/raspi

Verbose Output2:

Here is a more direct approach which isolates the mount to only the directory in question here:

$ sudo mount -v -o "vers=3" 192.168.7.1:/home/raspi/Documents/raspi/raspi /home/raspi/test
mount: no type was given - I'll assume nfs because of the colon
mount.nfs: timeout set for Fri Nov 15 07:47:38 2013
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

I had come across this rpc.statd is not running but is required for remote locking error before, but all the googling in the world did not help me at all...

EDIT:

Chaos asked me to execute the following commands and display the results:

First on the server:

SERVER$ sudo exportfs -rv
exporting raspi1:/home/raspi/Documents/raspi/raspi
exporting raspi2:/home/raspi/Documents/raspi/raspi
exporting raspi3:/home/raspi/Documents/raspi/raspi
exporting raspi4:/home/raspi/Documents/raspi/raspi
exporting raspi5:/home/raspi/Documents/raspi/raspi
exporting raspi6:/home/raspi/Documents/raspi/raspi
exporting raspi7:/home/raspi/Documents/raspi/raspi
exporting raspi8:/home/raspi/Documents/raspi/raspi
exporting raspi9:/home/raspi/Documents/raspi/raspi
exporting raspi10:/home/raspi/Documents/raspi/raspi
exporting raspi11:/home/raspi/Documents/raspi/raspi
exporting raspi12:/home/raspi/Documents/raspi/raspi
exporting raspi13:/home/raspi/Documents/raspi/raspi
exporting raspi14:/home/raspi/Documents/raspi/raspi
exporting raspi15:/home/raspi/Documents/raspi/raspi
exporting raspi16:/home/raspi/Documents/raspi/raspi
exporting raspi17:/home/raspi/Documents/raspi/raspi
exporting raspi18:/home/raspi/Documents/raspi/raspi
exporting raspi19:/home/raspi/Documents/raspi/raspi
exporting raspi20:/home/raspi/Documents/raspi/raspi
exporting raspi21:/home/raspi/Documents/raspi/raspi
exporting raspi22:/home/raspi/Documents/raspi/raspi
exporting raspi23:/home/raspi/Documents/raspi/raspi
exporting raspi24:/home/raspi/Documents/raspi/raspi
exporting raspi25:/home/raspi/Documents/raspi/raspi
exporting raspi26:/home/raspi/Documents/raspi/raspi
exporting raspi27:/home/raspi/Documents/raspi/raspi
exporting raspi28:/home/raspi/Documents/raspi/raspi
exporting raspi29:/home/raspi/Documents/raspi/raspi
exporting raspi30:/home/raspi/Documents/raspi/raspi
exporting raspi5:/home/raspi/Downloads
exporting 192.168.7.105:/home/raspi/Documents/raspi

And then on the client

CLIENT$ showmount -e 192.168.7.1
Export list for 192.168.7.1:
/home/raspi/Documents/raspi       192.168.7.105
/home/raspi/Documents/raspi/raspi raspi30,raspi29,raspi28,raspi27,raspi26,raspi25,raspi24,raspi23,raspi22,raspi21,raspi20,raspi19,raspi18,raspi17,raspi16,raspi15,raspi14,raspi13,raspi12,raspi11,raspi10,raspi9,raspi8,raspi7,raspi6,raspi5,raspi4,raspi3,raspi2,raspi1
/home/raspi/Downloads             raspi5

CLIENT$ sudo mount -a
mount.nfs: access denied by server while mounting 192.168.7.1:/home/raspi/Documents/raspi/raspi

CLIENT$ sudo tail -f /var/log/daemon.log 2>&1
Nov 15 21:35:55 raspi5 dhclient: bound to 192.168.7.105 -- renewal in 250 seconds.
Nov 15 21:40:05 raspi5 dhclient: DHCPREQUEST on eth0 to 192.168.7.1 port 67
Nov 15 21:40:05 raspi5 dhclient: DHCPACK from 192.168.7.1
Nov 15 21:40:05 raspi5 dhclient: bound to 192.168.7.105 -- renewal in 294 seconds.
Nov 15 21:44:59 raspi5 dhclient: DHCPREQUEST on eth0 to 192.168.7.1 port 67
Nov 15 21:44:59 raspi5 dhclient: DHCPACK from 192.168.7.1
Nov 15 21:44:59 raspi5 dhclient: bound to 192.168.7.105 -- renewal in 262 seconds.
Nov 15 21:49:21 raspi5 dhclient: DHCPREQUEST on eth0 to 192.168.7.1 port 67
Nov 15 21:49:21 raspi5 dhclient: DHCPACK from 192.168.7.1
Nov 15 21:49:21 raspi5 dhclient: bound to 192.168.7.105 -- renewal in 273 seconds.

EDIT 2:

Chaos asked me to further execute the following commands and display the results:

SERVER$ ping raspi5
PING raspi5 (192.168.7.105) 56(84) bytes of data.
64 bytes from raspi5 (192.168.7.105): icmp_seq=1 ttl=64 time=0.699 ms
64 bytes from raspi5 (192.168.7.105): icmp_seq=2 ttl=64 time=0.683 ms

SERVER$ sudo tail -f /var/log/daemon.log
tail: cannot open ‘/var/log/daemon.log’ for reading: No such file or directory

Here is my /etc/exports file on the server. As you can see I am trying to use its name as defined in /etc/hosts, its ip address, and I am also trying to different folders to export. All other lines I omitted as they are just duplicates of the last line, just redirected to different raspis.

# /etc/exports: the access control list for filesystems which may be exported
#       to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#

/home/raspi/Downloads raspi5(rw,sync,no_subtree_check,no_root_squash)
/home/raspi/Documents/raspi/ 192.168.7.105(rw,sync,no_subtree_check,no_root_squash,fsid=0)
/home/raspi/Documents/raspi/raspi raspi5(rw,sync,no_subtree_check,no_root_squash,fsid=0)

Here is my /etc/fstab file on my raspi5 client

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p5  /boot           vfat    defaults          0       2
/dev/mmcblk0p6 /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that

#automatically mount the shared raspi folder(s)
# Remember that NFS4 is AIDS, so use NFS3
192.168.7.1:/home/raspi/Documents/raspi/raspi /home/raspi/ nfs nfsvers=3,rsize=8192,wsize=8192,rw,auto,nolock 0 0

# Move highly used directories to RAM
tmpfs /var/tmp tmpfs nodev,nosuid,size=50M 0 0
tmpfs /var/log tmpfs nodev,nosuid,size=50M 0 0

EDIT 3:

Following some of the advice here I also set the insecure flag in exports, but to no avail, and here is the output of rpcinfo -p on the client and the server:

CLIENT$ sudo rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  53553  status
    100024    1   tcp  60026  status

SERVER$ rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  42430  status
    100024    1   tcp  49377  status
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049
    100227    3   tcp   2049
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049
    100227    3   udp   2049
    100021    1   udp  33106  nlockmgr
    100021    3   udp  33106  nlockmgr
    100021    4   udp  33106  nlockmgr
    100021    1   tcp  47922  nlockmgr
    100021    3   tcp  47922  nlockmgr
    100021    4   tcp  47922  nlockmgr
    100005    1   udp  50875  mountd
    100005    1   tcp  53329  mountd
    100005    2   udp  53583  mountd
    100005    2   tcp  42062  mountd
    100005    3   udp  36556  mountd
    100005    3   tcp  39984  mountd

EDIT 4:

Further following some of the advice here I have gotten the following debugging logs:

[ 2233.803852] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 2233.803885] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[ 3261.750455] nfsd: last server has exited, flushing export cache
[ 3262.912654] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 3262.912688] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[ 5231.798334] nfsd: last server has exited, flushing export cache
[ 5232.956633] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 5232.956668] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[ 5350.889640] nfsd: last server has exited, flushing export cache
[ 5352.023534] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 5352.023577] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[ 5517.837425] nfsd: last server has exited, flushing export cache
[ 5518.987793] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 5518.987826] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[56604.080162] Netfilter messages via NETLINK v0.30.
[56707.453535] nfsd: last server has exited, flushing export cache
[56708.605020] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[56708.605057] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[57338.926093] NFSD: laundromat service - starting
[57338.926105] NFSD: laundromat_main - sleeping for 90 seconds
[57363.216188] nfsd: freeing readahead buffers.
[57363.216244] nfsd: last server has exited, flushing export cache
[57364.351041] set_max_drc nfsd_drc_max_mem 6008832 
[57364.354764] nfsd: creating service
[57364.354771] nfsd: allocating 32 readahead buffers.
[57364.356023] nfsd4_umh_cltrack_upcall: cltrack_prog is disabled
[57364.356040] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[57364.356072] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[57364.369688] nfsd_dispatch: vers 3 proc 0
[57372.738225] nfsd_dispatch: vers 4 proc 0
[57372.753507] nfsd_dispatch: vers 4 proc 1
[57372.753521] nfsv4 compound op #1/3: 24 (OP_PUTROOTFH)
[57372.755550] exp_export: export of invalid fs type.
[57372.755679] found domain 192.168.7.105,raspi5
[57372.755685] found fsidtype 1
[57372.755690] found fsid length 4
[57372.755695] Path seems to be <>
[57372.755722] nfsv4 compound op ffff8801a47ec080 opcnt 3 #1: 24: status 2
[57372.755731] nfsv4 compound returned 2
[57372.779429] nfsd_dispatch: vers 3 proc 0
[57372.792595] exp_export: export of invalid fs type.

EDIT 5:

I've made some progress and have deduced the following

1. It's not a port issue as client can telnet into host telnet 192.168.7.1 2049
2. It's probably not a network issue as the server can not even mount to itself
3. The old version which supported (and still supports) mounting was running nfs-kernel-server 1:1.2.4-1 whereas the new server which does not support mounting is running nfs-kernel-server 1:1.2.8-2
4. The command sudo /etc/init.d/portmap restart works on the old server, but not the new server, which tells me that I am missing portmap on the new server. However, when I try to install it via sudo apt-get install portmap, it tells me Note, selecting 'rpcbind' instead of 'portmap', and thatrpcbind is already the newest version`

Edit 6:

Here is the output of iptables -L on my server:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
LOG        all  --  anywhere             anywhere             LOG level warning

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

and the correct ports are open:

CLIENT$ nmap -P0 192.168.7.1

Starting Nmap 6.00 ( http://nmap.org ) at 2013-11-16 06:44 UTC
Nmap scan report for raspiserver (192.168.7.1)
Host is up (0.0095s latency).
Not shown: 996 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
111/tcp  open  rpcbind
2049/tcp open  nfs

Nmap done: 1 IP address (1 host up) scanned in 1.55 seconds

Answer 1

I had the same issue as the OP. This used to work for me but then after an update it didn't. The solution was to add the "nfsvers=3" option.

Answer 2

It could be worth mentioning (even if it's probably not the case here) that if you're suddenly getting this error after renaming a folder or attempting to access a new one, you might have forgotten that you actually have to specify server-side what folders are shared and what aren't in /etc/exports -- this was the issue in my case.

After you've modified that file, you also need to do sudo service nfs-kernel-server restart

Answer 3

My two colleagues at work just solved a similar issue within Vagrant. AppArmor was the culprit, solution can be found here, citing:

mount: cannot mount block device /some/path read-only

and dmesg shows:

[ 6944.194280] type=1400 audit(1385049795.420:32): apparmor="DENIED" operation="mount" info="failed type match" error=-13 parent=6631 profile="lxc-container-default" name="/some/other/path" pid=6632 comm="mount" srcname="/some/path" flags="rw, bind"

AppArmor is blocking mount -o bind inside the LXC container.

To enable id add in /etc/apparmor.d/lxc/lxc-default:

profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
  ...
    mount options=(rw, bind),
  ...

Reload apparmor:

# /etc/init.d/apparmor reload

Answer 4

First the rpc.statd error occurs because the daemon is not running. Start the daemon by:

sudo service statd start

Secondly, the mount option vers=3 is wrong it should be

nfsvers=3

Answer 5

If server is not mounting on localhost, the problem is not the network.

Take a look at /etc/hosts.deny and /etc/hosts.allow.

Only for further problems...: I played this days with an old version of NFS. I had an intermediate switch between two hosts that allowed all TCP and UDP communications from ServerIP<->ClientIP, telnet worked on every port... but rpcinfo -p didn't.

We've found that the switch had a feature called "inspect" that inspected all the "sunrpc" packets and if the version was not in the "allowed" versions, dropped the packets without logging anything. It's called "Application Layer Protocol Inspection". http://www.cisco.com/c/en/us/td/docs/security/fwsm/fwsm31/configuration/guide/fwsm_cfg/inspct_f.html#wp1349924

After this we also found that we had to configure static ports for firewalling, but that's another issue.

Answer 6

This neither seems to be the case but when mounting from a non-root user, the noresvport may be useful in /etc/exports.

This use to happen in Finder (MacOS X) when trying to mount a nfs unit.

Answer 7

The issue can also happen to be as simple as the client machine's IP address has changed. (The machine you are getting the access denied error on.)

One can check the client's IP(s) with:

hostname -I

Then it's time to see whether this IP is among the ones authorized to access the share, revealed by:

showmount -e <REMOTE.NFS.SERVER.ADDRESS>