Troubleshooting NFS connectivity and server side issues

Symptoms:

Depending on the client and the issue, wide range of error messages can appear while trying to mount an NFS share, it might also take for ever to mount, or even mount normally but the mountpoints will be empty, in addition to one of the below messages:

mount.nfs: Connection refused

mount_nfs: can't mount / from 1.2.3.4 onto /mnt: RPC prog. not avail


In this article we will show how to generally troubleshoot connectivity and config issues, we usually start troubleshooting from the SoftNAS machine itself by using it as a client, to neutralize any firewall issues, so all the below steps should be performed on SoftNAS machine unless anything different is stated.


Troubleshooting Steps:


1-  NFS is running and listening


service  nfs status

netstat -ntlpu | grep 2049



  • If it is not up and running, please make sure you are on the primary node, as all storage services are stopped on the target node by default


2- Checking NFS config file

cat /etc/exports



  • If the first line does not exist [usually occures due to human error specially after migrating to a newer instance], you will not be able to mount using NFSv4, usually you will get the below error:

mount.nfs: access denied by server while mounting 1.2.3.4:/po1/vovo1/


  • If access to certain share is limited by specifying certain IP range in NFS config files [as below], and you are trying to mount from an out-of-range IP, actually it will mount normally without any issues, but the shares will be empty, so always check that you are allowed to mount


3- Bind mounts:

  mount | grep bind


  • Bind mounts are crutial for NFSv4 to work, if bind mounts are missing for certin reason, shares will be empty

4- Ability to list the mounts

showmount -e 172.31.30.125




  • Note that, if the previous command was tested on a client machine, and you are only enabling port 2049 [which is suffecient to mount NFSv4], it will not work, as it depends on other ports.

5- Ability to mount locally

mount 172.31.30.125:/pos3/vovo  /media/

mount | grep media

ls /media

6- Mount from the client


mount 172.31.30.125:/pos3/vovo  /media/

mount | grep media


  • If all the previous steps succeeded, but you failed to mount from your client, then this is likely a network layer issue such as a firewall acl, you may need to check with your security team that all the required ports are opened though all security layers in-betwen. The below article shows what the required ports would be.
    TCP/UDP ports required for accessing SoftNAS volumes
  • You may check traffic flow using tcpdump command. (NFS V4 for example, but you dont want to see your ssh session)
    • tcpdump -i eth0 port 2049 and not port 22