Troubleshooting NFS connectivity and server side issues
Symptoms:
Depending on the client and the issue, wide range of error messages can appear while trying to mount an NFS share, it might also take for ever to mount, or even mount normally but the mountpoints will be empty, in addition to one of the below messages:
mount.nfs: Connection refused
mount_nfs: can't mount / from 1.2.3.4 onto /mnt: RPC prog. not avail
In this article we will show how to generally troubleshoot connectivity and config issues, we usually start troubleshooting from the SoftNAS machine itself by using it as a client, to neutralize any firewall issues, so all the below steps should be performed on SoftNAS machine unless anything different is stated.
Troubleshooting Steps:
1- NFS is running and listening
service nfs status
netstat -ntlpu | grep 2049
- If it is not up and running, please make sure you are on the primary node, as all storage services are stopped on the target node by default
2- Checking NFS config file
cat /etc/exports
- If the first line does not exist [usually occures due to human error specially after migrating to a newer instance], you will not be able to mount using NFSv4, usually you will get the below error:
mount.nfs: access denied by server while mounting 1.2.3.4:/po1/vovo1/
- If access to certain share is limited by specifying certain IP range in NFS config files [as below], and you are trying to mount from an out-of-range IP, actually it will mount normally without any issues, but the shares will be empty, so always check that you are allowed to mount
3- Bind mounts:
mount | grep bind
Bind mounts are crutial for NFSv4 to work, if bind mounts are missing for certin reason, shares will be empty
4- Ability to list the mounts
showmount -e 172.31.30.125
- Note that, if the previous command was tested on a client machine, and you are only enabling port 2049 [which is suffecient to mount NFSv4], it will not work, as it depends on other ports.
5- Ability to mount locally
mount 172.31.30.125:/pos3/vovo /media/
mount | grep media
ls /media
6- Mount from the client
mount 172.31.30.125:/pos3/vovo /media/
mount | grep media
- If all the previous steps succeeded, but you failed to mount from your client, then this is likely a network layer issue such as a firewall acl, you may need to check with your security team that all the required ports are opened though all security layers in-betwen. The below article shows what the required ports would be.
TCP/UDP ports required for accessing SoftNAS volumes - You may check traffic flow using tcpdump command. (NFS V4 for example, but you dont want to see your ssh session)
- tcpdump -i eth0 port 2049 and not port 22