mounted NFS volumes not included in disk utilization graphs
list Paul Grondahl
On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs?
list Jeremy Laidman
▸
On 31 March 2015 at 09:49, Paul Grondahl <user-4616c79f3913@xymon.invalid> wrote:
On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs?
To monitor NFS-shared disks, you would typically monitor them on the NFS server, rather than one of the clients. What OS are you using? Can you show your [df] client data? J
list Paul Grondahl
Hi, Using CentOS 6.4 The last line in the df output shows the NFS mount. [df] Filesystem 1024-blocks Used Available Capacity Mounted on /dev/vda2 19766140 6449876 12312184 35% / /dev/vda1 99150 25418 68612 28% /boot [inode] Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vda2 1256640 192603 1064037 16% / /dev/vda1 25688 38 25650 1% /boot [mount] /dev/vda2 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/vda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) arrow81:/recordings/arrow82 on /mnt/nfs/recordings type nfs (rw,vers=4,clientaddr=172.16.6.167,addr=172.16.6.168)
▸
On Mar 30, 2015, at 10:17 PM, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote: On 31 March 2015 at 09:49, Paul Grondahl <user-4616c79f3913@xymon.invalid <mailto:user-4616c79f3913@xymon.invalid>> wrote: On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs? To monitor NFS-shared disks, you would typically monitor them on the NFS server, rather than one of the clients. What OS are you using? Can you show your [df] client data? J
list Jeremy Laidman
▸
On 1 April 2015 at 06:46, Paul Grondahl <user-4616c79f3913@xymon.invalid> wrote:
Using CentOS 6.4 The last line in the df output shows the NFS mount. [df] Filesystem 1024-blocks Used Available Capacity Mounted on /dev/vda2 19766140 6449876 12312184 35% / /dev/vda1 99150 25418 68612 28% /boot
It's the [df] section of the client data that is used to record disk
usage, not the [mount] section. For the Linux client script(xymonclient-linux.sh) the command is "df -Pl" where the "-l" means "local
file systems".
The script also looks for "nodev" devices and builds an exclude list of
filesystem times to remove from the output of "df". This is intended to
remove pseudo filesystems such as proc and tmpfs, but also removes
NFS-mounted filesystems.
There are various manipulations that you need to make in order for the "df"
command to show the extra filesystems you want. But in my opinion, the
easiest way to accomplish this is to simply add an extra "df" command into
the [df] part of the script, showing the extra filesystems you want. Such
as:
echo "[df]"
EXCLUDES=`cat /proc/filesystems | grep nodev | awk '{print $2}' | xargs
echo | sed -e 's! ! -x !g'`
{ df -Pl -x iso9660 -x $EXCLUDES; df -P -t nfs | tail +2; } | sed -e '/^[^
][^ ]*$/{
N
s/[ ]*\n[ ]*/ /
}'
The bit I added was "df -P -t nfs | tail +2" (along with the {braces} to
make it look to awk like a single df command). Give this a go and let me
know if it works for you.
HTH
Cheers
Jeremy
list Henrik Størner
Den 31-03-2015 kl. 00:49 skrev Paul Grondahl:
▸
On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs?
There is a potential problem with including network disks in the "disk" status: If your network connection to the NFS server dies, then the "df" command is likely to stall. So you won't get any client data sent over to the Xymon server, and your entire server goes purple. That is why the Xymon client uses the "-l" (local) flag for df to NOT report on network filesystems. Regards, Henrik
list Galen Johnson
I'll "me, too" Henrik's response (good to see he's still lurking). df on nfs should happen on the server side. Another problem, I've had the same thing happen to SAN disks that were "locally" attached when the server drops away unexpectedly. =G=
▸
From: Xymon <xymon-bounces at xymon.com> on behalf of Henrik Størner <user-ce4a2c883f75@xymon.invalid>
Sent: Wednesday, April 1, 2015 4:13 PM
To: xymon at xymon.com
Subject: Re: [Xymon] mounted NFS volumes not included in disk utilization graphs
Den 31-03-2015 kl. 00:49 skrev Paul Grondahl:On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs?
There is a potential problem with including network disks in the "disk" status: If your network connection to the NFS server dies, then the "df" command is likely to stall. So you won't get any client data sent over to the Xymon server, and your entire server goes purple. That is why the Xymon client uses the "-l" (local) flag for df to NOT report on network filesystems. Regards, Henrik
list Malcolm
At my work, my team developed some "safe" coding to list NFS filesystems in df. It uses the showmount command to list exports on the NFS server - if this succeeds, it should be safe to run df on the filesystem. The comment in the code references the URL http://osdir.com/ml/monitoring.hobbit/2007/msg02373.html but it doesn't seem to exist now. It's ugly and no doubt could be cleaned up and improved... but it does work, and it's handy to know which clients are affected when a NFS volume becomes full. Update to xymonclient-linux.sh : ================================ At the end of the code for "df", add these lines : # --- Start - Addition code, to check nfs mounts --- # Added following code, based on http://osdir.com/ml/monitoring.hobbit/2007/msg02373.html # /bin/bash -c "/bin/egrep $'[ \t]nfs[ \t]' /etc/mtab" | \ /bin/egrep -v "ignore|indirect" | \ /usr/bin/awk '{print $1, $2}' | \ while read line do set -- $line NFSHOST=`echo $1 | /usr/bin/awk -F: '{print $1}'` SHARE=`echo $1 | /usr/bin/awk -F: '{print $2}'` /bin/ping $NFSHOST -c 5 >/dev/null 2>&1 if [ $? -ne 0 ] then echo "$1 IS NOT RESPONDING(ping_failed) 999% $2" else # Ensure NFS connection is active BEFORE running the "showmount -e" command. /bin/df -k $2 > /dev/null 2>&1 & /bin/sleep 5 ; kill $! > /dev/null 2>&1 /usr/sbin/showmount -e $NFSHOST >/dev/null 2>&1 if [ $? -ne 0 ]; then echo "$1 IS NOT AVAILABLE(showmount_failed) 999% $2" else /usr/sbin/showmount -e $NFSHOST | /bin/egrep "^$SHARE " >/dev/null 2>&1 if [ $? -ne 0 ]; then # Remove "/share" prefix as required for QNAP NFS servers SHARE1=${SHARE#/share} /usr/sbin/showmount -e $NFSHOST | /bin/egrep "^$SHARE1 " >/dev/null 2>&1 fi if [ $? -ne 0 ]; then echo "$1 IS NOT AVAILABLE(showmount_failed_to_find_share) 999% $2" else /bin/df -kP $2 | tail -n +2 fi fi fi done # --- End of - Addition code, to check nfs mounts --- Regards Mal
▸
On 02/04/15 07:13, Henrik Størner wrote:Den 31-03-2015 kl. 00:49 skrev Paul Grondahl:On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs?There is a potential problem with including network disks in the "disk" status: If your network connection to the NFS server dies, then the "df" command is likely to stall. So you won't get any client data sent over to the Xymon server, and your entire server goes purple. That is why the Xymon client uses the "-l" (local) flag for df to NOT report on network filesystems. Regards, Henrik
list Paul Grondahl
Thanks all for your detailed responses. I will just monitor the NFS host.
▸
On Apr 1, 2015, at 6:41 PM, Malcolm <user-9734186ce057@xymon.invalid> wrote: At my work, my team developed some "safe" coding to list NFS filesystems in df. It uses the showmount command to list exports on the NFS server - if this succeeds, it should be safe to run df on the filesystem. The comment in the code references the URL http://osdir.com/ml/monitoring.hobbit/2007/msg02373.html but it doesn't seem to exist now. It's ugly and no doubt could be cleaned up and improved... but it does work, and it's handy to know which clients are affected when a NFS volume becomes full. Update to xymonclient-linux.sh : ================================ At the end of the code for "df", add these lines : # --- Start - Addition code, to check nfs mounts --- # Added following code, based on http://osdir.com/ml/monitoring.hobbit/2007/msg02373.html # /bin/bash -c "/bin/egrep $'[ \t]nfs[ \t]' /etc/mtab" | \ /bin/egrep -v "ignore|indirect" | \ /usr/bin/awk '{print $1, $2}' | \ while read line do set -- $line NFSHOST=`echo $1 | /usr/bin/awk -F: '{print $1}'` SHARE=`echo $1 | /usr/bin/awk -F: '{print $2}'` /bin/ping $NFSHOST -c 5 >/dev/null 2>&1 if [ $? -ne 0 ] then echo "$1 IS NOT RESPONDING(ping_failed) 999% $2" else # Ensure NFS connection is active BEFORE running the "showmount -e" command. /bin/df -k $2 > /dev/null 2>&1 & /bin/sleep 5 ; kill $! > /dev/null 2>&1 /usr/sbin/showmount -e $NFSHOST >/dev/null 2>&1 if [ $? -ne 0 ]; then echo "$1 IS NOT AVAILABLE(showmount_failed) 999% $2" else /usr/sbin/showmount -e $NFSHOST | /bin/egrep "^$SHARE " >/dev/null 2>&1 if [ $? -ne 0 ]; then # Remove "/share" prefix as required for QNAP NFS servers SHARE1=${SHARE#/share} /usr/sbin/showmount -e $NFSHOST | /bin/egrep "^$SHARE1 " >/dev/null 2>&1 fi if [ $? -ne 0 ]; then echo "$1 IS NOT AVAILABLE(showmount_failed_to_find_share) 999% $2" else /bin/df -kP $2 | tail -n +2 fi fi fi done # --- End of - Addition code, to check nfs mounts --- Regards Mal On 02/04/15 07:13, Henrik Størner wrote:Den 31-03-2015 kl. 00:49 skrev Paul Grondahl:On several systems with NFS mounts, the mounted volumes are not included in the disk utilization graphs. The volumes are visible in the client data. How does one get them included in the graphs?There is a potential problem with including network disks in the "disk" status: If your network connection to the NFS server dies, then the "df" command is likely to stall. So you won't get any client data sent over to the Xymon server, and your entire server goes purple. That is why the Xymon client uses the "-l" (local) flag for df to NOT report on network filesystems. Regards, Henrik