Help with disk monitoring
list Edward Bailey
Hello,
I have recently inherited a sizable Hobbit environment so please have
patience with my general ignorance of all things Hobbit. Our Hobbit
works really well except for one notable exception and I can't figure
out why. We have a series of boxes that run an application and for some
reason these systems will not monitor the /opt file system and alert us
when /opt goes over 90% even though these boxes have the same Hobbit
setup like the others on the server side and the client side.
The only difference I can see is that the /opt file system is mapped
differently on these systems. Most systems use something like
/dev/sda5 120G 82G 73% /opt
But the problem systems use
/dev/mapper/sysvg-opt 120G 82G 32G 73% /opt
Also the file systems from these problem servers show up differently as
well. The standard file system mounts are displayed without an issues
but the LVM maps display the mount point but nothing else
/dev/mapper/sysvg-opt
/dev/mapper/sysvg-var
/dev/cciss/c0d0p1 2016016 216060 1697544 12% /
/dev/cciss/c0d0p2 3020172 950024 1916728 34% /usr
The problem servers also have the follow error in the client side log
Mon Dec 11 15:53:21 2006 bb-local.sh /opt/Big_Brother/bb/bin/bb-disk.sh:
line 180: test: : integer expression expected
Mon Dec 11 15:58:21 2006 bb-local.sh /opt/Big_Brother/bb/bin/bb-disk.sh:
line 180: test: : integer expression expected
Mon Dec 11 15:58:21 2006 bb-local.sh /opt/Big_Brother/bb/bin/bb-disk.sh:
line 180: test: : integer expression expected
Mon Dec 11 16:03:21 2006 bb-local.sh /opt/Big_Brother/bb/bin/bb-disk.sh:
line 180: test: : integer expression expected
From bbdef.sh
#
# LOCAL CLIENT MONITORING CONFIGURATION FOR bb-local.sh
#
# WARNING AND PANIC LEVELS FOR LOCAL SYSTEM INFOMRAION
# YOU CAN SET VALUES ON A SPECIFIC FILESYSTEM BY USING
# THE etc/bb-dftab FILE
#
DFWARN=90 # (YELLOW) DISK % TO WARN
DFPANIC=95 # (RED) DISK % TO PANIC
export DFWARN DFPANIC
The lines above and below line 180 of bb-disk.sh
MARK=""
if test "$DISKPCT" -ge "$DFWARNVAL" #
ABOVE WARNING LEVEL
then
if test "$DISKPCT" -ge "$DFPANICVAL"
# ABOVE PANIC, RED ALERT !!!
then
COLOR="red"
MARK="PANIC"
MARKLEVEL=$DFPANICVAL
HIGHMARK="PANIC"
GIFCOLOR="&red"
REDLINE="${GIFCOLOR} ${DISKNAME}
(${DISKPCT}%) has reached the defined disk space ${MARK} le
vel (${MARKLEVEL}%)
${REDLINE}"
else
By default all servers are set to alert at 90% and 95% percent.
Thanks in advance for everyone's help
Ed Bailey
CONFIDENTIALITY NOTICE
This e-mail and any attachments contain information which may be confidential or privileged and exempt from disclosure under applicable law. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of the contents of this information is without authorization and is prohibited. If you have received this email in error, please immediately notify us by returning it to the sender and delete this copy from your computer system. Thank you.
list Henrik Størner
▸
On Mon, Dec 11, 2006 at 04:21:16PM -0500, Bailey, Edward wrote:
Also the file systems from these problem servers show up differently as well. The standard file system mounts are displayed without an issues but the LVM maps display the mount point but nothing else /dev/mapper/sysvg-opt /dev/mapper/sysvg-var /dev/cciss/c0d0p1 2016016 216060 1697544 12% / /dev/cciss/c0d0p2 3020172 950024 1916728 34% /usr
This is a classical problem with the Big Brother client - you are
apparently using the BB client instead of the Hobbit client. What's
happening is that the "df" command used on the monitored servers breaks
long lines in the output, so your "df" output looks like
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 579598 192382 356293 36% /
/dev/mapper/vg1-lv_usr
2568144 648964 1788724 27% /usr
/dev/mapper/vg1-lv_var
1048540 733448 315092 70% /var
and since the BB client does a 'grep "^/dev"' to weed out entries like
tmpfs, you only get the /dev/mapper line, not the next line with the
actual data in it.
The solution is to change the DFCMD setting on your servers to
"df -P", making it use the POSIX format output.
Regards,
Henrik
list Edward Bailey
Thanks for your quick response:) I did see that we have the big brother client installed so I need to plan on upgrading the clients first. Thanks again Ed
▸
-----Original Message----- From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] Sent: Monday, December 11, 2006 4:28 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Help with disk monitoring On Mon, Dec 11, 2006 at 04:21:16PM -0500, Bailey, Edward wrote:Also the file systems from these problem servers show up differently > as well. The standard file system mounts are displayed without an > issues but the LVM maps display the mount point but nothing else/dev/mapper/sysvg-opt/dev/mapper/sysvg-var /dev/cciss/c0d0p1 2016016 216060 1697544 12% / /dev/cciss/c0d0p2 3020172 950024 1916728 34% /usrThis is a classical problem with the Big Brother client - you are apparently using the BB client instead of the Hobbit client. What's happening is that the "df" command used on the monitored servers breaks long lines in the output, so your "df" output looks like Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 579598 192382 356293 36% / /dev/mapper/vg1-lv_usr 2568144 648964 1788724 27% /usr /dev/mapper/vg1-lv_var 1048540 733448 315092 70% /var and since the BB client does a 'grep "^/dev"' to weed out entries like tmpfs, you only get the /dev/mapper line, not the next line with the actual data in it. The solution is to change the DFCMD setting on your servers to "df -P", making it use the POSIX format output. Regards, Henrik
CONFIDENTIALITY NOTICE This e-mail and any attachments contain information which may be confidential or privileged and exempt from disclosure under applicable law. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of the contents of this information is without authorization and is prohibited. If you have received this email in error, please immediately notify us by returning it to the sender and delete this copy from your computer system. Thank you.
list Pnixon
In the meantime, before you get around to all your clients, you can edit the bbsys.local file and change DF="/bin/df -k" Line to: DF="/bin/df -kP" Which will take out the extra line return. *PS - Dangit Henrik! I knew this answer and was looking for it to see where to change it! Hehehe!
▸
-----Original Message-----
From: Bailey, Edward [mailto:user-dcede94722d2@xymon.invalid] Sent: Monday, December 11, 2006 4:42 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Help with disk monitoring
Thanks for your quick response:) I did see that we have the big brother
client installed so I need to plan on upgrading the clients first.
Thanks again
Ed -----Original Message----- From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] Sent: Monday, December 11, 2006 4:28 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Help with disk monitoring On Mon, Dec 11, 2006 at 04:21:16PM -0500, Bailey, Edward wrote:Also the file systems from these problem servers show up differently as well. The standard file system mounts are displayed without an > issues but the LVM maps display the mount point but nothing else/dev/mapper/sysvg-opt/dev/mapper/sysvg-var /dev/cciss/c0d0p1 2016016 216060 1697544 12% / /dev/cciss/c0d0p2 3020172 950024 1916728 34% /usrThis is a classical problem with the Big Brother client - you are apparently using the BB client instead of the Hobbit client. What's happening is that the "df" command used on the monitored servers breaks long lines in the output, so your "df" output looks like Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 579598 192382 356293 36% / /dev/mapper/vg1-lv_usr 2568144 648964 1788724 27% /usr /dev/mapper/vg1-lv_var 1048540 733448 315092 70% /var and since the BB client does a 'grep "^/dev"' to weed out entries like tmpfs, you only get the /dev/mapper line, not the next line with the actual data in it. The solution is to change the DFCMD setting on your servers to "df -P", making it use the POSIX format output. Regards, Henrik
CONFIDENTIALITY NOTICE This e-mail and any attachments contain information which may be confidential or privileged and exempt from disclosure under applicable law. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of the contents of this information is without authorization and is prohibited. If you have received this email in error, please immediately notify us by returning it to the sender and delete this copy from your computer system. Thank you.