Disk not showing
list Pnixon
Hey all, I just noticed that one of my systems disks isn't showing up in the graph for the disk utilization. Are there any steps to troubleshoot this? -- Text from disk status page-- Mon Jan 8 10:04:04 EST 2007 - Filesystems OK Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 6040288 1035648 4697804 19% / /dev/sda6 69804244 6892104 62912140 10% /home /dev/sdb1 720870988 406581792 277671040 60% /snapshot Status unchanged in 20 days,18 hours, 9 minutes -- Text from disk status page--
list Henrik Størner
▸
On Mon, Jan 08, 2007 at 10:09:55AM -0500, user-c102b8958c7a@xymon.invalid wrote:
I just noticed that one of my systems disks isn't showing up in the graph for the disk utilization. Are there any steps to troubleshoot this? -- Text from disk status page-- Mon Jan 8 10:04:04 EST 2007 - Filesystems OK Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 6040288 1035648 4697804 19% / /dev/sda6 69804244 6892104 62912140 10% /home /dev/sdb1 720870988 406581792 277671040 60% /snapshot
Which one is missing ? And is it the *graph only* that is missing, or is there a filesystem which isn't included at all in the above df output? Note that NFS mounts are NOT reported by the client. Henrik
list Pnixon
It's the /dev/sdb1 that's not graphing appropriately. It is graphing on the occasional /mnt that the system does overnight, but not sdb1. --Pat
▸
-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Monday, January 08, 2007 12:25 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Disk not showing
On Mon, Jan 08, 2007 at 10:09:55AM -0500, user-c102b8958c7a@xymon.invalid wrote:I just noticed that one of my systems disks isn't showing up in the graph for the disk utilization. Are there any steps to troubleshoot this? -- Text from disk status page-- Mon Jan 8 10:04:04 EST 2007 - Filesystems OK Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 6040288 1035648 4697804 19% / /dev/sda6 69804244 6892104 62912140 10% /home /dev/sdb1 720870988 406581792 277671040 60% /snapshot
Which one is missing ? And is it the *graph only* that is missing, or is there a filesystem which isn't included at all in the above df output? Note that NFS mounts are NOT reported by the client. Henrik
list Richard Leon
Hey everyone Today we switched time servers and my hobbit server and clients went crazy. Rrd graphs where not updating and I was getting some kind of time "error in the logs" I decided to back up the rrd and data directory and and then delete everything. By doing this the graphs started to show up again. But now I am showing negative time in my unchanged status. For example: Status unchanged in -8 hours, -9 minutes How can I fix that? And was there another fix besides deleting my rrd and host data? This time it was ok because I was going to have to do that anyways after the rest of my production hosts have been added. But in the future that would not really be an option. Thanks Richard
list Charles Jones
I had a similar issue when running the hobbit server on a VMware instance once. The OS under VMWare doesn't accurately keep time - sometimes the time would shift backwards or forwards. When this happens Hobbit does not record the data, because RRD files are time sensitive in that they will only accept updates within a specific time interval. As for your problem I suspect that the system time on your server changed due to the different NTP server. Perhaps it was wrong before and now it's right, or maybe it was right before and now it's wrong. Either way the jump in time is what caused the problem. I'm not sure how you can fix it, short of manually editing the timestamps in the rrd files. Maybe someone else has a better suggestion :) -Charles
▸
Richard Leon wrote:Hey everyone Today we switched time servers and my hobbit server and clients went crazy. Rrd graphs where not updating and I was getting some kind of time "error in the logs" I decided to back up the rrd and data directory and and then delete everything. By doing this the graphs started to show up again. But now I am showing negative time in my unchanged status. For example: Status unchanged in -8 hours, -9 minutes How can I fix that? And was there another fix besides deleting my rrd and host data? This time it was ok because I was going to have to do that anyways after the rest of my production hosts have been added. But in the future that would not really be an option.
list Richard Leon
To change my status back to normal I disabled everything for a 10 minute time limit. Now my unchanged status shows normal. Richard
▸
-----Original Message-----
From: Charles Jones [mailto:user-e86b4aeade4e@xymon.invalid] Sent: Monday, January 08, 2007 2:43 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Time Issue
I had a similar issue when running the hobbit server on a VMware instance once. The OS under VMWare doesn't accurately keep time - sometimes the time would shift backwards or forwards. When this happens Hobbit does not record the data, because RRD files are time sensitive in that they will only accept updates within a specific time interval.
As for your problem I suspect that the system time on your server changed due to the different NTP server. Perhaps it was wrong before and now it's right, or maybe it was right before and now it's wrong. Either way the jump in time is what caused the problem. I'm not sure how you can fix it, short of manually editing the timestamps in the rrd files. Maybe someone else has a better suggestion :)
-Charles
Richard Leon wrote:Hey everyone Today we switched time servers and my hobbit server and clients went crazy. Rrd graphs where not updating and I was getting some kind of time "error in the logs" I decided to back up the rrd and data directory and and then delete everything. By doing this the graphs started to show up again. But now I am showing negative time in my unchanged status. For example: Status unchanged in -8 hours, -9 minutes How can I fix that? And was there another fix besides deleting my rrd and host data? This time it was ok because I was going to have to do that anyways after the rest of my production hosts have been added. But in the future that would not really be an option.
list Henrik Størner
▸
On Mon, Jan 08, 2007 at 01:34:30PM -0500, user-c102b8958c7a@xymon.invalid wrote:
It's the /dev/sdb1 that's not graphing appropriately. It is graphing on the occasional /mnt that the system does overnight, but not sdb1.
Do you have an RRDDISKS or NORRDDISKS setting in hobbitserver.cfg ? Is there a "disk,snapshot.rrd" file in the ~hobbit/data/rrd/HOSTNAME/ directory ? Does the graph show up on the "trends" column ? Regards, Henrik
list Pnixon
▸
-----Original Message----- From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid] Sent: Tuesday, January 09, 2007 5:09 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Disk not showing On Mon, Jan 08, 2007 at 01:34:30PM -0500, user-c102b8958c7a@xymon.invalid wrote:
It's the /dev/sdb1 that's not graphing appropriately. It is graphing on the occasional /mnt that the system does overnight, but not sdb1.
Do you have an RRDDISKS or NORRDDISKS setting in hobbitserver.cfg ?
--- Neither are configured in hobbitserver.cfg
Is there a "disk,snapshot.rrd" file in the ~hobbit/data/rrd/HOSTNAME/
directory ?
--- Yes, the file is there with a current timestamp.
Does the graph show up on the "trends" column ?
--- Yeah, but it's the seventh disk monitored.
Overnight, several remote disks are cifs mounted to be able to be copied
locally.
list Pnixon
Okay, I hate replying to my own emails, but I deleted the disk,mnt* files, about five of them, from the rrd/bigfoot directory and now snapshot is on the disk graph, and all the trends.
▸
-----Original Message-----
From: user-c102b8958c7a@xymon.invalid [mailto:user-c102b8958c7a@xymon.invalid]
Sent: Wednesday, January 10, 2007 12:13 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Disk not showing
-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Tuesday, January 09, 2007 5:09 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Disk not showing
On Mon, Jan 08, 2007 at 01:34:30PM -0500, user-c102b8958c7a@xymon.invalid wrote:It's the /dev/sdb1 that's not graphing appropriately. It is graphing on the occasional /mnt that the system does overnight, but not sdb1.
Do you have an RRDDISKS or NORRDDISKS setting in hobbitserver.cfg ? --- Neither are configured in hobbitserver.cfg Is there a "disk,snapshot.rrd" file in the ~hobbit/data/rrd/HOSTNAME/ directory ? --- Yes, the file is there with a current timestamp. Does the graph show up on the "trends" column ? --- Yeah, but it's the seventh disk monitored. Overnight, several remote disks are cifs mounted to be able to be copied locally.
list Henrik Størner
▸
On Wed, Jan 10, 2007 at 12:13:22PM -0500, user-c102b8958c7a@xymon.invalid wrote:
On Mon, Jan 08, 2007 at 01:34:30PM -0500, user-c102b8958c7a@xymon.invalid wrote:It's the /dev/sdb1 that's not graphing appropriately. It is graphing on the occasional /mnt that the system does overnight, but not sdb1.Is there a "disk,snapshot.rrd" file in the ~hobbit/data/rrd/HOSTNAME/ directory ? --- Yes, the file is there with a current timestamp.
OK, that means it's "only" an issue with getting the graph displayed. Hobbit does pick up the data for this filesystem.
▸
Does the graph show up on the "trends" column ? --- Yeah, but it's the seventh disk monitored. Overnight, several remote disks are cifs mounted to be able to be copied locally.
That's the reason. Filesystems that "come and go" tend to confuse the "disk" graph display. I'll bet that your "disk" graph currently has three graphs showing, but it's just not the right three graphs - there are one or more of these remote filesystems showing up instead. (What's happening is that when building the disk graph, Hobbit looks at the number of filesystems reported in the disk status message. It sees 3 filesystems, so it builds a graph for three filesystems - but when there are more disk*.rrd files than that, it will pick the first three RRD files - in alphabetical order - which were modified within the past 24 hours. Since "mnt" goes before "snapshot", you get a graph for the /mnt filesystem instead of the one for /snapshot). The best solution would be to put configuration entries for these come-and-go filesystems into hobbit-clients.cfg with an IGNORE setting. Then you can delete the disk*.rrd files that refer to these cifs filesystems. That removes them from the "disk" status, and so they will no longer confuse the disk graph. Note that this only works if the server is running the Hobbit client (not the BB client). Regards, Henrik
list Pnixon
Yeah, sorta stumbled onto that. I'll see how it behaves tonight and make sure it's happy. I think I did add the IGNORE statement to hobbit-client.cfg earlier this week in an attempt to clear out the temporary mnt information.
▸
--Pat
-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Wednesday, January 10, 2007 12:26 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Disk not showing
On Wed, Jan 10, 2007 at 12:13:22PM -0500, user-c102b8958c7a@xymon.invalid wrote:On Mon, Jan 08, 2007 at 01:34:30PM -0500, user-c102b8958c7a@xymon.invalid wrote:It's the /dev/sdb1 that's not graphing appropriately. It is graphing on the occasional /mnt that the system does overnight, but not sdb1.Is there a "disk,snapshot.rrd" file in the ~hobbit/data/rrd/HOSTNAME/ directory ? --- Yes, the file is there with a current timestamp.
OK, that means it's "only" an issue with getting the graph displayed. Hobbit does pick up the data for this filesystem.
Does the graph show up on the "trends" column ? --- Yeah, but it's the seventh disk monitored. Overnight, several remote disks are cifs mounted to be able to be copied locally.
That's the reason. Filesystems that "come and go" tend to confuse the "disk" graph display. I'll bet that your "disk" graph currently has three graphs showing, but it's just not the right three graphs - there are one or more of these remote filesystems showing up instead. (What's happening is that when building the disk graph, Hobbit looks at the number of filesystems reported in the disk status message. It sees 3 filesystems, so it builds a graph for three filesystems - but when there are more disk*.rrd files than that, it will pick the first three RRD files - in alphabetical order - which were modified within the past 24 hours. Since "mnt" goes before "snapshot", you get a graph for the /mnt filesystem instead of the one for /snapshot). The best solution would be to put configuration entries for these come-and-go filesystems into hobbit-clients.cfg with an IGNORE setting. Then you can delete the disk*.rrd files that refer to these cifs filesystems. That removes them from the "disk" status, and so they will no longer confuse the disk graph. Note that this only works if the server is running the Hobbit client (not the BB client). Regards, Henrik