disk graph page limits total file systems 🔗 link

On Tue, May 17, 2016 11:26 am, user-15513f33c451@xymon.invalid wrote:

I'm having some trouble with the disk graph pages for hosts that have
numerous file systems reporting in.  The limit seems to be 85.  There are
5
file systems displayed on each graph and there are 17 graphs max on this
page.  If I view the trends page for those host, all of the missing
filesystems are graphed.  85 seems pretty arbitrary.

I've gone through and cleaned up the RRD files from $XYMON/data/rrd/$HOST
that are "stale".  Cleaning some of these out, which would otherwise not
be
listed, permit the filesystems that were trimmed to then be listed on the
graph page.  Still, not all of the filesystems make it on the graphs.

I searched the mail archive, and didn't find a solutions.  I found a
discussion of changing the [disk] section to [disk::10] for 10 filesystems
per graph... that prevented the graphs from being generated at all.  My
thinking was, perhaps if i increase the number of filesystems per graph,
17
graphs would be enough for this particular system.

In any event, I'm wondering if this is a bug or a configurable.  It
actually works fine in a previous version of xymon. (v4.2.2 works,
v.4.3.24).  Thanks!

Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Place | Austin, MN XXXXX
Phone: (XXX) XXX-XXXX
user-15513f33c451@xymon.invalid | www.hormelfoods.com

From:	"J.C. Cleaver" <user-87556346d4af@xymon.invalid>
To:	user-15513f33c451@xymon.invalid
Cc:	"Xymon Mailing List" <xymon at xymon.com>
Date:	05/17/2016 04:53 PM
Subject:	Re: [Xymon] disk graph page limits total file systems


Erik,

85 isn't an intentional hard limit here. I've been scanning through the
showgraph code and it seems like the reallocation should be able to
continue as needed (whether not having a hard limit at all is a good idea
is a separate question...). There's a reference to 16 arguments to
rrd_graph, however that's a per-graph value and I don't believe it would
affect the number here.

A couple of next steps:
- Can you increase to disk::6 or 7 and see if there's a point where the
parsing of that number breaks?
- Does it constantly die at the same partition being graphed?
- Are there any errors coming out in /logs/ or the httpd error log, or
core files left?
- Are there any unusual file conditions in that directory?

I'd definitely suggest upgrading to a new version for security purposes,
but I don't think any fixes addressing this area specifically are present.

There were a lot of changes between 4.2.2 and 4.3.x, so it's hard to say
exactly what might be contributing there.


HTH,
-jc

From:	"J.C. Cleaver" <user-87556346d4af@xymon.invalid>
To:	user-15513f33c451@xymon.invalid
Cc:	"Xymon Mailing List" <xymon at xymon.com>
Date:	05/17/2016 04:53 PM
Subject:	Re: [Xymon] disk graph page limits total file systems


On Tue, May 17, 2016 11:26 am, user-15513f33c451@xymon.invalid wrote:

I'm having some trouble with the disk graph pages for hosts that have
numerous file systems reporting in.  The limit seems to be 85.  There are
5
file systems displayed on each graph and there are 17 graphs max on this
page.  If I view the trends page for those host, all of the missing
filesystems are graphed.  85 seems pretty arbitrary.

I've gone through and cleaned up the RRD files from $XYMON/data/rrd/$HOST
that are "stale".  Cleaning some of these out, which would otherwise not
be
listed, permit the filesystems that were trimmed to then be listed on the
graph page.  Still, not all of the filesystems make it on the graphs.

I searched the mail archive, and didn't find a solutions.  I found a
discussion of changing the [disk] section to [disk::10] for 10
filesystems
per graph... that prevented the graphs from being generated at all.  My
thinking was, perhaps if i increase the number of filesystems per graph,
17
graphs would be enough for this particular system.

In any event, I'm wondering if this is a bug or a configurable.  It
actually works fine in a previous version of xymon. (v4.2.2 works,
v.4.3.24).  Thanks!

Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Place | Austin, MN XXXXX
Phone: (XXX) XXX-XXXX
user-15513f33c451@xymon.invalid | www.hormelfoods.com

Erik,

85 isn't an intentional hard limit here. I've been scanning through the
showgraph code and it seems like the reallocation should be able to
continue as needed (whether not having a hard limit at all is a good idea
is a separate question...). There's a reference to 16 arguments to
rrd_graph, however that's a per-graph value and I don't believe it would
affect the number here.

A couple of next steps:
- Can you increase to disk::6 or 7 and see if there's a point where the
parsing of that number breaks?
- Does it constantly die at the same partition being graphed?
- Are there any errors coming out in /logs/ or the httpd error log, or
core files left?
- Are there any unusual file conditions in that directory?

I'd definitely suggest upgrading to a new version for security purposes,
but I don't think any fixes addressing this area specifically are present.

There were a lot of changes between 4.2.2 and 4.3.x, so it's hard to say
exactly what might be contributing there.


HTH,
-jc

From:	John Palys <user-6ab3068b00ee@xymon.invalid>
To:	user-15513f33c451@xymon.invalid
Date:	05/18/2016 05:17 PM
Subject:	Re: [Xymon] disk graph page limits total file systems

On Tue, May 17, 2016 at 11:26 AM, <user-15513f33c451@xymon.invalid> wrote:

  I'm having some trouble with the disk graph pages for hosts that have
  numerous file systems reporting in.  The limit seems to be 85.  There are
  5
  file systems displayed on each graph and there are 17 graphs max on this
  page.  If I view the trends page for those host, all of the missing
  filesystems are graphed.  85 seems pretty arbitrary.

  I've gone through and cleaned up the RRD files from $XYMON/data/rrd/$HOST
  that are "stale".  Cleaning some of these out, which would otherwise not
  be
  listed, permit the filesystems that were trimmed to then be listed on the
  graph page.  Still, not all of the filesystems make it on the graphs.

  I searched the mail archive, and didn't find a solutions.  I found a
  discussion of changing the [disk] section to [disk::10] for 10
  filesystems
  per graph... that prevented the graphs from being generated at all.  My
  thinking was, perhaps if i increase the number of filesystems per graph,
  17
  graphs would be enough for this particular system.

  In any event, I'm wondering if this is a bug or a configurable.  It
  actually works fine in a previous version of xymon. (v4.2.2 works,
  v.4.3.24).  Thanks!

  Erik D. Schminke | Associate Systems Programmer
  Hormel Foods Corporation | One Hormel Place | Austin, MN XXXXX
  Phone: (XXX) XXX-XXXX
  user-15513f33c451@xymon.invalid | www.hormelfoods.com


--

From:	Ron Cohen <user-f26e06d1e992@xymon.invalid>
To:	user-15513f33c451@xymon.invalid
Date:	05/18/2016 03:59 PM
Subject:	Re: [Xymon] disk graph page limits total file systems

On 18 May 2016 21:13, <user-15513f33c451@xymon.invalid> wrote:
  update: i upgraded v4.3.24 to v4.3.27.  this had no effect on my problem.
  85 filesystem "limit" still exists.

  Erik D. Schminke | Associate Systems Programmer
  Hormel Foods Corporation | One Hormel Place | Austin, MN XXXXX
  Phone: (XXX) XXX-XXXX
  user-15513f33c451@xymon.invalid | www.hormelfoods.com


  From:   "J.C. Cleaver" <user-87556346d4af@xymon.invalid>
  To:     user-15513f33c451@xymon.invalid
  Cc:     "Xymon Mailing List" <xymon at xymon.com>
  Date:   05/17/2016 04:53 PM
  Subject:        Re: [Xymon] disk graph page limits total file systems


  On Tue, May 17, 2016 11:26 am, user-15513f33c451@xymon.invalid wrote:

I'm having some trouble with the disk graph pages for hosts that have
numerous file systems reporting in.  The limit seems to be 85.  There
  are
5
file systems displayed on each graph and there are 17 graphs max on
  this
page.  If I view the trends page for those host, all of the missing
filesystems are graphed.  85 seems pretty arbitrary.

I've gone through and cleaned up the RRD files from

  $XYMON/data/rrd/$HOST

that are "stale".  Cleaning some of these out, which would otherwise
  not
be
listed, permit the filesystems that were trimmed to then be listed on
  the
graph page.  Still, not all of the filesystems make it on the graphs.

I searched the mail archive, and didn't find a solutions.  I found a
discussion of changing the [disk] section to [disk::10] for 10
  filesystems
per graph... that prevented the graphs from being generated at all.  My
thinking was, perhaps if i increase the number of filesystems per
  graph,
17
graphs would be enough for this particular system.

In any event, I'm wondering if this is a bug or a configurable.  It
actually works fine in a previous version of xymon. (v4.2.2 works,
v.4.3.24).  Thanks!

Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Place | Austin, MN XXXXX
Phone: (XXX) XXX-XXXX
user-15513f33c451@xymon.invalid | www.hormelfoods.com

  Erik,

  85 isn't an intentional hard limit here. I've been scanning through the
  showgraph code and it seems like the reallocation should be able to
  continue as needed (whether not having a hard limit at all is a good idea
  is a separate question...). There's a reference to 16 arguments to
  rrd_graph, however that's a per-graph value and I don't believe it would
  affect the number here.

  A couple of next steps:
  - Can you increase to disk::6 or 7 and see if there's a point where the
  parsing of that number breaks?
  - Does it constantly die at the same partition being graphed?
  - Are there any errors coming out in /logs/ or the httpd error log, or
  core files left?
  - Are there any unusual file conditions in that directory?

  I'd definitely suggest upgrading to a new version for security purposes,
  but I don't think any fixes addressing this area specifically are
  present.

  There were a lot of changes between 4.2.2 and 4.3.x, so it's hard to say
  exactly what might be contributing there.


  HTH,
  -jc

Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Place | Austin, MN XXXXX
Phone: (XXX) XXX-XXXX
user-15513f33c451@xymon.invalid | www.hormelfoods.com

On Mon, May 23, 2016 12:46 pm, user-15513f33c451@xymon.invalid wrote:

Another update for this topic:

I added 100 file systems to a couple systems to see what would happen with
the graphs.  The target systems were different from the one that spawned
this topic.

When I added 100 file systems to a Linux (RHEL 6.6) system, all file
systems were reported/graphed on the disk page.
When I added 100 file systems to an AIX (v7.1) system, file systems were
truncated; although at a different point.  120 of 132 file systems were
represented..

I don't understand why they're so different.  Comparing the df portions of
the messages from each of the systems do not reveal any obvious
differences.

I'd really appreciate some suggestions for debugging.  What commands I can
run manually that the disk page is running internally.  I've looked at the
code, but haven't quite figured out what's going on in there; my C skills
are rubbish.

From:	"J.C. Cleaver" <user-87556346d4af@xymon.invalid>
To:	user-15513f33c451@xymon.invalid
Cc:	"Xymon Mailing List" <xymon at xymon.com>
Date:	05/23/2016 10:32 PM
Subject:	Re: [Xymon] disk graph page limits total file systems


Hi Erik,

This actually helps a great deal, as it implies there's a distinction in
parsing code ... and potentially not an issue on the display side at all
(which I've been pouring over with little success).

Can you confirm whether the RRD files themselves are being properly
updated for both the AIX and Linux systems? (It might help to disable
caching in xymond_rrd during this process, if your system has enough space
I/O capacity.) In theory all partitions that are coming in should have
their .rrd files updated continually, but if there's a parsing issue then
that might explain one aspect of the failure.

Alternatively, can you try adding and removing partition values in the
client report and see if going above and below the 85-parition value
reliably enables the 86th?

It might be helpful to manually edit the xymonclient-$OS.sh script to grep
out (or include additional) lines of the 'df' output.

Can you also confirm that the remainder of the client report
(CPU/memory/etc.) is being handled OK, even on the AIX system?


So far I've been unable to duplicate this, but I was primarily testing on
x86_64 Linux VMs.


Regards,
-jc

On Wed, May 25, 2016 11:40 am, user-15513f33c451@xymon.invalid wrote:

JC,

I think I'm starting to see a pattern emerge, and a theory develop, here.
Hope everyone is able to follow this... here goes:

I think there may potentially be a disconnect between how the disk page
determines how many filesystems SHOULD be graphed and the number of RRD
files that are available TO BE graphed.  I think the reason the trends
page
seems to work OK is it just graphs all data that it has available without
condition.  It seems like the disk page determines 1) the number of
filesystems to graph and,  2) based on that number, the number of
filesystems per image.  These numbers seem to be determined BEFORE it
generates the HTML that produces the link HREFs and image SRCs.  It then
seems to produce just enough graphs to satisfy the predetermined number,
plus enough to satisfy the predetermined "multiple",

The predetermined number of filesystems seems to come from the number of
filesystems reported in the previous message from the client.   I believe
the assumption was made those numbers should always match.  And for the
most part they do.  It's not everyday sys admins remove filesystems from
their systems.  Until now, it may not have been so easy to spot.  It's a
little more obvious to me, being primarily an AIX administrator.  We have
a
daily process that creates an "alt_disk_copy" of our rootvg so that we
always have a hot backup of the OS.  This process causes a lot of
transient
filesystems to be created.  Those filesystems get reported and recorded
during the brief window that this process is running.  On closer
examination of my AIX systems, it is not just the one with 85 filesystems
getting truncated... it's all of them.
When I view HTML source on the disk page, I see that just ahead of the
HTML
code that displays the graph images and links, there is an HTML comment
line: "<!-- linecount=x -->" Where x equals the number of filesystems that
were reported in the previous message from the client.  (Count number of
lines, excluding header, from [df] section.) I went through each of my
systems, Linux and AIX, and determined that to always be the case.  There
must also be some range at which it determines the number "y" that
determines how many filesystems to display on each graph.  It seems like
that number is x<80, y=4 and x>=80, y=5.  (If y changes to 6 at some
point,
I haven't done enough testing to determine where that threshold is.)

The request to showgraph.cgi includes the parameters first=z and count=y.
If there are no more RRD files to graph, it stops and the graph shows
fewer
filesystems than the count parameter.  But, if you have a situation where
you have more data available than the predetermined number of filesystems,
it will continue to graph them.

On the system that previously seem limited to 85 file systems, I modified
the "hobbitclient-$os.sh" script and grepped out a certain number of file
systems.  After doing this, I had 77 filesystems reported.  That number
was
reflected in the "linecount=" HTML comment, and I also began seeing 4
filesystems per graph (instead of 5, previously) and 20 graphs being
displayed (instead of 17, previously) for a total of 80 filesystems being
graphed.  It graphed 80 because it still had enough data from RRD files to
round out the last graph.  Also, the filesystems that were grepped out of
the message from the client, were still graphed.

I also went back and checked my Linux systems; the ones where I added 100
filesystems.  On those systems, I created enough filesystems to push past
that 85 filesystem "limit".  Since those all graphed successfully, I had
previously thought that it was the difference between AIX and Linux.  That
no longer seems to be the case.  Now that I have removed all of those test
file systems, and since it's only reporting 10 filesystems, only 10
filesystems are being graphed.  File systems like /, /boot, and /home are
graphed... but the test ones that I removed, are still being graphed, and
filesystems that you would expect to see at the end alphabetically,
(e.g. /usr, /var, /opt, /tmp, etc) are not displayed.

A lot of speculation, I realize that, but the theory seems to fit reality
in all cases.  I haven't examined the code to prove it out since, as I've
said before, my C skills are rubbish.  But if my theory proves to be true,
the suggestion for improvement that I would offer is, make sure that at
least every file system from the most recent message is represented, plus
any additional file systems that might have data available in the time
period requested; between "graph_start" and "graph_end".


Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Place | Austin, MN XXXXX
Phone: (XXX) XXX-XXXX
user-15513f33c451@xymon.invalid | www.hormelfoods.com


From:	"J.C. Cleaver" <user-87556346d4af@xymon.invalid>
To:	user-15513f33c451@xymon.invalid
Cc:	"Xymon Mailing List" <xymon at xymon.com>
Date:	05/23/2016 10:32 PM
Subject:	Re: [Xymon] disk graph page limits total file systems


Hi Erik,

This actually helps a great deal, as it implies there's a distinction in
parsing code ... and potentially not an issue on the display side at all
(which I've been pouring over with little success).

Can you confirm whether the RRD files themselves are being properly
updated for both the AIX and Linux systems? (It might help to disable
caching in xymond_rrd during this process, if your system has enough space
I/O capacity.) In theory all partitions that are coming in should have
their .rrd files updated continually, but if there's a parsing issue then
that might explain one aspect of the failure.

Alternatively, can you try adding and removing partition values in the
client report and see if going above and below the 85-parition value
reliably enables the 86th?

It might be helpful to manually edit the xymonclient-$OS.sh script to grep
out (or include additional) lines of the 'df' output.

Can you also confirm that the remainder of the client report
(CPU/memory/etc.) is being handled OK, even on the AIX system?


So far I've been unable to duplicate this, but I was primarily testing on
x86_64 Linux VMs.


Regards,
-jc

On Sun, May 29, 2016 2:20 pm, J.C. Cleaver wrote:

Hi Ed,

Apologies for the delay, there've been some RL issues getting in the way
here.