upgrade to 4.1.2 breaks netapp qtree graphing 🔗 link

Upgraded from hobbit 4.0.4 -> 4.1.2 and hit my first bug:  the data collected from my filerstats2bb.pl ext script, which gets data from my NetApp filer and allows me to graph the qtrees, volumes & inodes, no longer works.  The data collection still works fine, as well as the display of the results, but only one the first qtree/inode/disk is graphed.  The rest are not.

hobbitgraph.cfg definitions are still the same as before, so I poked around the hobbitd/rrd/do_disk.c and compared it to the one used in 4.0.4, but I think that is working well since it populates the rrd files...?

hobbitgraph.c is where I should look, right?

I'm kinda lost here, and of course we need to see our filer graphs to check on a problem, which is how a co-worker disovered the bug in the first place!

Tom

Tom Georgoulias wrote:

Doh, forgot to add:

the graphs displayed under the "trends" column are fine, just the graphs under the specific parameter are having this problem (i.e. graph in qtree column shows first qtree, qtree graphs in trends shows all qtrees on filer.)

Tom

On Fri, Oct 14, 2005 at 09:04:54AM -0400, Tom Georgoulias wrote:

Is this on both the individual status display and the one in the
"trends" column, or just one of them ?

hobbitgraph.cfg definitions are still the same as before, so I poked around the hobbitd/rrd/do_disk.c and compared it to the one used in 4.0.4, but I think that is working well since it populates the rrd files...?

If the RRD files are being updated, then rrd/do_disk is working OK.

hobbitgraph.c is where I should look, right?

Sounds like it, but there are some complicating factors.


I did some testing here, and copying a set of disk*rrd files to
inode-ditto's and bringing up an "inode" status looks OK. So I
think it's some upgrade problem you've run into.

Could you try and compare the following files:

- bb-hostsvc.sh (from the installed cgi-bin directory) with the   hobbitd/bb-hostsvc.sh from the hobbit-4.1.2 build directory ?

- hobbitgraph.sh (from the installed cgi-bin directory) with the
  hobbitd/hobbitgraph.sh from the hobbit-4.1.2 build directory ?

- hobbitgraph.cfg (from the installed server/etc/ directory) with
  the hobbitd/etcfiles/hobbitgraph.cfg from the hobbit-4.1.2 build
  directory ?

The two cgi-scripts - if there are any differences - can be copied
over directly from the build directory. The hobbitgraph.cfg may
need some hand-tweaking, if you have defined any graphs yourself.


Regards,
Henrik

On Fri, Oct 14, 2005 at 09:17:18AM -0400, Tom Georgoulias wrote:

BTW, could you check the timestamps on your inode*rrd files ? Hobbit
4.1.2 has a new feature where it ignores stale rrd files. If some of
your inode rrd files are not being updated, then you would see only the recently modified ones on the "inode" status page.


Henrik

Henrik Stoerner wrote:

md5sum bb-hostsvc.sh /home/bb/cgi-bin/bb-hostsvc.sh
5c17106fd83c3c5eda051592b945f497  bb-hostsvc.sh
5c17106fd83c3c5eda051592b945f497  /home/bb/cgi-bin/bb-hostsvc.sh

md5sum hobbitgraph.sh /home/bb/cgi-bin/hobbitgraph.sh
f778ab8f6fd0e0e3cc5af665628d0889  hobbitgraph.sh
f778ab8f6fd0e0e3cc5af665628d0889  /home/bb/cgi-bin/hobbitgraph.sh

md5sum etcfiles/hobbitgraph.cfg /home/bb/server/etc/hobbitgraph.cfg
a65dc5f1a199a684010cc97712277025  etcfiles/hobbitgraph.cfg
f70f5a1f180fc59cd9b2f71ffb0fb2be  /home/bb/server/etc/hobbitgraph.cfg


I have some custom edits to this file, so that is why they won't match. 
  Since we're using inodes in troubleshooting, here is the def I'm using:

[inode]
         FNPATTERN inode(.*).rrd
         TITLE inode Utilization
         YAXIS % Full
         DEF:p at RRDIDX@=@RRDFN@:pct:AVERAGE
         LINE2:p at RRDIDX@#@COLOR@:@RRDPARAM@
         -u 100
         -l 0
         GPRINT:p at RRDIDX@:LAST: \: %5.1lf (cur)
         GPRINT:p at RRDIDX@:MAX: \: %5.1lf (max)
         GPRINT:p at RRDIDX@:MIN: \: %5.1lf (min)
         GPRINT:p at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

I will check this file further and see if my custom edits are still 
necessary, but would it make sense to blame the def if the graphs in the 
trends column are right while the inode column is not?  I would think 
that since they likely use the same def, the graph would be broken in 
both places.

Sure, they look fine (example below).  The graphs in the trends column 
have all the current data and qtrees/inode/disks, and I can query the 
rrd files themselves with rrdtool and the data looks fine.

ls -l inode,vol,*
-rw-r--r--    1 bb       bb          38532 Oct 14 09:34 
inode,vol,prod01,.rrd
-rw-r--r--    1 bb       bb          38532 Oct 14 09:34 
inode,vol,prod02,.rrd
-rw-r--r--    1 bb       bb          38532 Oct 14 09:34 inode,vol,vol0,.rrd

Tom Georgoulias wrote:

Poked around some more and noticed that the when I view the graph within the inode column display, the URL referenced by that graph with a single value has the parameter "count=1" in it.  If I follow that link, it show the historical graphs with 1 filesystem in each.  If I make the count=2+, more filesystems show up.  Is the count=1 being passed along somehow (maybe as a default) when I first drill down into the column?

Tom

Henrik,

Just wanted to let you know that this bug (Netapp qtree/vol/inode 
graphing) was fixed in 4.1.2p1.  I guess whatever changes you made to 
fix the disk graphing issue for normal systems also took care of this.


Thanks for the good work on Hobbit.

Tom