New version of beastat.pl netapp.pl and dbcheck.pl 🔗 link

Hi all... 
Those are just released on sourceforge at
http://sourceforge.net/projects/hobbit-perl-cl/


Those are the Changelog:

Beastat.pl v1.02
1.00
- Initial Release
1.01
- Display timing information at the bottom of the test
- Updated some check for environment variable and put a defbbvar in the
perl file for the default BBVAR directory.
- $debug=1 will not report all the message to the standard output. If
you want the entire content of message while debugging use $debug=2
- Changed the location of .cfg and .key file. Now they go in a directory
called "beastat.pw" into the etc directory. You have to manually move
the files into that directory if you are using the encrypted
authentication.
- Added an option to change the domain part of the name reported to
bb/hobbit during the configuration(see README).
- Change the Multithreading code. MultiThread = 0 (sequential run), = 1
(run as much process as the number of wls domains to test and wait the
for the last one to exit), = n (with n>1, run a maximum of n different
thread in parallel)
1.02
- Changed the check of TEST in bb-hosts to open the file for BB standard
client instead of using a grep that could not workwith some OS.
- Changed how data are read from the java client to understand better
multiline on some errors or fields
- Changed "<text>" into the message read from the java client into
"-text-" because they'll not displayed by hobbit
- Solved a bug with Attribute checking that was reporting errors in some
conditions on the Health field
- Changed the code to reformat data with word wrap using Text::Wrap
- Added a "---- Full Status Report ----" line before reporting full data
(useful to get data and report them in rrd files)
- Added a flag to beastat.pl to choose between using bbhostgrep
(preferred) or reading directly the bb-hosts file.
- Corrected some bugs related to the "notest" setting for single tests
- Added the possibility to include files in the configuration with a
include parameters in the ini file (more info in the beastat.ini file)

Netapp.pl v1.05
- Initial Release
1.00
- Corrected the launch of the bbhostgrep or grep command (bbhostgrep was
always launched)
- Check for the rsh command to work on the filer (return clear tests if
it don't work)
- Report Filer version in cpu column
1.01
- Corrected various condition where the script will exit without
examining other hosts (if one host is down and after reporting error on
the first host in not MULTITHREADING running)
- Added graphs for temperature from environment in a separate column
- Corrected the false reporting of "information condition" as error in
the environment
- Changed the quotas report to report volume directory on the left side
and users on the right side.
- Don't execute snap delta on data ontap version < 7.0
- If we are running Hobbit execute it with "bb hostname @" passing all
data via stdin
- Corrected a bug reporting data for CIFS with data ontap version < 7.0
- Changed some logic in the snapshot/inode/disk check and corrected some
reporting bugs
- Added default in the perl code to prevent errors for new columns added
without change to the config file
- Added check to sysconfig report for: spare disks not zeroed, no spare
disk available, disk in prefail
- state, disk failed (sysconfig -r and sysconfig -d), misconfiguration
(sysreport -c)
- Some bug fix on various functions
1.02
- Added check for snapmirror volume state
1.03
- Display timing information at the bottom of the test
- Updated some check for environment variable and put a defbbvar in the
perl file for the default BBVAR directory.
- $debug=1 will not report all the message to the standard output. If
you want the entire content of message while debugging use $debug=2
- Renamed the default configuration file from filerstats2bb.ini to
netapp.ini you need to manually rename it before launching the new
version of netapp.pl script
- Change the Multithreading code. MultiThread = 0 (sequential run), = 1
(run as much process as the number of filers to test and wait the for
the last one to exit), = n (with n>1, run a maximum of n different
thread in parallel)
1.04
- netapp.ini file was missing from the distribution
1.05
- Changed the check of TEST in bb-hosts to open the file for BB standard
client instead of using a grep that could not work with some OS.
- Added a flag to netapp.pl to choose between using bbhostgrep
(preferred) or reading directly the bb-hosts file.
- Change to disk/inode/snapshot/qtree check to accept absolute free
diskspace/free inode number instead of the % used. Check the netapp.ini
file for instruction
- Change to statistics check (sometimes they got errors if the command
could not finish correctly)
- Added a check for duplicate quota entry
- Corrected some problems with environment check and SATA shelf reported
as errors
- Corrected some bugs related to the "notest" setting for single tests
- Added the possibility to include files in the configuration with a
include parameters in the ini file (more info in the netapp.ini file)

Dbcheck.pl v1.03
1.00
- Initial Release
1.01
- Corrected a condition when if the dbtype was not defined the program
will exit without further tests
- The rollback check are now performed normally
- Corrected a problem with Oracle rowcache reporting
- Display timing information at the bottom of the test
1.02
- Updated some check for environment variable and put a defbbvar in the
perl file for the default BBVAR directory.
- Updated the extent check. I was using the wrong column name so the
check was wrong.
- $debug=1 will not report all the message to the standard output. If
you want the entire content of message while debugging use $debug=2
- Change the Multithreading code. MultiThread = 0 (sequential run), = 1
(run as much process as the number of database to test and wait the for
the last one to exit), = n (with n>1, run a maximum of n different
thread in parallel)
1.03
- Changed the check of TEST in bb-hosts to open the file for BB standard
client instead of using a grep that could not workwith some OS.
- Added a flag to dbcheck.pl to choose between using bbhostgrep
(preferred) or reading directly the bb-hosts file.
- Lock check is now using gv$ tables to comply with RAC check
- Change to invalid object check: there are now 2 parameters, invobjwarn
and invobjignore. Those ara a list of ";" separated list of
"type:user:object" to check. If the object are matched by invobjwarn
they're reported as yellow warning, if they're matched by invobjignore
they're reported as green.
- Change to Tablespace check and parameters to accept absolute value
(followed by k/m/g/t) instead of the % used. Check thedbcheck.ini file
for instructions
- Added the possibility to send all data regarding an instance in a
single column instead of multiple column or to send thesingle and the
multiple check. Check the dbcheck.ini file for instructions
- Add of Request Failed and Last Request Failed Size to the data
reported via MemReq
- Add a parameters in the ini file to check GV$ tables for RAC instead
of V$ tables to get data about locks, processes and sessions. Check
dbcheck.ini and README for more information about RAC check.
- Added some delay to medium (20 seconds) and slow (40 seconds) check
before running them to avoid double data reporting tohobbit for rrd
complaining about double data reporting.
- Added a detail regarding SessionID and Object Locked in the locks
check for Oracle
- Corrected some bugs related to the "notest" setting for single tests
- Added the possibility to include files in the configuration with a
include parameters in the ini file (more info in the dbcheck.ini file)
- Added lifetime of test in the last line of test output


Francesco

Hello,

I got a strange behaviour on a server with solaris 7, I got this output 
when I launch client :

$ ./runclient.sh start
Hobbit client for sunos started on edsvideo
$ ps -efd|grep hobbit
  hobbit 21806 21805  0 11:46:21 ?        0:00 /usr/lib/sa/sadc 300 2
  hobbit 21898 21897  0 11:46:24 ?        0:00 vmstat 300 2
  hobbit 21780     1  0 11:46:20 ?        0:00 
/opt/hobbit/client/bin/hobbitlaunch 
--config=/opt/hobbit/client/etc/clientlaunc
  hobbit 21785 21780  0                   0:00 <defunct>
  hobbit 21781 21780  0                   0:00 <defunct>
  hobbit 21784 21780  0                   0:00 <defunct>
  hobbit 21787 21780  0                   0:00 <defunct>
  hobbit 21783 21780  0 11:46:20 ?        0:00 /usr/bin/sh 
/opt/hobbit/client/ext/bb-sar.sh
  hobbit 21897     1  0 11:46:24 ?        0:00 sh -c vmstat 300 2 
1>/opt/hobbit/client/tmp/hobbit_vmstat.edsvideo.21811 2>&1; 
  hobbit 21904 21903  0 11:46:24 ?        0:00 iostat -c 300 2
  hobbit 21903     1  0 11:46:24 ?        0:00 sh -c iostat -c 300 2 
1>/opt/hobbit/client/tmp/hobbit_iostatcpu.edsvideo.21811 
  hobbit 15431 15421  0 10:51:57 pts/2    0:00 -ksh
  hobbit 21805 21783  0 11:46:21 ?        0:00 /usr/bin/sar -A -o 
/opt/hobbit/client/tmp/sar.21783 300 1
  hobbit 21786 21780  0                   0:00 <defunct>
  hobbit 21782 21780  0                   0:01 <defunct>

Few minutes later, I got this :

$ ps -efd|grep hobbit
  hobbit 21780     1  0 11:46:20 ?        0:00 
/opt/hobbit/client/bin/hobbitlaunch 
--config=/opt/hobbit/client/etc/clientlaunc
  hobbit 21785 21780  0                   0:00 <defunct>
  hobbit 21781 21780  0                   0:00 <defunct>
  hobbit 21784 21780  0                   0:00 <defunct>
  hobbit 21787 21780  0                   0:00 <defunct>
  hobbit 21783 21780  0                   0:00 <defunct>
  hobbit 15431 15421  0 10:51:57 pts/2    0:00 -ksh
  hobbit 21786 21780  0                   0:00 <defunct>
  hobbit 21782 21780  0                   0:01 <defunct>

Thus, nothing is graphed on hobbit server and all tests are purple. I got 
several servers with solaris 7 and everything goes fine on these, so I'm 
just posting here to know if someone got this problem too.
Sincerly,

Thomas


Ce message (et toutes ses pieces jointes eventuelles) est confidentiel et etabli a l'intention exclusive de ses destinataires.
Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est
interdite, sauf autorisation expresse.
L'internet ne permettant pas d'assurer l'integrite de ce message, CNP Assurances et ses filiales declinent toute responsabilite
au titre de ce message, s'il a ete altere, deforme ou falsifie.

*****

This message and any attachments (the "message") are confidential and intended solely for the addressees.
Any unauthorised use or dissemination is prohibited.
E-mails are susceptible to alteration.
Neither CNP Assurances nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified.

Francesco Duranti wrote:

Those are just released on sourceforge at
http://sourceforge.net/projects/hobbit-perl-cl/

Netapp.pl v1.05

This is a great script!  I've been using a modified version of filerstats2bb for sometime, and I'm in the process of switching to netapp.pl instead.  THe ability to track NFS ops is worth the upgrade alone, even though I haven't been able to get that to work just yet?

I was going to try and get the snapshot column to display graphs, so if I get that working I'll pass it along.

Is there an easy way to disable some of the tests, such as quotas and cifs?

Tom

Thanks :D
There's an error in the README ... 
If you're using hobbit 4.2 and NetApp with Ontap version 6.5+ just do
this:

In hobbitserver.cfg

Add those line to the end of the file
NCV_ops="NFSops:GAUGE,CIFSops:GAUGE,iSCSIops:GAUGE,HTTPops:GAUGE,FCPops:
GAUGE,Totalops:GAUGE,*:NONE"
NCV_stats="NETread:GAUGE,NETwrite:GAUGE,DISKread:GAUGE,DISKwrite:GAUGE,T
APEread:GAUGE,TAPEwrite:GAUGE,FCPin:GAUGE,FCPout:GAUGE,*:NONE"

edit the TEST2RRD and GRAPHS columns and add :
",ops=ncv,stats=ncv" to the end of the TEST2RRD line
",ops,stats" to the end of the GRAPHS line

Edit the hobbitgraphs.cfg and add this to the end of file:
[ops]
        TITLE NetApp Operations/Sec
        YAXIS Ops/Sec
        DEF:NFS=ops.rrd:NFSops:AVERAGE
        DEF:CIFS=ops.rrd:CIFSops:AVERAGE
        DEF:iSCSI=ops.rrd:iSCSIops:AVERAGE
        DEF:HTTP=ops.rrd:HTTPops:AVERAGE
        DEF:FCP=ops.rrd:FCPops:AVERAGE
        DEF:Total=ops.rrd:Totalops:AVERAGE
        LINE1:NFS#0000FF:NFS ops
        LINE1:CIFS#FF0000:CIFS ops
        LINE1:iSCSI#00CC00:iSCSI ops
        LINE1:HTTP#FF00FF:HTTP ops
        LINE1:FCP#555555:FCP ops
        LINE1:Total#880000:Total ops
        COMMENT:\n
        GPRINT:NFS:LAST:NFS   ops \: %5.1lf%s (cur)
        GPRINT:NFS:MAX: \: %5.1lf%s (max)
        GPRINT:NFS:MIN: \: %5.1lf%s (min)
        GPRINT:NFS:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:CIFS:LAST:CIFS  ops \: %5.1lf%s (cur)
        GPRINT:CIFS:MAX: \: %5.1lf%s (max)
        GPRINT:CIFS:MIN: \: %5.1lf%s (min)
        GPRINT:CIFS:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:iSCSI:LAST:iSCSI ops \: %5.1lf%s (cur)
        GPRINT:iSCSI:MAX: \: %5.1lf%s (max)
        GPRINT:iSCSI:MIN: \: %5.1lf%s (min)
        GPRINT:iSCSI:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:HTTP:LAST:HTTP  ops \: %5.1lf%s (cur)
        GPRINT:HTTP:MAX: \: %5.1lf%s (max)
        GPRINT:HTTP:MIN: \: %5.1lf%s (min)
        GPRINT:HTTP:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:FCP:LAST:FCP   ops \: %5.1lf%s (cur)
        GPRINT:FCP:MAX: \: %5.1lf%s (max)
        GPRINT:FCP:MIN: \: %5.1lf%s (min)
        GPRINT:FCP:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:Total:LAST:Total ops \: %5.1lf%s (cur)
        GPRINT:Total:MAX: \: %5.1lf%s (max)
        GPRINT:Total:MIN: \: %5.1lf%s (min)
        GPRINT:Total:AVERAGE: \: %5.1lf%s (avg)\n

[stats]
        TITLE NetApp Transfer Rates
        YAXIS Tranfer Rates
        -b 1024
        DEF:Ni=stats.rrd:NETread:AVERAGE
        DEF:No=stats.rrd:NETwrite:AVERAGE
        DEF:Di=stats.rrd:DISKread:AVERAGE
        DEF:Do=stats.rrd:DISKwrite:AVERAGE
        DEF:Ti=stats.rrd:TAPEread:AVERAGE
        DEF:To=stats.rrd:TAPEwrite:AVERAGE
        DEF:Fi=stats.rrd:FCPin:AVERAGE
        DEF:Fo=stats.rrd:FCPout:AVERAGE
        CDEF:Nin=Ni,1024,*
        CDEF:Nout=No,1024,*
        CDEF:Din=Di,1024,*
        CDEF:Dout=Do,1024,*
        CDEF:Tin=Ti,1024,*
        CDEF:Tout=To,1024,*
        CDEF:Fin=Fi,1024,*
        CDEF:Fout=Fo,1024,*
        LINE1:Nin#0000FF:NETin
        LINE1:Nout#FF0000:NETout
        LINE1:Din#00CC00:DISKrd
        LINE1:Dout#FF00FF:DISKwr
        LINE1:Tin#555555:TAPErd
        LINE1:Tout#880000:TAPEwr
        LINE1:Fin#00FF00:FCPin
        LINE1:Fout#000088:FCPout
        COMMENT:\n
        GPRINT:Nin:LAST:NET   in   \: %5.1lf%s (cur)
        GPRINT:Nin:MAX: \: %5.1lf%s (max)
        GPRINT:Nin:MIN: \: %5.1lf%s (min)
        GPRINT:Nin:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:Nout:LAST:NET   out  \: %5.1lf%s (cur)
        GPRINT:Nout:MAX: \: %5.1lf%s (max)
        GPRINT:Nout:MIN: \: %5.1lf%s (min)
        GPRINT:Nout:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:Din:LAST:Disk Read  \: %5.1lf%s (cur)
        GPRINT:Din:MAX: \: %5.1lf%s (max)
        GPRINT:Din:MIN: \: %5.1lf%s (min)
        GPRINT:Din:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:Dout:LAST:Disk Write \: %5.1lf%s (cur)
        GPRINT:Dout:MAX: \: %5.1lf%s (max)
        GPRINT:Dout:MIN: \: %5.1lf%s (min)
        GPRINT:Dout:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:Tin:LAST:Tape Read  \: %5.1lf%s (cur)
        GPRINT:Tin:MAX: \: %5.1lf%s (max)
        GPRINT:Tin:MIN: \: %5.1lf%s (min)
        GPRINT:Tin:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:Tout:LAST:Tape Write \: %5.1lf%s (cur)
        GPRINT:Tout:MAX: \: %5.1lf%s (max)
        GPRINT:Tout:MIN: \: %5.1lf%s (min)
        GPRINT:Tout:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:Fin:LAST:FCP  In    \: %5.1lf%s (cur)
        GPRINT:Fin:MAX: \: %5.1lf%s (max)
        GPRINT:Fin:MIN: \: %5.1lf%s (min)
        GPRINT:Fin:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:Fout:LAST:FCP  Out   \: %5.1lf%s (cur)
        GPRINT:Fout:MAX: \: %5.1lf%s (max)
        GPRINT:Fout:MIN: \: %5.1lf%s (min)
        GPRINT:Fout:AVERAGE: \: %5.1lf%s (avg)\n


At this point restart the hobbit server or kill the hobbitd_rrd file and
check if data are inside your $BBHOME/data/rrd/filersname/ directory.
You should find a ops.rrd and stats.rrd files there

For snapshot there's not a simple way, you've to edit the source code of
hobbit and some will get wrong. At this moment I'm finishing to work to
module for rrd implementation inside the hobbit_rrd for them and I've
something done but I'm waiting for some nice feature regarding custom
rrd implementation from Henrik so you can have something in the next few
days (snapshot, quotas, qtree, and something more in the next future
like full nfsstatistics and interface/network statistics)...

Just put in your [default] section or under the [storage] section of
your netapp.ini this one and it will not check those columns (it's valid
for all columns).
bbuserquotacolumn       = quotas:notest
bbcifscolumn            = cifs:notest

If you want data to be reported silently without having a column just to
create a graph in the trend you can also do:
bbopscolumn             = ops:data
bbstatscolumn           = stats:data
And those columns will be sent as data and not showed in the hobbit web
pages (but the data will still go in the rrd and will be graphed in the
trends column).

Francesco Duranti wrote:

I've got it working now, and made a few tweaks so that the nfs ops (the only thing we use) were tracked on their own graph, instead of as a part of a group.  Makes the graph scaling easier on the eyes.  ;)

I was going to just hack up the do_disk.c file and see what I could do, but sounds like waiting is the better option.  filerstats2bb used to put the .snapshots filesystem alongside those in the disk column, which worked out pretty well except for segregating the snapshot alerts to separate users.

Perfect.

Again, very cool. I now have an ops column on my front page, displaying NFS data, and it is a welcome site.  :)

I was going to try and get a way to collect the disk %utilization values for my volumes (those are displayed when you collect data with statit for a period of time).  That would also be a extremely useful data point.

Tom

I'll create some more definition for the next version so the user can
just choose what to use... I'm also using only NFS on some filers and
NFS/CIFS on some others or just ISCSI on another one...

I've preferred to put the snapshot out of disk space report to have a
clean situation ... A warning on the snapshot can be a problem related to an oracle
snapshot/backup that didn't worked so I've to send alert to db admin to
check them... If you want to give it a try I can send a patch working with 4.2.0 (it
should work from 4.2.0 to the last snapshot) but this will include also
the graphs rrd function for dbcheck.pl and beastat.pl script and is a
work in progress so it could work but it could also do bad thing :P
(well it's running now by one week on my test and production system
without problems).

I'm trying to define what data to get next... The ifinfo also seems
interesting (it also seems the only method to get byte counter related
to network interface). I think the next think I'll get will be: cluster
status, nfsstat, graphs for snapmirrored volumes and data related to
snapvaulted volumes. I've saw the statit command and it's nice the only thing not so good is
the fact that it work in background and there can be some problem
getting good data out of it because running wrong command could get
wrong data on that. I was also checking the "stats show" command that
seems to report the same kind of data and can be run like the sysstat so
with a 300 seconds interval that will let me get the data at the next
run of the script. I should try to understand what of all those data
will really be usefull to monitor and how to graph them .... 

Francesco

Francesco Duranti wrote:

That's cool.  I just made several stats defs to do the same thing, so I 
have disk reads/writes on one graph, net in/out on another.

As soon as I wrap up work on my main project, I'm going to get your 
script working on my secondary hobbit server and start messing around. 
I might be interested in that patch at that tiem.

Those sound good.  I have most of those functions in use, so I can help test

I agree.  I would not want to use statit to get the data from within 
this script, but I'm not sure there is another method.  I was 
researching that when I sent my email.

Other info I was thinking of adding to your script is aggregate status.

Tom