export usage reports for management
list Steve Holmes
I know this isn't really what Xymon is designed to do, but I've been asked to produce data for a text based report of usage (cpu, memory, and disk weekly averages (which I've yet to convince them might not make sense)) from Xymon history or rrd data for management. The requester will put the data I provide into a spreadsheet. Before I go re-inventing the wheel I'm wondering if anyone has done something like this already. I was unable to find anything on the list archive and the only thing I found on xymonton.org was a pdf report generator which from the description doesn't look like it will work for what I want. I've tested some rrdtool queries (fetch, xport, etc.), but I'm having some difficulty seeing how the data I get corresponds to what I'm seeing on the Xymon page. Any pointers welcome. Thanks, Steve Holmes Purdue University -- If they give you ruled paper, write the other way. -Juan Ramon Jimenez, poet, Nobel Prize in literature (1881-1958) Truth never damages a cause that is just. -Mohandas Karamchand Gandhi (1869-1948)
list Henrik Størner
▸
On 11-01-2012 17:43, Steve Holmes wrote:
I know this isn't really what Xymon is designed to do, but I've been asked to produce data for a text based report of usage (cpu, memory, and disk weekly averages (which I've yet to convince them might not make sense)) from Xymon history or rrd data for management. The requester will put the data I provide into a spreadsheet. Before I go re-inventing the wheel I'm wondering if anyone has done something like this already.
perfdata.cgi - ships with Xymon 4.3.x. Dumps all of the data in the RRD files to a comma-separated file (CSV) that you can import into Excel for further processing.
Hmm, I see it isn't documented. Anyway, I do the extract once a month with this script:
#!/bin/sh
START=`date '+%Y%m01' --date="1 month ago"`
END=`date '+%Y%m01'`
xymoncmd perfdata.cgi $START $END 2>/dev/null | \
gzip >~ftp/perfmonthly/data-$END.gz
If you only want to export data from a particular group of Xymon servers, perfdata.cgi takes a "--page=REGEXP" option to only extract data for hosts that appear on a page matching REGEXP (a regular expression). But it is surprisingly fast to export data from even a very large number of hosts.
There's also a "--csv=CHAR" option, in case your local Excel version uses another character than comma as the delimiter in CSV-files (around here, we use semi-colon...)
Here's an example of the data:
"hostname";"datasource";"rrdcolumn";"measurement";"time";"value"
"myserver01";"la.rrd";"la";"pctbusy";"20111201010000";2.753750
"myserver01";"la.rrd";"la";"pctbusy";"20111201030000";2.359861
"myserver01";"la.rrd";"la";"pctbusy";"20111201050000";3.428056
"myserver01";"la.rrd";"la";"pctbusy";"20111201070000";2.363194
"myserver01";"la.rrd";"la";"pctbusy";"20111201090000";3.246250
As you can see, it is a rather raw export of the RRD data with a timestamp on RRD datapoint - it doesn't do any aggregation/averaging over the past week. You'll have to do that yourself in Excel.
"pctbusy" means the "percent CPU busy". For Unix-systems, perfdata.cgi pulls data from the "vmstat" data, so you will also get a "percent busy" reading for those systems - not a "load average".
There are some other data items:
"myserver01";"memory.actual.rrd";"realmempct";"Virtual";"20111201010000";0.000000
"myserver01";"memory.real.rrd";"realmempct";"RAM";"20111201010000";49.708333
"myserver01";"memory.swap.rrd";"realmempct";"Swap";"20111201010000";23.416667
"Virtual", "RAM" and "Swap" correspond to the data-items you'll see on the "Memory" status in Xymon.
"myserver01";"disk,C.rrd";"pct";"/C";"20111201010000";77.000000
How full a disk is.
Note that since this pulls data directly from the RRD files, the granularity of the data depends on how old they are. If you pull data from the past week, you'll get 5-minute or 30-minute readings; if you pull data more than 48 days old, you'll only get 1-day averages.
Regards,
Henrik
list Steve Holmes
Excellent! Thanks. One more reason to upgrade. But this will save me LOTS of time. Thanks Henrik. Steve
▸
On Wed, Jan 11, 2012 at 12:08 PM, Henrik Størner <user-ce4a2c883f75@xymon.invalid> wrote:
On 11-01-2012 17:43, Steve Holmes wrote:I know this isn't really what Xymon is designed to do, but I've been asked to produce data for a text based report of usage (cpu, memory, and disk weekly averages (which I've yet to convince them might not make sense)) from Xymon history or rrd data for management. The requester will put the data I provide into a spreadsheet. Before I go re-inventing the wheel I'm wondering if anyone has done something like this already.perfdata.cgi - ships with Xymon 4.3.x. Dumps all of the data in the RRD files to a comma-separated file (CSV) that you can import into Excel for further processing. Hmm, I see it isn't documented. Anyway, I do the extract once a month with this script: #!/bin/sh START=`date '+%Y%m01' --date="1 month ago"` END=`date '+%Y%m01'` xymoncmd perfdata.cgi $START $END 2>/dev/null | \ gzip >~ftp/perfmonthly/data-$END.gz If you only want to export data from a particular group of Xymon servers, perfdata.cgi takes a "--page=REGEXP" option to only extract data for hosts that appear on a page matching REGEXP (a regular expression). But it is surprisingly fast to export data from even a very large number of hosts. There's also a "--csv=CHAR" option, in case your local Excel version uses another character than comma as the delimiter in CSV-files (around here, we use semi-colon...) Here's an example of the data: "hostname";"datasource";"**rrdcolumn";"measurement";"**time";"value" "myserver01";"la.rrd";"la";"**pctbusy";"20111201010000";2.**753750 "myserver01";"la.rrd";"la";"**pctbusy";"20111201030000";2.**359861 "myserver01";"la.rrd";"la";"**pctbusy";"20111201050000";3.**428056 "myserver01";"la.rrd";"la";"**pctbusy";"20111201070000";2.**363194 "myserver01";"la.rrd";"la";"**pctbusy";"20111201090000";3.**246250 As you can see, it is a rather raw export of the RRD data with a timestamp on RRD datapoint - it doesn't do any aggregation/averaging over the past week. You'll have to do that yourself in Excel. "pctbusy" means the "percent CPU busy". For Unix-systems, perfdata.cgi pulls data from the "vmstat" data, so you will also get a "percent busy" reading for those systems - not a "load average". There are some other data items:
"myserver01";"memory.actual.**rrd";"realmempct";"Virtual";"**
20111201010000";0.000000
"myserver01";"memory.real.rrd"**;"realmempct";"RAM";"**
20111201010000";49.708333
"myserver01";"memory.swap.rrd"**;"realmempct";"Swap";"**
20111201010000";23.416667
▸
"Virtual", "RAM" and "Swap" correspond to the data-items you'll see on the
"Memory" status in Xymon.
"myserver01";"disk,C.rrd";"**pct";"/C";"20111201010000";77.**000000
How full a disk is.
Note that since this pulls data directly from the RRD files, the
granularity of the data depends on how old they are. If you pull data from
the past week, you'll get 5-minute or 30-minute readings; if you pull data
more than 48 days old, you'll only get 1-day averages.
Regards,
Henrik
______________________________**
Xymon at xymon.com<
▸
-- If they give you ruled paper, write the other way. -Juan Ramon Jimenez, poet, Nobel Prize in literature (1881-1958) Truth never damages a cause that is just. -Mohandas Karamchand Gandhi (1869-1948)
list Henrik Størner
On 11-01-2012 18:17, Steve Holmes wrote:
Excellent! Thanks. One more reason to upgrade.
It actually has very few 4.3-dependencies - I think you could run it on
a 4.2.x setup if you just do
HOSTSCFG=$BBHOSTS
XYMONRRDS=$BBRRDS
export HOSTSCFG XYMONRRDS
before running perfdata.cgi ...
Regards,
Henrik
list Steve Holmes
Thanks again. I did get it to work. However, there must be something hard coded in the source looking for hosts.cfg because even with the environment variables set as below, we were getting errors in the hobbitd column for our 4.2.3 server saying "hosts.cfg not found". We were unable to find the string in the source so I can't tell you where it is coming from. We put a symlink in the server/etc directory so hosts.cfg points to bb-hosts and that eliminated that problem. Thanks,
▸
Steve
On Wed, Jan 11, 2012 at 12:26 PM, Henrik Størner <user-ce4a2c883f75@xymon.invalid> wrote:
On 11-01-2012 18:17, Steve Holmes wrote:Excellent! Thanks. One more reason to upgrade.It actually has very few 4.3-dependencies - I think you could run it on a 4.2.x setup if you just do HOSTSCFG=$BBHOSTS XYMONRRDS=$BBRRDS export HOSTSCFG XYMONRRDS before running perfdata.cgi ... Regards, Henrik ______________________________** Xymon at xymon.com<
-- If they give you ruled paper, write the other way. -Juan Ramon Jimenez, poet, Nobel Prize in literature (1881-1958) Truth never damages a cause that is just. -Mohandas Karamchand Gandhi (1869-1948)
list Jeremy Laidman
▸
On Thu, Jan 12, 2012 at 8:10 AM, Steve Holmes <user-ec1bf77b1b44@xymon.invalid> wrote:
Thanks again. I did get it to work. However, there must be something hard coded in the source looking for hosts.cfg because even with the environment variables set as below, we were getting errors in the hobbitd column for our 4.2.3 server saying "hosts.cfg not found". We were unable to find the string in the source so I can't tell you where it is coming from.
It uses the "HOSTCFG" variable, and is not hard-coded anywhere.
list Hermann-Josef Beckers
xymon-bounces at xymon.com schrieb am 11.01.2012 18:08:36:
Von:
▸
...
perfdata.cgi - ships with Xymon 4.3.x. Dumps all of the data in the RRD files to a comma-separated file (CSV) that you can import into Excel for
further processing. Hmm, I see it isn't documented. Anyway, I do the extract once a month with this script:
Are you sure? My slight modified version
"#!/bin/sh
START=`date '+%Y%m01' --date="1 month ago"`
END=`date '+%Y%m01'`
/home/xymon/server/bin/xymoncmd perfdata.cgi $START $END 2> fehlers | \
gzip > data-$END.gz
exit"
gives
"2012-01-12 08:50:00 Using default environment file /home/xymon/server/etc/xymonserver.cfg
2012-01-12 08:50:00 Usage:
perfdata.cgi HOSTNAME-PATTERN STARTTIME ENDTIME"
Using a HOST variable as HOSTNAME-PATTERN in different combinations
HOST=%lscopy02 HOST=lscopy02(.local.domain) HOST="--page=lscopy02"
gives no errors, but only an empty gz file. Tested with xymon versions
4.3.5/7.
Any hints?
Yours
hjb
list Henrik Størner
▸
On 12-01-2012 09:02, Hermann-Josef Beckers wrote:
Hmm, I see it isn't documented. Anyway, I do the extract once a month with this script:Are you sure? My slight modified version "#!/bin/sh START=`date '+%Y%m01' --date="1 month ago"` END=`date '+%Y%m01'` /home/xymon/server/bin/xymoncmd perfdata.cgi $START $END 2> fehlers | \ gzip > data-$END.gz exit" gives "2012-01-12 08:50:00 Using default environment file /home/xymon/server/etc/xymonserver.cfg 2012-01-12 08:50:00 Usage: perfdata.cgi HOSTNAME-PATTERN STARTTIME ENDTIME"
I should know better than to show commands without trying them ... This one has been tested: xymoncmd perfdata.cgi --page=. $START $END --csv=',' Sorry 'bout that. Regards, Henrik
list Hermann-Josef Beckers
xymon-bounces at xymon.com schrieb am 12.01.2012 09:26:29:
▸
..."2012-01-12 08:50:00 Using default environment file /home/xymon/server/etc/xymonserver.cfg 2012-01-12 08:50:00 Usage: perfdata.cgi HOSTNAME-PATTERN STARTTIME ENDTIME"I should know better than to show commands without trying them ... This one has been tested: xymoncmd perfdata.cgi --page=. $START $END --csv=','
Works as designed.
Sorry 'bout that.
No problem. Yours hjb