monitor a series of files for absolute size (alert & trending)
list Jerry Yu
greetings, I need to monitor some os and db backup files for their sizes for alert based on absolute sizes as well as rrd trending. Does Hobbit do this now? I am running 4.2-RC-20060712 on CentOS 4.3/i386. A twist is the file names are timestamped (os-backup-YYYYMMDD-HHmm.star.gz). any suggestions/tricks? I thought of making a copy to a fixed name for monitoring, but it is kinda expensive due to the size of the backups themselves.
list Greg L Hubbard
you can probably write a custom script that will do what is needed, and have the Hobbit client run it wherever you need it. Then you can write another script on the server side to "catch" the data and set up the RRD calls. There are several moving parts, but Henrik has done a nice job explaining what needs to be done, and provides a helpful example. I use this function for a number of custom measurements, and it ticks like a watch. Took me a few hours to get the first one working, and the rest were pretty easy after that. GLH From: Jerry Yu [mailto:user-764c1f364fe0@xymon.invalid] Sent: Thursday, August 03, 2006 8:04 AM To: user-ae9b8668bcde@xymon.invalid Subject: [hobbit] monitor a series of files for absolute size (alert & trending)
▸
greetings,
I need to monitor some os and db backup files for their sizes
for alert based on absolute sizes as well as rrd trending. Does Hobbit
do this now? I am running 4.2-RC-20060712 on CentOS 4.3/i386.
A twist is the file names are timestamped
(os-backup-YYYYMMDD-HHmm.star.gz). any suggestions/tricks? I thought of
making a copy to a fixed name for monitoring, but it is kinda expensive
due to the size of the backups themselves.
list Ralph Mitchell
▸
On 8/3/06, Jerry Yu <user-764c1f364fe0@xymon.invalid> wrote:
greetings, I need to monitor some os and db backup files for their sizes for alert based on absolute sizes as well as rrd trending. Does Hobbit do this now? I am running 4.2-RC-20060712 on CentOS 4.3/i386. A twist is the file names are timestamped (os-backup-YYYYMMDD-HHmm.star.gz). any suggestions/tricks? I thought of making a copy to a fixed name for monitoring, but it is kinda expensive due to the size of the backups themselves.
Why not make a hardlink to each file and change it at midnight with a cron entry?? That gives you the fixed names without the overhead of copying the files. Ralph Mitchell
list Henrik Størner
▸
On Thu, Aug 03, 2006 at 09:04:23AM -0400, Jerry Yu wrote:
I need to monitor some os and db backup files for their sizes for alert based on absolute sizes as well as rrd trending. Does Hobbit do this now? I am running 4.2-RC-20060712 on CentOS 4.3/i386. A twist is the file names are timestamped (os-backup-YYYYMMDD-HHmm.star.gz). any suggestions/tricks? I thought of making a copy to a fixed name for monitoring, but it is kinda expensive due to the size of the backups themselves.
First thing is to get a "file:" entry in client-local.cfg to grab the
data for the latest file. Something like:
file:`ls -t -1 /backup(os-backup-*.tar.gz|head -1`
This runs the "ls -t...." command to determine the filename. Since it
uses a time-sort and grabs only the first line, it should give you the
name of the latest file.
Next you want to track the size of it. In hobbit-clients.cfg
define a FILE entry to track this - it needs to use a regex to match
the filename, and an explicit RRD id to make it always use a specific
RRD file. Perhaps you want to alert if they get bigger than 1 GB. So:
FILE %^/backup/os-backup.*.tar.gz SIZE<1G TRACK=osbackup
Other interesting options for the FILE entry might be "MTIME<86400" to
check that the latest backup file is at most 24 hours old.
[10 minutes later]
OK, I've learnt to test things before sending mails like this. The TRACK
setting for files and directories currently ignores the ID you may pass
to it, and uses the current filename when deciding on the RRD filename.
So to use this, you'll need to grab either the current snapshot and
build that, or the current "all-in-one" patch from
http://www.hswn.dk/hobbitsw/betapatches/
Regards,
Henrik
list Jerry Yu
thanks for verifying, Henrik. I'll wait for 4.2 release to rebuild my RPMs. Right now, I am still settling in with my first Hobbit installation. used to work with bb-1.3/5/9.
▸
On 8/3/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:On Thu, Aug 03, 2006 at 09:04:23AM -0400, Jerry Yu wrote:I need to monitor some os and db backup files for their sizes for alert based on absolute sizes as well as rrd trending. Does Hobbit do this now? I am running 4.2-RC-20060712 on CentOS 4.3/i386. A twist is the file names are timestamped ( os-backup-YYYYMMDD-HHmm.star.gz). any suggestions/tricks? I thought of making a copy to a fixed name for monitoring, but it is kinda expensive due to the size of the backups themselves.First thing is to get a "file:" entry in client-local.cfg to grab the data for the latest file. Something like: file:`ls -t -1 /backup(os-backup-*.tar.gz|head -1` This runs the "ls -t...." command to determine the filename. Since it uses a time-sort and grabs only the first line, it should give you the name of the latest file. Next you want to track the size of it. In hobbit-clients.cfg define a FILE entry to track this - it needs to use a regex to match the filename, and an explicit RRD id to make it always use a specific RRD file. Perhaps you want to alert if they get bigger than 1 GB. So: FILE %^/backup/os-backup.*.tar.gz SIZE<1G TRACK=osbackup Other interesting options for the FILE entry might be "MTIME<86400" to check that the latest backup file is at most 24 hours old. [10 minutes later] OK, I've learnt to test things before sending mails like this. The TRACK setting for files and directories currently ignores the ID you may pass to it, and uses the current filename when deciding on the RRD filename. So to use this, you'll need to grab either the current snapshot and build that, or the current "all-in-one" patch from http://www.hswn.dk/hobbitsw/betapatches/ Regards, Henrik
list Jerry Yu
Henrik, I set it up per the steps above. the correct files now show up under the 'files' column. however, no rrd of any files.*.rrd gets created under the server:/var/lib/hobbit/DB09p/rrd. I do understand if the rrd name doesn't use the track id under 4.2-RC-20060712, then it is not useful anyway. FILE %^/backup/*trans*cp size<500M mtime<3600 track=transDbDump FILE %^/backup/*full*dp size<50G track=fullDbDump
▸
On 8/3/06, Jerry Yu <user-764c1f364fe0@xymon.invalid> wrote:thanks for verifying, Henrik. I'll wait for 4.2 release to rebuild my RPMs. Right now, I am still settling in with my first Hobbit installation. used to work with bb-1.3/5/9. On 8/3/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:On Thu, Aug 03, 2006 at 09:04:23AM -0400, Jerry Yu wrote:I need to monitor some os and db backup files for their sizes for alert based on absolute sizes as well as rrd trending. Does Hobbit do this now? I am running 4.2-RC-20060712 on CentOS 4.3/i386. A twist is the file names are timestamped ( os-backup-YYYYMMDD-HHmm.star.gz). any suggestions/tricks? I thought of making a copy to a fixed name formonitoring, but it is kinda expensive due to the size of the backups themselves.First thing is to get a "file:" entry in client-local.cfg to grab the data for the latest file. Something like: file:`ls -t -1 /backup(os-backup-*.tar.gz|head -1` This runs the "ls -t...." command to determine the filename. Since it uses a time-sort and grabs only the first line, it should give you the name of the latest file. Next you want to track the size of it. In hobbit-clients.cfg define a FILE entry to track this - it needs to use a regex to match the filename, and an explicit RRD id to make it always use a specific RRD file. Perhaps you want to alert if they get bigger than 1 GB. So: FILE %^/backup/os-backup.*.tar.gz SIZE<1G TRACK=osbackup Other interesting options for the FILE entry might be "MTIME<86400" to check that the latest backup file is at most 24 hours old. [10 minutes later] OK, I've learnt to test things before sending mails like this. The TRACK setting for files and directories currently ignores the ID you may pass to it, and uses the current filename when deciding on the RRD filename. So to use this, you'll need to grab either the current snapshot and build that, or the current "all-in-one" patch from http://www.hswn.dk/hobbitsw/betapatches/ Regards, Henrik
list Jerry Yu
eh, neither does the size check generate alerts either. I changed it to size<1G to coerce it to fail as the file is around 20G FILE %^/backup/*full*cmp* size<1G mtime<86400 track=fullDbDump Click on the file as listed under 'files' gives the following: [file:/backup/db_full.cmp_080406.101245] type:100000 (file) mode:644 (-rw-r-----) linkcount:1 owner:506 (sql) group:506 (dbas) size:22155416434 clock:1154705700 (2006/08/04-11:35:00) atime:1154702075 (2006/08/04-10:34:35) ctime:1154701823 (2006/08/04-10:30:23) mtime:1154701823 (2006/08/04-10:30:23)
▸
On 8/4/06, Jerry Yu <user-764c1f364fe0@xymon.invalid> wrote:Henrik, I set it up per the steps above. the correct files now show up under the 'files' column. however, no rrd of any files.*.rrd gets created under the server:/var/lib/hobbit/DB09p/rrd. I do understand if the rrd name doesn't use the track id under 4.2-RC-20060712, then it is not useful anyway. FILE %^/backup/*trans*cp size<500M mtime<3600 track=transDbDump FILE %^/backup/*full*dp size<50G track=fullDbDump On 8/3/06, Jerry Yu <user-764c1f364fe0@xymon.invalid> wrote:thanks for verifying, Henrik. I'll wait for 4.2 release to rebuild my RPMs. Right now, I am still settling in with my first Hobbit installation. used to work with bb-1.3/5/9. On 8/3/06, Henrik Stoerner < user-ce4a2c883f75@xymon.invalid> wrote:On Thu, Aug 03, 2006 at 09:04:23AM -0400, Jerry Yu wrote:I need to monitor some os and db backup files for their sizes for alert based on absolute sizes as well as rrd trending. Does Hobbit do this now? I am running 4.2-RC-20060712 on CentOS 4.3/i386. A twist is the file names are timestamped ( os-backup-YYYYMMDD-HHmm.star.gz). any suggestions/tricks? I thought of making a copy to a fixed name for monitoring, but it is kinda expensive due to the size of the backups themselves.First thing is to get a "file:" entry in client-local.cfg to grab the data for the latest file. Something like: file:`ls -t -1 /backup(os-backup-*.tar.gz|head -1` This runs the "ls -t...." command to determine the filename. Since it uses a time-sort and grabs only the first line, it should give you the name of the latest file. Next you want to track the size of it. In hobbit-clients.cfg define a FILE entry to track this - it needs to use a regex to match the filename, and an explicit RRD id to make it always use a specific RRD file. Perhaps you want to alert if they get bigger than 1 GB. So: FILE %^/backup/os-backup.*.tar.gz SIZE<1G TRACK=osbackup Other interesting options for the FILE entry might be "MTIME<86400" to check that the latest backup file is at most 24 hours old. [10 minutes later] OK, I've learnt to test things before sending mails like this. The TRACK setting for files and directories currently ignores the ID you may pass to it, and uses the current filename when deciding on the RRD filename. So to use this, you'll need to grab either the current snapshot and build that, or the current "all-in-one" patch from http://www.hswn.dk/hobbitsw/betapatches/ Regards, Henrik
list Henrik Størner
▸
On Fri, Aug 04, 2006 at 11:44:17AM -0400, Jerry Yu wrote:
eh, neither does the size check generate alerts either. I changed it to size<1G to coerce it to fail as the file is around 20G FILE %^/backup/*full*cmp* size<1G mtime<86400 track=fullDbDump
Your regex is wrong. "*" by itself doesn't do what you want; it must be ".*". So it should be FILE %^/backup/.*full.*cmp.* size<1G mtime<86400 track=fullDbDump Regards, Henrik
list Jerry Yu
hmm, I suspected that. however, the file is now listed under 'files'. I guess that's not enough?
▸
On 8/4/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:On Fri, Aug 04, 2006 at 11:44:17AM -0400, Jerry Yu wrote:eh, neither does the size check generate alerts either. I changed it to size<1G to coerce it to fail as the file is around 20G FILE %^/backup/*full*cmp* size<1G mtime<86400 track=fullDbDumpYour regex is wrong. "*" by itself doesn't do what you want; it must be ".*". So it should be FILE %^/backup/.*full.*cmp.* size<1G mtime<86400 track=fullDbDump Regards, Henrik
list Henrik Størner
▸
On Fri, Aug 04, 2006 at 11:58:12AM -0400, Jerry Yu wrote:
Your regex is wrong. "*" by itself doesn't do what you want; it must be ".*". So it should be FILE %^/backup/.*full.*cmp.* size<1G mtime<86400 track=fullDbDump hmm, I suspected that. however, the file is now listed under 'files'. I
guess that's not enough?
It goes on the "files" display as soon as the client reports anything about it. However, to apply any kind of action to the file data it must be recognized by hobbit-clients.cfg. That includes size checks and tracking, which is why your size check doesn't trigger and the RRD file doesn't exist. Regards, Henrik
list Jerry Yu
you are absolutely correct, Henrik. corrected the RE and the alerts were triggered. can't wait to have track rrd working as well..
▸
On 8/4/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:On Fri, Aug 04, 2006 at 11:58:12AM -0400, Jerry Yu wrote:Your regex is wrong. "*" by itself doesn't do what you want; it must be ".*". So it should be FILE %^/backup/.*full.*cmp.* size<1G mtime<86400 track=fullDbDump hmm, I suspected that. however, the file is now listed under'files'. Iguess that's not enough?It goes on the "files" display as soon as the client reports anything about it. However, to apply any kind of action to the file data it must be recognized by hobbit-clients.cfg. That includes size checks and tracking, which is why your size check doesn't trigger and the RRD file doesn't exist. Regards, Henrik
list Jordan Mendler
At work we use strict iptables rules, so what exceptions need to be made to get hobbit working? Do I need to allow traffic on port 1984 from the server to the client, client to the server, or both? Also do I need to allow both TCP and UDP traffic on that connection? Thanks, Jordan
list Charles Jones
▸
Jordan Mendler wrote:
At work we use strict iptables rules, so what exceptions need to be made to get hobbit working? Do I need to allow traffic on port 1984 from the server to the client, client to the server, or both? Also do I need to allow both TCP and UDP traffic on that connection?
In "normal" mode, the hobbit clients send information TO the server on TCP port 1984. If you use hobbitfetch then you need to allow outbound TCP port 1984 to the clients. Hobbit does not use UDP for anything. -Charles