Email alert based on specific process 🔗 link

Hi All

I am using bb-proctab to monitor a list of processes and if any process
a whole group get alert. Is there a way I can setup the alert with
process specific. So if proc 1 out the list dies only user 1 get alert
and so on. I know I could use the SCRIPT feature but like to keep that
as the last resort. I am using hobbit 4.1.2 on Solaris 8. The process
list I am concern about is running bb1.9c client on Solaris 8 as well.

Sorry if it were already discussed in the mailing list.

Thanks

-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
"..there are two kinds of people: those who work and those who take the credit...try
 to be in the first group;...less competition there."  - Indira Gandhi

On Fri, Nov 11, 2005 at 04:22:32PM -0500, Asif Iqbal wrote:

Short answer: No.

Longer answer: Alerts in both BB and Hobbit trigger on a status column.
So you can define an alert that triggers when the "procs" column goes
red, but you cannot define alerts that distinguish between the different
processes that cause the procs column to go red.

I've been thinking about implementing some way of doing this with the
Hobbit client - it would be useful not only for the process-checks, but
also for disk-checks and (once that gets implemented) for the logfile
checks. My idea is that you can associate some random "label" with
the rules you define in the hobbit-clients.cfg file. E.g. if you have
two process checks for a host, the check for the "httpd" process can 
be associated with a "web" label, and the check for the "mysqld" 
process gets a "database" label. Then you can define alerts that use
these labels just like we currently use e.g. the service names, i.e.

   HOST=* 
       MAIL user-e3177f2b6ed0@xymon.invalid LABEL=database
       MAIL user-dd28c5cb1c5a@xymon.invalid LABEL=web


Regards,
Henrik

On Fri, 11 Nov 2005, (Henrik Stoerner) wrote:

Beautiful.  And it would be ideal if those 'labels' also ended up as
columns in the display, not just for notifications.

So for the the 'database' label, you would like to see mysql up, nothing
bad in the log file, and the /mysql partition is OK.


-- 
Scott Walters
-PacketPusher

It would be nice if Hobbit came with a tool, that you could run on your bb servers rrd directory, or a copy of it, and it would automagically copy the data to the host-based subdirs that hobbit uses.

I have a BB server that has been running for about 1.5 years. I have a Hobbit server I have been running in unison for several months, (via the BBRELAY option on the bb server).  Soon I am going to totally decommission the bb server and use only the Hobbit server.

However, I do not wish to lose the trends history that the bb server has. Is there any sort of tool to migrate the rrd data from bb to hobbit? I started doing it manually, and it wasn't very fun.

The method I used to move the memory rrd data over is:

First off, I noticed that the directory structure of the ~hobbit/data/rrd directories and files  is different from the ~bb/bbvar/rrd directories and files:

[*hobbit*]$ ls data/rrd/host.my-domain.net  #Note that hobbit stores the rrds in subdirs
**disk,bigdisk.rrd   *memory.physical.rrd*  tcp.http.host.my-domain.net,.rrd
disk,boot.rrd      *memory.real.rrd*      tcp.http.host.my-domain.net.rrd
disk,root.rrd      *memory.swap.rrd*      tcp.ssh.rrd
la.rrd             procs.rrd            users.rrd
*memory.actual.rrd*  tcp.conn.rrd
(Hmm it looks like there is a duplicate tcp.http.host.my-domain.net,|.rrd? No idea where that came from, but I digress.)

[*bbhost*]$ ls bbvar/rrd |grep host     #Note that bb stores all of the rrds in one dir
host.my-domain.net.disk,bigdisk.rrd
host.my-domain.net.disk,boot.rrd
host.my-domain.net.disk,root.rrd
host.my-domain.net.la.rrd
*host,my-domain,net.memory.actual.rrd
host,my-domain,net.memory.physical.rrd
host,my-domain,net.memory.real.rrd
host,my-domain,net.memory.swap.rrd*
host.my-domain.net.procs.rrd
host.my-domain.net.tcp.conn.rrd
host.my-domain.net.tcp.http.host.my-domain.net.rrd
host.my-domain.net.tcp.ssh.rrd
host.my-domain.net.users.rrd

So it looks like on the bbhost, for some reason, the host name is comma delimited....but only for the memory rrd files? This is strange. Ah well I just need to get them to the hobbit server, and with the proper filenames, and in the proper subdirs. So I scp'd all of the memory rrd files to the hobbit server into /tmp/mem, and then:

[hobbit] $ cd ~hobbit/data/rrd; for hostdir in `ls`; do BBHOST=`echo $hostdir|sed 's/\./,/g'`; cp -vf /tmp/mem/$BBHOST.memory.real.rrd $hostdir/memory.real.rrd; done 
 I repeated this command for the "swap", "actual", and "real" files.
The sed command was to generate the comma-delimited filenames I needed to find the files. This command copied all of the memory rrd files into the proper subdirs.

I'm not sure why only the memory rrd files on my bb server are comma-delimited.  Maybe some misconfiguration I did in the past? The memory graphs look normal in the trends column though...ah well.

It would be nice if Hobbit came with a tool, that you could run on your bb servers rrd directory, or a copy of it, and it would automagically copy the data to the host-based subdirs that hobbit uses.

I have a BB server that has been running for about 1.5 years. I have a Hobbit server I have been running in unison for several months, (via the BBRELAY option on the bb server).  Soon I am going to totally decommission the bb server and use only the Hobbit server.

However, I do not wish to lose the trends history that the bb server has. Is there any sort of tool to migrate the rrd data from bb to hobbit? I started doing it manually, and it wasn't very fun.

The method I used to move the memory rrd data over is:

First off, I noticed that the directory structure of the ~hobbit/data/rrd directories and files  is different from the ~bb/bbvar/rrd directories and files:

[*hobbit*]$ ls data/rrd/host.my-domain.net  #Note that hobbit stores the rrds in subdirs
**disk,bigdisk.rrd   *memory.physical.rrd*  tcp.http.host.my-domain.net,.rrd
disk,boot.rrd      *memory.real.rrd*      tcp.http.host.my-domain.net.rrd
disk,root.rrd      *memory.swap.rrd*      tcp.ssh.rrd
la.rrd             procs.rrd            users.rrd
*memory.actual.rrd*  tcp.conn.rrd
(Hmm it looks like there is a duplicate tcp.http.host.my-domain.net,|.rrd? No idea where that came from, but I digress.)

[*bbhost*]$ ls bbvar/rrd |grep host     #Note that bb stores all of the rrds in one dir
host.my-domain.net.disk,bigdisk.rrd
host.my-domain.net.disk,boot.rrd
host.my-domain.net.disk,root.rrd
host.my-domain.net.la.rrd
*host,my-domain,net.memory.actual.rrd
host,my-domain,net.memory.physical.rrd
host,my-domain,net.memory.real.rrd
host,my-domain,net.memory.swap.rrd*
host.my-domain.net.procs.rrd
host.my-domain.net.tcp.conn.rrd
host.my-domain.net.tcp.http.host.my-domain.net.rrd
host.my-domain.net.tcp.ssh.rrd
host.my-domain.net.users.rrd

So it looks like on the bbhost, for some reason, the host name is comma delimited....but only for the memory rrd files? This is strange. Ah well I just need to get them to the hobbit server, and with the proper filenames, and in the proper subdirs. So I scp'd all of the memory rrd files to the hobbit server into /tmp/mem, and then:

[hobbit] $ cd ~hobbit/data/rrd; for hostdir in `ls`; do BBHOST=`echo $hostdir|sed 's/\./,/g'`; cp -vf /tmp/mem/$BBHOST.memory.real.rrd $hostdir/memory.real.rrd; done 
 I repeated this command for the "swap", "actual", and "real" files.
The sed command was to generate the comma-delimited filenames I needed to find the files. This command copied all of the memory rrd files into the proper subdirs.

I'm not sure why only the memory rrd files on my bb server are comma-delimited.  Maybe some misconfiguration I did in the past? The memory graphs look normal in the trends column though...ah well.

Sounds like you did not read the migration guide ? There is a moverrd.sh script in the source hobbitd dir that will move all your old RRD files to the hobbit installation and reorder them for you.

Regards, Thomas

Charles Jones wrote:

Thomas,

Thanks for pointing out what should have been obvious to me.  Man, I gotta start reading more thoroughly before I ask questions :)

-Charles

Thomas wrote:

Does anyone know if this ever got implemented, I was scanning through a
thread back in Nov 2005?  I need something similar based on ports,  if a
specific port is down I need a specific group notified.

 
Referenced Post:

http://www.hswn.dk/hobbiton/2005/11/msg00170.html

 
Thanks

Trent

 
Trent Melcher
Unix/Linux Systems Administrator
Securities America Financial Corporation
(XXX) XXX-XXXX x2774
user-a7d23e07dd5c@xymon.invalid 

Securities America, Inc., a Registered Broker/Dealer, Member NASD/SIPC.
Advisory services offered through Securities America Advisors, Inc., an
SEC Registered Investment Advisory

The text of this communication is confidential, and use by any person
who is not the intended recipient is prohibited.  Any person who
receives this communication in error is requested to immediately destroy
the text of this communication without copying or further dissemination.
Your cooperation is appreciated.

On Thu, Oct 19, 2006 at 04:03:56PM -0500, Trent Melcher wrote:

See the "Directing alerts to groups" section of the hobbit-clients.cfg
man-page, and the corresponding "GROUP=foo" setting in hobbit-alerts.cfg.


Henrik

Thanks Henrik,  I understand the group piece for notifications.  Your idea
of Labeling a process or port so you can send the specific alert to a
specific group is great and wondered if it ever got implemented.  I couldn't
find anything on it in the docs.

Thanks
Trent

-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] 
Sent: Thursday, October 19, 2006 4:19 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Email alert based on specific process

On Thu, Oct 19, 2006 at 04:03:56PM -0500, Trent Melcher wrote:

See the "Directing alerts to groups" section of the hobbit-clients.cfg
man-page, and the corresponding "GROUP=foo" setting in hobbit-alerts.cfg.


Henrik

Hi all,

Since 4.2.0 Hobbit officially supports the GROUP-label mechanism for
distinguishing processes, disks, files, etc. When I quote Henrik
correctly "alerts in both BB and Hobbit trigger on a status column.",
this gives me the following problem.

Let's say we have the following config:

hobbit-clients.cfg:

SERVER=name
  DISK / 90 95 GROUP=unix
  DISK /database 90 95 GROUP=dba

hobbit-alerts.cfg:

GROUP=dba
  MAIL user-ff23a14cc6fc@xymon.invalid bla bla

GROUP=unix
  MAIL user-c51360387a10@xymon.invalid bla bla

The threshold for /database is exceeded, status is yellow, mail is
sent to the dba. Great! Now the /-partition starts to grow, the status
remains yellow because the dba's are investigating the problem but
have not solved the issue, but the unix-team is not alerted because no
status change has occured.

Any hints???

Below I found a discussion from last year, when I think the idea was
born and the term LABEL was introduced but is called GROUP now.

Thanks,
Peter


2005/11/13, Scott Walters <user-2c405ccfe1ee@xymon.invalid>: