Ack events sent to multiple BBDISP (issue with bback.cgi and cookies) 🔗 link

Hi,

Hi,

Hi,

Hi,

Hi,

Hi,

Hi,

A co-worker worked on the multiple BBDISP ack problem i posted here in
February, and i'd like to share his work :-)

Here his investigation results and the fix :

1.  Hobbit does not allow ack's without the correct acknowledgement number
as it  is this acknum that is used to look up the {HOST}.{TEST} to be
acknowledged.  So there was no way that the simple trick of using the main/
area section in  hobbitserver.cfg was going to work. A side note....
-{ACKNUM} will acknowledge  all events for {HOST}
  
 2. The code only ever forwards to the first IP listed in  BBDISPLAYS if the
messages type is not one of those as defined in lib/sendmsg.c:multircptcmds.

static char *multircptcmds[] = { "status", "combo",  "meta", "data",
"notify", "enable", "disable", "drop", "rename", "client",  NULL };

3. The ordering of the parameters in hobbitcfg.conf is  critical. The --area
must be prior to the --env file parameter so that the  correct environment
is  selected.
  
4.  We need to use --nopin
  
5.  We can keep the cookies enabled


The Fix :D


The new bb-ack.cgi now does the following:

1.  Grab a list of all acknowledgable events from the list of Hobbit
BBDISPLAY  servers.

2.  Present the normal looking "Acknowledge Alert" page, showing the list of
events and entry fields to send acknowledgements.

3.  Send the acknowledgements to the relevant BBDISPLAY  servers.

4.  Provide feedback as normal for the acknowledgements sent.

  
The  Configuration:

1.  Change the CGI_ACK_OPTS setting in /opt/hobbit/server/etc/hobbitcgi.conf
to:-

CGI_ACK_OPTS="--area=maint  --env=/opt/hobbit/server/etc/hobbitserver.cfg
--no-pin"
 
2.  Add to ~hobbit/server/etc/hobbitserver.conf

#  acks for others  hobbit servers
maint/BBDISP="0.0.0.0"
maint/BBDISPLAYS="$BBSERVERIP  10.0.0.2 10.0.0.3"
#  acks end
 
Known Issues:
 
1.  Not fully tested =-o
  
     Could be issues with performance if the  BBDISPLAY list is long.
  
     If a BBDISPLAY does not respond (IP is unbound, etc.)  then there is a
delay while the TCP connect timesout [order of a few  seconds].
  
     Theoretical possibility of timing issue between alerts  being 'raised'
and  acknowledged while the alerts are being propogated  between BBDISPLAYS.
  
2.  CGI code is not fantastically 'safe' but I think that's constant with
the rest  of the code (i.e. I haven't made it any worse!) :/.


Find the new bb-ack.c in attachment. The modifications done are commented
between /* Begin Changes for multiBBDISPLAYS for acks */ tag.

Henrik, if this fix is ok for you, it could be good to add it into xymon.

Regards,

Nico


On 11/02/09 20:19, "nico" <user-2c7a80acbe2b@xymon.invalid> wrote:

Hi,
Sorry, i'm sending again this email because when i wrote the last, i didn't
see it has been merged in another thread :-(
I'd like to send ack events to multiple BBDISPLAY.

To do that, i did a change on hobbitserver.cfg with this configuration:

http://www.hswn.dk/hobbiton/2005/10/msg00203.html

1) In hobbitserver.cfg on both display servers, after the BBDISPLAY
   setting add
        # For the enable/disable and acknowledge scripts.
        maint/BBDISP="0.0.0.0"
        maint/BBDISPLAYS="10.0.0.1 172.16.12.10"
   (replace with your display-server IP-adresses)
2) In hobbitcgi.cfg, change the line for the bb-ack.cgi tool:
        # bb-ack.cgi options
        CGI_ACK_OPTS="--area=maint
--env=/usr/lib/hobbit/server/etc/hobbitserver.cfg"

The first problem is that it seems to not send acks events to the multiples
IP but only to local.

The second problem is that when i modified BBDISP with the IP of another
hobbit server, we got an error:

2009-02-11 17:02:52 Cookie 338777 not found, dropping ack
2009-02-11 17:06:34 Cookie 338777 not found, dropping ack

The reason is that the cookie is different for the same host/test:

serverA$ ${BB} ${BBSERVERIP}:${BBPORT} "hobbitdboard color=red,yellow
fields=hostname,testname,cookie"
cusaa-ws02|ftp|640796

serverB$ ${BB} ${BBSERVERIP}:${BBPORT} "hobbitdboard color=red,yellow
fields=hostname,testname,cookie"
cusaa-ws02|ftp|338777

We tried to disable cookie by adding "--no-cookies" into CGI_ACK_OPTS
variable of hobbitcgi.cfg but the problem still happens.

Is there any way to fix it please ?
Thanks.
Regards,
nicolas

Hi all,
I¹ve been trying to create graphs with Devmon, and a few internet howtos out
there, and for the likes of me, I can¹t seem to figure out how to get them to
show up.

Maybe someone can point out to me what I¹m doing wrong?

I followed the advice here:


http://permalink.gmane.org/gmane.comp.monitoring.hobbit/10245

I¹m not to sure what they mean about #4

With putting an entry in the hobbitgraph.cfg: [if_load]

is there only 1 entry, or 1 entry for every test?
one entry for [if_load}, one entry for [if_error] etc etc...

Also with regards to #6, what is this patch they are talking about?

and I got the extra-rrd.pl script from here:

http://www.hswn.dk/hobbiton/2007/09/msg00148.html

any one?

thanks

William

-- 
This message has been scanned for dangerous content by MailScanner, and is believed to be clean.

On Wednesday 01 April 2009 21:42:16 William Ottley wrote:

Why do you consult out-of-date howtos, instead of the documentation supplied with the software (see docs/GRAPHING, or http://devmon.svn.sourceforge.net/viewvc/devmon/trunk/docs/GRAPHING)

(this documentation originated shortly after the mailing list entry you refer to below, and hasn't been updated that much since the devmon rrd collector was merged into Xymon, so I am updating it a bit to reflect that, and reduce the amount of detail on the implementation).

If you believe there are errors with that document, please feel free to supply a patch to improve it.

Please supply the URLs of the howtos, so I can try and have the author note that it is outdated and refer to the authoritative documentation.

Since you haven't listed what you have done, or what the results are, no.

None of your mails to the mailing list have actually stated which versions of anything you are running, so maybe:
1)Are you using Hobbit 4.2.0, or Xymon 4.2.2 or later ?
2)What version of devmon are you running?

The advice may differ based on the answers (hint, it's much easier using the latest software).

I followed the advice here:


http://permalink.gmane.org/gmane.comp.monitoring.hobbit/10245

This advice is outdated (which is to be expected, considering it is over 18 months old), and applies to software for which there have been two subsequent releases (each).

I am not aware of any devmon templates currently enabling RRD graphs for if_error (although, it can be done quite easily, but maybe you should stick to what more or less works out-the-box, if_load). Current devmon (0.3.0 final or later) ships with the graph definitions in docs/devmon-graph.cfg.

Also with regards to #6, what is this patch they are talking about?

An outdated one.

and I got the extra-rrd.pl script from here:

http://www.hswn.dk/hobbiton/2007/09/msg00148.html

any one?

If you are running Xymon 4.2.2 or later, this is what you need to do to get graphs (let's concentrate on if_load first) working:

1)Ensure you have 'if_load=devmon' in the TEST2RRD variable in hobbitserver.cfg (should be there by default AFAIK)
2)Ensure you have "if_load::1" and "devmon" in the GRAPHS variable in hobbitserver.cfg (should be there by default AFAIK)

At this point, you should have RRD files for any hosts that have devmon if_load tests, e.g. $BBVAR/rrd/<hostname>/if_load.<interface_name>.rrd

3)Make sure Hobbit/Xymon knows how to draw the graph, by ensuring that there is a graph definition for if_load. The easiest way (IMHO), is to add "directory /etc/xymon/hobbitgraph.d" to hobbitgraph.cfg, and drop the devmon-graph.cfg from Devmon's extras directory in /etc/xymon/hobbitgraph.d.

Hi again Buchan,

Your instructions say:

2)Ensure you have "if_load::1" and "devmon" in the GRAPHS variable in
hobbitserver.cfg (should be there by default AFAIK)

What I have is: devmon::1,if_load::1

Slightly different. Is this ok?

With regards to the rrd files,
All I have is one file for each of the cisco's I'm monitoring:

$BBVAR/rrd/<hostname>/tcp.conn.rrd

I have a few servers that are being monitored, and there are several
different .rrd files.....

With regards to your #3. I'm a bit confused.
My structure is different.
I have everything in:

/home/xymon/ data
/home/xymon/ cgi-secure
/home/xymon/ cgi-bin
/home/xymon/ client
/home/xymon/ server /etc...
/home/xymon/ devmon

I don't see the hobbitgraph.d
I do see the devmon-graph.cfg in /home/xymon/devmon/extras/
So I just simply did a cat devmon-graph.cfg >> hobbitgraph.cfg

I restarted everything, and I still only get the one graph: tcp.conn.rrd


So maybe I should show you how my web page looks like, and an example of my
bb-hosts file?


My Web page has:


Switches / Firewall
                                conn                                    info
trends
firewall    conn:green:2d12h51m     info:green:10.136.209.150
trends:green:
walmart    conn:green:2d15h22m     info:green:161.174.32.2     trends:green:


Each of these sections only has 1 graph.

Bb-hosts:

page switches Switches / Firewall
group-compress <H3><I>Switches / Firewall</I></H3>
10.136.209.150 firewall         #
DEVMON:model(cisco;asa),cid(cmisnmp),tests(memory,if_stat,if_load,if_err,if_
dsc,cpu,cluster,serial,specs,connects)
161.174.32.2 walmart            # DEVMON:model(cisco;2600),cid(cmisnmp)


Does this help?

Will

-- 
This message has been scanned for dangerous content by MailScanner, and is believed to be clean.

On Thursday 02 April 2009 16:21:50 William Ottley wrote:

OK, so devmon isn't actually polling these devices. Having (e.g.) an if_load 
test displayed on the hobbit server is the first step (you won't get rrd files 
for the if_load test if Hobbit isn't getting any if_load data) ....

What do you get from running devmon with the -readbbhosts option? Have these 
hosts (walmart, firewall) been added to the devmon hosts.db file (you can use 
'strings /path/to/hosts.db' to see what text is in the db file)? Are there any 
relevant errors in the devmon log file?

Hi Buchan,
I guess what you say makes sense.
I only get the "default" xymon page.

Now you say I have to put in the tests in bb-hosts, but how is this done?
When you say "an if_load test displayed on the hobbit server"

To me I would put that test in the bb-hosts file like this:??

HOW I HAVE IT NOW:
161.174.32.2 walmart            # DEVMON:model(cisco;2600),cid(cmisnmp)

HOW I SHOULD HAVE IT??:
161.174.32.2 walmart            # if_load
DEVMON:model(cisco;2600),cid(cmisnmp)
{I tried it this way, and nothing comes up}


Output from /home/xymon/devmon/hosts.db:

firewall^[10.136.209.150^[cisco^[asa^[memory,if_stat,if_load,if_err,if_dsc,c
pu,cluster,serial,specs,connects^[cmisnmp^[^[
router-main^[10.136.209.1^[cisco^[2600^[all^[cmisnmp^[^[
switch-233^[10.136.209.233^[cisco^[2960^[all^[public^[^[
walmart^[161.174.32.2^[cisco^[2600^[all^[cmisnmp^[^[


When I do devmon ‹readbbhosts

Nothing omes up, I need to use the ­vv, and this comes up:

[xymon at centos devmon]$ ./devmon --readbbhosts -vv
[09-04-02 at 07:11:22] Option 'bblocation' defaulting to:
[09-04-02 at 07:11:23] SNMP querying all hosts in bb-hosts file, please wait...
[09-04-02 at 07:11:23] Checking if           # bbd http://centos/ apache
matches NET:.
[09-04-02 at 07:11:23] Checking if         #  matches NET:.
[09-04-02 at 07:11:23] Checking if            # http://10.136.209.209/ matches
NET:.
[09-04-02 at 07:11:23] Checking if            # matches NET:.
[09-04-02 at 07:11:23] Checking if       # DEVMON:model(cisco;2960) matches
NET:.
[09-04-02 at 07:11:23] Checking if         #
DEVMON:model(cisco;2600),cid(cmisnmp) matches NET:.
[09-04-02 at 07:11:23] Checking if                 #
DEVMON:model(cisco;asa),cid(cmisnmp),tests(memory,if_stat,if_load,if_err,if_
dsc,cpu,cluster,serial,specs,connects) matches NET:.
[09-04-02 at 07:11:23] Checking if                 # if_load if_dsc connects
temp DEVMON:model(cisco;2600),cid(cmisnmp) matches NET:.
[09-04-02 at 07:11:23] Option 'bbdateformat' defaulting to: .
[09-04-02 at 07:11:23] Option 'dispport' defaulting to: 1984.
[09-04-02 at 07:11:23] Option 'dispserv' defaulting to: localhost.
[09-04-02 at 07:11:23] Querying pre-existing hosts
[09-04-02 at 07:11:23] Discovered switch-233 as a cisco 2960
[09-04-02 at 07:11:23] Discovered walmart as a cisco 2600
[09-04-02 at 07:11:23] Discovered router-main as a cisco 2600
[09-04-02 at 07:11:23] Discovered firewall as a cisco asa

-- 
This message has been scanned for dangerous content by MailScanner, and is believed to be clean.

On Thursday 02 April 2009 18:11:32 William Ottley wrote:

Devmon must send it a status report, for which devmon needs to be running 
(nothing else is required by default).

[...]

Output from /home/xymon/devmon/hosts.db:

firewall^[10.136.209.150^[cisco^[asa^[memory,if_stat,if_load,if_err,if_dsc,
c pu,cluster,serial,specs,connects^[cmisnmp^[^[
router-main^[10.136.209.1^[cisco^[2600^[all^[cmisnmp^[^[
switch-233^[10.136.209.233^[cisco^[2960^[all^[public^[^[
walmart^[161.174.32.2^[cisco^[2600^[all^[cmisnmp^[^[

So, devmon has discovered them, good.

Ok, but have you actually started devmon ? As in (in your case). Running it 
with --readbbhosts doesn't start devmon, it just updates the hosts.db file ...

$ ./devmon

or

$ ./devmon -vv

BTW, there are RPMS of xymon and devmon for RHEL/Centos, which have init 
scripts and everyhing ....

If devmon is running, it should be logging to it's log file (even with no 
verbosity, it logs that it starting (IIRC).

Regards,
Buchan