hobbit monitoring 🔗 link

Hi,

Some time ago I promised information about aix monitoring and possible updates to hobbit.  We are going to use hobbit for monitoring, but most of the monitor scripts for AIX will be integrated in a custom written perl library that is also used for making and monitoring backups.
And I can not publish the full library (I wrote it myself, but my boss will not be happy if I do this) and so I also can not publish the monitor scripts :(

Same for trending, hobbit is not powerfull enough for our needs.  And we don't have the C knowledge to change hobbit.  One of the problems is that there is 1 rrd file / check.  But we have checks where the number on 1 page we want to graph can change over time.


Stef

Hi,

Stef Coene wrote :

I made a patch that lets you choose between "one rrd file with multiple
datasources in it per status "or "one rrd per datasource with one DS in
each file ".

Just specify "SPLITNCV_foo" instead of "NCV_foo" in you config file.

I posted it on the list on  "28 Dec 2006 19:36:08 +0100" :

http://www.hswn.dk/hobbiton/2006/12/msg00357.html

Note that you must apply the all-in-one patch before, or at the very
least the ncv-ignoretext patch.

-- 
Charles Goyard - user-98f9625a7a59@xymon.invalid - (+33) 1 45 38 01 31

On Wednesday 17 January 2007 12:22, Charles Goyard wrote:

I once changed the BigBrother scripts so you can control messages in the message that are used for rrd.  It's like ncv, but much more powerfull.  I also hide it with html code.  So you get something like:
<!--RRD EBCT EBCT1 20585056.0512-->
This was picked up by the larrd script.  I also changed the script creates the graphs so I had a "fall-back" entry (I called it blanco) that I used to create graphs from any rrd file I wanted:
/cgi-bin/larrd-grapher.cgi?host=server&service=blanco&part=EBCT,EBCT1

Ok, I missed this one ;)

The biggest problem we have is that we are going to combine lots of different scripts (monitoring, trending, capacity planning, ...) in 1 tool.  And the first usable version needs to be finished before end february.
So, once we have something, we can change what we want.
The idea is that we have 1 server / customer (this can be a vmware image or a dedicated server) that we can use for deployment, monitoring, trending, alerting, inventory, ....  We have most of the stuff, but we need to "glue" them together.  Hobbit for monitoring is clear ;), but we also needs a framework for trending.


The second problem is the fact that I want to distribute the same client and scripts to all my servers.  So the clienlaunch config file needs a parameter where I can specify on which server I want to run the tests (in the old BigBrother days, there was a config file for this).
I don't think this is hard to do, but my C knowledge is really that bad that I don't even try to do this :(


Stef

On Wed, Jan 17, 2007 at 01:02:21PM +0100, Stef Coene wrote:

Try this modification for hobbitlaunch; it adds an "ONHOST" setting to
hobbitlaunch.cfg, so a task will be disabled unless the local hostname
matches this expression. E.g. if you have a task that should only run on
hosts whose hostname begins with "web", then

   [webchecks]
        ENVFILE /usr/lib/hobbit/server/etc/hobbitserver.cfg
	CMD $BBHOME/ext/webchecks.sh
	INTERVAL 5m
	ONHOST ^web


Regards,
Henrik

-------------- next part --------------
--- common/hobbitlaunch.cfg.5	2006/08/09 19:49:47	1.47
+++ common/hobbitlaunch.cfg.5	2007/01/17 12:46:54
@@ -55,6 +55,12 @@
 tasks back into the file, so unless you have them listed as DISABLED
 then tasks may re-appear unexpectedly after an upgrade.
 
+The \fBONHOST\fR keyword tells hobbitlaunch that this task should
+only run on specific hosts. After the ONHOST keyword, you must
+provide a "regular expression"; if the hostname where hobbitlaunch
+runs matches this expression, then the task will run. If it doesn't
+match, then the task is treated as if it were DISABLED.
• The \fBNEEDS\fR instructs hobbitlaunch not to run this task unless
 the task defined by the NEEDS keyword is already running. This
 is used e.g. to delay the start of some application until the
--- common/hobbitlaunch.c	2006/07/20 16:06:41	1.41
+++ common/hobbitlaunch.c	2007/01/17 12:44:12
@@ -26,6 +26,7 @@
 #include <unistd.h>
 #include <errno.h>
 #include <ctype.h>
+#include <regex.h>
 
 #include "libbbgen.h"
 
@@ -56,7 +57,7 @@
 	char *cmd;
 	int interval;
 	char *logfile;
-	char *envfile, *envarea;
+	char *envfile, *envarea, *onhostptn;
 	pid_t pid;
 	time_t laststart;
 	int exitcode;
@@ -80,6 +81,7 @@
 	int freeit = 1;
 	int logfilechanged = 0;
 	int envfilechanged = 0;
+	int disablechanged = 0;
 
 	for (twalk = taskhead; (twalk && (strcmp(twalk->key, newtask->key))); twalk = twalk->next);
 
@@ -113,6 +115,11 @@
 		else {
 			envfilechanged = 0;
 		}
• +		if (twalk->disabled != newtask->disabled)
+			disablechanged = 1;
+		else
+			disablechanged = 0;
 	}
 
 	if (newtask->cmd == NULL) {
@@ -128,16 +135,18 @@
 		tasktail = newtask;
 		freeit = 0;
 	}
-	else if (strcmp(twalk->cmd, newtask->cmd) || logfilechanged || envfilechanged) {
+	else if (strcmp(twalk->cmd, newtask->cmd) || logfilechanged || envfilechanged || disablechanged) {
 		/* Task changed. */
 		xfree(twalk->cmd); 
 		if (twalk->logfile) xfree(twalk->logfile);
 		if (twalk->envfile) xfree(twalk->envfile);
 		if (twalk->envarea) xfree(twalk->envarea);
+		if (twalk->onhostptn) xfree(twalk->onhostptn);
 		twalk->cmd = strdup(newtask->cmd);
 		if (newtask->logfile) twalk->logfile = strdup(newtask->logfile); else twalk->logfile = NULL;
 		if (newtask->envfile) twalk->envfile = strdup(newtask->envfile); else twalk->envfile = NULL;
 		if (newtask->envarea) twalk->envarea = strdup(newtask->envarea); else twalk->envarea = NULL;
+		if (newtask->onhostptn) twalk->onhostptn = strdup(newtask->onhostptn); else twalk->onhostptn = NULL;
 
 		/* Must bounce the task */
 		twalk->cfload = 1;
@@ -161,6 +170,7 @@
 		if (newtask->logfile) xfree(newtask->logfile);
 		if (newtask->envfile) xfree(newtask->envfile);
 		if (newtask->envarea) xfree(newtask->envarea);
+		if (newtask->onhostptn) xfree(newtask->onhostptn);
 		xfree(newtask);
 	}
 }
@@ -172,6 +182,7 @@
 	FILE *fd;
 	strbuffer_t *inbuf;
 	char *p;
+	char myhostname[256];
 
 	/* First check if there were no modifications at all */
 	if (configfiles) {
@@ -186,6 +197,10 @@
 	}
 
 	errprintf("Loading tasklist configuration from %s\n", conffn);
+	if (gethostname(myhostname, sizeof(myhostname)) != 0) {
+		errprintf("Cannot get the local hostname, using 'localhost' (error: %s)\n", strerror(errno));
+		strcpy(myhostname, "localhost");
+	}
 
 	/* The cfload flag: -1=delete task, 0=old task unchanged, 1=new/changed task */
 	for (twalk = taskhead; (twalk); twalk = twalk->next) {
@@ -290,6 +305,25 @@
 		else if (curtask && (strcasecmp(p, "DISABLED") == 0)) {
 			curtask->disabled = 1;
 		}
+		else if (curtask && (strncasecmp(p, "ONHOST ", 7) == 0)) {
+			regex_t cpattern;
+			int status;
• +			p += 7;
+			p += strspn(p, " \t");
• +			curtask->onhostptn = strdup(p);
• +			/* Match the hostname against the pattern; if it doesnt match then disable the task */
+			status = regcomp(&cpattern, curtask->onhostptn, REG_EXTENDED|REG_ICASE|REG_NOSUB);
+			if (status == 0) {
+				status = regexec(&cpattern, myhostname, 0, NULL, 0);
+				if (status == REG_NOMATCH) curtask->disabled = 1;
+			}
+			else {
+				errprintf("ONHOST pattern '%s' is invalid\n", p);
+			}
+		}
 	}
 	if (curtask) update_task(curtask);
 	stackfclose(fd);
@@ -310,6 +344,7 @@
 			if (twalk->logfile) xfree(twalk->logfile);
 			if (twalk->envfile) xfree(twalk->envfile);
 			if (twalk->envarea) xfree(twalk->envarea);
+			if (twalk->onhostptn) xfree(twalk->onhostptn);
 			break;
 
 		  case 0:
@@ -437,12 +472,14 @@
 			for (twalk = taskhead; (twalk); twalk = twalk->next) {
 				printf("[%s]\n", twalk->key);
 				printf("\tCMD %s\n", twalk->cmd);
+				if (twalk->disabled) printf("\tDISABLED\n");
 				if (twalk->group)    printf("\tGROUP %s\n", twalk->group->groupname);
 				if (twalk->depends)  printf("\tNEEDS %s\n", twalk->depends->key);
 				if (twalk->interval) printf("\tINTERVAL %d\n", twalk->interval);
 				if (twalk->logfile)  printf("\tLOGFILE %s\n", twalk->logfile);
 				if (twalk->envfile)  printf("\tENVFILE %s\n", twalk->envfile);
 				if (twalk->envarea)  printf("\tENVAREA %s\n", twalk->envarea);
+				if (twalk->onhostptn) printf("\tONHOST %s\n", twalk->onhostptn);
 				printf("\n");
 			}
 			return 0;

On Wednesday 17 January 2007 10:38, Stef Coene wrote:

Just wondering, how hard would it be to create an extra channel for trending?
So you can use the bb client to send "numbers" to the hobbit server together 
with some extra control information.  

bb <bb server> trends <server name>
<rrd file name> <ds name> <number> <options>
...

<rrd file name> is the name of the rrd file.
<ds name> is the name you want to store in the rrd file
<number> is the number you want to store in the rrd file
<options> can be used to specify rrd settings (for instance type GAUGE)

Maybe you can also specify somehow that you want to use 1 rrd for multiple 
counters.

This can be very powerfull because you can put any number in a rrd file.


Stef

Stef Coene wrote :

It's a bad idea, because you hook the underlying data storage format
into the protocol.

You can always add something like that :

hobbitd_channel --channel=status --log=/var/log/hobbit/rrd-customengine-status.log hobbitd_rrdcustomengine --rrddir=/usr/local/hobbit/data/rrd
hobbitd_channel --channel=data --log=/var/log/hobbit/rrd-customengine-data.log hobbitd_rrdcustomengine --rrddir=/usr/local/hobbit/data/rrd

It's up to the reader to write the hobbitd_rrdcustomengine program (can
be Perl, Shell, whatever suits you).


-- 
Charles Goyard - user-98f9625a7a59@xymon.invalid - (+33) 1 45 38 01 31

On Wed, Jan 17, 2007 at 02:04:29PM +0100, Stef Coene wrote:

You wouldn't need a dedicated channel for this, you can use the "data"
type message.

I've made a very quick implementation that takes this input

   data SERVERNAME.trends

   [file1.rrd]
   DS:set1:GAUGE:600:0:U 10
   DS:set2:DERIVE:600:0:U 20

   [file2.rrd]
   DS:anotherset1:COUNTER:600:0:U 12345

and then generates/updates the two RRD files. file1.rrd has
two datasets ("set1" and "set2") with the values "10" and "20",
and file2.rrd has one dataset "anotherset1" with the value "12345".

Would that fulfill your needs ?


Regards,
Henrik

On Wed, Jan 17, 2007 at 02:16:49PM +0100, Charles Goyard wrote:

True, but I can see it being useful for custom trends tracking. 
The "NCV" module also has a heavy coupling between the way the client
formats the raw data, and how it gets stored in the RRD files.

I would do it differently (because I hate having to update clients),
but I won't insist that people shouldn't aim at their own feet when
pulling the trigger :-)

Anyway, a reasonable implementation takes less than 100 lines of code.
It compiles, and with a bit of luck it may even work! Apply the attached
patch with

   cd hobbit-4.2.0
   patch -p0 </tmp/trends.patch

Then run "make" and install (at least) the new hobbitd/hobbitd_rrd
binary and restart Hobbit.


Regards,
Henrik

-------------- next part --------------
--- /dev/null	2006-10-26 23:11:53.000000000 +0200
+++ hobbitd/rrd/do_trends.c	2007-01-17 15:20:55.184807439 +0100
@@ -0,0 +1,127 @@
+/*----------------------------------------------------------------------------*/
+/* Hobbit RRD handler module.                                                 */
+/*                                                                            */
+/* This module handles custom "trends" data.                                  */
+/*                                                                            */
+/* Copyright (C) 2007 Henrik Storner <user-ce4a2c883f75@xymon.invalid>                         */
+/*                                                                            */
+/* This program is released under the GNU General Public License (GPL),       */
+/* version 2. See the file "COPYING" for details.                             */
+/*                                                                            */
+/*----------------------------------------------------------------------------*/
• +static char trends_rcsid[] = "$Id: do_counts.c,v 1.5 2006/06/09 22:23:49 henrik Rel $";
• +/*
+ * This module was inspired by a mail from Stef Coene:
+ *
+ * Date: Wed, 17 Jan 2007 14:04:29 +0100
+ * From: Stef Coene
+ * Subject: Re: [hobbit] hobbit monitoring

+ * 
+ * On Wednesday 17 January 2007 10:38, Stef Coene wrote:
+ * > Same for trending, hobbit is not powerfull enough for our needs.  And we
+ * > don't have the C knowledge to change hobbit.  One of the problems is that
+ * > there is 1 rrd file / check.  But we have checks where the number on 1 page
+ * > we want to graph can change over time.
+ * Just wondering, how hard would it be to create an extra channel for trending?
+ * So you can use the bb client to send "numbers" to the hobbit server together
+ * with some extra control information.
+ * 
+ * bb <bb server> trends <server name>
+ * <rrd file name> <ds name> <number> <options>
+ *
+ */
• +/*

+ * To use this, send a "data" message to hobbit like this:
+ *
+ *    data $MACHINE.trends
+ *    [filename.rrd]
+ *        DS-definition1 VALUE2
+ *        DS-definition2 VALUE2
+ *
+ * E.g. to create/update a custom RRD file "weather.rrd" with two 
+ * GAUGE datasets "temp" and "wind", with current values "21" and 
+ * "8" respectively, send this message:
+ *
+ *    [weather.rrd]
+ *    DS:temp:GAUGE:600:0:U 21
+ *    DS:wind:GAUGE:600:0:U 8
+ */
• +static int do_trends_rrd(char *hostname, char *testname, char *msg, time_t tstamp)
+{ 
+	char *boln, *eoln, *p;
+	char *rrdfn = NULL;
+	int dscount = 0;
+	char **creparams;
• +	creparams = (char **)calloc(7, sizeof(char *));
• +	boln = strchr(msg, '\n'); if (boln) boln++;
+	while (boln && *boln) {
+		eoln = strchr(boln, '\n'); if (eoln) *eoln = '\0';
• +		if (*boln == '[') {
+			/* Flush the current RRD file */
+			if (rrdfn && (dscount > 0)) {
+				/* Setup the static RRD create-parameters */
+				creparams[0] = "rrdcreate";
+				creparams[1] = rrdfn;
+				creparams[2+dscount] = rra1; dscount++;
+				creparams[2+dscount] = rra2; dscount++;
+				creparams[2+dscount] = rra3; dscount++;
+				creparams[2+dscount] = rra4; dscount++;
+				creparams[2+dscount] = NULL; dscount++;
• +				create_and_update_rrd(hostname, rrdfn, creparams, NULL);
• +				xfree(rrdfn); rrdfn = NULL;
+				creparams = (char **)realloc(creparams, 7*sizeof(char *));
+				memset(creparams, 0, 7*sizeof(char *));
+				dscount = 0;
+			}
• +			/* Get the RRD filename */
+			p = strchr(boln+1, ']'); if (p) *p = '\0';
+			rrdfn = strdup(boln+1);
• +			/* And setup the initial rrdvalues string */
+			sprintf(rrdvalues, "%d", (int)tstamp);
+		}
+		else if (strncmp(boln, "DS:", 3) == 0) {
+			char *valptr = boln + strcspn(boln, " \t");
• +			if ((*valptr == ' ') || (*valptr == '\t')) {
+				*valptr = '\0'; valptr += 1 + strspn(valptr+1, " \t");
+				dscount++;
+				creparams = (char **)realloc(creparams, (7+dscount)*sizeof(char **));
+				creparams[1+dscount] = boln;
+				sprintf(rrdvalues+strlen(rrdvalues), ":%s", valptr);
+			}
+		}
• +		boln = (eoln ? eoln+1 : NULL);
+	}
• +	/* Do the last RRD set */
+	if (rrdfn && (dscount > 0)) {
+		/* Setup the static RRD create-parameters */
+		creparams[0] = "rrdcreate";
+		creparams[1] = rrdfn;
+		creparams[2+dscount] = rra1; dscount++;
+		creparams[2+dscount] = rra2; dscount++;
+		creparams[2+dscount] = rra3; dscount++;
+		creparams[2+dscount] = rra4; dscount++;
+		creparams[2+dscount] = NULL; dscount++;
• +		create_and_update_rrd(hostname, rrdfn, creparams, NULL);
+	}
• +	if (rrdfn) xfree(rrdfn);
+	xfree(creparams);
• +	return 0;
+}
• --- ../hobbit-4.2.0/hobbitd/do_rrd.c	2006-08-09 22:10:05.000000000 +0200
+++ hobbitd/do_rrd.c	2007-01-17 15:21:19.516361306 +0100
@@ -309,6 +309,7 @@
 #include "rrd/do_external.c"
 #include "rrd/do_filesizes.c"
 #include "rrd/do_counts.c"
+#include "rrd/do_trends.c"
 
 #ifdef USE_BEA2
 #include "rrd/do_bea2.c"
@@ -368,6 +369,7 @@
 	else if (strcmp(id, "proccounts") == 0)  res = do_counts_rrd("processes", hostname, testname, msg, tstamp);
 	else if (strcmp(id, "portcounts") == 0)  res = do_counts_rrd("ports", hostname, testname, msg, tstamp);
 	else if (strcmp(id, "linecounts") == 0)  res = do_derives_rrd("lines", hostname, testname, msg, tstamp);
+	else if (strcmp(id, "trends") == 0)      res = do_trends_rrd(hostname, testname, msg, tstamp);
 
 #ifdef DO_ORCA
 	else if (strcmp(id, "orca") == 0)        res = do_orca_rrd(hostname, testname, msg, tstamp);

On Wednesday 17 January 2007 15:05, Henrik Stoerner wrote:

Can you send me your program?  I suppose this is a program that reads the data 
channel and picks up the .trends messages ?

Maybe.  The problem is that I know what I want the result to be, but I don't 
know how to get there.  So I'm trying different things.


Stef

On Wednesday 17 January 2007 13:54, Henrik Stoerner wrote:

Pff.  Help.  It's not working.  But that's not why I'm saying help.  I'm 
saying help because I know that all the changes I need are easy to implement, 
but my C knowledge is 0, nada, zero :((
(implementing this in perl will take me 2 minutes)


Stef

On Wednesday 17 January 2007 15:05, Henrik Stoerner wrote:

update: found the "program" in your other mail.


Stef

On Wednesday 17 January 2007 15:27, Henrik Stoerner wrote:

The "problem" with the ncv module is that I want to use hobbit to create 
graphs.  Lots of graphs (you would be surprised how many usefull counters an 
oracle database has).
This is too complex to handly it with the ncv worker module.


Stef

On Wednesday 17 January 2007 15:40, Stef Coene wrote:

Update: it is working.

That's one of my problems solved:)


Stef

We are beginning to run Oracle under SLES9 on an IBM mainframe and I use
hobbit to monitor some of these linux systems. I will be including the
Oracle server soon. I would be interested in what Oracle values you are
going to monitor and how you got them out of Oracle.

/Thomas Kern
/XXX-XXX-XXXX  

-----Original Message-----
From: Stef Coene The "problem" with the ncv module is that I want to use hobbit to create graphs.  Lots of graphs (you would be surprised how many usefull counters an oracle database has).

On Wednesday 17 January 2007 16:31, Kern, Thomas wrote:

That's the job of my collega, the oracle guru ;)

For the data getting out of oracle, we wrote our own perl library to do fancy 
stuff.  This is primarly for backup, but can also be used to query oracle.  
The problem is I can not publish these libraries :(


Stef

Okay. Thanks. 
I will discuss with my Oracle guru what he wants to monitor and how HE
will be getting the numbers out of Oracle.

/Thomas Kern
/XXX-XXX-XXXX  

On Wednesday 17 January 2007 18:45, Kern, Thomas wrote:

Basically, I wrote a perl module so I have a sqlplus function that I can use.  
It executes sqlplus as user oracle.  Once you can execute sqlplus as a 
function, you can get any information you want in perl.
You can also try to link perl to oracle, but that's not so easy.


Stef

Hobbit monitoring

I managed to make the installation of a server Hobbit.

I have create my bb-hosts and I can see on my web page the result of my bb-hosts.

My problem is that I cannot see the history of my equipments.

Be that the service of hobbit restarts automatically during the starting up of the computer.

How we make to receive alerts resulting from the server via e-mails.

Be that it would be possible to have a paper text to be able to translate him with a translator.

I am not an expert of Linux and my first language is French.

Thank you.