Another stellar release 🔗 link

I've finished implementing Hobbit 4.2 at a customer site, the last little problem being the rrd issue that Henrik (and others) confirmed the problem that I was having with moving graphs between architectures.

When the release notes indicated that there was up to a 50% reduction in CPU Utilization, I really took notice.  This customer runs Hobbit in a shared, virtualized environment (in a Linux virtual machine on the mainframe).  In this environment we like applications that get the work done while using the least amount of CPU and memory possible.

Before 4.2 at it's peak the Hobbit virtual machine would use ~13% of the CPU. I wasn't worried too much about that, since the peak comes at a consistent interval.

After 4.2 I've noticed that the Hobbit virtual machine now barely gets about 1% of CPU Utilization.  I was very surprised at the significant reduction in resources required for this release.

I must commend Henrik, not only on this wonderful achievement, but also on the amazing new features in 4.2.  May all your releases be so good!

Regards,
-- 
Rich Smrcina
VM Assist, Inc.
Phone: XXX-XXX-XXXX
Ans Service:  XXX-XXX-XXXX
user-61add9955ef9@xymon.invalid

Catch the WAVV!  http://www.wavv.org
WAVV 2007 - Green Bay, WI - May 18-22, 2007

And after much time spent RTFM'ing and asking on this mailing list,
I've finally figured out the log monitoring.  It works a little
differently than I thought, but after getting used to it, it's so much
better than BB's implementation.  I look forward to see what future
releases will bring.

On Sun, Aug 27, 2006 at 08:12:08PM -0500, Rich Smrcina wrote:

Sounds like it was every 5 minutes when the webpage update kicked in.

After 4.2 I've noticed that the Hobbit virtual machine now barely gets about 1% of CPU Utilization.  I was very surprised at the significant reduction in resources required for this release.

Nice! Well, it just goes to show how embarassingly non-optimal some of
my early coding was :-) It's good that there are open source profiling
tools available to point out where the hotspots are in your program.

I must commend Henrik, not only on this wonderful achievement, but also on the amazing new features in 4.2.  May all your releases be so good!

I'll do my very best. 

Thanks,
Henrik

Hi Gary,

On Sun, Aug 27, 2006 at 11:19:47PM -0400, Gary B. wrote:

It would be interesting to hear what it was that confused you. If you
have any suggestions for extra documentation or such that would make
it easier to setup, I'm all ears.

The problem with writing docs is that you only do it after you have been
working with something for a long time. So it's difficult to forget
everything you know, and try catch everything you must describe to
someone without that knowledge.


Regards,
Henrik

On 8/28/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:

I will be putting together some internal documentation on log file
monitoring--a "cheat-sheet" of sorts.  Once I complete it, I'll post
what I have here.

I think part of the confusion on my part was having a different idea
in my head of how it should work than what was actually written, and
thus was partly confusing myself.  But I understand what you mean, as
I've often run in to similar problems when trying to explain things to
others who know less about the given subject than I do.

On 8/28/06, Gary B. <user-33b796116d5f@xymon.invalid> wrote:

I've attached two plaintext files.  One is the internal documentation
I wrote for setting up clients for monitoring and alerting.  The other
is the internal documentation I just wrote for setting up log
monitoring.  Both are meant to be used as supplements to the full
documentation.
-------------- next part --------------
Configuring Clients in Hobbit
=============================


*) There are three files of interest, all in /var/hobbit/server/etc:
   bb-hosts		=> Same function as bb-hosts on the BB server
   hobbit-clients.cfg	=> Same function a bb-def.sh on the BB clients
   hobbit-alerts.cfg	=> Same function as bbwarnrules.cfg on the BB server

*) Unlike BB, all client configuration is done on the Hobbit server; no
   need to do anything on the client except install the Hobbit client.

*) Hobbit will accept input from the BB client, as well as use the BB client's
   configuration.  However, installation of the Hobbit client is recommended,
   especially if you plan on making use of Hobbit's message file monitoring.

*) Adding new hosts to Hobbit is a 3-step process:
   1) Add the host to bb-hosts
   2) Configure any custom tests in hobbit-clients
   3) Configure any custom alert rules in hobbit-alerts
      -or-
      Accept the default alerts, and add the host to the appropriate section in
      hobbit-alerts


Adding hosts to Hobbit (bb-hosts)
*) In general, the format of this file is similar to bb-hosts in BB.
*) A few keywords of note:
   1) title <main group title>
      This controls the title you see for the major groups on the main
      Hobbit page
   2) page <page-name> <page title>
      This controls the names of each individual page (i.e. The icons you
      click on to see more details), as well as the title of the page you
      see on the main Hobbit page.
   3) title <page title>  (when this is located below a "page" line)
      This controls the title of the page when you click on it to see more
      details.
   4) group <group name>
      Allows you to separate hosts into individual groups.  This has the same
      function as the group-compress keyword in BB.
*) When adding a host, follow this format:
   <host IP>	<host name>	# <monitoring options>
*) The following options are useful:
   1) conn		=> Not usually needed; enables ping tests
   2) ssh		=> Enables SSH tests
   3) noconn		=> Disables ping tests
   4) telnet		=> Enables telnet tests
   5) dns		=> Enables DNS tests
   6) smtp		=> Enables SMTP tests
   7) ftp		=> Enables FTP tests
   8) http <url>	=> Enables http tests to <url>
   9) CLIENT:<name>	=> Used when the client reports back a different name
                           than "<host name>" as used on the client definition
   10) NAME:<name>	=> Used to display a different name than "<host name>"
                           as used on the client definition


Adding custom host monitoring (hobbit-clients)
*) Unless a host has its own custom tests defined, it will use the defaults:
   1) cpu load:		yellow @ 5, red @ 10
   2) disk usage:	yellow @ 90%, red @ 95%
   3) physical memory:	always green
   4) swap memory:	yellow @ 50%, red @ 80%
*) Host tests are processed from top to bottom.  If using regular expression
   host matching, try to make sure multiple expressions don't match the same
   host, or unexpected things may happen.
*) A few keywords of note:
   1) HOST=<host list>		=> Defines a test for the specified hosts.
				   <host list> may be a single host, comma-
				   separated hosts, or a regular expression
				   prefixed by %
   2) PROC <process> <min> <max> => Check for <process>, and optionally, for
				    a minimum and maximum process count.  If
				    only <min> is defined, check for at least
				    <min> process count.  If no values are
				    defined, default is <min>=0
   3) LOAD <yellow> <red>	=> Go yellow at a load of <yellow>, and red at
				   a load of <red>
   4) MEMSWAP <yellow> <red>	=> Go yellow at <yellow>% swap usage, and red at
				   <red>% swap usage.
   5) DISK <slice> <yellow> <red> => Go yellow at <yellow>% <slice> usage, and
				     red at <red>% usage
*) All other keywords are described in the actual hobbit-clients file
*) To effectively disable memory/disk usage percentage monitoring, define
   both yellow and red tests as 101
*) All host tests can be appended with HOST=<host list> to apply only to
   a specific host(s).  This is useful when a group of hosts have similar
   tests defined, but only a couple of them have additional tests.
   *) NOTE: Tests are evaluated top-to-bottom, so put the host-specific tests
	    above the general tests for the other hosts in a group


Adding host alerts (hobbit-alerts)
*) ** No alerts will be sent out by default, unless the new host(s) are added
   to this file **
*) Alert rules are processed from top to bottom, processing ALL matches for
   any given host
*) A few keywords of note:
   1) HOST=<host/variable>	=> The host(s) that will use the defined rules
   2) RECOVERED			=> Send an alert when the test has recovered
   3) MAIL <address(es)>	=> Send an email to the list of addresses
				   following, comma-separating addresses
   4) REPEAT=<value>		=> Repeat the defined alert every "value" time
				   periods (h=hour, m=min)
   5) SCRIPT <script> <vars>	=> Run the defined script with the defined
				   variables.  Currently, only the
				   /var/hobbit/server/ext/oursms script is
				   defined, which sends a specially-formatted
				   message to our pagers
   6) FORMAT=SMS		=> Currently, only needed when using the above
				   SCRIPT definition
   7) STOP			=> Don't process any additional alert rules for
				   the matching host
   8) COLOR=<colors>		=> Match tests for the given color
   9) SERVICE=<service/*>	=> Match tests for the given service, or all
				   services if "*" is used
   10) TIME=<days>:<start>:<end> => Used to send alerts only during the
			            specified time period
*) All other keywords are described in the actual hobbit-alerts file
*) The format of an alert rule is:
   HOST=<host/variable> <alert type (optional)
	--followed by one or more of the below--
	MAIL <address(es)> <alert type / period (optional)
	SCRIPT <script> <script vars> <alert type / period (optional)
	STOP (optional, and only if needed)
*) There are various custom alerts defined; you should be able to figure out
   anything else you may need to do by looking at some of the rules already
   defined.
-------------- next part --------------
Configuring Log Monitoring
==========================
$Id: logs.txt,v 1.2 2006/08/28 13:58:07 gbaluha Exp $


*) There are two files of interest, all in /var/hobbit/server/etc:
   client-local.cfg	=> Controls which files the client will report on
   hobbit-clients.cfg	=> Controls what to look for in the client's logs

*) The above files are RCS'ed as user "hobbit".  Be sure to "su - hobbit"
   before making any changes to the above files, and/or restarting hobbit.
   This goes for the clients as well.

*) Unlike BB, all client configuration is done on the Hobbit server; no
   need to do anything on the client except install the Hobbit client.


Format of client-local.cfg
*) The appropriate section of this file will get pushed to the individual
   clients being monitored.
*) There are three ways to configure which section will be sent to a client.
   Only *one* of these methods may be used for any given host.
   1) By OS type (from "uname -s" on the client)
   2) By "class" (set when Hobbit starts on the client)
   3) By hostname
*) Individual log files are set with the following format
   [<OS / class / hostname>]
   log:<log file name>:<max size in bytes>
   ignore <string / regex - no spaces>
   trigger <string / regex - no spaces>

	*) Where "<>" is not meant literally
	*) "ignore" and "trigger" are optional, and with v4.2, only *one*
	   "ignore" and *one* "trigger" are allowed with any given "log"
	   statement.
		a) "ignore" will ignore any log entry with the string/regex,
		   and won't send it to the Hobbit server
		b) "trigger" will specifically send any log entry with the
		   string/regex to the Hobbit server, even if it contains a
		   string/regex in the [optional] "ignore" line.
	*) Multiple "log" statements are allowed per OS/class/hostname section.
	*) Hobbit is picky about the format - be sure it is written correctly.


Format of LOG entries in hobbit-clients.cfg
*) The "LOG" keyword format is as such:
   LOG <log file from client-local.cfg> <string / regex> COLOR=<color> IGNORE=<string / regex>
*) If "<string / regex>" is a regex, it needs to be prefixed by "%"
*) The default is to be CASE-INSENSITIVE.  If case-sensitivity is preferred,
   prefix "<string / regex>" by "%(?-i)"
*) Use IGNORE if you want to ignore specific entries that match the first
   "<string / regex>" but contain the IGNORE "<string / regex>".
*) COLOR and IGNORE are optional

good write-up, Gary. maybe this should be added to the Hobbit Wiki.

Question, which takes precedence, if a line matches both IGNORE and the
alert string/reg. One line  of particular interests to me is the SELinux's
pts relabeling warning in /var/log/messages, which is harmless but annoying.
I am on 4.2-RC1-20060712 and have seen such a log line causing this log test
to go red.

In 'hobbit-clients.cfg', I have
log /var/log/messages %WARNING|NOTICE|ERROR IGNORE=relabeling

The offending log lines goes line:
Jul 30 18:22:25 SRV01 su[31747]: Warning!  Could not relabel /dev/pts/0 with
user_u:object_r:devpts_t, not relabeling.Operation not permitted

On 8/28/06, Gary B. <user-33b796116d5f@xymon.invalid> wrote:

On Mon, Aug 28, 2006 at 11:08:20AM -0400, Jerry Yu wrote:

IGNORE takes precedence over the matching rule. So this line would not
trigger an error status.


Regards,
Henrik

Hello,

I can't compile the new version of hobbit client (4.2 stable) on AIX 4.2. I've uncommented the 'CC' and 'CFLAGS' lines in 'build/Makefile.AIX' to use gcc and not the IBM compiler.
There is no problem on AIX 4.3.3. Can you help me ?
Best regards,

Thomas


sigma </opt/gnu/hobbit-4.2.0 >#oslevel
4.2.1.0

sigma </opt/gnu/hobbit-4.2.0 >#gmake 
CC="gcc" CFLAGS="-g -DDEBUG -D_REENTRANT  -DAIX -I. -I`pwd`/include -DCLIENTONLY=1" LDFLAGS="" OSDEF="-DAIX" RPATHOPT="" PCREINCDIR="" SSLFLAGS="" SSLINCDIR="" SSLLIBS="" NETLIBS="" BBTOPDIR="/opt/hobbit" BBLOGDIR="" BBHOSTNAME="" BBHOSTIP="158.157.156.91" BBHOSTOS="" LOCALCLIENT="no" gmake -C lib client
gmake[1]: Entering directory `/opt/gnu/hobbit-4.2.0/lib'
gcc -g -DDEBUG -D_REENTRANT  -DAIX -I. -I/opt/gnu/hobbit-4.2.0/include -DCLIENTONLY=1 -I. -I../include  -o test-endianness test-endianness.c
cpp: installation problem, cannot exec `cpp': The parameter or environment lists are too long.
gmake[1]: *** [test-endianness] Error 1
gmake[1]: Leaving directory `/opt/gnu/hobbit-4.2.0/lib'
gmake: *** [lib-client] Error 2

sigma </opt/gnu/hobbit-4.2.0 >#which cpp
/opt/gnu/usr/local/bin/cpp

sigma </opt/gnu/hobbit-4.2.0 >#gcc --version
2.95.2


Ce message (et toutes ses pieces jointes eventuelles) est confidentiel et etabli a l'intention exclusive de ses destinataires.
Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est
interdite, sauf autorisation expresse.
L'internet ne permettant pas d'assurer l'integrite de ce message, CNP Assurances et ses filiales declinent toute responsabilite
au titre de ce message, s'il a ete altere, deforme ou falsifie.

*****

This message and any attachments (the "message") are confidential and intended solely for the addressees.
Any unauthorised use or dissemination is prohibited.
E-mails are susceptible to alteration.
Neither CNP Assurances nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified.