Hobbit Average Data 🔗 link

Hello,

 
I've got some folks here that want to keep data all the time.

ie. They don't want to use a Round Robin Database that averages

the data over time. They want the 5 minute data forever.

 
I know we can get into a debate as to the reality of this, but

it's want the performance team here wants. If hobbit can't do it,

they want me to use sar and input to another database.

 
Is there a way to do this with Hobbit?

 
Anyone else had this come up?

 
Thanks..James

 
James Wade

James Wade wrote:

I went through an exercise  several years ago, where I  made some tweaks
to big brother to log alerts to a mysql database, in addition to the
other alerting mechanisms. I declared it a success, and then forgot
about it since management wasn't really interested in the solution,
since they were not aware of any problem.  At any rate, it was fairly
quick and easy to whip up something with perl and DBI.

Joe

On Mon, Nov 26, 2007 at 02:49:27PM -0600, James Wade wrote:

Not out-of-the-box, but doing it wouldn't require a lot of work.
I assume you want to store the data that Hobbit normally extracts
and puts into the RRD files ? I.e. you want to save e.g. just the
current utilisation of the /home filesystem (96%, usually), and not
the full status message ?

In that case, you will want to make some changes to the
hobbit-4.2.0/hobbitd/do_rrd.c file, specifically the
"create_and_update_rrd()" routine in there. This function handles ALL of
the RRD updating, so it is the place you want to put your code to grab
the RRD data and save somewhere permanent. In that routine, it ends up
calling the rrd_update() function in the RRD library:
      result = rrd_update(pcount, updparams);
"updparams" is a table of "pcount" strings, which are really the same
strings that you would feed into an "rrd update" command that you run 
from the commandline. See the rrdupdate(1) man-page - Hobbit always
uses the "-t" option to specify which datasets are being updated and the
sequence in which the datavalues is provided (for RRD files that contain
more than one set of data).

The simplest solution is probably to setup a pipe to a child process,
and then write a line with the RRD filename, the template and the data
to that pipe - the child process can then take care of splitting the
data into individual records, and store them in the database.


Henrik

On Mon, Nov 26, 2007 at 10:22:01PM +0100, Henrik Stoerner wrote:

There's an implementation of this in the current snapshot, including
a new hobbitd_rrd manpage that describes how to run it, and what the
input to your custom script looks like:

 --processor=COMMAND
     Feed the raw RRD data into COMMAND via standard  input.  COMMAND
     must  be a simple command with no options; if necessary, COMMAND
     can be a script that runs the real command  with  the  necessary
     options.  The  data sent to COMMAND consists of lines of text of
     the form
        dsname1[:dsname2] time:value1[:value2] rrdtype [key1] [key2]
     First the dataset names, corresponding to the DS definitions  in
     the  RRD  files.   Next the timestamp of the update, followed by
     the values in the same order as the dsname-list. Third  item  is
     the  type of RRD file. After that there may be between 0-2 addi-
     tional keys: For "disk" this is the filesystem  mountpoint,  for
     "tcp" tests it is the service name, for "tcp" "http" tests there
     is also the URL checked. Note that  filesystem  names  and  URLs
     have  forwardslash substituted by comma. See the hobbitd/rrd/*.c
     sourcecode for details, look for the "setupfn"  function  calls.
     An example:

     la 1196115478:7 la
     la 1196115478:169 procs
     la 1196115478:1 users
     la 1196115478:244 clock
     pct:used 1196115478:63:62338144 disk ,root
     realmempct 1196115478:76 memory real
     realmempct 1196115478:0 memory swap
     realmempct 1196115478:20 memory actual
     sec 1196115478:0.03 tcp conn
     sec 1196115478:1.38 tcp http osiris.hswn.dk,


Regards,
Henrik

I'll ask a question first, what is it that is wrong with the current
implementation. If the "people" want the ability to zoom into a five
minute period from a year back, then you need to modify the rrd files.

We had a sort of similar requirement for capacity planning in that for
servers that peak during the day, and are completely idle at night.
After a period of time, the average down to one sample per day would
not give a good indication of how the server is performing over time.
What we did was create an rrd database that stored MAX values, that
way when the data is reduced to one sample per day, it was the max
value for the day. We only did this for a 15 minute uptime value chart
and from memory, we set up a script that created the rrd file to our
spec the first time a sample flowed through based on current
functionality within hobbit to add custom processing for different
alerts. I don't have access to the system right now to check how it
was done.

Cheers

Phil

On 27/11/2007, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:

-- 

Tel: XXXX XXX XXX
Fax: XXXX XXX XXX
email: user-e365c1418192@xymon.invalid

Hi Phil,

On Tue, Nov 27, 2007 at 03:08:22PM +0900, Phil Wild wrote:

Good question.

There's really nothing wrong with the current implementation, but it
does have some limitations. You *can* tune the RRD definitions to save
more 5-minute measurements, or store MIN/MAX values (in fact, the
snapshots have had the capability for some time to configure the RRD 
file layout so you don't have to dig into the C code).

The main limitation is that RRD will *always* normalize data. It is
quite rare for updates to happen exactly at 300 second intervals, and
when they are slightly off then RRD will not store the raw data;
instead, it will compute what the value probably would have been, if the
measurement had happened exactly at the 300-second mark. So you really
cannot get the exact measurements (the observed data) from the RRD
files.

In most cases this doesn't matter since the differences are small, and
measurements do happen roughly at the "right" time. But it is something
you should consider when trying to tap into the Hobbit measurements.


The main reason I added this was because it would allow users an easy 
hook into the raw data that Hobbit collects. The data that is fed into 
the RRD files is extremely standardized, as opposed to most of the other 
data available in Hobbit. So if you do want to add some custom processing 
of monitoring data, this is a good place to do it. And I could see some
uses for it myself.

Hobbit is intended to provide a good basis for your monitoring, but it
cannot solve everyones needs. If I can add a few lines of code that lets
users connect their own custom tools so they are fed by the data that
Hobbit collects, then there is a much better chance that they will stick
with Hobbit instead of picking some other tool.


So I added it, because it makes Hobbit much more flexible and allows
you to make more use of the data that Hobbit collects.


Regards,
Henrik

On Monday 26 November 2007, James Wade wrote:

I created a perl script that updates all rrd files.  It adds MAX and MIN.
It also adds more data point.  It uses more space to store, but the MAX and 
MIN can be handy to find out peaks.  The extra data points are handy to zoom 
in on older data.


Stef