[patch] split-ncv and maximum value tracking
list Charles Goyard
Hi all,
here is a patch that provides two new features to the NCV graph backend.
It answers to a request I did on August 2nd (subject: "Dynamic or Split
NCV").
- split-ncv : Lets you store an ever-changing set of data on a status
column basis. Say, you want to keep track of who is accessing a
database. It works by creating a rrd file per ncv line instead of
packing a lot of DS in a single rrd file. I use it to keep track of
tomcat session number per context, harddrive temperature, database
connexions, and a whole bunch of other things, such a the repartition of
loaded classes on a jvm (useful to find memory leaks).
- trackmax : it lets you keep track of the maximum value of a dataset,
along with the average. It's useful when the maximum is somehow as
significant than the average over time. (say, to track a hard-limited
ressource, such as telephone lines). I use it to keep track of the
maximum number of busy X25 lines on a specific application. It only
works for NCV at the time, but I may add it to other tests.
The patch includes the documentation for all this. It applies on today's
snapshot and on october' all-in-one. It does not work with the plain
release because of a nasty bug in do_ncv.c.
Well, here's the patch. Consider it beta, but yet it works on my
production server. I strongly hope it will be included in the
mainstream hobbit (I don't like maintaining local patches :)).
Regards,
--
Charles Goyard - user-98f9625a7a59@xymon.invalid - (+33) 1 45 38 01 31
-------------- next part --------------
diff -u -x .svn -x '*~' -x Makefile -rw upstream-2006-10-03/hobbitd/do_rrd.c cvf/hobbitd/do_rrd.c
--- upstream-2006-10-03/hobbitd/do_rrd.c 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/do_rrd.c 2006-12-15 15:47:58.000000000 +0100
@@ -29,6 +29,7 @@
#include "do_rrd.h"
char *rrddir = NULL;
+char *trackmax = NULL;
static char *exthandler = NULL;
static char **extids = NULL;
@@ -37,11 +38,16 @@
static char rra2[] = "RRA:AVERAGE:0.5:6:576";
static char rra3[] = "RRA:AVERAGE:0.5:24:576";
static char rra4[] = "RRA:AVERAGE:0.5:288:576";
+static char rra5[] = "RRA:MAX:0.5:1:576";
+static char rra6[] = "RRA:MAX:0.5:6:576";
+static char rra7[] = "RRA:MAX:0.5:24:576";
+static char rra8[] = "RRA:MAX:0.5:288:576";
static char *senderip = NULL;
static char rrdfn[PATH_MAX]; /* This one used by the modules */
static char filedir[PATH_MAX]; /* This one used here */
• void setup_exthandler(char *handlerpath, char *ids)
{
char *p;
@@ -100,6 +106,24 @@
rrdfn[sizeof(rrdfn)-1] = '\0';
}
+static int has_trackmax(char *testname)
+{
+ char testnamecoma[strlen(testname)+3];
• + if(trackmax == NULL) {
+ return 0;
+ }
• + sprintf(testnamecoma, ",%s,", testname);
+ if(strstr(trackmax, testnamecoma)) {
+ return 1;
+ }
+ else {
+ return 0;
+ }
+}
• +
static int create_and_update_rrd(char *hostname, char *fn, char *creparams[], char *template)
{
struct stat st;
diff -u -x .svn -x '*~' -x Makefile -rw upstream-2006-10-03/hobbitd/do_rrd.h cvf/hobbitd/do_rrd.h
--- upstream-2006-10-03/hobbitd/do_rrd.h 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/do_rrd.h 2006-11-24 17:55:53.000000000 +0100
@@ -16,6 +16,7 @@
#include "libbbgen.h"
extern char *rrddir;
+extern char *trackmax;
extern void setup_exthandler(char *handlerpath, char *ids);
extern void update_rrd(char *hostname, char *testname, char *restofmsg, time_t tstamp, char *sender, hobbitrrd_t *ldef);
diff -u -x .svn -x '*~' -x Makefile -rw upstream-2006-10-03/hobbitd/hobbitd.c cvf/hobbitd/hobbitd.c
--- upstream-2006-10-03/hobbitd/hobbitd.c 2006-11-24 17:18:57.000000000 +0100
+++ cvf/hobbitd/hobbitd.c 2006-12-01 18:30:49.000000000 +0100
@@ -1910,7 +1910,7 @@
strbuffer_t *inbuf, *result;
dbgprintf("-> get_config %s\n", fn);
- sprintf(fullfn, "%s/etc/%s", xgetenv("BBHOME"), fn);
+ sprintf(fullfn, "%s/%s", xgetenv("CLIENTEXTCFG"), fn);
fd = stackfopen(fullfn, "r", NULL);
if (fd == NULL) {
errprintf("Config file %s not found\n", fn);
Only in cvf/hobbitd/: hobbitd_cvfinventaire.c
diff -u -x .svn -x '*~' -x Makefile -rw upstream-2006-10-03/hobbitd/hobbitd_rrd.8 cvf/hobbitd/hobbitd_rrd.8
--- upstream-2006-10-03/hobbitd/hobbitd_rrd.8 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/hobbitd_rrd.8 2006-11-24 17:55:53.000000000 +0100
@@ -65,6 +65,16 @@
Defines the types of data collected by the "ncv" module in hobbitd_rrd.
See below for more information.
+.IP SPLITNCV_testname
+The same as NCV_testname, but keeps the data into separate files. That
+is, it creates one rrd file per "NAME : value" line found in the
+status message. It is useful when the list of NCV lines is varying.
• +.IP TRACKMAX
+Comma-separated list of columnname for which you want to keep the
+maximum values along with the default average values. This only works
+ for the NCV backend.
• .SH COLLECTED DATA
The following RRD-file datasets are generated by hobbitd_rrd:
@@ -152,18 +162,22 @@
or "NAME = value". So a generic module in hobbitd_rrd allows for
easy tracking of this type of data.
-The "ncv" module will automatically detect all occurrences of
-a "NAME : value" or "NAME = value" string in a status message, and
-generate an RRD file holding all of the name/value data found in
-the message. The colon- or equal-sign must be present - if there is
-only whitespace, this module will fail.
+ The "ncv" module will automatically detect all occurrences of a "NAME
+ : value" or "NAME = value" string in a status message, and generate an
+ RRD file holding all of the name/value data found in the message
+ (unless you use SPLITNCV, see above). The colon- or equal-sign must be
+ present - if there is only whitespace, this module will fail.
Only the valid letters (A-Z, a-z) and digits (0-9) are used in the
dataset names; whitespace and other characters are stripped off
-automatically. Only the first 19 characters of a dataset name are
-used (this is an RRD limitation). Underscore '_' is not allowed,
-even though RRDtool permits this, and will be stripped from the
-name.
+ automatically. Only the first 19 characters of a dataset name are used
+ (this is an RRD limitation). Underscore '_' is not allowed, even
+ though RRDtool permits this, and will be stripped from the name.
• + When using the alternative SPLITNCV_testname, the dataset name is not
+ limited in length, and non-valid characters are changed to underscores
+ instead of being stripped off. The dataset inside the resulting rrd
+ file is always "lambda.
Note that each "NAME : value" must be on a line by itself. If you have
a custom script generating the status- or data-message that is fed
@@ -177,12 +191,13 @@
all status-messages for the column COLUMNNAME through the hobbitd_rrd
ncv-handler.
-The name of the RRD file will be COLUMNNAME.rrd.
+The name of the RRD file will be COLUMNNAME.rrd. When using SPLITNCV,
+the name of the RRD file will be COLUMNAME,DATASETNAME.rrd.
By default, all of the datasets are generated as the RRD type "DERIVE"
which works for all types of monotonically increasing counters. If you
have data that are of the type GAUGE, you can override the default via
-an environment variable NCV_COLUMNNAME.
+an environment variable NCV_COLUMNNAME (or SPLITNCV_COLUMNAME).
E.g. if you are using the bb-mysqlstatus script from www.deadcat.net to
collect data about your MySQL server, it generates a report in the column
diff -u -x .svn -x '*~' -x Makefile -rw upstream-2006-10-03/hobbitd/hobbitd_rrd.c cvf/hobbitd/hobbitd_rrd.c
--- upstream-2006-10-03/hobbitd/hobbitd_rrd.c 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/hobbitd_rrd.c 2006-12-28 18:35:27.000000000 +0100
@@ -74,6 +74,13 @@
rrddir = strdup(xgetenv("BBRRDS"));
}
+ /* trackmax initialization */
+ if(getenv("TRACKMAX")) {
+ trackmax = (char*)malloc(strlen(getenv("TRACKMAX"))+3);
+ sprintf(trackmax, ",%s,", getenv("TRACKMAX"));
+ dbgprintf("Will track max for: %s\n", trackmax);
+ }
• save_errbuf = 0;
setup_signalhandler("hobbitd_rrd");
memset(&sa, 0, sizeof(sa));
@@ -139,7 +146,7 @@
break;
default:
- /* Ignore reports with purple, blue or clear - they have no data we want. */
+ /* Ignore reports with purple or clear - they have no data we want. */
break;
}
}
diff -u -x .svn -x '*~' -x Makefile -rw upstream-2006-10-03/hobbitd/rrd/do_ncv.c cvf/hobbitd/rrd/do_ncv.c
--- upstream-2006-10-03/hobbitd/rrd/do_ncv.c 2006-11-24 17:18:57.000000000 +0100
+++ cvf/hobbitd/rrd/do_ncv.c 2006-12-28 18:42:58.000000000 +0100
@@ -5,6 +5,7 @@
/* NAME: VALUE */
/* */
/* Copyright (C) 2004-2006 Henrik Storner <user-ce4a2c883f75@xymon.invalid> */
+/* split-ncv added by Charles Goyard November 2006 */
/* */
/* This program is released under the GNU General Public License (GPL), */
/* version 2. See the file "COPYING" for details. */
@@ -17,21 +18,30 @@
{
char **params = NULL;
int paridx;
- char dsdef[1024];
+ char dsdef[1024]; /* destination DS syntax for rrd engine */
char *l, *name, *val;
char *envnam;
- char *dstypes = NULL;
• - setupfn("%s.rrd", testname);
+ char *dstypes = NULL; /* contain NCV_testname value*/
+ int split_ncv = 0;
+ int dslen;
sprintf(rrdvalues, "%d", (int)tstamp);
• params = (char **)calloc(8, sizeof(char *));
params[0] = "rrdcreate";
- params[1] = rrdfn;
paridx = 1;
- envnam = (char *)malloc(4 + strlen(testname) + 1); sprintf(envnam, "NCV_%s", testname);
+ envnam = (char *)malloc(9 + strlen(testname) + 1);
+ sprintf(envnam, "SPLITNCV_%s", testname);
+ l = getenv(envnam);
+ if (l) {
+ split_ncv = 1;
+ dslen = 200;
+ }
+ else {
+ dslen = 19;
+ setupfn("%s.rrd", testname);
+ sprintf(envnam, "NCV_%s", testname);
l = getenv(envnam);
+ }
if (l) {
dstypes = (char *)malloc(strlen(l)+3);
sprintf(dstypes, ",%s,", l);
@@ -84,62 +94,128 @@
strtod(val, &endptr);
if (isspace((int)*endptr) || (*endptr == '\0')) {
- char dsname[20];
- char dskey[22];
- char *dstype = NULL;
+ char dsname[250]; /* name of ncv in status message (with space and al) */
+ char dskey[252]; /* name of final DS key (stripped) */
+ char *dstype = NULL; /* type of final DS */
char *inp;
int outidx;
• /* val contains a valid number */
- /* rrdcreate(1) says: ds must be in the set [a-zA-Z0-9_] */
- for (inp=name,outidx=0; (*inp && (outidx < 19)); inp++) {
+ /* rrdcreate(1) says: ds must be in the set [a-zA-Z0-9_] ... */
+ for (inp=name,outidx=0; (*inp && (outidx < dslen)); inp++) {
if ( ((*inp >= 'A') && (*inp <= 'Z')) ||
((*inp >= 'a') && (*inp <= 'z')) ||
((*inp >= '0') && (*inp <= '9')) ) {
dsname[outidx++] = *inp;
}
+ /* ... however, for split ncv, we replace anything else */
+ /* with an underscore, compacting successive invalid */
+ /* characters into a single one */
+ else if (split_ncv && (dsname[outidx - 1] != '_')) {
+ dsname[outidx++] = '_';
+ }
}
+ if(dsname[outidx-1] == '_') {
+ dsname[outidx-1] = '\0';
+ }
+ else {
dsname[outidx] = '\0';
+ }
sprintf(dskey, ",%s:", dsname);
+ if(split_ncv) {
+ /* setupfn("%s,%s.rrd", testname, dsname); */
+ snprintf(rrdfn, sizeof(rrdfn)-1, "%s,%s.rrd", testname,dsname);
+ rrdfn[sizeof(rrdfn)-1] = '\0';
• + params[1] = rrdfn;
+ paridx = 1;
+ }
if (dstypes) {
dstype = strstr(dstypes, dskey);
- if (!dstype) { strcpy(dskey, ",*:"); dstype = strstr(dstypes, dskey); }
+ if (!dstype) {
+ strcpy(dskey, ",*:");
+ dstype = strstr(dstypes, dskey);
+ }
}
- if (dstype) {
+ if (dstype) { /* if ds type is forced */
char *p;
dstype += strlen(dskey);
p = strchr(dstype, ','); if (p) *p = '\0';
+ if(split_ncv) {
+ sprintf(dsdef, "DS:lambda:%s:600:0:U", dstype);
+ }
+ else {
sprintf(dsdef, "DS:%s:%s:600:0:U", dsname, dstype);
+ }
if (p) *p = ',';
}
+ else { /* nothing specified in the environnement, and no '*:' default */
+ if(split_ncv) {
+ strcpy(dsdef, "DS:lambda:DERIVE:600:0:U");
+ }
else {
sprintf(dsdef, "DS:%s:DERIVE:600:0:U", dsname);
}
+ }
- if (!dstype || (strncasecmp(dstype, "NONE", 4) != 0)) {
+ if (!dstype || (strncasecmp(dstype, "NONE", 4) != 0)) { /* if we have something */
paridx++;
params = (char **)realloc(params, (7 + paridx)*sizeof(char *));
params[paridx] = strdup(dsdef);
params[paridx+1] = NULL;
• sprintf(rrdvalues+strlen(rrdvalues), ":%s", val);
}
}
• + if(split_ncv && (paridx > 1)) {
+ int i;
+ params[++paridx] = strdup(rra1);
+ params[++paridx] = strdup(rra2);
+ params[++paridx] = strdup(rra3);
+ params[++paridx] = strdup(rra4);
• + if(has_trackmax(testname)) {
+ params = (char **)realloc(params, (11 + paridx)*sizeof(char *));
+ params[++paridx] = strdup(rra5);
+ params[++paridx] = strdup(rra6);
+ params[++paridx] = strdup(rra7);
+ params[++paridx] = strdup(rra8);
+ }
• + params[++paridx] = NULL;
+ create_and_update_rrd(hostname, rrdfn, params, NULL);
+ for(i = 2 ; i<paridx ; i++) {
+ params[i] = NULL;
+ }
+ sprintf(rrdvalues, "%d", (int)tstamp);
}
}
- if (paridx > 1) {
+ } /* end of while */
• + if (split_ncv) {
+ for (paridx=2; (params[paridx] != NULL); paridx++) {
+ xfree(params[paridx]);
+ }
+ }
+ else if(paridx > 1) {
params[++paridx] = strdup(rra1);
params[++paridx] = strdup(rra2);
params[++paridx] = strdup(rra3);
params[++paridx] = strdup(rra4);
- params[++paridx] = NULL;
- create_and_update_rrd(hostname, rrdfn, params, NULL);
+ if(has_trackmax(testname)) {
+ params = (char **)realloc(params, (11 + paridx)*sizeof(char *));
+ params[++paridx] = strdup(rra5);
+ params[++paridx] = strdup(rra6);
+ params[++paridx] = strdup(rra7);
+ params[++paridx] = strdup(rra8);
+ }
+ params[++paridx] = NULL;
+ create_and_update_rrd(hostname, rrdfn, params, NULL);
for (paridx=2; (params[paridx] != NULL); paridx++)
xfree(params[paridx]);
}
list Charles Goyard
Hi again, of course I blew it up :) The second hunk is a bit from another new feature, so here's another patch. Sorry for the inconvenience. -- Charles Goyard - user-98f9625a7a59@xymon.invalid - (+33) 1 45 38 01 31 -------------- next part -------------- diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/do_rrd.c cvf/hobbitd/do_rrd.c
▸
--- upstream-2006-10-03/hobbitd/do_rrd.c 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/do_rrd.c 2006-12-15 15:47:58.000000000 +0100
@@ -29,6 +29,7 @@
#include "do_rrd.h"
char *rrddir = NULL;
+char *trackmax = NULL;
static char *exthandler = NULL;
static char **extids = NULL;
@@ -37,11 +38,16 @@
static char rra2[] = "RRA:AVERAGE:0.5:6:576";
static char rra3[] = "RRA:AVERAGE:0.5:24:576";
static char rra4[] = "RRA:AVERAGE:0.5:288:576";
+static char rra5[] = "RRA:MAX:0.5:1:576";
+static char rra6[] = "RRA:MAX:0.5:6:576";
+static char rra7[] = "RRA:MAX:0.5:24:576";
+static char rra8[] = "RRA:MAX:0.5:288:576";
static char *senderip = NULL;
static char rrdfn[PATH_MAX]; /* This one used by the modules */
static char filedir[PATH_MAX]; /* This one used here */
• void setup_exthandler(char *handlerpath, char *ids)
{
char *p;
@@ -100,6 +106,24 @@
rrdfn[sizeof(rrdfn)-1] = '\0';
}
+static int has_trackmax(char *testname)
+{
+ char testnamecoma[strlen(testname)+3];
• + if(trackmax == NULL) {
+ return 0;
+ }
• + sprintf(testnamecoma, ",%s,", testname);
+ if(strstr(trackmax, testnamecoma)) {
+ return 1;
+ }
+ else {
+ return 0;
+ }
+}
• +
static int create_and_update_rrd(char *hostname, char *fn, char *creparams[], char *template)
{
struct stat st;
diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/do_rrd.h cvf/hobbitd/do_rrd.h
▸
--- upstream-2006-10-03/hobbitd/do_rrd.h 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/do_rrd.h 2006-11-24 17:55:53.000000000 +0100
@@ -16,6 +16,7 @@
#include "libbbgen.h"
extern char *rrddir;
+extern char *trackmax;
extern void setup_exthandler(char *handlerpath, char *ids);
extern void update_rrd(char *hostname, char *testname, char *restofmsg, time_t tstamp, char *sender, hobbitrrd_t *ldef);
Only in cvf/hobbitd/: hobbitd_cvfinventaire.c
diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/hobbitd_rrd.8 cvf/hobbitd/hobbitd_rrd.8
▸
--- upstream-2006-10-03/hobbitd/hobbitd_rrd.8 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/hobbitd_rrd.8 2006-11-24 17:55:53.000000000 +0100
@@ -65,6 +65,16 @@
Defines the types of data collected by the "ncv" module in hobbitd_rrd.
See below for more information.
+.IP SPLITNCV_testname
+The same as NCV_testname, but keeps the data into separate files. That
+is, it creates one rrd file per "NAME : value" line found in the
+status message. It is useful when the list of NCV lines is varying.
• +.IP TRACKMAX
+Comma-separated list of columnname for which you want to keep the
+maximum values along with the default average values. This only works
+ for the NCV backend.
• .SH COLLECTED DATA
The following RRD-file datasets are generated by hobbitd_rrd:
@@ -152,18 +162,22 @@
or "NAME = value". So a generic module in hobbitd_rrd allows for
easy tracking of this type of data.
-The "ncv" module will automatically detect all occurrences of
-a "NAME : value" or "NAME = value" string in a status message, and
-generate an RRD file holding all of the name/value data found in
-the message. The colon- or equal-sign must be present - if there is
-only whitespace, this module will fail.
+ The "ncv" module will automatically detect all occurrences of a "NAME
+ : value" or "NAME = value" string in a status message, and generate an
+ RRD file holding all of the name/value data found in the message
+ (unless you use SPLITNCV, see above). The colon- or equal-sign must be
+ present - if there is only whitespace, this module will fail.
Only the valid letters (A-Z, a-z) and digits (0-9) are used in the
dataset names; whitespace and other characters are stripped off
-automatically. Only the first 19 characters of a dataset name are
-used (this is an RRD limitation). Underscore '_' is not allowed,
-even though RRDtool permits this, and will be stripped from the
-name.
+ automatically. Only the first 19 characters of a dataset name are used
+ (this is an RRD limitation). Underscore '_' is not allowed, even
+ though RRDtool permits this, and will be stripped from the name.
• + When using the alternative SPLITNCV_testname, the dataset name is not
+ limited in length, and non-valid characters are changed to underscores
+ instead of being stripped off. The dataset inside the resulting rrd
+ file is always "lambda.
Note that each "NAME : value" must be on a line by itself. If you have
a custom script generating the status- or data-message that is fed
@@ -177,12 +191,13 @@
all status-messages for the column COLUMNNAME through the hobbitd_rrd
ncv-handler.
-The name of the RRD file will be COLUMNNAME.rrd.
+The name of the RRD file will be COLUMNNAME.rrd. When using SPLITNCV,
+the name of the RRD file will be COLUMNAME,DATASETNAME.rrd.
By default, all of the datasets are generated as the RRD type "DERIVE"
which works for all types of monotonically increasing counters. If you
have data that are of the type GAUGE, you can override the default via
-an environment variable NCV_COLUMNNAME.
+an environment variable NCV_COLUMNNAME (or SPLITNCV_COLUMNAME).
E.g. if you are using the bb-mysqlstatus script from www.deadcat.net to
collect data about your MySQL server, it generates a report in the column
diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/hobbitd_rrd.c cvf/hobbitd/hobbitd_rrd.c
▸
--- upstream-2006-10-03/hobbitd/hobbitd_rrd.c 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/hobbitd_rrd.c 2006-12-28 18:35:27.000000000 +0100
@@ -74,6 +74,13 @@
rrddir = strdup(xgetenv("BBRRDS"));
}
+ /* trackmax initialization */
+ if(getenv("TRACKMAX")) {
+ trackmax = (char*)malloc(strlen(getenv("TRACKMAX"))+3);
+ sprintf(trackmax, ",%s,", getenv("TRACKMAX"));
+ dbgprintf("Will track max for: %s\n", trackmax);
+ }
• save_errbuf = 0;
setup_signalhandler("hobbitd_rrd");
memset(&sa, 0, sizeof(sa));
@@ -139,7 +146,7 @@
break;
default:
- /* Ignore reports with purple, blue or clear - they have no data we want. */
+ /* Ignore reports with purple or clear - they have no data we want. */
break;
}
}
diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/rrd/do_ncv.c cvf/hobbitd/rrd/do_ncv.c
▸
--- upstream-2006-10-03/hobbitd/rrd/do_ncv.c 2006-11-24 17:18:57.000000000 +0100
+++ cvf/hobbitd/rrd/do_ncv.c 2006-12-28 18:42:58.000000000 +0100
@@ -5,6 +5,7 @@
/* NAME: VALUE */
/* */
/* Copyright (C) 2004-2006 Henrik Storner <user-ce4a2c883f75@xymon.invalid> */
+/* split-ncv added by Charles Goyard November 2006 */
/* */
/* This program is released under the GNU General Public License (GPL), */
/* version 2. See the file "COPYING" for details. */
@@ -17,21 +18,30 @@
{
char **params = NULL;
int paridx;
- char dsdef[1024];
+ char dsdef[1024]; /* destination DS syntax for rrd engine */
char *l, *name, *val;
char *envnam;
- char *dstypes = NULL;
• - setupfn("%s.rrd", testname);
+ char *dstypes = NULL; /* contain NCV_testname value*/
+ int split_ncv = 0;
+ int dslen;
sprintf(rrdvalues, "%d", (int)tstamp);
• params = (char **)calloc(8, sizeof(char *));
params[0] = "rrdcreate";
- params[1] = rrdfn;
paridx = 1;
- envnam = (char *)malloc(4 + strlen(testname) + 1); sprintf(envnam, "NCV_%s", testname);
+ envnam = (char *)malloc(9 + strlen(testname) + 1);
+ sprintf(envnam, "SPLITNCV_%s", testname);
+ l = getenv(envnam);
+ if (l) {
+ split_ncv = 1;
+ dslen = 200;
+ }
+ else {
+ dslen = 19;
+ setupfn("%s.rrd", testname);
+ sprintf(envnam, "NCV_%s", testname);
l = getenv(envnam);
+ }
if (l) {
dstypes = (char *)malloc(strlen(l)+3);
sprintf(dstypes, ",%s,", l);
@@ -84,62 +94,128 @@
strtod(val, &endptr);
if (isspace((int)*endptr) || (*endptr == '\0')) {
- char dsname[20];
- char dskey[22];
- char *dstype = NULL;
+ char dsname[250]; /* name of ncv in status message (with space and al) */
+ char dskey[252]; /* name of final DS key (stripped) */
+ char *dstype = NULL; /* type of final DS */
char *inp;
int outidx;
• /* val contains a valid number */
- /* rrdcreate(1) says: ds must be in the set [a-zA-Z0-9_] */
- for (inp=name,outidx=0; (*inp && (outidx < 19)); inp++) {
+ /* rrdcreate(1) says: ds must be in the set [a-zA-Z0-9_] ... */
+ for (inp=name,outidx=0; (*inp && (outidx < dslen)); inp++) {
if ( ((*inp >= 'A') && (*inp <= 'Z')) ||
((*inp >= 'a') && (*inp <= 'z')) ||
((*inp >= '0') && (*inp <= '9')) ) {
dsname[outidx++] = *inp;
}
+ /* ... however, for split ncv, we replace anything else */
+ /* with an underscore, compacting successive invalid */
+ /* characters into a single one */
+ else if (split_ncv && (dsname[outidx - 1] != '_')) {
+ dsname[outidx++] = '_';
+ }
}
+ if(dsname[outidx-1] == '_') {
+ dsname[outidx-1] = '\0';
+ }
+ else {
dsname[outidx] = '\0';
+ }
sprintf(dskey, ",%s:", dsname);
+ if(split_ncv) {
+ /* setupfn("%s,%s.rrd", testname, dsname); */
+ snprintf(rrdfn, sizeof(rrdfn)-1, "%s,%s.rrd", testname,dsname);
+ rrdfn[sizeof(rrdfn)-1] = '\0';
• + params[1] = rrdfn;
+ paridx = 1;
+ }
if (dstypes) {
dstype = strstr(dstypes, dskey);
- if (!dstype) { strcpy(dskey, ",*:"); dstype = strstr(dstypes, dskey); }
+ if (!dstype) {
+ strcpy(dskey, ",*:");
+ dstype = strstr(dstypes, dskey);
+ }
}
- if (dstype) {
+ if (dstype) { /* if ds type is forced */
char *p;
dstype += strlen(dskey);
p = strchr(dstype, ','); if (p) *p = '\0';
+ if(split_ncv) {
+ sprintf(dsdef, "DS:lambda:%s:600:0:U", dstype);
+ }
+ else {
sprintf(dsdef, "DS:%s:%s:600:0:U", dsname, dstype);
+ }
if (p) *p = ',';
}
+ else { /* nothing specified in the environnement, and no '*:' default */
+ if(split_ncv) {
+ strcpy(dsdef, "DS:lambda:DERIVE:600:0:U");
+ }
else {
sprintf(dsdef, "DS:%s:DERIVE:600:0:U", dsname);
}
+ }
- if (!dstype || (strncasecmp(dstype, "NONE", 4) != 0)) {
+ if (!dstype || (strncasecmp(dstype, "NONE", 4) != 0)) { /* if we have something */
paridx++;
params = (char **)realloc(params, (7 + paridx)*sizeof(char *));
params[paridx] = strdup(dsdef);
params[paridx+1] = NULL;
• sprintf(rrdvalues+strlen(rrdvalues), ":%s", val);
}
}
• + if(split_ncv && (paridx > 1)) {
+ int i;
+ params[++paridx] = strdup(rra1);
+ params[++paridx] = strdup(rra2);
+ params[++paridx] = strdup(rra3);
+ params[++paridx] = strdup(rra4);
• + if(has_trackmax(testname)) {
+ params = (char **)realloc(params, (11 + paridx)*sizeof(char *));
+ params[++paridx] = strdup(rra5);
+ params[++paridx] = strdup(rra6);
+ params[++paridx] = strdup(rra7);
+ params[++paridx] = strdup(rra8);
+ }
• + params[++paridx] = NULL;
+ create_and_update_rrd(hostname, rrdfn, params, NULL);
+ for(i = 2 ; i<paridx ; i++) {
+ params[i] = NULL;
+ }
+ sprintf(rrdvalues, "%d", (int)tstamp);
}
}
- if (paridx > 1) {
+ } /* end of while */
• + if (split_ncv) {
+ for (paridx=2; (params[paridx] != NULL); paridx++) {
+ xfree(params[paridx]);
+ }
+ }
+ else if(paridx > 1) {
params[++paridx] = strdup(rra1);
params[++paridx] = strdup(rra2);
params[++paridx] = strdup(rra3);
params[++paridx] = strdup(rra4);
- params[++paridx] = NULL;
- create_and_update_rrd(hostname, rrdfn, params, NULL);
+ if(has_trackmax(testname)) {
+ params = (char **)realloc(params, (11 + paridx)*sizeof(char *));
+ params[++paridx] = strdup(rra5);
+ params[++paridx] = strdup(rra6);
+ params[++paridx] = strdup(rra7);
+ params[++paridx] = strdup(rra8);
+ }
+ params[++paridx] = NULL;
+ create_and_update_rrd(hostname, rrdfn, params, NULL);
for (paridx=2; (params[paridx] != NULL); paridx++)
xfree(params[paridx]);
}
list Gildas le Nadan
Hi, The TRACKMAX feature is really interesting as the max values get "diluted" in the week/month/year views. Alas, it seem to only work for NCV values at the moment (and I wish I could use it for cpu/memory/network/disk). Charles, Henrik, do you think this is feasible? Cheers, Gildas
▸
Charles Goyard wrote:Hi again,
of course I blew it up :)
The second hunk is a bit from another new feature, so here's another
patch.
Sorry for the inconvenience.
diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/do_rrd.c cvf/hobbitd/do_rrd.c
--- upstream-2006-10-03/hobbitd/do_rrd.c 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/do_rrd.c 2006-12-15 15:47:58.000000000 +0100
@@ -29,6 +29,7 @@
#include "do_rrd.h"
char *rrddir = NULL;
+char *trackmax = NULL;
static char *exthandler = NULL;
static char **extids = NULL;
@@ -37,11 +38,16 @@
static char rra2[] = "RRA:AVERAGE:0.5:6:576";
static char rra3[] = "RRA:AVERAGE:0.5:24:576";
static char rra4[] = "RRA:AVERAGE:0.5:288:576";
+static char rra5[] = "RRA:MAX:0.5:1:576";
+static char rra6[] = "RRA:MAX:0.5:6:576";
+static char rra7[] = "RRA:MAX:0.5:24:576";
+static char rra8[] = "RRA:MAX:0.5:288:576";
static char *senderip = NULL;
static char rrdfn[PATH_MAX]; /* This one used by the modules */
static char filedir[PATH_MAX]; /* This one used here */
• void setup_exthandler(char *handlerpath, char *ids)
{
char *p;
@@ -100,6 +106,24 @@
rrdfn[sizeof(rrdfn)-1] = '\0';
}
+static int has_trackmax(char *testname)
+{
+ char testnamecoma[strlen(testname)+3];
• + if(trackmax == NULL) {
+ return 0;
+ }
• + sprintf(testnamecoma, ",%s,", testname);
+ if(strstr(trackmax, testnamecoma)) {
+ return 1;
+ }
+ else {
+ return 0;
+ }
+}
• +
static int create_and_update_rrd(char *hostname, char *fn, char *creparams[], char *template)
{
struct stat st;
diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/do_rrd.h cvf/hobbitd/do_rrd.h
--- upstream-2006-10-03/hobbitd/do_rrd.h 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/do_rrd.h 2006-11-24 17:55:53.000000000 +0100
@@ -16,6 +16,7 @@
#include "libbbgen.h"
extern char *rrddir;
+extern char *trackmax;
extern void setup_exthandler(char *handlerpath, char *ids);
extern void update_rrd(char *hostname, char *testname, char *restofmsg, time_t tstamp, char *sender, hobbitrrd_t *ldef);
Only in cvf/hobbitd/: hobbitd_cvfinventaire.c
diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/hobbitd_rrd.8 cvf/hobbitd/hobbitd_rrd.8
--- upstream-2006-10-03/hobbitd/hobbitd_rrd.8 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/hobbitd_rrd.8 2006-11-24 17:55:53.000000000 +0100
@@ -65,6 +65,16 @@
Defines the types of data collected by the "ncv" module in hobbitd_rrd.
See below for more information.
+.IP SPLITNCV_testname
+The same as NCV_testname, but keeps the data into separate files. That
+is, it creates one rrd file per "NAME : value" line found in the
+status message. It is useful when the list of NCV lines is varying.
• +.IP TRACKMAX
+Comma-separated list of columnname for which you want to keep the
+maximum values along with the default average values. This only works
+ for the NCV backend.
• .SH COLLECTED DATA
The following RRD-file datasets are generated by hobbitd_rrd:
@@ -152,18 +162,22 @@
or "NAME = value". So a generic module in hobbitd_rrd allows for
easy tracking of this type of data.
-The "ncv" module will automatically detect all occurrences of
-a "NAME : value" or "NAME = value" string in a status message, and
-generate an RRD file holding all of the name/value data found in
-the message. The colon- or equal-sign must be present - if there is
-only whitespace, this module will fail.
+ The "ncv" module will automatically detect all occurrences of a "NAME
+ : value" or "NAME = value" string in a status message, and generate an
+ RRD file holding all of the name/value data found in the message
+ (unless you use SPLITNCV, see above). The colon- or equal-sign must be
+ present - if there is only whitespace, this module will fail.
Only the valid letters (A-Z, a-z) and digits (0-9) are used in the
dataset names; whitespace and other characters are stripped off
-automatically. Only the first 19 characters of a dataset name are
-used (this is an RRD limitation). Underscore '_' is not allowed,
-even though RRDtool permits this, and will be stripped from the
-name.
+ automatically. Only the first 19 characters of a dataset name are used
+ (this is an RRD limitation). Underscore '_' is not allowed, even
+ though RRDtool permits this, and will be stripped from the name.
• + When using the alternative SPLITNCV_testname, the dataset name is not
+ limited in length, and non-valid characters are changed to underscores
+ instead of being stripped off. The dataset inside the resulting rrd
+ file is always "lambda.
Note that each "NAME : value" must be on a line by itself. If you have
a custom script generating the status- or data-message that is fed
@@ -177,12 +191,13 @@
all status-messages for the column COLUMNNAME through the hobbitd_rrd
ncv-handler.
-The name of the RRD file will be COLUMNNAME.rrd.
+The name of the RRD file will be COLUMNNAME.rrd. When using SPLITNCV,
+the name of the RRD file will be COLUMNAME,DATASETNAME.rrd.
By default, all of the datasets are generated as the RRD type "DERIVE"
which works for all types of monotonically increasing counters. If you
have data that are of the type GAUGE, you can override the default via
-an environment variable NCV_COLUMNNAME.
+an environment variable NCV_COLUMNNAME (or SPLITNCV_COLUMNAME).
E.g. if you are using the bb-mysqlstatus script from www.deadcat.net to
collect data about your MySQL server, it generates a report in the column
diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/hobbitd_rrd.c cvf/hobbitd/hobbitd_rrd.c
--- upstream-2006-10-03/hobbitd/hobbitd_rrd.c 2006-11-24 17:18:09.000000000 +0100
+++ cvf/hobbitd/hobbitd_rrd.c 2006-12-28 18:35:27.000000000 +0100
@@ -74,6 +74,13 @@
rrddir = strdup(xgetenv("BBRRDS"));
}
+ /* trackmax initialization */
+ if(getenv("TRACKMAX")) {
+ trackmax = (char*)malloc(strlen(getenv("TRACKMAX"))+3);
+ sprintf(trackmax, ",%s,", getenv("TRACKMAX"));
+ dbgprintf("Will track max for: %s\n", trackmax);
+ }
• save_errbuf = 0;
setup_signalhandler("hobbitd_rrd");
memset(&sa, 0, sizeof(sa));
@@ -139,7 +146,7 @@
break;
default:
- /* Ignore reports with purple, blue or clear - they have no data we want. */
+ /* Ignore reports with purple or clear - they have no data we want. */
break;
}
}
diff -u -x .svn -x '*~' -x hobbitd.c -x Makefile -rw upstream-2006-10-03/hobbitd/rrd/do_ncv.c cvf/hobbitd/rrd/do_ncv.c
--- upstream-2006-10-03/hobbitd/rrd/do_ncv.c 2006-11-24 17:18:57.000000000 +0100
+++ cvf/hobbitd/rrd/do_ncv.c 2006-12-28 18:42:58.000000000 +0100
@@ -5,6 +5,7 @@
/* NAME: VALUE */
/* */
/* Copyright (C) 2004-2006 Henrik Storner <user-ce4a2c883f75@xymon.invalid> */
+/* split-ncv added by Charles Goyard November 2006 */
/* */
/* This program is released under the GNU General Public License (GPL), */
/* version 2. See the file "COPYING" for details. */
@@ -17,21 +18,30 @@
{
char **params = NULL;
int paridx;
- char dsdef[1024];
+ char dsdef[1024]; /* destination DS syntax for rrd engine */
char *l, *name, *val;
char *envnam;
- char *dstypes = NULL;
• - setupfn("%s.rrd", testname);
+ char *dstypes = NULL; /* contain NCV_testname value*/
+ int split_ncv = 0;
+ int dslen;
sprintf(rrdvalues, "%d", (int)tstamp);
• params = (char **)calloc(8, sizeof(char *));
params[0] = "rrdcreate";
- params[1] = rrdfn;
paridx = 1;
- envnam = (char *)malloc(4 + strlen(testname) + 1); sprintf(envnam, "NCV_%s", testname);
+ envnam = (char *)malloc(9 + strlen(testname) + 1);
+ sprintf(envnam, "SPLITNCV_%s", testname);
+ l = getenv(envnam);
+ if (l) {
+ split_ncv = 1;
+ dslen = 200;
+ }
+ else {
+ dslen = 19;
+ setupfn("%s.rrd", testname);
+ sprintf(envnam, "NCV_%s", testname);
l = getenv(envnam);
+ }
if (l) {
dstypes = (char *)malloc(strlen(l)+3);
sprintf(dstypes, ",%s,", l);
@@ -84,62 +94,128 @@
strtod(val, &endptr);
if (isspace((int)*endptr) || (*endptr == '\0')) {
- char dsname[20];
- char dskey[22];
- char *dstype = NULL;
+ char dsname[250]; /* name of ncv in status message (with space and al) */
+ char dskey[252]; /* name of final DS key (stripped) */
+ char *dstype = NULL; /* type of final DS */
char *inp;
int outidx;
• /* val contains a valid number */
- /* rrdcreate(1) says: ds must be in the set [a-zA-Z0-9_] */
- for (inp=name,outidx=0; (*inp && (outidx < 19)); inp++) {
+ /* rrdcreate(1) says: ds must be in the set [a-zA-Z0-9_] ... */
+ for (inp=name,outidx=0; (*inp && (outidx < dslen)); inp++) {
if ( ((*inp >= 'A') && (*inp <= 'Z')) ||
((*inp >= 'a') && (*inp <= 'z')) ||
((*inp >= '0') && (*inp <= '9')) ) {
dsname[outidx++] = *inp;
}
+ /* ... however, for split ncv, we replace anything else */
+ /* with an underscore, compacting successive invalid */
+ /* characters into a single one */
+ else if (split_ncv && (dsname[outidx - 1] != '_')) {
+ dsname[outidx++] = '_';
+ }
}
+ if(dsname[outidx-1] == '_') {
+ dsname[outidx-1] = '\0';
+ }
+ else {
dsname[outidx] = '\0';
+ }
sprintf(dskey, ",%s:", dsname);
+ if(split_ncv) {
+ /* setupfn("%s,%s.rrd", testname, dsname); */
+ snprintf(rrdfn, sizeof(rrdfn)-1, "%s,%s.rrd", testname,dsname);
+ rrdfn[sizeof(rrdfn)-1] = '\0';
• + params[1] = rrdfn;
+ paridx = 1;
+ }
if (dstypes) {
dstype = strstr(dstypes, dskey);
- if (!dstype) { strcpy(dskey, ",*:"); dstype = strstr(dstypes, dskey); }
+ if (!dstype) {
+ strcpy(dskey, ",*:");
+ dstype = strstr(dstypes, dskey);
+ }
}
- if (dstype) {
+ if (dstype) { /* if ds type is forced */
char *p;
dstype += strlen(dskey);
p = strchr(dstype, ','); if (p) *p = '\0';
+ if(split_ncv) {
+ sprintf(dsdef, "DS:lambda:%s:600:0:U", dstype);
+ }
+ else {
sprintf(dsdef, "DS:%s:%s:600:0:U", dsname, dstype);
+ }
if (p) *p = ',';
}
+ else { /* nothing specified in the environnement, and no '*:' default */
+ if(split_ncv) {
+ strcpy(dsdef, "DS:lambda:DERIVE:600:0:U");
+ }
else {
sprintf(dsdef, "DS:%s:DERIVE:600:0:U", dsname);
}
+ }
- if (!dstype || (strncasecmp(dstype, "NONE", 4) != 0)) {
+ if (!dstype || (strncasecmp(dstype, "NONE", 4) != 0)) { /* if we have something */
paridx++;
params = (char **)realloc(params, (7 + paridx)*sizeof(char *));
params[paridx] = strdup(dsdef);
params[paridx+1] = NULL;
• sprintf(rrdvalues+strlen(rrdvalues), ":%s", val);
}
}
• + if(split_ncv && (paridx > 1)) {
+ int i;
+ params[++paridx] = strdup(rra1);
+ params[++paridx] = strdup(rra2);
+ params[++paridx] = strdup(rra3);
+ params[++paridx] = strdup(rra4);
• + if(has_trackmax(testname)) {
+ params = (char **)realloc(params, (11 + paridx)*sizeof(char *));
+ params[++paridx] = strdup(rra5);
+ params[++paridx] = strdup(rra6);
+ params[++paridx] = strdup(rra7);
+ params[++paridx] = strdup(rra8);
+ }
• + params[++paridx] = NULL;
+ create_and_update_rrd(hostname, rrdfn, params, NULL);
+ for(i = 2 ; i<paridx ; i++) {
+ params[i] = NULL;
+ }
+ sprintf(rrdvalues, "%d", (int)tstamp);
}
}
- if (paridx > 1) {
+ } /* end of while */
• + if (split_ncv) {
+ for (paridx=2; (params[paridx] != NULL); paridx++) {
+ xfree(params[paridx]);
+ }
+ }
+ else if(paridx > 1) {
params[++paridx] = strdup(rra1);
params[++paridx] = strdup(rra2);
params[++paridx] = strdup(rra3);
params[++paridx] = strdup(rra4);
- params[++paridx] = NULL;
- create_and_update_rrd(hostname, rrdfn, params, NULL);
+ if(has_trackmax(testname)) {
+ params = (char **)realloc(params, (11 + paridx)*sizeof(char *));
+ params[++paridx] = strdup(rra5);
+ params[++paridx] = strdup(rra6);
+ params[++paridx] = strdup(rra7);
+ params[++paridx] = strdup(rra8);
+ }
+ params[++paridx] = NULL;
+ create_and_update_rrd(hostname, rrdfn, params, NULL);
for (paridx=2; (params[paridx] != NULL); paridx++)
xfree(params[paridx]);
}
list Charles Goyard
Hi,
▸
Gildas Le Nadan wrote :The TRACKMAX feature is really interesting as the max values get "diluted" in the week/month/year views. Alas, it seem to only work for NCV values at the moment (and I wish I could use it for cpu/memory/network/disk). Charles, Henrik, do you think this is feasible?
I agree it would be nice. I have a question for Henrik prior to implement it. The RRAS are added individually in each /rrd/*.c backend, and they get calculated for every status message coming in, (file creation parameters calculated even if the file already exists. That part can be moved inside create_and_update_rrd(), making Hobbit faster. I'll do it that way only if you don't have future plans underneath that duplication of code. If you do, I'll add the trackmax test in every individual rrd backend. (and I'd love to see that feature in the mainstream hobbit, too :) Regards, -- Charles Goyard - user-98f9625a7a59@xymon.invalid - (+33) 1 45 38 01 31
list Henrik Størner
▸
On Fri, Jan 19, 2007 at 10:55:05AM +0100, Charles Goyard wrote:
Gildas Le Nadan wrote :The TRACKMAX feature is really interesting as the max values get "diluted" in the week/month/year views. Alas, it seem to only work for NCV values at the moment (and I wish I could use it for cpu/memory/network/disk). Charles, Henrik, do you think this is feasible?I agree it would be nice.
It would. Only problem is that it would increase the size of the RRD files (perhaps not a big issue), and it would only have effect on new RRD files that are created, not any existing ones. So to add this to an existing setup, you would have to dump all of your RRD files, then create new ones, and import the data from the dump. Not sure how much work would be involved in automating this. However, there has also been some requests to increase the granularity of the stored data, e.g. to keep 7 days worth of 5-minute averages as opposed to the current 2 days. And likewise for the other RRA's. (For those not familiar with how RRDtool works, the RRA's define how each measurement gets averaged over time. Hobbit uses 4 RRA's in each RRD file - one RRA tracks data (almost) without averaging it, using the 5-minute interval; the next averages across 6 measurements - 30 minutes; the third averages across 24 measurements - 2 hours; the last averages across 288 measurements - 1 day. For each of these 4 groups of data-averages, the RRD file contains 576 values. So the first RRA covers 576*5 minutes=2 days, the second covers 12 days, the third 48 days and the last 576 days. When Hobbit generates a graph from the RRD data, RRDtool will automatically grab the data for the graph from the best group of data which has data for the period requested). So if we're going to add MAX/MIN tracking to the RRD-files, we might as well do it at the same time that we change the granularity. The numbers I've been thinking of are to keep * 30 days of 5-minute averages * 90 days of 15-minute averages * 360 days on 1-hour averages * 1080 days of 3-hour averages That alone would cause the size of the RRD files to increase 15 times. Adding MIN+MAX tracking would mean tripling the size. An "average" host in my setup uses ~400 KB of diskspace for RRD-files, so increasing that 15x3 times means it would grow to ~16 MB per host. It's a significant increase (I'd have to get more disk space for my production systems to handle that), but disks are getting bigger and cheaper - and I'd still be storing data for 4000 hosts on less than 60 GB. An simple RRD file would grow from ~19 KB to 855 KB. I don't think there would be much of a performance hit from this. The RRD update spends most of its time opening and locking the file, whereas the actual data-update doesn't take long.
▸
I have a question for Henrik prior to implement it. The RRAS are added individually in each /rrd/*.c backend, and they get calculated for every status message coming in
Actually, they don't. All of the rrd/*.c files use logic like this:
static char *la_params[] = { "rrdcreate", rrdfn, "DS:la:GAUGE:600:0:U",
rra1, rra2, rra3, rra4, NULL };
static char *la_tpl = NULL;
if (la_tpl == NULL) la_tpl = setup_template(la_params);
The keyword here is the declaration of the variables as "static".
The "la_params[]" is a static table, so this is initialized at
compile-time with the static values. "la_tpl" is the RRD "template",
which basically is a list of the dataset names in the order which
Hobbit is feeding data values in. This template is only calculated
the first time this type of RRD file is updated - that's what the
"if (la_tpl == NULL) ... " does.
Inside the create_and_update_rrd() function, the "la_params"
with the RRA's are only used when creating a new RRD.
Regards,
Henrik
list Charles Goyard
Hi,
▸
Henrik Stoerner wrote :On Fri, Jan 19, 2007 at 10:55:05AM +0100, Charles Goyard wrote:Gildas Le Nadan wrote :The TRACKMAX feature is really interesting as the max values get "diluted" in the week/month/year views. Alas, it seem to only work for NCV values at the moment (and I wish I could use it for cpu/memory/network/disk). Charles, Henrik, do you think this is feasible?I agree it would be nice.[...] and it would only have effect on new RRD files that are created, not any existing ones.
However, there has also been some requests to increase the granularity of the stored data, e.g. to keep 7 days worth of 5-minute averages as opposed to the current 2 days. And likewise for the other RRA's.
So if we're going to add MAX/MIN tracking to the RRD-files, we might as well do it at the same time that we change the granularity. The numbers I've been thinking of are to keep * 30 days of 5-minute averages * 90 days of 15-minute averages * 360 days on 1-hour averages * 1080 days of 3-hour averages
The "la_params[]" is a static table, so this is initialized at compile-time with the static values.
Very nice design, great (thanks for the quick course). And d'oh!, too. I expected run-time setting, not compile-time. I had in mind a config file option. It won't be that bad to have the values of RRAS in the hobbitserver.cfg, read once at start. That would let you provide a few typical setups (with the current one enabled by default). At the very least, it can be a setting in the Makefile. We also have to keep in mind that TRACKMAX (and TRACKMIN) is a per-statusname option, not a global setting. I can take some time next week to add these features into hobbit, if we agree on how to do it. I propose a setting in hobbitserver.cfg for RRAS like : # hobbitserver.cfg sample # standard setup -- fits most needs rra = "RRA:AVERAGE:0.5:1:576"; rra = "$rra RRA:AVERAGE:0.5:6:576"; rra = "$rra RRA:AVERAGE:0.5:24:576"; rra = "$rra RRA:AVERAGE:0.5:288:576"; rramax = "RRA:MAX:0.5:1:576"; rramax = "$rramax RRA:MAX:0.5:6:576"; rramax = "$rramax RRA:MAX:0.5:24:576"; rramax = "$rramax RRA:MAX:0.5:288:576"; rramin = "RRA:MIN:0.5:1:576"; rramin = "$rramin RRA:MIN:0.5:6:576"; rramin = "$rramin RRA:MIN:0.5:24:576"; rramin = "$rramin RRA:MIN:0.5:288:576"; # larger data retention ( for today the numbers are provided as an example ) #rra = "RRA:AVERAGE:0.5:1:8640"; #rra = "$rra RRA:AVERAGE:0.5:6:8640"; #rra = "$rra RRA:AVERAGE:0.5:24:8640"; #rra = "$rra RRA:AVERAGE:0.5:288:8640"; # #rramax = "RRA:MAX:0.5:1:576"; #rramax = "$rramax RRA:MAX:0.5:6:8640"; #rramax = "$rramax RRA:MAX:0.5:24:8640"; #rramax = "$rramax RRA:MAX:0.5:288:8640"; # #rramin = "RRA:MIN:0.5:1:576"; #rramin = "$rramin RRA:MIN:0.5:6:8640"; #rramin = "$rramin RRA:MIN:0.5:24:8640"; #rramin = "$rramin RRA:MIN:0.5:288:8640"; I (or anybody) can provide dump/restore tools along with hobbit, too. Regards, -- Charles Goyard - user-98f9625a7a59@xymon.invalid - (+33) 1 45 38 01 31
list Stef Coene
▸
On Friday 19 January 2007 13:23, Henrik Stoerner wrote:
So if we're going to add MAX/MIN tracking to the RRD-files, we might as well do it at the same time that we change the granularity. The numbers I've been thinking of are to keep * 30 days of 5-minute averages * 90 days of 15-minute averages * 360 days on 1-hour averages * 1080 days of 3-hour averages
Even 1080 days is not enough for me. I have rrd files of backup systems that are 6 years old. I also have scripts that monitors devices every hour. So specifying the --step parameter per rrd can also be handy.
▸
That alone would cause the size of the RRD files to increase 15 times. Adding MIN+MAX tracking would mean tripling the size. An "average" host in my setup uses ~400 KB of diskspace for RRD-files, so increasing that 15x3 times means it would grow to ~16 MB per host. It's a significant increase (I'd have to get more disk space for my production systems to handle that), but disks are getting bigger and cheaper - and I'd still be storing data for 4000 hosts on less than 60 GB.
I agree. We are monitoring TB's of oracle databases. I'm sure they can miss 100 GB for monitoring purpose. Btw, the trends patch your wrote is working perfect. The next step is finding a way to create graphs. Stef
list Gildas le Nadan
▸
Hi,
The TRACKMAX feature is really interesting as the max values get "diluted" in the week/month/year views.
[...SNIP...]
▸
However, there has also been some requests to increase the granularity of the stored data, e.g. to keep 7 days worth of 5-minute averages as opposed to the current 2 days. And likewise for the other RRA's. So if we're going to add MAX/MIN tracking to the RRD-files, we might as well do it at the same time that we change the granularity. The numbers I've been thinking of are to keep * 30 days of 5-minute averages * 90 days of 15-minute averages * 360 days on 1-hour averages * 1080 days of 3-hour averages
[...SNIP...]
▸
We also have to keep in mind that TRACKMAX (and TRACKMIN) is a per-statusname option, not a global setting. I can take some time next week to add these features into hobbit, if we agree on how to do it. I propose a setting in hobbitserver.cfg for RRAS like : # hobbitserver.cfg sample # standard setup -- fits most needs rra = "RRA:AVERAGE:0.5:1:576"; rra = "$rra RRA:AVERAGE:0.5:6:576"; rra = "$rra RRA:AVERAGE:0.5:24:576"; rra = "$rra RRA:AVERAGE:0.5:288:576";
[...SNIP...]
I (or anybody) can provide dump/restore tools along with hobbit, too.
I like this approach where one can set the values that fits for his particular setup. Some of us don't really need to keep 5 minutes average forever, but keeping the absolute maximum is mandatory, so the ability to tweek the value to the needs seems like a definite win. Of course, adjusting the rra setup for existing rrd, as well as adding min/max value makes the dump/restore tools an absolute necessity. There's probably some caveats concerning the dump/restore when it happens while hobbitd_rrd tries to update the values of the rrd being recreated, no? If so, maybe the dump/restore tool could be integrated in hobbitd_rdd or do_rrd or in the future "rrd distribution backend"? Also, I think the setup syntax is a bit confusing, I would prefer either rra1 = "RRA:AVERAGE:0.5:1:576"; rra2 = "RRA:AVERAGE:0.5:6:576"; rra3 = "RRA:AVERAGE:0.5:24:576"; rra4 = "RRA:AVERAGE:0.5:288:576"; or rra = "RRA:AVERAGE:0.5:1:576 \ RRA:AVERAGE:0.5:6:576 \ RRA:AVERAGE:0.5:24:576 \ RRA:AVERAGE:0.5:288:576";
list Henrik Størner
▸
On Thu, Dec 28, 2006 at 07:07:07PM +0100, Charles Goyard wrote:
here is a patch that provides two new features to the NCV graph backend. It answers to a request I did on August 2nd (subject: "Dynamic or Split NCV"). - split-ncv : Lets you store an ever-changing set of data on a status column basis.
[snip]
- trackmax : it lets you keep track of the maximum value of a dataset,
I've merged Charles' split-NCV patch now. Regarding the "trackmax" feature, this was really a more general issue of being able to configure the way the RRD databases are generated. So you can configure it to include the MAX data values, or change the granularity of the data stored etc. So I've decided to do this all-out, and just have a config file where you can setup all of the RRD definitions you want to use when creating new RRD files. So I haven't merged Charles' trackmax feature, but done something else that will let you do the same thing (and more). Regards, Henrik