Status, NCV and time stamps 🔗 link

On Mon, October 5, 2015 3:52 pm, Boldt, David wrote:

I'm in the process of converting Big Brother tests to XYMon and would like
to enable RRD/graphics for these tests.

Several of these tests present date/time information which is important.

As a workaround for one test I have replaced the colon on a HH:MM:DD
formatted date with a dot, but it doesn't really look like a time anymore.

* Is there a way to have NCV ignore colons (and only use "=")?

* Is there a way to have NCV ignore a line (which might contain colons)?

* Can I use Data messages in such a way that the corresponding Status
message is not parsed?

On Mon, Oct 5, 2015 at 7:36 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid> wrote:

On Mon, October 5, 2015 3:52 pm, Boldt, David wrote:

I'm in the process of converting Big Brother tests to XYMon and would
like
to enable RRD/graphics for these tests.

Several of these tests present date/time information which is important.

As a workaround for one test I have replaced the colon on a HH:MM:DD
formatted date with a dot, but it doesn't really look like a time
anymore.

* Is there a way to have NCV ignore colons (and only use "=")?

* Is there a way to have NCV ignore a line (which might contain colons)?

Unfortunately, there isn't a way to do either of these at the moment. I'd
considered a data parser that looks for a hidden HTML comment marker for
lines to process (and ignores the rest), but it's not quite present yet.

* Can I use Data messages in such a way that the corresponding Status
message is not parsed?

The easiest way to do this would be to add a --filter= option to the
xymond_channel command line for the xymond_rrd process that's handling the
*status* channel, but not the data channel.

There's a CPU load hit for doing so (since now you're doing a PCRE on each
message), however it may not have much of an impact depending on your
message volume.

* Might there be some other mechanism entirely?

Generally, speaking for more advanced RRD submission, the 'trends' message
can be a useful data payload which is read directly by xymond_rrd. The
advantage there (aside from more direct control) is the ability to send
data points for multiple graphs at once, which could outweigh the expense
of another message transmission.


HTH,

-jc

-- 
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "Ah, this is obviously some strange usage of the word 'safe' that I
wasn't previously aware of."
    --Douglas Adams (1952 - 2001), Arthur Dent in "The Hitchhikers Guide to
the Galaxy"

From: Xymon <xymon-bounces at xymon.com> on behalf of Boldt, David <user-945c7be1b8e4@xymon.invalid>
Sent: Monday, October 5, 2015 7:48 PM
To: J.C. Cleaver
Cc: xymon at xymon.com
Subject: Re: [Xymon] Status, NCV and time stamps

Thanks for these leads. Will looks at both the filter and "trends" message options.

As a request for enhancement, I think it would be useful to have an option to use a regexp to extract data from the status, one for each variable.


On Mon, Oct 5, 2015 at 7:36 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid<mailto:user-87556346d4af@xymon.invalid>> wrote:
On Mon, October 5, 2015 3:52 pm, Boldt, David wrote:

I'm in the process of converting Big Brother tests to XYMon and would like
to enable RRD/graphics for these tests.

Several of these tests present date/time information which is important.

As a workaround for one test I have replaced the colon on a HH:MM:DD
formatted date with a dot, but it doesn't really look like a time anymore.

* Is there a way to have NCV ignore colons (and only use "=")?

* Is there a way to have NCV ignore a line (which might contain colons)?

Unfortunately, there isn't a way to do either of these at the moment. I'd
considered a data parser that looks for a hidden HTML comment marker for
lines to process (and ignores the rest), but it's not quite present yet.

* Can I use Data messages in such a way that the corresponding Status
message is not parsed?

The easiest way to do this would be to add a --filter= option to the
xymond_channel command line for the xymond_rrd process that's handling the
*status* channel, but not the data channel.

There's a CPU load hit for doing so (since now you're doing a PCRE on each
message), however it may not have much of an impact depending on your
message volume.

* Might there be some other mechanism entirely?

Generally, speaking for more advanced RRD submission, the 'trends' message
can be a useful data payload which is read directly by xymond_rrd. The
advantage there (aside from more direct control) is the ability to send
data points for multiple graphs at once, which could outweigh the expense
of another message transmission.


HTH,

-jc


--
                                         -- David Boldt

   "Ah, this is obviously some strange usage of the word 'safe' that I wasn't previously aware of."
    --Douglas Adams (1952 - 2001), Arthur Dent in "The Hitchhikers Guide to the Galaxy"

On Mon, Oct 5, 2015 at 11:03 PM, Galen Johnson <user-87f955643e3d@xymon.invalid> wrote:

When I ran into this, someone suggested that I replace the colon with it's
html character code equivalent (%3A).  Since xymon parses the colon, it
should ignore the the html encoded representation but the browser will still
display it...it's a complete hack but it should work (ultimately, I chose
not to include the time stamp since that is in the page already).


=G=


From: Xymon <xymon-bounces at xymon.com> on behalf of Boldt, David
<user-945c7be1b8e4@xymon.invalid>
Sent: Monday, October 5, 2015 7:48 PM
To: J.C. Cleaver
Cc: xymon at xymon.com
Subject: Re: [Xymon] Status, NCV and time stamps

Thanks for these leads. Will looks at both the filter and "trends" message
options.

As a request for enhancement, I think it would be useful to have an option
to use a regexp to extract data from the status, one for each variable.


On Mon, Oct 5, 2015 at 7:36 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid> wrote:

On Mon, October 5, 2015 3:52 pm, Boldt, David wrote:

I'm in the process of converting Big Brother tests to XYMon and would
like
to enable RRD/graphics for these tests.

Several of these tests present date/time information which is important.

As a workaround for one test I have replaced the colon on a HH:MM:DD
formatted date with a dot, but it doesn't really look like a time
anymore.

* Is there a way to have NCV ignore colons (and only use "=")?

* Is there a way to have NCV ignore a line (which might contain colons)?

Unfortunately, there isn't a way to do either of these at the moment. I'd
considered a data parser that looks for a hidden HTML comment marker for
lines to process (and ignores the rest), but it's not quite present yet.

* Can I use Data messages in such a way that the corresponding Status
message is not parsed?

The easiest way to do this would be to add a --filter= option to the
xymond_channel command line for the xymond_rrd process that's handling the
*status* channel, but not the data channel.

There's a CPU load hit for doing so (since now you're doing a PCRE on each
message), however it may not have much of an impact depending on your
message volume.

* Might there be some other mechanism entirely?

Generally, speaking for more advanced RRD submission, the 'trends' message
can be a useful data payload which is read directly by xymond_rrd. The
advantage there (aside from more direct control) is the ability to send
data points for multiple graphs at once, which could outweigh the expense
of another message transmission.


HTH,

-jc

--
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "Ah, this is obviously some strange usage of the word 'safe' that I
wasn't previously aware of."
    --Douglas Adams (1952 - 2001), Arthur Dent in "The Hitchhikers Guide to
the Galaxy"

-- 
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "Ah, this is obviously some strange usage of the word 'safe' that I
wasn't previously aware of."
    --Douglas Adams (1952 - 2001), Arthur Dent in "The Hitchhikers
Guide to the Galaxy"

On Mon, Oct 5, 2015 at 7:48 PM, Boldt, David <user-945c7be1b8e4@xymon.invalid> wrote:

Thanks for these leads. Will looks at both the filter and "trends"
message options.

As a request for enhancement, I think it would be useful to have an
option to use a regexp to extract data from the status, one for each
variable.


On Mon, Oct 5, 2015 at 7:36 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid>
wrote:

On Mon, October 5, 2015 3:52 pm, Boldt, David wrote:

I'm in the process of converting Big Brother tests to XYMon and would
like
to enable RRD/graphics for these tests.

Several of these tests present date/time information which is
important.

As a workaround for one test I have replaced the colon on a HH:MM:DD
formatted date with a dot, but it doesn't really look like a time
anymore.

* Is there a way to have NCV ignore colons (and only use "=")?

* Is there a way to have NCV ignore a line (which might contain
colons)?

Unfortunately, there isn't a way to do either of these at the moment. I'd
considered a data parser that looks for a hidden HTML comment marker for
lines to process (and ignores the rest), but it's not quite present yet.

* Can I use Data messages in such a way that the corresponding Status
message is not parsed?

The easiest way to do this would be to add a --filter= option to the
xymond_channel command line for the xymond_rrd process that's handling
the
*status* channel, but not the data channel.

There's a CPU load hit for doing so (since now you're doing a PCRE on
each
message), however it may not have much of an impact depending on your
message volume.

* Might there be some other mechanism entirely?

Generally, speaking for more advanced RRD submission, the 'trends'
message
can be a useful data payload which is read directly by xymond_rrd. The
advantage there (aside from more direct control) is the ability to send
data points for multiple graphs at once, which could outweigh the expense
of another message transmission.


HTH,

-jc

--
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "Ah, this is obviously some strange usage of the word 'safe' that I
wasn't previously aware of."
    --Douglas Adams (1952 - 2001), Arthur Dent in "The Hitchhikers Guide
to the Galaxy"

On Tue, October 6, 2015 7:42 am, Matt Vander Werf wrote:

(Sending this again to the list, as I sent it using the wrong e-mail
previously (non-subscriber e-mail))

I second this request for enhancement. It would be nice to be able to send
more data from the client to be displayed on the column/test page, but not
get parsed to be included in the RRD graph. Only parsing certain lines
that
are in the NCV format would be a nice feature. Something like what J.C.
mentioned ("a data parser that looks for a hidden HTML comment marker for
lines to process (and ignores the rest)") would work well for this, IMO.

Or is there a way to do this already? (even through a hack?)

Nothing urgent, just something that would be nice to be included in some
future Xymon release.

Thanks.

--
Matt Vander Werf

On Tue, Oct 6, 2015 at 10:39 AM, Matt Vander Werf <user-dfc3cf2ca434@xymon.invalid>
wrote:

I second this request for enhancement. It would be nice to be able to
send
more data from the client to be displayed on the column/test page, but
not
get parsed to be included in the RRD graph. Only parsing certain lines
that
are in the NCV format would be a nice feature. Something like what J.C.
mentioned ("a data parser that looks for a hidden HTML comment marker
for
lines to process (and ignores the rest)") would work well for this, IMO.

Or is there a way to do this already? (even through a hack?)

Nothing urgent, just something that would be nice to be included in some
future Xymon release.

Thanks.

--
Matt Vander Werf

On Mon, Oct 5, 2015 at 7:48 PM, Boldt, David <user-945c7be1b8e4@xymon.invalid> wrote:

Thanks for these leads. Will looks at both the filter and "trends"
message options.

As a request for enhancement, I think it would be useful to have an
option to use a regexp to extract data from the status, one for each
variable.


On Mon, Oct 5, 2015 at 7:36 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid>
wrote:

On Mon, October 5, 2015 3:52 pm, Boldt, David wrote:

I'm in the process of converting Big Brother tests to XYMon and
would
like
to enable RRD/graphics for these tests.

Several of these tests present date/time information which is
important.

As a workaround for one test I have replaced the colon on a HH:MM:DD
formatted date with a dot, but it doesn't really look like a time
anymore.

* Is there a way to have NCV ignore colons (and only use "=")?

* Is there a way to have NCV ignore a line (which might contain
colons)?

Unfortunately, there isn't a way to do either of these at the moment.
I'd
considered a data parser that looks for a hidden HTML comment marker
for
lines to process (and ignores the rest), but it's not quite present
yet.

* Can I use Data messages in such a way that the corresponding

Status

message is not parsed?

The easiest way to do this would be to add a --filter= option to the
xymond_channel command line for the xymond_rrd process that's handling
the
*status* channel, but not the data channel.

There's a CPU load hit for doing so (since now you're doing a PCRE on
each
message), however it may not have much of an impact depending on your
message volume.

* Might there be some other mechanism entirely?

Generally, speaking for more advanced RRD submission, the 'trends'
message
can be a useful data payload which is read directly by xymond_rrd. The
advantage there (aside from more direct control) is the ability to
send
data points for multiple graphs at once, which could outweigh the
expense
of another message transmission.


HTH,

-jc

--
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "Ah, this is obviously some strange usage of the word 'safe' that I
wasn't previously aware of."
    --Douglas Adams (1952 - 2001), Arthur Dent in "The Hitchhikers
Guide
to the Galaxy"

wrote:

FYI, all,

4.3.22 will have comment markers that can be used for both skipping a line
and for ending processing. Something along the lines of this will work:
<pre>
ip6_dst_cache : 7680
ndisc_cache : 7680
<!-- ncv_skip -->ip6_mrt_cache : 0
ext4_inode_cache : 7960000
<!-- ncv_end -->
scsi_sense_cache : 15360
</pre>

Not entirely sure if that will render properly :) But essentially:
pre-pend lines with "ncv_skip" in a regular HTML comment to skip it, and a
bare "ncv_end" in a comment will abort further processing for that
message.


I'd considered adding regex capabilities directly into the NCV module, but
it's a fairly time sensitive operation on more heavily-loaded servers, so
it's important to keep existing modules like that as efficient as
possible.

The process of RRD module writing here is pretty simple though (simpler
than I'd recalled it), and it shouldn't be too hard to add other types of
automated/regular processing besides NCV that could be called up.


HTH,

-jc


On Tue, October 6, 2015 7:42 am, Matt Vander Werf wrote:

(Sending this again to the list, as I sent it using the wrong e-mail
previously (non-subscriber e-mail))

I second this request for enhancement. It would be nice to be able to
send
more data from the client to be displayed on the column/test page, but
not
get parsed to be included in the RRD graph. Only parsing certain lines
that
are in the NCV format would be a nice feature. Something like what J.C.
mentioned ("a data parser that looks for a hidden HTML comment marker for
lines to process (and ignores the rest)") would work well for this, IMO.

Or is there a way to do this already? (even through a hack?)

Nothing urgent, just something that would be nice to be included in some
future Xymon release.

Thanks.

--
Matt Vander Werf

On Tue, Oct 6, 2015 at 10:39 AM, Matt Vander Werf <user-dfc3cf2ca434@xymon.invalid>
wrote:

I second this request for enhancement. It would be nice to be able to
send
more data from the client to be displayed on the column/test page, but
not
get parsed to be included in the RRD graph. Only parsing certain lines
that
are in the NCV format would be a nice feature. Something like what J.C.
mentioned ("a data parser that looks for a hidden HTML comment marker
for
lines to process (and ignores the rest)") would work well for this, IMO.

Or is there a way to do this already? (even through a hack?)

Nothing urgent, just something that would be nice to be included in some
future Xymon release.

Thanks.

--
Matt Vander Werf

On Mon, Oct 5, 2015 at 7:48 PM, Boldt, David <user-945c7be1b8e4@xymon.invalid> wrote:

Thanks for these leads. Will looks at both the filter and "trends"
message options.

As a request for enhancement, I think it would be useful to have an
option to use a regexp to extract data from the status, one for each
variable.


On Mon, Oct 5, 2015 at 7:36 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid>
wrote:

On Mon, October 5, 2015 3:52 pm, Boldt, David wrote:

I'm in the process of converting Big Brother tests to XYMon and
would
like
to enable RRD/graphics for these tests.

Several of these tests present date/time information which is
important.

As a workaround for one test I have replaced the colon on a HH:MM:DD
formatted date with a dot, but it doesn't really look like a time
anymore.

* Is there a way to have NCV ignore colons (and only use "=")?

* Is there a way to have NCV ignore a line (which might contain
colons)?

Unfortunately, there isn't a way to do either of these at the moment.
I'd
considered a data parser that looks for a hidden HTML comment marker
for
lines to process (and ignores the rest), but it's not quite present
yet.

* Can I use Data messages in such a way that the corresponding

Status

message is not parsed?

The easiest way to do this would be to add a --filter= option to the
xymond_channel command line for the xymond_rrd process that's handling
the
*status* channel, but not the data channel.

There's a CPU load hit for doing so (since now you're doing a PCRE on
each
message), however it may not have much of an impact depending on your
message volume.

* Might there be some other mechanism entirely?

Generally, speaking for more advanced RRD submission, the 'trends'
message
can be a useful data payload which is read directly by xymond_rrd. The
advantage there (aside from more direct control) is the ability to
send
data points for multiple graphs at once, which could outweigh the
expense
of another message transmission.


HTH,

-jc

--
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "Ah, this is obviously some strange usage of the word 'safe' that I
wasn't previously aware of."
    --Douglas Adams (1952 - 2001), Arthur Dent in "The Hitchhikers
Guide
to the Galaxy"

On Sat, October 31, 2015 4:11 am, Matt Vander Werf wrote:

Would it be possible to also make it so that you can exclude parts of
lines/multiple lines (not just at the end of the message) from being
parsed
using a block form -like method, in addition to what you've already put in
place?

Something like this:

<!-- ncv_skip --> *this should be ignored* </--> ip6_mrt_cache : 0
ext4_inode_cache <!-- ncv_skip --> *this should also be ignored* </--> :
7960000

Basically, the parsing tool would just see this to parse for the graphing:

ip6_mrt_cache : 0
ext4_inode_cache : 7960000

Is this something that would be possible to also put in place? I'm not
sure
how the new functionality is done now, so I don't know if this is
something
that's feasible to add in or not.

If it's not very easy or very feasible or even possible at all (without
major re-work of the code), then that's fine. The existing new
functionality for parsing skipping will be very useful to me as it is!

But if it could be added, that would be really awesome and even more
useful
for me!!

If this is something that is possible, but probably won't be ready for the
actual 4.3.22 release, then that's perfectly fine with me as well! I'm
willing to wait until the next release if that's what is necessary! :)

Thanks as always!

--
Matt Vander Werf

On Sat, Oct 17, 2015 at 10:54 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid>
wrote:

FYI, all,

4.3.22 will have comment markers that can be used for both skipping a
line
and for ending processing. Something along the lines of this will work:
<pre>
ip6_dst_cache : 7680
ndisc_cache : 7680
<!-- ncv_skip -->ip6_mrt_cache : 0
ext4_inode_cache : 7960000
<!-- ncv_end -->
scsi_sense_cache : 15360
</pre>

Not entirely sure if that will render properly :) But essentially:
pre-pend lines with "ncv_skip" in a regular HTML comment to skip it, and
a
bare "ncv_end" in a comment will abort further processing for that
message.


I'd considered adding regex capabilities directly into the NCV module,
but
it's a fairly time sensitive operation on more heavily-loaded servers,
so
it's important to keep existing modules like that as efficient as
possible.

The process of RRD module writing here is pretty simple though (simpler
than I'd recalled it), and it shouldn't be too hard to add other types
of
automated/regular processing besides NCV that could be called up.


HTH,

-jc


On Tue, October 6, 2015 7:42 am, Matt Vander Werf wrote:

(Sending this again to the list, as I sent it using the wrong e-mail
previously (non-subscriber e-mail))

I second this request for enhancement. It would be nice to be able to
send
more data from the client to be displayed on the column/test page, but
not
get parsed to be included in the RRD graph. Only parsing certain lines
that
are in the NCV format would be a nice feature. Something like what

J.C.

mentioned ("a data parser that looks for a hidden HTML comment marker
for
lines to process (and ignores the rest)") would work well for this,

IMO.

Or is there a way to do this already? (even through a hack?)

Nothing urgent, just something that would be nice to be included in
some
future Xymon release.

Thanks.

--
Matt Vander Werf

On Tue, Oct 6, 2015 at 10:39 AM, Matt Vander Werf <user-dfc3cf2ca434@xymon.invalid>
wrote:

I second this request for enhancement. It would be nice to be able to
send
more data from the client to be displayed on the column/test page,
but
not
get parsed to be included in the RRD graph. Only parsing certain
lines
that
are in the NCV format would be a nice feature. Something like what

J.C.

mentioned ("a data parser that looks for a hidden HTML comment marker
for
lines to process (and ignores the rest)") would work well for this,

IMO.

Or is there a way to do this already? (even through a hack?)

Nothing urgent, just something that would be nice to be included in
some
future Xymon release.

Thanks.

--
Matt Vander Werf

On Mon, Oct 5, 2015 at 7:48 PM, Boldt, David <user-945c7be1b8e4@xymon.invalid> wrote:

Thanks for these leads. Will looks at both the filter and "trends"
message options.

As a request for enhancement, I think it would be useful to have an
option to use a regexp to extract data from the status, one for each
variable.


On Mon, Oct 5, 2015 at 7:36 PM, J.C. Cleaver

<user-87556346d4af@xymon.invalid>

wrote:

On Mon, October 5, 2015 3:52 pm, Boldt, David wrote:

I'm in the process of converting Big Brother tests to XYMon and
would
like
to enable RRD/graphics for these tests.

Several of these tests present date/time information which is
important.

As a workaround for one test I have replaced the colon on a

HH:MM:DD

formatted date with a dot, but it doesn't really look like a time
anymore.

* Is there a way to have NCV ignore colons (and only use "=")?

* Is there a way to have NCV ignore a line (which might contain
colons)?

Unfortunately, there isn't a way to do either of these at the
moment.
I'd
considered a data parser that looks for a hidden HTML comment
marker
for
lines to process (and ignores the rest), but it's not quite present
yet.

* Can I use Data messages in such a way that the corresponding

Status

message is not parsed?

The easiest way to do this would be to add a --filter= option to
the
xymond_channel command line for the xymond_rrd process that's
handling
the
*status* channel, but not the data channel.

There's a CPU load hit for doing so (since now you're doing a PCRE
on
each
message), however it may not have much of an impact depending on
your
message volume.

* Might there be some other mechanism entirely?

Generally, speaking for more advanced RRD submission, the 'trends'
message
can be a useful data payload which is read directly by xymond_rrd.

The

advantage there (aside from more direct control) is the ability to
send
data points for multiple graphs at once, which could outweigh the
expense
of another message transmission.


HTH,

-jc

--
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "Ah, this is obviously some strange usage of the word 'safe' that

I

wasn't previously aware of."
    --Douglas Adams (1952 - 2001), Arthur Dent in "The Hitchhikers
Guide
to the Galaxy"