Xymon version 5

9 messages in this thread

list James Louis · Sat, 31 Oct 2015 12:44:56 -0500 ·

Hi all,

Maybe I'm out of the loop on this but is there going to be a version 5 of
Xymon? Is there a roadmap page?

Thanks,
Jim

-- 


*----------------------------------------[image:
http://action.lung.org/site/TR/Climb?px=5751753&pg=personal&fr_id=12850&s_src=BF_emailbadge]
<http://action.lung.org/site/TR/Climb?px=5751753&pg=personal&fr_id=12850&s_src=BF_emailbadge>;
Jim Louis       \\\\||////       \ ~ ~  /       | @ @ |*


*--oOo---(_)---oOo--*

"The software isn't finished until the last user is dead." ~ Anonymous

list Ryan Novosielski · Sat, 31 Oct 2015 14:52:47 -0400 ·

Check the list archives. There was a discussion not too long ago about it, as I recall.

--
____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
|| \\UTGERS      |---------------------*O*---------------------
||_// Biomedical | Ryan Novosielski - Senior Technologist
|| \\ and Health | user-46c89e614701@xymon.invalid<mailto:user-46c89e614701@xymon.invalid>- 973/972.0922 (2x0922)
||  \\  Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark

▸ quoted from James Louis

    `'

On Oct 31, 2015, at 11:45, James Louis <user-518fefde45bd@xymon.invalid<mailto:user-518fefde45bd@xymon.invalid>> wrote:

Hi all,

Maybe I'm out of the loop on this but is there going to be a version 5 of Xymon? Is there a roadmap page?

Thanks,
Jim

--


[http://bfapps1.boundlessfundraising.com/badge/alaclimb/display/5751753/12850/status.jpg]<http://action.lung.org/site/TR/Climb?px=5751753&pg=personal&fr_id=12850&s_src=BF_emailbadge>;

▸ quoted from James Louis


    Jim Louis

       \\\\||////
       \ ~ ~  /
       | @ @ |
--oOo---(_)---oOo--


"The software isn't finished until the last user is dead." ~ Anonymous

list Japheth Cleaver · Sun, 1 Nov 2015 02:08:02 -0800 ·

▸ quoted from Ryan Novosielski

On Sat, October 31, 2015 10:52 am, Novosielski, Ryan wrote:

Check the list archives. There was a discussion not too long ago about it,
as I recall.

I believe http://lists.xymon.com/archive/2014-January/038897.html is most
recent.

▸ quoted from Ryan Novosielski

On Oct 31, 2015, at 11:45, James Louis
<user-518fefde45bd@xymon.invalid<mailto:user-518fefde45bd@xymon.invalid>> wrote:

Hi all,

Maybe I'm out of the loop on this but is there going to be a version 5 of
Xymon? Is there a roadmap page?


I was actually planning on a roadmap update during the 4.3.22 announcement
this week, but might as well post now. :)

After 4.3.22 is released, I'll be opening up the 4.4 release. This is
primarily focused on high performance and core features, including a
back-port of the compression support from Xymon 5 (a/k/a "trunk"). It's a
fairly large patch set that incorporates a lot of the work that had been
done in the Terabithia RPMs. The potential for migration gotchas combined
with the sheer size of the delta warrants a minor version bump, but it's
still identifiably "Xymon 4".


Xymon 5 is the trunk update Henrik has been working on for a while now.
The biggest core features are compression, sqlite as a data store, IPv6
and direct SSL communication support throughout, a new poller (xymonnet2)
and page generator, and more streamlined client->RRD processing.
Development on it's been somewhat stalled with Henrik being busy at his
new position, but it's still definitely "on the roadmap". For a more
specific ETA on it, I'd have to defer to him.


Performance-wise, there's a lot of improvement in 4.4 which gives
continued flexibility to the "roll your own" solutions for both
DB-type/automated access and SSL wrapping of payloads, which are probably
the two largest fundamental core features (outside of IPv6)... As
mentioned, compression will be backported into 4.4 -- with support for lzo
and lz4 on top of the original zlib -- but the rest hasn't made it in in a
testable form.


My plan is to get a 4.4 beta release out by the end of December.


I'm also looking to complete full removal of display HTML and an easier
way to deploy "themes" at the front end (which can, of course, link in to
whatever custom display solutions are desired). Some of that might be 4.4,
some might be 4.5, but IMO it's additional ways of visualizing / browsing
data at the web layer that would most help the project's reach.


Regards,

-jc

list Adam Goryachev · Sun, 1 Nov 2015 21:18:49 +1100 ·

▸ quoted from Japheth Cleaver

On 01/11/15 21:08, J.C. Cleaver wrote:

I was actually planning on a roadmap update during the 4.3.22 announcement
this week, but might as well post now. :)

After 4.3.22 is released, I'll be opening up the 4.4 release. This is
primarily focused on high performance and core features, including a
back-port of the compression support from Xymon 5 (a/k/a "trunk"). It's a
fairly large patch set that incorporates a lot of the work that had been
done in the Terabithia RPMs. The potential for migration gotchas combined
with the sheer size of the delta warrants a minor version bump, but it's
still identifiably "Xymon 4".

Xymon 5 is the trunk update Henrik has been working on for a while now.
The biggest core features are

Would it perhaps be useful to break down the above enhancements, and release them incrementally?
compression - version 4.4
sqlite as a data store - version 4.5
IPv6 - version 4.6
direct SSL communication support throughout - version 4.7
a new poller (xymonnet2) - version 4.8
and page generator - version 4.9
more streamlined client->RRD processing - version 5.0 (because now we have done everything that was needed for 5.0 :)

▸ quoted from Japheth Cleaver

Performance-wise, there's a lot of improvement in 4.4 which gives
continued flexibility to the "roll your own" solutions for both
DB-type/automated access and SSL wrapping of payloads, which are probably
the two largest fundamental core features (outside of IPv6)... As
mentioned, compression will be backported into 4.4 -- with support for lzo
and lz4 on top of the original zlib -- but the rest hasn't made it in in a
testable form.


My plan is to get a 4.4 beta release out by the end of December.


I'm also looking to complete full removal of display HTML and an easier
way to deploy "themes" at the front end (which can, of course, link in to
whatever custom display solutions are desired). Some of that might be 4.4,
some might be 4.5, but IMO it's additional ways of visualizing / browsing
data at the web layer that would most help the project's reach.

This could be good. One thing that might help improve performance of the
pages is to use some sort of persistent connection between javascript on
the client (browser) and the server. Then any status change can be
communicated in real time to the browser which can simply update the
icon and background colour as needed. This could also be used to read
real time status updates to pass to other systems/services.

Finally, separating the "theme" from the code will allow much greater
innovation with how the data is presented.

So, overall, I'm just wondering if we should create smaller targets such
that it is possible to add one feature at a time without as long a delay
between new feature release?

Regards,
Adam


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

list Japheth Cleaver · Sun, 1 Nov 2015 07:09:26 -0800 ·

▸ quoted from Adam Goryachev

On Sun, November 1, 2015 2:18 am, Adam Goryachev wrote:

On 01/11/15 21:08, J.C. Cleaver wrote:

I was actually planning on a roadmap update during the 4.3.22
announcement
this week, but might as well post now. :)

After 4.3.22 is released, I'll be opening up the 4.4 release. This is
primarily focused on high performance and core features, including a
back-port of the compression support from Xymon 5 (a/k/a "trunk"). It's
a
fairly large patch set that incorporates a lot of the work that had been
done in the Terabithia RPMs. The potential for migration gotchas
combined
with the sheer size of the delta warrants a minor version bump, but it's
still identifiably "Xymon 4".

Xymon 5 is the trunk update Henrik has been working on for a while now.
The biggest core features are

Would it perhaps be useful to break down the above enhancements, and
release them incrementally?
compression - version 4.4
sqlite as a data store - version 4.5
IPv6 - version 4.6
direct SSL communication support throughout - version 4.7
a new poller (xymonnet2) - version 4.8
and page generator - version 4.9
more streamlined client->RRD processing - version 5.0 (because now we have
done everything that was needed for 5.0 :)


That type of thing might work, though that does mean committing to the
order somewhat ;)

sqlite in xymond in particular (rather than the channel workers) is a
pretty large change that could have some performance impact over the tree
structure in use now.

▸ quoted from Adam Goryachev

I'm also looking to complete full removal of display HTML and an easier
way to deploy "themes" at the front end (which can, of course, link in
to
whatever custom display solutions are desired). Some of that might be
4.4,
some might be 4.5, but IMO it's additional ways of visualizing /
browsing
data at the web layer that would most help the project's reach.

This could be good. One thing that might help improve performance of the
pages is to use some sort of persistent connection between javascript on
the client (browser) and the server. Then any status change can be
communicated in real time to the browser which can simply update the
icon and background colour as needed. This could also be used to read
real time status updates to pass to other systems/services.


Possibly. The problem there is that keeping an active channel open
directly into xymond (for what are essentially push notifications) has the
potential for impact to the core, which we generally want to avoid. It'd
be safer to isolate that by having a --channel=stachg worker writing to a
location that can be queried by other processes, optionally notifying, so
that message processing isn't affected.*

Generally speaking, though, yes. An AJAX-y dashboard can dynamically
update the page without reloading the view, and that's exactly the type of
thing that can be provided.


(Intermittently querying for changes can actually be done now with an
appropriate "appfeed" filter command, like
http://www.example.com/xymon-cgi/appfeed.sh?filter=color=red,yellow,green+lastchange>1445783474
, which will give you XML for anything that's changed in the last week or
so.

*This does also raise the possibility of providing HA/mirroring directly
within xymond. A read-only xymond replicated mirror can be used for report
generation and whatnot.)

▸ quoted from Adam Goryachev

Finally, separating the "theme" from the code will allow much greater
innovation with how the data is presented.

This is definitely the goal. The display we have now has certainly stood
the test of time, but can be off-putting to those expecting e.g., a
Nagios-type experience.

The existing CGIs now can be replaced with whatever one likes, header
customization can be used to wrap xymongen-generated page to match, and
channel workers can be written to process data into any sort of message
bus or DB (all RRD data at SNC gets written to a rotating tmpfs log, which
is ingested by a Big Data system for further analysis and a distinct
dashboard)... The problem is that doing all that requires a fair amount of
xymon experience to architect something customized to what your site
requires; lowering the barrier to entry for that will help.

▸ quoted from Adam Goryachev

So, overall, I'm just wondering if we should create smaller targets such
that it is possible to add one feature at a time without as long a delay
between new feature release?

It's definitely something I'll keep in mind.

Part of the problem this year was that I really only intended to pause
development for about a month after 4.3.21 was released -- there'd been a
lot of updates leading up to that and I wanted a chance to let it settle a
bit. Unfortunately, one month became one full quarter :/ The pace between
new feature releases is definitely going to pick back up going forward,
one way or the other.


Regards,
-jc

list Ralph Mitchell · Sun, 1 Nov 2015 12:56:30 -0500 ·

How big of a thing is "direct SSL communication support throughout" going
to be?  I mean, what will that cover?  I'd really like to see encrypted
client-to-server connections, because I'm required to encrypt all network
traffic.  At present I'm handlng it by using curl to send to
xymoncgimsg.cgi.  It works well enough for a small number of clients, but
is becoming unreliable as more are added.

Ralph Mitchell


On Sun, Nov 1, 2015 at 10:09 AM, J.C. Cleaver <user-87556346d4af@xymon.invalid>

▸ quoted from Japheth Cleaver

wrote:

On Sun, November 1, 2015 2:18 am, Adam Goryachev wrote:

On 01/11/15 21:08, J.C. Cleaver wrote:

I was actually planning on a roadmap update during the 4.3.22
announcement
this week, but might as well post now. :)

After 4.3.22 is released, I'll be opening up the 4.4 release. This is
primarily focused on high performance and core features, including a
back-port of the compression support from Xymon 5 (a/k/a "trunk"). It's
a
fairly large patch set that incorporates a lot of the work that had been
done in the Terabithia RPMs. The potential for migration gotchas
combined
with the sheer size of the delta warrants a minor version bump, but it's
still identifiably "Xymon 4".

Xymon 5 is the trunk update Henrik has been working on for a while now.
The biggest core features are

Would it perhaps be useful to break down the above enhancements, and
release them incrementally?
compression - version 4.4
sqlite as a data store - version 4.5
IPv6 - version 4.6
direct SSL communication support throughout - version 4.7
a new poller (xymonnet2) - version 4.8
and page generator - version 4.9
more streamlined client->RRD processing - version 5.0 (because now we
have
done everything that was needed for 5.0 :)


That type of thing might work, though that does mean committing to the
order somewhat ;)

sqlite in xymond in particular (rather than the channel workers) is a
pretty large change that could have some performance impact over the tree
structure in use now.

I'm also looking to complete full removal of display HTML and an easier
way to deploy "themes" at the front end (which can, of course, link in
to
whatever custom display solutions are desired). Some of that might be
4.4,
some might be 4.5, but IMO it's additional ways of visualizing /
browsing
data at the web layer that would most help the project's reach.

This could be good. One thing that might help improve performance of the
pages is to use some sort of persistent connection between javascript on
the client (browser) and the server. Then any status change can be
communicated in real time to the browser which can simply update the
icon and background colour as needed. This could also be used to read
real time status updates to pass to other systems/services.


Possibly. The problem there is that keeping an active channel open
directly into xymond (for what are essentially push notifications) has the
potential for impact to the core, which we generally want to avoid. It'd
be safer to isolate that by having a --channel=stachg worker writing to a
location that can be queried by other processes, optionally notifying, so
that message processing isn't affected.*

Generally speaking, though, yes. An AJAX-y dashboard can dynamically
update the page without reloading the view, and that's exactly the type of
thing that can be provided.


(Intermittently querying for changes can actually be done now with an
appropriate "appfeed" filter command, like


http://www.example.com/xymon-cgi/appfeed.sh?filter=color=red,yellow,green+lastchange

1445783474

▸ quoted from Japheth Cleaver

, which will give you XML for anything that's changed in the last week or
so.

*This does also raise the possibility of providing HA/mirroring directly
within xymond. A read-only xymond replicated mirror can be used for report
generation and whatnot.)

Finally, separating the "theme" from the code will allow much greater
innovation with how the data is presented.

This is definitely the goal. The display we have now has certainly stood
the test of time, but can be off-putting to those expecting e.g., a
Nagios-type experience.

The existing CGIs now can be replaced with whatever one likes, header
customization can be used to wrap xymongen-generated page to match, and
channel workers can be written to process data into any sort of message
bus or DB (all RRD data at SNC gets written to a rotating tmpfs log, which
is ingested by a Big Data system for further analysis and a distinct
dashboard)... The problem is that doing all that requires a fair amount of
xymon experience to architect something customized to what your site
requires; lowering the barrier to entry for that will help.

So, overall, I'm just wondering if we should create smaller targets such
that it is possible to add one feature at a time without as long a delay
between new feature release?

It's definitely something I'll keep in mind.

Part of the problem this year was that I really only intended to pause
development for about a month after 4.3.21 was released -- there'd been a
lot of updates leading up to that and I wanted a chance to let it settle a
bit. Unfortunately, one month became one full quarter :/ The pace between
new feature releases is definitely going to pick back up going forward,
one way or the other.


Regards,
-jc

list Henrik Størner · Sun, 01 Nov 2015 20:15:47 +0100 ·

Hi,

J.C. Cleaver:

▸ quoted from Japheth Cleaver

Xymon 5 is the trunk update Henrik has been working on for a while now.
The biggest core features are compression, sqlite as a data store, IPv6
and direct SSL communication support throughout, a new poller (xymonnet2)
and page generator, and more streamlined client->RRD processing.
Development on it's been somewhat stalled with Henrik being busy at his
new position, but it's still definitely "on the roadmap". For a more
specific ETA on it, I'd have to defer to him.

First, I must say that I am pretty impressed (and happy) with the way JC has managed the v4 codebase. I think I made the right choice when I handed it over to him.

Re. version 5 ... well, most of the code is there. One of the last releases I did was an alfa/beta/preview/whatever with the new networking code. But it does need some cleanup, and re-adapting into the current v4 code - it was originally branched off 4.3.7 I think, and since JC took over I haven't fitted any of his fixes and enhancements into v5-code.

And unfortunately I don't have the time to do any serious work on it. Version 5 just turned out to include too many changes and enhancements, it should never have gone for so long without a release.

I think the best way forward is to slice the version-5 code into smaller pieces, and then gently slip them into the current v4 code. There are some generic changes that could easily move back into v4, and then some more serious changes for the new networking code. But even that could (I think) me nudged into the v4 code, and live alongside the current net-tool. That would probably be a slightly less intimidating set of changes.


Regards,
Henrik

list Henrik Størner · Sun, 01 Nov 2015 20:17:20 +0100 ·

Hmm sorry about that, seems the drafts of this message were also posted to the list. How that happened I honestly don't know.

Regards,
Henrik


Den 01-11-2015 kl. 20:15 skrev Henrik Størner:

▸ quoted from Henrik Størner

Hi,

J.C. Cleaver:

Xymon 5 is the trunk update Henrik has been working on for a while now.
The biggest core features are compression, sqlite as a data store, IPv6
and direct SSL communication support throughout, a new poller (xymonnet2)
and page generator, and more streamlined client->RRD processing.
Development on it's been somewhat stalled with Henrik being busy at his
new position, but it's still definitely "on the roadmap". For a more
specific ETA on it, I'd have to defer to him.

First, I must say that I am pretty impressed (and happy) with the way JC has managed the v4 codebase. I think I made the right choice when I handed it over to him.

Re. version 5 ... well, most of the code is there. One of the last releases I did was an alfa/beta/preview/whatever with the new networking code. But it does need some cleanup, and re-adapting into the current v4 code - it was originally branched off 4.3.7 I think, and since JC took over I haven't fitted any of his fixes and enhancements into v5-code.

And unfortunately I don't have the time to do any serious work on it. Version 5 just turned out to include too many changes and enhancements, it should never have gone for so long without a release.

I think the best way forward is to slice the version-5 code into smaller pieces, and then gently slip them into the current v4 code. There are some generic changes that could easily move back into v4, and then some more serious changes for the new networking code. But even that could (I think) me nudged into the v4 code, and live alongside the current net-tool. That would probably be a slightly less intimidating set of changes.


Regards,
Henrik

list Japheth Cleaver · Sun, 1 Nov 2015 19:08:14 -0800 ·

▸ quoted from Henrik Størner

On Sun, November 1, 2015 11:15 am, Henrik Størner wrote:

Hi,

J.C. Cleaver:

Xymon 5 is the trunk update Henrik has been working on for a while now.
The biggest core features are compression, sqlite as a data store, IPv6
and direct SSL communication support throughout, a new poller
(xymonnet2)
and page generator, and more streamlined client->RRD processing.
Development on it's been somewhat stalled with Henrik being busy at his
new position, but it's still definitely "on the roadmap". For a more
specific ETA on it, I'd have to defer to him.

First, I must say that I am pretty impressed (and happy) with the way JC
has managed the v4 codebase. I think I made the right choice when I
handed it over to him.

Thank you :) It's been an honor to help out with such a useful project.

▸ quoted from Henrik Størner

Re. version 5 ... well, most of the code is there. One of the last
releases I did was an alfa/beta/preview/whatever with the new networking
code. But it does need some cleanup, and re-adapting into the current v4
code - it was originally branched off 4.3.7 I think, and since JC took
over I haven't fitted any of his fixes and enhancements into v5-code.

And unfortunately I don't have the time to do any serious work on it.
Version 5 just turned out to include too many changes and enhancements,
it should never have gone for so long without a release.

I think the best way forward is to slice the version-5 code into smaller
pieces, and then gently slip them into the current v4 code. There are
some generic changes that could easily move back into v4, and then some
more serious changes for the new networking code. But even that could (I
think) me nudged into the v4 code, and live alongside the current
net-tool. That would probably be a slightly less intimidating set of
changes.


Regards,
Henrik

I think this would work out really well... Migrating library functions
together would help ease the transition over to the new code, which could
be done in gradually and in parallel. With core support, the various
worker modules and daemons could be brought to parity as testing permits.

Having a good version-specific baseline to compare the trunk to really
helps, too. It'll make the commit-analyzing and diff-wrangling a lot
easier.

The more I think about it, the more I also like Adam's idea of assigning
point release versions to various functional milestones. We can focus on
the core compatibility and flesh out the component support one block at a
time.


Regards,
-jc

Xymon version 5 🔗 link

Xymon version 5