urlplus.pl http/https monitoring extension script 🔗 link

 
From: Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid] 
Sent: Monday, January 28, 2008 10:55 AM
To: user-ae9b8668bcde@xymon.invalid; user-e594ddd6c03a@xymon.invalid
Subject: [hobbit] urlplus.pl http/https monitoring extension script

 
We are in the process of replacing several sitescope monitors with
Hobbit, and I have found some of the built-in Hobbit http checks
inadequate.  As a result, I have been working on an extension script
which at its core is essentially a wrapper for curl.  However, it
provides for more robust user-customization than is easily possible with
Hobbit's built-in tests.  It is still a work-in-progress, but I believe
it has matured enough to be worth posting on The Shire.  I wrote it
using perl, but I have a feeling that some point in the future it may
need to be converted to C (mostly for the speed).  However, it seems to
work pretty good for me now.

The main feature I required from this script was the addition of a
per-host customizable timeout period.  Some of the hosts I'm testing
take up to a minute to return back any data, and this is considered
normal.  However, I didn't want to bump up the timeout for _all_ hosts.
Admittedly, this script uses an additional configuration file and
doesn't completely integrate with the bb-hosts file, but it works for me
and helps keep the bb-hosts file clean.  I'd be interested in what other
people think of this script, which is the main reason I'm contributing
it now.  I have attached it to this post (I haven't yet subscribed to
theshire-developers, so I'm not able to upload it myself yet).

On Jan 28, 2008 10:54 AM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

We are in the process of replacing several sitescope monitors with Hobbit,
and I have found some of the built-in Hobbit http checks inadequate.  As a
result, I have been working on an extension script which at its core is
essentially a wrapper for curl.  However, it provides for more robust
user-customization than is easily possible with Hobbit's built-in tests.  It
is still a work-in-progress, but I believe it has matured enough to be worth
posting on The Shire.  I wrote it using perl, but I have a feeling that some
point in the future it may need to be converted to C (mostly for the
speed).  However, it seems to work pretty good for me now.

The main feature I required from this script was the addition of a
per-host customizable timeout period.  Some of the hosts I'm testing take up
to a minute to return back any data, and this is considered normal.
However, I didn't want to bump up the timeout for _all_ hosts.  Admittedly,
this script uses an additional configuration file and doesn't completely
integrate with the bb-hosts file, but it works for me and helps keep the
bb-hosts file clean.  I'd be interested in what other people think of this
script, which is the main reason I'm contributing it now.  I have attached
it to this post (I haven't yet subscribed to theshire-developers, so I'm not
able to upload it myself yet).

On Jan 28, 2008 9:54 AM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

We are in the process of replacing several sitescope monitors with Hobbit,
and I have found some of the built-in Hobbit http checks inadequate.  As a
result, I have been working on an extension script which at its core is
essentially a wrapper for curl.  However, it provides for more robust
user-customization than is easily possible with Hobbit's built-in tests.  It
is still a work-in-progress, but I believe it has matured enough to be worth
posting on The Shire.  I wrote it using perl, but I have a feeling that some
point in the future it may need to be converted to C (mostly for the
speed).  However, it seems to work pretty good for me now.

The main feature I required from this script was the addition of a
per-host customizable timeout period.  Some of the hosts I'm testing take up
to a minute to return back any data, and this is considered normal.
However, I didn't want to bump up the timeout for _all_ hosts.  Admittedly,
this script uses an additional configuration file and doesn't completely
integrate with the bb-hosts file, but it works for me and helps keep the
bb-hosts file clean.  I'd be interested in what other people think of this
script, which is the main reason I'm contributing it now.  I have attached
it to this post (I haven't yet subscribed to theshire-developers, so I'm not
able to upload it myself yet).

That looks like the kind of thing I've been doing in bash scripts using
curl.  I've thought about switching to a different language, but I don't
know enough Perl to get started.  Maybe I'll take another look...  :)

On Jan 28, 2008 11:07 AM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

That looks like the kind of thing I've been doing in bash scripts using

curl.  I've thought about switching to a different language, but I don't
know enough Perl to get started.  Maybe I'll take another look...  :)

For the kind of work that this script is doing, there weren't really any
reasons to require perl over shell script.  I picked perl because I've been
doing a lot of perl scripting lately, but also because it should be easier
to convert the perl code to C at some point, if I feel that is necessary.

As far as speed goes, I've found there isn't much of a difference between
perl and shell, though in certain cases I've seen shell script a little
quicker.  I'm sure the usual "your mileage may vary" quote is worth throwing
out at this point ;-)

I'm not doing anything really perl-crazy with this script, and I think I

have it pretty well commented, so it should be a decent reference if you
want to get in to perl.  Either way, I'd be interested in comparing notes in
what our two scripts do, and possibly combining features from yours into
this one.

 
From: Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid] 
Sent: Monday, January 28, 2008 11:07 AM
To: user-ae9b8668bcde@xymon.invalid; user-e594ddd6c03a@xymon.invalid
Subject: [hobbit] Re: urlplus.pl http/https monitoring extension script

 
Whoops, forgot to include the template information...


==== urlplus.pl ====

Summary

  * Download:      [[monitors:monitor|monitor]]

  * Author:        user-ae3e15c22de1@xymon.invalid|Gary Baluha


  * Compatible with: Linux, OpenBSD

  * Last Update:   2008-01-28


\\--------------------------------------------------------------------


====== Monitor ======


^ Compatibility | Hobbit 4.2 |

^ Requirements | Perl, unix, curl |


^ Download | None |

^ Last Update | 2008-01-28 |


===== Description =====

Provides a slightly more robust form of URL content checking that Hobbit
does out of the box.

Per-host user-configuration is provided.


===== Installation =====

=== Client side ===

N/A


=== Server side ===

1) Copy urlplus.pl to $BBHOME/ext

2) Create an new configuration file: $BBHOME/etc/cont-check.cfg


===== Source =====

==== Monitor ====


<code>

</code>


===== Known  Bugs and Issues =====

None at this time


===== To Do =====

Need more testing on a larger variety of hosts, mostly for inclusion of
new features


===== Credits =====


===== Changelog =====


  * **2008-01-02**

    * Initial creation

  * **2008-01-28**

    * Initial release to The Shire

 
On Jan 28, 2008 10:54 AM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

We are in the process of replacing several sitescope monitors with
Hobbit, and I have found some of the built-in Hobbit http checks
inadequate.  As a result, I have been working on an extension script
which at its core is essentially a wrapper for curl.  However, it
provides for more robust user-customization than is easily possible with
Hobbit's built-in tests.  It is still a work-in-progress, but I believe
it has matured enough to be worth posting on The Shire.  I wrote it
using perl, but I have a feeling that some point in the future it may
need to be converted to C (mostly for the speed).  However, it seems to
work pretty good for me now.

The main feature I required from this script was the addition of a
per-host customizable timeout period.  Some of the hosts I'm testing
take up to a minute to return back any data, and this is considered
normal.  However, I didn't want to bump up the timeout for _all_ hosts.
Admittedly, this script uses an additional configuration file and
doesn't completely integrate with the bb-hosts file, but it works for me
and helps keep the bb-hosts file clean.  I'd be interested in what other
people think of this script, which is the main reason I'm contributing
it now.  I have attached it to this post (I haven't yet subscribed to
theshire-developers, so I'm not able to upload it myself yet).

On Jan 28, 2008 10:54 AM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

We are in the process of replacing several sitescope monitors with Hobbit,
and I have found some of the built-in Hobbit http checks inadequate.  As a
result, I have been working on an extension script which at its core is
essentially a wrapper for curl.  However, it provides for more robust
user-customization than is easily possible with Hobbit's built-in tests.  It
is still a work-in-progress, but I believe it has matured enough to be worth
posting on The Shire.  I wrote it using perl, but I have a feeling that some
point in the future it may need to be converted to C (mostly for the
speed).  However, it seems to work pretty good for me now.

The main feature I required from this script was the addition of a
per-host customizable timeout period.  Some of the hosts I'm testing take up
to a minute to return back any data, and this is considered normal.
However, I didn't want to bump up the timeout for _all_ hosts.  Admittedly,
this script uses an additional configuration file and doesn't completely
integrate with the bb-hosts file, but it works for me and helps keep the
bb-hosts file clean.  I'd be interested in what other people think of this
script, which is the main reason I'm contributing it now.  I have attached
it to this post (I haven't yet subscribed to theshire-developers, so I'm not
able to upload it myself yet).

On Feb 6, 2008 1:40 PM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

I've partially completed the end-to-end transaction monitoring part of
urlplus.pl.  Currently, it only works with a 2-part test: content check
initial page, submit content, content check resulting page.  I haven't yet
completed it beyond 2 pages.  I've also been re-prioritized, so I'm not sure
when I'll get around to getting it to work past 2 pages.  But in any case,
I've also cleaned up some of the code, making it a little more user-friendly
to modify for their own needs.

On Feb 6, 2008 4:17 PM, Ralph Mitchell <user-00a5e44c48c0@xymon.invalid> wrote:

On Feb 6, 2008 1:40 PM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

I've partially completed the end-to-end transaction monitoring part of
urlplus.pl.  Currently, it only works with a 2-part test: content check
initial page, submit content, content check resulting page.  I haven't
yet
completed it beyond 2 pages.  I've also been re-prioritized, so I'm not
sure
when I'll get around to getting it to work past 2 pages.  But in any
case,
I've also cleaned up some of the code, making it a little more
user-friendly
to modify for their own needs.

Can I interest you in a diff that adds proxy userid/password??  Most
of the checks I do go through a proxy that requires authentication.
And then there are some occasions when I use a *different* proxy that
doesn't require authentication at all - I suppose that would mean
moving the proxy url out to the config file, or maybe pick the proxy
via p1/p2/P1/P2??

Ralph Mitchell

On Feb 7, 2008 8:44 AM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

Cool, thanks!  I only just looked over the diff file, but so far it looks
pretty good.  You did a good job at keeping the same formatting I used too,
which is nice :-)  Only one thing was missed: you didn't add your initials
and a comment to the revision history section ;-)

Regarding the proxy, I think it would be a good idea to have the proxy url
in the config file.  But I think in general, the config file could use a
little overhauling, as some of the monitors I set up have long urls and
especially with the "post" type, long post content as well.  It makes the
single-line format a bit unwieldy to edit.  I'll have to come up with
something when I get a chance.