URLPlus interest - looking for feedback 🔗 link

From: Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid]
Sent: Thursday, 30 July 2009 12:35 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] [Hobbit] URLPlus interest - looking for feedback

For those of you who have been around here for a while, you will have noticed that I've been somewhat absent of late.  The short story is, I have changed jobs and just don't get as many opportunities to check in as I used to.  This also means I have left my reasons for working on URLPlus behind, which is why development of it has stopped.

The purpose of this post is because I have gotten a few requests from time to time about URLPlus, and I'm curious what the interest in me getting back to developing it would be.  I've been out of the loop as far as new features in the latest version of Hobbit being worked on, but from what I understand, while there are more web page monitoring features built-in, there are still some things where URLPlus could be useful.

So, I would be interested in the following feedback:
1) Is there still demand for more robust website content checking features than what Hobbit supports out of the box?
2) If so, what are the features missing in Hobbit that people are looking for?
3) For those who have chosen to use URLPlus, what are the reasons for that, versus using some other Hobbit plugin (or for that matter, some other monitoring tool)?
4) Finally, for those that are using URLPlus, what additional features would you like to see it be capable of doing?


-Gary

user-9da1a1882f49@xymon.invalid> wrote:

 Hi Gary

I have seen URLPlus, but never delved into it.
Instead, I have been messing about with scripts using curl to try and get
web sites checked.
It gets a bit tricky here, because we have internal and external web sites
we need to check, some need to go through our proxy, some not.
And then we also need to check external "service" web sites that we require
to be up.
And it's all a nice mix of http and https.
Does URLPlus handle proxy servers OK?

Not sure what it needs, since I haven't used it yet, but I would encourage
you to continue working on it.

We have a major project deadline we are working towards right now, so time
is a premium, but in the near future, I will give URLPlus a go, and let you
know what I think.

Regards
     Vernon


*From:* Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid]
*Sent:* Thursday, 30 July 2009 12:35 AM
*To:* user-ae9b8668bcde@xymon.invalid
*Subject:* [hobbit] [Hobbit] URLPlus interest - looking for feedback

For those of you who have been around here for a while, you will have
noticed that I've been somewhat absent of late.  The short story is, I have
changed jobs and just don't get as many opportunities to check in as I used
to.  This also means I have left my reasons for working on URLPlus behind,
which is why development of it has stopped.

The purpose of this post is because I have gotten a few requests from time
to time about URLPlus, and I'm curious what the interest in me getting back
to developing it would be.  I've been out of the loop as far as new features
in the latest version of Hobbit being worked on, but from what I understand,
while there are more web page monitoring features built-in, there are still
some things where URLPlus could be useful.

So, I would be interested in the following feedback:
1) Is there still demand for more robust website content checking features
than what Hobbit supports out of the box?
2) If so, what are the features missing in Hobbit that people are looking
for?
3) For those who have chosen to use URLPlus, what are the reasons for that,
versus using some other Hobbit plugin (or for that matter, some other
monitoring tool)?
4) Finally, for those that are using URLPlus, what additional features
would you like to see it be capable of doing?


-Gary

NOTICE: This email and any attachments are confidential.
They may contain legally privileged information or
copyright material. You must not read, copy, use or
disclose them without authorisation. If you are not an
intended recipient, please contact us at once by return
email and then delete both messages and all attachments.

 

From: Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid] 
Sent: mercoledì 29 luglio 2009 18.35
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] [Hobbit] URLPlus interest - looking for feedback


For those of you who have been around here for a while, you will have noticed that I've been somewhat absent of late.  The short story is, I have changed jobs and just don't get as many opportunities to check in as I used to.  This also means I have left my reasons for working on URLPlus behind, which is why development of it has stopped.

The purpose of this post is because I have gotten a few requests from time to time about URLPlus, and I'm curious what the interest in me getting back to developing it would be.  I've been out of the loop as far as new features in the latest version of Hobbit being worked on, but from what I understand, while there are more web page monitoring features built-in, there are still some things where URLPlus could be useful.

So, I would be interested in the following feedback:
1) Is there still demand for more robust website content checking features than what Hobbit supports out of the box?
2) If so, what are the features missing in Hobbit that people are looking for?
3) For those who have chosen to use URLPlus, what are the reasons for that, versus using some other Hobbit plugin (or for that matter, some other monitoring tool)?
4) Finally, for those that are using URLPlus, what additional features would you like to see it be capable of doing?


-Gary

user-32025d8bd22e@xymon.invalid> wrote:

 Hi Gary,

nice to hear that you're in the loop again.

 I'm currently using v1.21 ($Id: url-plus.pl,v 1.21 2008/04/30 13:56:07) and
I'm happy with it.

My main reason to use it is:
- per host customizable response time

My feature request is the capability to do something like the following:
1. go to a web page
2. insert username/password to login
3. surf the web site following specific links
4. grab some data from a specific web page
5. check the grabbed data (ie if GRABBED_DATA < MY_THRESHOLD then alert)


Regards.

Massimo Morsiani
Information Technology Dept.
Gilbarco S.r.l.
via de' Cattani, 220/G
50145 Firenze, Italy
tel:    +XX-XXX-XXXXX
fax:    +XX-XXX-XXXXXX
email:  user-32025d8bd22e@xymon.invalid
web:    http://www.gilbarco.it


*From:* Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid]
*Sent:* mercoledì 29 luglio 2009 18.35
*To:* user-ae9b8668bcde@xymon.invalid
*Subject:* [hobbit] [Hobbit] URLPlus interest - looking for feedback

For those of you who have been around here for a while, you will have
noticed that I've been somewhat absent of late.  The short story is, I have
changed jobs and just don't get as many opportunities to check in as I used
to.  This also means I have left my reasons for working on URLPlus behind,
which is why development of it has stopped.

The purpose of this post is because I have gotten a few requests from time
to time about URLPlus, and I'm curious what the interest in me getting back
to developing it would be.  I've been out of the loop as far as new features
in the latest version of Hobbit being worked on, but from what I understand,
while there are more web page monitoring features built-in, there are still
some things where URLPlus could be useful.

So, I would be interested in the following feedback:
1) Is there still demand for more robust website content checking features
than what Hobbit supports out of the box?
2) If so, what are the features missing in Hobbit that people are looking
for?
3) For those who have chosen to use URLPlus, what are the reasons for that,
versus using some other Hobbit plugin (or for that matter, some other
monitoring tool)?
4) Finally, for those that are using URLPlus, what additional features
would you like to see it be capable of doing?


-Gary

Please be advised that this email may contain confidential information.
If you are not the intended recipient, please do not read, copy or
re-transmit this email. If you have received this email in error,
please notify us by email by replying to the sender and by telephone
(call us collect at +X XXX-XXX-XXXX) and delete this message and any
attachments. Thank you in advance for your cooperation and assistance.

In addition, Danaher and its subsidiaries disclaim that the content of
this email constitutes an offer to enter into, or the acceptance of,
any
contract or agreement or any amendment thereto; provided that the
foregoing disclaimer does not invalidate the binding effect of any
digital or other electronic reproduction of a manual signature that is
included in any attachment to this email.

On Fri, Jul 31, 2009 at 2:15 AM, Ralph Mitchell <user-00a5e44c48c0@xymon.invalid>wrote:

Heh - that looks like exactly what I was doing for the last 8 years.  Do
you have any vacancies??  :)
URLPlus uses curl, so it should be able to do what your existing scripts
do.  This is the line in the showUrl subroutine:

    $ret=qx(curl -m $tout -s -k -S $pstr "$url" 2>&1);


I guess that's one place some changes could be made - adding options for
user authentication, proxy url and proxy authentication.

Ralph Mitchell


On Thu, Jul 30, 2009 at 9:14 PM, Everett, Vernon <
user-9da1a1882f49@xymon.invalid> wrote:

 Hi Gary

I have seen URLPlus, but never delved into it.
Instead, I have been messing about with scripts using curl to try and get
web sites checked.
It gets a bit tricky here, because we have internal and external web sites
we need to check, some need to go through our proxy, some not.
And then we also need to check external "service" web sites that we
require to be up.
And it's all a nice mix of http and https.
Does URLPlus handle proxy servers OK?

Not sure what it needs, since I haven't used it yet, but I would encourage
you to continue working on it.

We have a major project deadline we are working towards right now, so time
is a premium, but in the near future, I will give URLPlus a go, and let you
know what I think.

Regards
     Vernon

On Fri, Jul 31, 2009 at 4:57 AM, Ralph Mitchell <user-00a5e44c48c0@xymon.invalid>wrote:

I could really have used something like your feature request about 6 years
ago.  Instead I spent a lot of time handcrafting bash scripts to login to
web pages.

Don't get me started on the sites that hit you with 5 different types of
redirects before reaching the front page, or the sites where each input
field is held in it's own personal form. and the submit button executes
javascript to copy the values into form full of hidden fields for the actual
submittal.

I'm pretty sure you're going to need to break out the url stuff into some
kind of config file and just leave an "urlplus" flag in bb-hosts.

On Fri, Jul 31, 2009 at 11:34 AM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

On Fri, Jul 31, 2009 at 4:57 AM, Ralph Mitchell <user-00a5e44c48c0@xymon.invalid>wrote:

I could really have used something like your feature request about 6 years
ago.  Instead I spent a lot of time handcrafting bash scripts to login to
web pages.

Yep, that's kind of how URLPlus got started in the first place ;-)

Don't get me started on the sites that hit you with 5 different types of
redirects before reaching the front page, or the sites where each input
field is held in it's own personal form. and the submit button executes
javascript to copy the values into form full of hidden fields for the actual
submittal.

The redirect issue actually isn't too difficult to work around.  I have
been working on a perl program that is capable of more in-depth session
management than URLPlus is currently capable of, and the solution I'm using
now seems to work pretty well.  My goal is to eventually convert URLPlus
from using a command-line curl solution, to my current one.  This new method
deals with multi-page redirects better.

It's not so much the multi-page redirects using the standard "302: page is now elsewhere" format, as the other weird ways redirects are sometimes done.  The one that irritated me the most did all of these, in no particular order:

   1) meta-refresh with zero time delay and a new url
   2) self-submitting form - i.e. a preloaded form with "form.submit();" at the end of the html, between script tags

   3) self-submitting form - another preloaded form, but with "onLoad=form.submit();" in the html BODY tag
   4) in script tags, change the page location via:   top.location="newurl"

   5) as above, but use "top.href", or "page.href" or something similar.
I'm not knocking your efforts - you've already done more than I ever did towards a generic webpage check.  I just think that the above are going to be tricky to handle in an automated way without replicating a large fraction of a web browser.  But, now at least they're documented in the mailing list for anyone interested in doing their own web checks...  :)

As for the javascript part, that is a bit more difficult.


Especially when the page you just downloaded creates the form POST url on-the-fly from some of the form elements filled in by the user.  Yep, saw that happen too...  Another weird page ran a java function to generate a random character string to include in the url - luckily the function wasn't too hard to extract and shove through the spidermonkey javascript interpreter...  :)