URLplus

On Wed, Apr 30, 2008 at 12:35 PM, Gary Baluha <user-ae3e15c22de1@xymon.invalid> wrote:

On Wed, Apr 30, 2008 at 1:28 PM, Ralph Mitchell <user-00a5e44c48c0@xymon.invalid>
wrote:

Unfortunately it's not just one script.  I've found many variations in
login pages, so I've had to customize the scripts for each different
target.  Some logins require a form to be posted back, others are handled
via WWW-Authenticate pop-up boxes and sometimes redirects have to be
followed.  That's not hard if the site uses Location headers, because curl
can handle those, but I've also found meta-refresh tags and several types of
javascript "go-elsewhere" redirects that have to be individually scripted.

And these are some of the same issues I've run in to while writing the
URLPlus script.  I've been trying to slowly morph it into something that can
configured solely through the configuration file without having to change
any code, but that's easier said than done.  Especially since the original
reason I was writing the script got removed, and is supposed to be converted
to yet another monitoring system (which so far hasn't progressed, probably
because of the fact that some of the monitors are non-trivial).

Seems like we have a "get-rid-of-Hobbit" project about every 2 years or so.


I think our National Tools Team has some kind of click-through tools that
records page requests, but  I'm pretty sure it doesn't do too well with
forms that include some variable(s) sent back by the server.  For example,
one thing I've seen is a form variable called __VIEWSTATE that seems to get
a new, longer value with each successive page loaded.  I've no idea what the
value is as it's encoded some way, but I suppose it tracks where you are in
the page.

I don't think the script recorder can handle variable inputs either.  I'm
watching a number of web pages where I have to enter a date that's days or
weeks ahead of today's date.  The recorder will just see the post containing
the future date and faithfully re-post it every time, right up until it
becomes a past date and the monitor breaks...

I won't be too surprised if in the end I get to keep my scripts, but I'll
probably have to send traps to Unicenter as well as sending Hobbit reports.
That way the "best shore" eyes-on-glass folks in Brazil or Singapore or
wherever won't have to deal with Hobbit, while the 2nd-level folks at home
still get the detail they're used to.

In the meantime, attached are the two files I sent previously, on 6/23/06.
Here's what I wrote then:


bb-siteminder.sh is an example script to demonstrate how to login

formextract.pl is a slightly modified copy of Daniel Stenberg's
formfind.pl, which can be found here:

   http://curl.haxx.se/docs/programs.html

I'm not a perl programmer, so any screwups in formextract are mine, not
Daniel's... :)

It's doubtful that the example script will work out of the box.  I've
found that I generally have to take it in easy stages - fetch the first
page, eyeball it, adjust the script, lather, rinse, repeat.

LiveHTTPHeaders is a really useful Firefox extension for tracking what
happens when navigating web pages.

Ralph Mitchell