On Mon, May 25, 2009 at 10:23 AM, Ralph Mitchell <user-00a5e44c48c0@xymon.invalid>wrote:
It was titled "Hobbit content check script". Here's the body of the
message, and the attachments:
Sorry I didn't get back to you earlier - I've been fighting with perl
& SOAP, trying to get some status messages out of a bunch of load
balancers. I think I finally got it now....
Attached (hopefully :) are a couple of files I tried to post to the
list some time last year. There's a shell script that does a login to
a Siteminder site, and a perl script evolved from something Daniel
Stenberg wrote - he's the original author of curl. Daniel's very
active on the curl mailing list, by the way, along with a bunch of
other helpful folks, if it starts going weird on you.
The Siteminder login probably won't be much use to you as is, but it
demonstrates a few things I've worked out over the years. Like, the
standard options string to pass to curl, collecting messages along the
way, and loading up the message string with a Manual Check url at the
end for ops to try the check for themselves before waking people up...
You can put almost *anything* into the message string that gets sent
to Hobbit. Right now I have scripts that check car rentals at travel
sites for a particular company and emit a table of the current prices.
Another script hits a cruise monitor and puts out a list using the
&red and &green dots to mark cruise lines as reachable or not. You
can push out font color and size changes, urls to saved pages, etc.
As far as cookies go, I generally just have curl save them whenever it
finishes:
curl -b cookies -c cookies .....
If you do that you can go look at the cookies and see what came in.
Other things become possible too, such as editing cookies in between
curl calls. If you do keep cookies, though, the *first* thing you
should do in the script is delete the cookie file. That's like
clearing the browser cookie cache, and can avoid a lot of mysterious
errors.
Finally, if you think you might spend some time on web site scripts,
there's a Firefox extension that is absolutely invaluable -
LiveHTTPheaders. It shows you *exactly* what the browser sends to the
server, as well as what it gets back - not the page content, just the
headers, but that's all you need to recreate the sequence. It even
works twith secure sites. If you see something going out that you
can't account for, the usual suspect is javascript. I've seen
javascript that assembles a url on the fly from form elements, fiddles
with some of the form elements, then posts the form. That kind of
thing can be difficult to nail down, and almost impossible to
replicate...
Ralph Mitchell
The attachments may not have gone through, so here they are again:
========== bb-siteminder.sh ==========
#!/bin/sh
# In case of disaster, uncomment the next line and watch the output
# set -x
#
# This script runs out of cron. Tell it where to find Big Brother.
BBHOME=/home/bb/bb18b3
export BBHOME
TESTHOST=system.domain.com
URL="http://$TESTHOST/";
ORIGURL="$URL"
USERNAME=bbusername
PASSWORD=bbsecretword
# Where to stash the downloaded html files. These get patched into the BB
# report so that the monitoring guys can click through and see what we got.
BBDIR="$BBHOME/www/siteminder"
# Gonna need a cookie jar. Start off empty, so each checkout starts clean.
COOKIES="$BBDIR/$TESTHOST.cookies"
rm -f $COOKIES
# Some standard curl flags. See the curl man page for details
STDOPTS="-w %{url_effective} -L --connect-timeout 60 --max-time 120 -s -S -b
$COOKIES -c $COOKIES -A Mozilla/4.79"
# This is just a tag for identifying environment dumps.
BBPROG=bb-siteminder.sh; export BBPROG
# The next block came from the BB ext example script. If BBHOME is not set
# there's no point continuing. If BBTMP is not set, load up the standard
# environment from bbdef.sh.
if test "$BBHOME" = ""
then
echo "BBHOME is not set... exiting"
exit 1
fi
if test ! "$BBTMP" # GET DEFINITIONS IF NEEDED
then
# echo "*** LOADING BBDEF ***"
. $BBHOME/etc/bbdef.sh # INCLUDE STANDARD DEFINITIONS
fi
# This is used to label the result column
TEST="login"
# Make sure bail out code is set.
bailout=0
# ========== step 1 ==========
# Grab the first page
STEP=step1
# Name the file for the host, test and step, to make it unique
FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html
# remove any previous copy to avoid misunderstandings
rm -f $FILENAME
# Grab the first page. The effective url is recorded for patching
# into the action url is necessary
effect=`$CURL $STDOPTS \
-o $FILENAME \
$URL`
# Save the return code, it might be important
ret=$?
# code 52 means "the server sent nothing"
if [ "$ret" -eq "52" ]; then
# empty reply, try it again
MESSAGE="$STEP: failed with curl error code: 52 - Empty reply from server.
Url: <a href=$URL>$URL</a>"
MESSAGE="$MESSAGE
$STEP: retrying..."
sleep 2
effect=`$CURL $STDOPTS \
-o /dev/null \
$URL`
ret=$?
fi
if [ -s $FILENAME ]; then
# file exists and has size greater than zero. look for something to
validate
TESTSTR="Something from the Login Page"
check=`$GREP -ic "$TESTSTR" $FILENAME`
if [ "$check" -gt "0" ]; then
# Found the test string, we're good to continue
MESSAGE="$MESSAGE
$STEP: Got Login page - found <b>$TESTSTR</b>"
else
# Didn't find the test string, bail out of subsequent steps
MESSAGE="$MESSAGE
$STEP: failed to find <b>$TESTSTR</b>. curl error code: $ret. Url: <a
href=$URL>$URL</a>"
bailout=1
fi
else
# Got nothing useful from the server, give up
MESSAGE-"$MESSAGE
$STEP: failed with empty file. curl error code: $ret. Url: <a
href=$URL>$URL</a>"
bailout=1
fi
# If previous step completed OK, perform next step
if [ "$bailout" -eq "0" ]; then
# formextract.pl strips out the form elements and formats them like this:
# -d username=
# -d password=
# http://server.domain.com/path/to/form/processor
# On some pages, the action url is not fully qualified.
FORM=`cat $FILENAME | $HOME/bin/formextract.pl`
# Pull out the form variable lines and patch in the userid & password
FORMVARS=`echo "$FORM" | $GREP -- '-d' | $SED \
-e "s/txtUsername=/txtUsername=$USERNAME/" \
-e "s/Password=/Password=$PASSWORD/"`
# extract the action url. Adjustments can be to the url in the pipeline
ACTION=`echo "$FORM" | $TAIL -1`
# Break up the effective url based on the /'s
OFS="$IFS"
IFS="/"
set $effect
IFS="$OFS"
# Assemble the proper action url. It should be one of these forms:
# Complete URL: http://system.domain.com/path/to/form/processor
# URL="$ACTION"
# Absolute path, no hostname: /path/to/form/processor
# URL="$1//$3$ACTION"
# Relative path, no hostname: path/to/form/processor
# URL="$1//$3/$ACTION"
# pick up $1 (protocol) & $3 (hostname), because the // in the effective
url makes $2=null.
URL="$1//$3$ACTION"
STEP=step2
FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html
rm -f $FILENAME
# Post the form back to the action url. The returned file should
# contain the page a browser would show after logging in.
effect=`$CURL $STDOPTS \
-w "%{url_effective}" \
-o $FILENAME \
$FORMVARS \
$URL`
ret=$?
# As before, make sure the file is not empty, and that it contains
# something we recognise from a successful login. You can also look
# for signs of login failure, such as a repeat of the login page, or
# some message such as "login failed".
if [ -s $FILENAME ]; then
if [ "$ret" -eq "0" ]; then
TESTSTR="Big Brother Test Page"
check=`$GREP -ic "$TESTSTR" $FILENAME`
if [ "$check" -gt "0" ]; then
MESSAGE="$MESSAGE
$STEP: Found <b>$TESTSTR</b>. Looking good"
fi
else
MESSAGE="$MESSAGE
$STEP: failed to find <b>$TESTSTR</b>. curl error code: $ret. Url: <a
href=$URL>$URL</a>"
bailout=1
fi
else
MESSAGE-"$MESSAGE
$STEP: failed with empty file. curl error code: $ret. Url: <a
href=$URL>$URL</a>"
bailout=1
fi
fi
# If bailout is non-zero, something unexpected happened.
if [ "$bailout" -eq "1" ]; then
COLOR="red"
elif [ "$bailout" -eq "2" ]; then
# This is useful for non-critical-but-interesting conditions
# that may be worth looking for. There are none in this example.
COLOR="yellow"
else
# WooHoo!!
COLOR="green"
fi
MESSAGE="$MESSAGE
<P>
See the saved files in <a href=$BBWEBHOST/bb/siteminder/>the checkout
directory</a>
<P>
<a href=$ORIGURL>Manual Check</a>"
#
# The script runs from cron, with output directed to a file in /tmp.
# This echo is not strictly necessary, but it can be useful to validate
# that it ran correctly
echo "$TESTHOST: $TEST: $COLOR"
# convert the dots to commas in the hostname
MACHINE=`echo $TESTHOST | $SED -e 's/\./,/g'`
# Assemble the status message as per normal
LINE="status $MACHINE.$TEST $COLOR `date`
$MESSAGE"
# fire it off to BB/Hobbit
$BB $BBDISP "$LINE" # SEND IT TO BBDISPLAY
========== formextract.pl ==========
#!/usr/bin/env perl
# $Id: formfind,v 1.5 2003/04/28 13:48:16 bagder Exp $
#
# formfind.pl
#
# This script gets a HTML page on stdin and presents form information on
# stdout.
#
# Author: Daniel Stenberg <user-047629c2b550@xymon.invalid>
# Version: 0.2 Nov 18, 2002
#
# HISTORY
#
# 0.1 - Nov 12 1998 - Created now!
# 0.2 - Nov 18 2002 - Enhanced. Removed URL support, use only stdin.
# sometime - echo the first form in a format suitable for dropping
# into a curl command line. - Ralph Mitchell
#
$in="";
if($ARGV[0] eq "-h") {
print "Usage: $0 < HTML\n";
exit;
}
sub namevalue {
my ($tag)=@_;
my $name=$tag;
if($name =~ /name *=/i) {
if($name =~ /name *= *([^\"\']([^ \">]*))/i) {
$name = $1;
}
elsif($name =~ /name *= *(\"|\')([^\"\']*)(\"|\')/i) {
$name=$2;
}
else {
# there is a tag but we didn't find the contents
$name="[weird]";
}
}
else {
# no name given
$name="";
}
# get value tag
my $value= $tag;
if($value =~ /[^\.a-zA-Z0-9]value *=/i) {
if($value =~ /[^\.a-zA-Z0-9]value *= *([^\"\']([^ \">]*))/i) {
$value = $1;
}
elsif($value =~ /[^\.a-zA-Z0-9]value *= *(\"|\')([^\"\']*)(\"|\')/i)
{
$value=$2;
}
else {
# there is a tag but we didn't find the contents
$value="[weird]";
}
}
else {
$value="";
}
return ($name, $value);
}
while(<STDIN>) {
$line = $_;
push @indoc, $line;
$line=~ s/\n//g;
$line=~ s/\r//g;
$in=$in.$line;
}
while($in =~ /[^<]*(<[^>]+>)/g ) {
# we have a tag in $1
$tag = $1;
if($tag =~ /^<!--/) {
# this is a comment tag, ignore it
}
else {
if(!$form &&
($tag =~ /^< *form/i )) {
$method= $tag;
if($method =~ /method *=/i) {
$method=~ s/.*method *= *(\"|)([^ \">]*).*/$2/gi;
}
else {
$method="get"; # default method
}
$action= $tag;
$action=~ s/.*action *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;
$method=uc($method);
$enctype=$tag;
if ($enctype =~ /enctype *=/) {
$enctype=~ s/.*enctype *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;
if($enctype eq "multipart/form-data") {
$enctype="multipart form upload [use -F]"
}
$enctype = "\n--- type: $enctype";
}
else {
$enctype="";
}
# print "URL=\"$action\"$enctype\n";
$form=1;
}
elsif($form &&
($tag =~ /< *\/form/i )) {
# print "--- end of FORM\n";
print "$action\n";
$form=0;
if( 0 ) {
print "*** Fill in all or any of these: (default assigns may
be shown)\n";
for(@vars) {
$var = $_;
$def = $value{$var};
print "$var=$def\n";
}
print "*** Pick one of these:\n";
for(@alts) {
print "$_\n";
}
}
undef @vars;
undef @alts;
}
elsif($form &&
($tag =~ /^< *(input|select)/i)) {
$mtag = $1;
($name, $value)=namevalue($tag);
if($mtag =~ /select/i) {
print "Select: NAME=\"$name\"\n";
push @vars, "$name";
$select = 1;
}
else {
$type=$tag;
if($type =~ /type *=/i) {
$type =~ s/.*type *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;
}
else {
$type="text"; # default type
}
$type=uc($type);
if(lc($type) eq "reset") {
# reset types are for UI only, ignore.
}
elsif($name eq "") {
# let's read the value parameter
print "Button: \"$value\" ($type)\n";
push @alts, "$value";
}
else {
$xname = s/\ /\+/g;
print " -d $name=";
if($value ne "") {
# print "\"$value\"";
print "$value";
}
print " \n";
push @vars, "$name";
# store default value:
$value{$name}=$value;
}
}
}
elsif($form &&
($tag =~ /^< *textarea/i)) {
my ($name, $value)=namevalue($tag);
print "Textarea: NAME=\"$name\"\n";
}
elsif($select) {
if($tag =~ /^< *\/ *select/i) {
print "[end of select]\n";
$select = 0;
}
elsif($tag =~ /[^\/] *option/i ) {
my ($name, $value)=namevalue($tag);
my $s;
if($tag =~ /selected/i) {
$s= " (SELECTED)";
}
print " Option VALUE=\"$value\"$s\n";
}
}
}
}
====================