Any way to do http tests with HEAD request?
list Andrew Rakowski
I've been converting from running an installation of BigBrother professional to Xymon (4.3.18), and I've had some complaints that the HTTP test is impacting a few of our servers (lots of back-end database requests and calculations for a generated page that is being tested for availability.)
I didn't understand why nobody was complaining previously, until I saw that BigBro was set up to use the "curl" command with the "-I" option (aka the "--head" option) to send a "HEAD" request to the server. The BigBro web test used this setting:
....
hawkeye{root}169: grep curl etc/bbdef-server.sh
# OR USE curl, WHICH IS BETTER...
LYNX="/usr/bin/curl -g -s -S -m 45 -I -k --connect-timeout 15 --user-agent 'Mozilla/5.0' "
hawkeye{root}170:
....
Is there some equivalent ability in Xymon to just issue a "HEAD" request when testing http and https services?
I expected I'd find an equivalent option for use in the hosts.cfg file in Xymon, but haven't been able to find it if it's there. Any ideas on how to
-Andrew
list Jeremy Laidman
▸
On 10 April 2015 at 11:19, Andrew Rakowski <user-00c59fc5f1d5@xymon.invalid> wrote:
I didn't understand why nobody was complaining previously, until I saw that BigBro was set up to use the "curl" command with the "-I" option (aka the "--head" option) to send a "HEAD" request to the server. The BigBro web test used this setting:
This was one of the problems with BigBro - it forked to external programs to do the probes which doesn't scale, whereas xymonnet does it all internally. Alas, xymonnet sends either a POST or a GET request, and this cannot be modified by configuration. I can see two options for you. One is to do your own curl/wget call with a HEAD request, in a script. The other option is to define your own "http" protocol definition in protocols.cfg and use that. You would probably need to name it something other than "http" otherwise xymonnet will use its internal protocol code rather than what's defined in protocols.cfg. J
list Andrew Rakowski
Thanks for the reply Jeremy. I thought about making a "httphead" test in the protocols.cfg file, and using the "curl" utility to do what BigBrother used to do, but I don't see how to pass the URL to the test.
Is there some implicit passing of test URL to these protocols.cfg defined tests (like the string from the hosts.cfg file is stuck at the end of the invoked command taken from protocols.cfg?) I don't see any examples in the file, and it looks like everything just sends canned strings.
In this case, I'd want to run the "curl" command like this:
/usr/bin/curl -g -s -S -m 45 -I -k --connect-timeout 15
with the URL taken from the hosts.cfg line, which looks like this:
0.0.0.0 xyz-web2 # noconn --timeout=15 https://xyz.somewhere.com/xyz/portal/user/anon/js_peid/123456789/panel/Main/template/content
so that webpage could be tested by the curl utility, which is configured to send only a "HEAD" request to the web server (the '-I' option.) That curl command is what was being used in BigBrother.
Cheers,
-Andrew
▸
On Thu, 9 Apr 2015, Jeremy Laidman wrote:
On 10 April 2015 at 11:19, Andrew Rakowski <user-00c59fc5f1d5@xymon.invalid> wrote:
I didn't understand why nobody was complaining previously, until I saw that BigBro
was set up to use the "curl" command with the "-I" option (aka the "--head" option)
to send a "HEAD" request to the server. The BigBro web test used this setting:
This was one of the problems with BigBro - it forked to external programs to do the probes which
doesn't scale, whereas xymonnet does it all internally.
Alas, xymonnet sends either a POST or a GET request, and this cannot be modified by
configuration.
I can see two options for you. One is to do your own curl/wget call with a HEAD request, in a
script. The other option is to define your own "http" protocol definition in protocols.cfg and
use that. You would probably need to name it something other than "http" otherwise xymonnet
will use its internal protocol code rather than what's defined in protocols.cfg.
J
list Japheth Cleaver
Thinking about it, this might be something that's not too difficult to add into xymonnet either. A HEAD is basically just a GET from the poller's perspective, and any evaluation on body contents (or lack thereof) is semantically identical to the configuration options available now. An open question is what the best way to indicate that this is a "HEAD-only" test would be. It could be something applied to the entire host line (like browser=), made into a separate column (similar to httpstatus= or post=, except without the... POST), or another per-URL protocol specifier similar to how the TLS/SSL configuration is handled. (I could see use cases for all three, although I think the middle option might be the most manageable.) For now, however, this is something that would need to be handled via an external script. Regards, -jc
▸
On Mon, April 13, 2015 1:45 pm, Andrew Rakowski wrote:Thanks for the reply Jeremy. I thought about making a "httphead" test in the protocols.cfg file, and using the "curl" utility to do what BigBrother used to do, but I don't see how to pass the URL to the test. Is there some implicit passing of test URL to these protocols.cfg defined tests (like the string from the hosts.cfg file is stuck at the end of the invoked command taken from protocols.cfg?) I don't see any examples in the file, and it looks like everything just sends canned strings. In this case, I'd want to run the "curl" command like this: /usr/bin/curl -g -s -S -m 45 -I -k --connect-timeout 15 with the URL taken from the hosts.cfg line, which looks like this: 0.0.0.0 xyz-web2 # noconn --timeout=15 https://xyz.somewhere.com/xyz/portal/user/anon/js_peid/123456789/panel/Main/template/content so that webpage could be tested by the curl utility, which is configured to send only a "HEAD" request to the web server (the '-I' option.) That curl command is what was being used in BigBrother. Cheers, -Andrew On Thu, 9 Apr 2015, Jeremy Laidman wrote:On 10 April 2015 at 11:19, Andrew Rakowski <user-00c59fc5f1d5@xymon.invalid> wrote: I didn't understand why nobody was complaining previously, until I saw that BigBro was set up to use the "curl" command with the "-I" option (aka the "--head" option)
to send a "HEAD" request to the server. The BigBro web test used
▸
this setting: This was one of the problems with BigBro - it forked to external programs to do the probes which doesn't scale, whereas xymonnet does it all internally. Alas, xymonnet sends either a POST or a GET request, and this cannot be modified by configuration.
I can see two options for you. One is to do your own curl/wget call with a HEAD request, in a script. The other option is to define your own "http" protocol definition in protocols.cfg and use that. You would probably need to name it something other than
▸
"http" otherwise xymonnet will use its internal protocol code rather than what's defined in protocols.cfg. J
list Sean MacGuire
Could always do something ugly like httphead://blah.com since we have to parse http/https/etc...
▸
J.C. Cleaver wrote:Thinking about it, this might be something that's not too difficult to add into xymonnet either. A HEAD is basically just a GET from the poller's perspective, and any evaluation on body contents (or lack thereof) is semantically identical to the configuration options available now. An open question is what the best way to indicate that this is a "HEAD-only" test would be. It could be something applied to the entire host line (like browser=), made into a separate column (similar to httpstatus= or post=, except without the... POST), or another per-URL protocol specifier similar to how the TLS/SSL configuration is handled. (I could see use cases for all three, although I think the middle option might be the most manageable.) For now, however, this is something that would need to be handled via an external script. Regards, -jc On Mon, April 13, 2015 1:45 pm, Andrew Rakowski wrote:Thanks for the reply Jeremy. I thought about making a "httphead" test in the protocols.cfg file, and using the "curl" utility to do what BigBrother used to do, but I don't see how to pass the URL to the test. Is there some implicit passing of test URL to these protocols.cfg defined tests (like the string from the hosts.cfg file is stuck at the end of the invoked command taken from protocols.cfg?) I don't see any examples in the file, and it looks like everything just sends canned strings. In this case, I'd want to run the "curl" command like this: /usr/bin/curl -g -s -S -m 45 -I -k --connect-timeout 15 with the URL taken from the hosts.cfg line, which looks like this: 0.0.0.0 xyz-web2 # noconn --timeout=15 https://xyz.somewhere.com/xyz/portal/user/anon/js_peid/123456789/panel/Main/template/content so that webpage could be tested by the curl utility, which is configured to send only a "HEAD" request to the web server (the '-I' option.) That curl command is what was being used in BigBrother. Cheers, -Andrew On Thu, 9 Apr 2015, Jeremy Laidman wrote:On 10 April 2015 at 11:19, Andrew Rakowski <user-00c59fc5f1d5@xymon.invalid> wrote: I didn't understand why nobody was complaining previously, until I saw that BigBro was set up to use the "curl" command with the "-I" option (aka the "--head" option) to send a "HEAD" request to the server. The BigBro web test used this setting: This was one of the problems with BigBro - it forked to external programs to do the probes which doesn't scale, whereas xymonnet does it all internally. Alas, xymonnet sends either a POST or a GET request, and this cannot be modified by configuration. I can see two options for you. One is to do your own curl/wget call with a HEAD request, in a script. The other option is to define your own "http" protocol definition in protocols.cfg and use that. You would probably need to name it something other than "http" otherwise xymonnet will use its internal protocol code rather than what's defined in protocols.cfg. J
--
Sean MacGuire user-4915795a2617@xymon.invalid
Key West +X XXX XXX XXXX
The best way to predict the future is to invent it. - Alan Kay
list Jeremy Laidman
▸
On 14 April 2015 at 06:45, Andrew Rakowski <user-00c59fc5f1d5@xymon.invalid> wrote:
Thanks for the reply Jeremy. I thought about making a "httphead" test in the protocols.cfg file, and using the "curl" utility to do what BigBrother used to do,
You would do one or the other.
but I don't see how to pass the URL to the test.
To use protocols.cfg, you would have to specify it in the protocols.cfg file. For example: [httphead] port 80 send "HEAD /xyz/portal/user/anon/js_peid/123456790/panel/Main/template/content\r\nHost: xyz.somewhere.com\r\n\r\n" expect "HTTP/1.1 200 OK" Actually, my protocols.cfg (and probably the installed version) already had a similar definition as [cuspd] which queries a CUPS protocol print server, which is essentially a web service running on port 631.
▸
Is there some implicit passing of test URL to these protocols.cfg defined tests (like the string from the hosts.cfg file is stuck at the end of the invoked command taken from protocols.cfg?)
No. But you can do this with a script. For example: 0.0.0.0 xyz-web2 # noconn HTTPHEAD:TIMEOUT=15,URL= https://xyz.somewhere.com/xyz/portal/user/anon/js_peid/123456789/panel/Main/template/content Then: #!/bin/sh [ "$XYMONHOME" ] || exit 1 TEST=httphead CURL=/usr/bin/curl $XYMONHOME/bin/xymongrep "HTTPHEAD:*" | sed 's/ *# */ /;s/ dialup//;s/ testip//'| while read IP HOST STRING; do PART=`IFS=":"; set - $STRING; shift; echo "$*"` ( IFS=, set - $PART TIMEOUT=15 URL="" while [ "$1" ]; do VAR=`IFS="="; set - $1; echo $1` VAL=`IFS="="; set - $1; shift; echo "$*"` case "$VAR" in TIMEOUT) TIMEOUT="$VAL";; URL) URL="$VAL";; esac shift done RESULT=`$CURL -g -s -S -m 45 -I -k --connect-timeout $TIMEOUT "$URL" 2>&1` [ $? -eq 0 ] && COL=green || COL=red #printf "status $HOST.$TEST $COL `date` HEAD status $COL\n\n$RESULT\n" | $XYMON $XYMSRV "@" printf "status $HOST.$TEST $COL `date` HEAD status $COL\n\n$RESULT\n" ) done
list Andrew Rakowski
Hi jc,
▸
On Mon, 13 Apr 2015, J.C. Cleaver wrote:
Thinking about it, this might be something that's not too difficult to add into xymonnet either. A HEAD is basically just a GET from the poller's perspective, and any evaluation on body contents (or lack thereof) is semantically identical to the configuration options available now. An open question is what the best way to indicate that this is a "HEAD-only" test would be. It could be something applied to the entire host line (like browser=), made into a separate column (similar to httpstatus= or post=, except without the... POST), or another per-URL protocol specifier similar to how the TLS/SSL configuration is handled.
Unless there's significant savings in overhead one way over the other, I see no problem with doing it whichever way is simplest to implement. From my point of view, if I'm testing multiple URLs on the same server, it would typically be for different applications, and I'd probably make a separate host line for each app (so I could have different notifications to the app maintainer.) If the web server itself is failing, then I'd see the problem across multiple app URLs and know the problem was with the web server itself. However, if per URL is easier, that would work just as well. For the time being, I guess I'll just have to make a special case external script to test using curl (and try to convince the developers that they should always have a 'test' URL that doesn't take large amounts of processing just to verify things are working...) -Andrew
▸
(I could see use cases for all three, although I think the middle option might be the most manageable.) For now, however, this is something that would need to be handled via an external script. Regards, -jc
[...snip...]
list John Thurston
▸
On 4/16/2015 5:00 PM, Andrew Rakowski wrote: ...
For the time being, I guess I'll just have to make a special case external script to test using curl (and try to convince the developers that they should always have a 'test' URL that doesn't take large amounts of processing just to verify things are working...)
I would expect that this is something the business owner of the
application would want to have included in their design requirements.
"A page which, when requested, exercises the web server, application
engine, authentication services, and basic database functions."
--
Do things because you should, not just because you can.
John Thurston XXX-XXX-XXXX
user-ce4d79d99bab@xymon.invalid
Enterprise Technology Services
Department of Administration
State of Alaska
list Japheth Cleaver
▸
On Thu, April 16, 2015 6:00 pm, Andrew Rakowski wrote:
Hi jc, On Mon, 13 Apr 2015, J.C. Cleaver wrote:Thinking about it, this might be something that's not too difficult to add into xymonnet either. A HEAD is basically just a GET from the poller's perspective, and any evaluation on body contents (or lack thereof) is semantically identical to the configuration options available now. An open question is what the best way to indicate that this is a "HEAD-only" test would be. It could be something applied to the entire host line (like browser=), made into a separate column (similar to httpstatus= or post=, except without the... POST), or another per-URL protocol specifier similar to how the TLS/SSL configuration is handled.Unless there's significant savings in overhead one way over the other, I see no problem with doing it whichever way is simplest to implement. From my point of view, if I'm testing multiple URLs on the same server, it would typically be for different applications, and I'd probably make a separate host line for each app (so I could have different notifications to the app maintainer.) If the web server itself is failing, then I'd see the problem across multiple app URLs and know the problem was with the web server itself. However, if per URL is easier, that would work just as well. For the time being, I guess I'll just have to make a special case external script to test using curl (and try to convince the developers that they should always have a 'test' URL that doesn't take large amounts of processing just to verify things are working...) -Andrew
Hi Andrew, HEAD support using the syntax "httphead[=COL];http://www.example.com/blah/"; has been added in, and should be available in the next release. HTH, -jc