Xymon and Lighttpd 500 Internal Error
list Guy
Hello, The Xymon index page is loading; however, when I attempt to drill down into an individual service check (invoking svcstatus.sh), lighttpd returns a 500 internal server error. OS: Devuan Ascii 2.0 Kernel: Linux pihole 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux Lighttpd: Version: 1.4.45-1 Xymon: Version: 4.3.28-2 Xymon and lighttpd were both installed from the official Devuan repos using apt-get. Here's my lighttpd config: https://clbin.com/F2LzF Here's an strace: https://clbin.com/EuesM Thanks, Guy
list Japheth Cleaver
▸
On 8/29/2019 11:15 AM, Guy wrote:
Hello, The Xymon index page is loading; however, when I attempt to drill down into an individual service check (invoking svcstatus.sh), lighttpd returns a 500 internal server error. OS: Devuan Ascii 2.0 Kernel: Linux pihole 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux Lighttpd: Version: 1.4.45-1 Xymon: Version: 4.3.28-2 Xymon and lighttpd were both installed from the official Devuan repos using apt-get. Here's my lighttpd config: https://clbin.com/F2LzF Here's an strace: https://clbin.com/EuesM Thanks, Guy
Hi, Can you try running svcstatus.sh by hand as the web user, with the appropriate QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http", and see what happens? From the stack trace it seems to get as far as the bash fork, but there's not much on the return to go on. -jc
list Guy
I'm not entirely sure how to pass argument to svcstatus.sh on the command line. The web server runs as www-data. Here's what I tried. www-data at pihole:~$ /usr/lib/xymon/cgi-bin/svcstatus.sh pihole.xx.lan http Status: 403 Refresh: 30 Content-type: text/html <html><head><title>Invalid request</title></head> <body>Invalid request</body></html> www-data at pihole:~$ id uid=33(www-data) gid=33(www-data) groups=33(www-data),107(xymon) www-data at pihole:~$
▸
On Thu, Aug 29, 2019 at 2:28 PM Japheth Cleaver <user-87556346d4af@xymon.invalid> wrote:On 8/29/2019 11:15 AM, Guy wrote:Hello, The Xymon index page is loading; however, when I attempt to drill down into an individual service check (invoking svcstatus.sh), lighttpd returns a 500 internal server error. OS: Devuan Ascii 2.0 Kernel: Linux pihole 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux Lighttpd: Version: 1.4.45-1 Xymon: Version: 4.3.28-2 Xymon and lighttpd were both installed from the official Devuan repos using apt-get. Here's my lighttpd config: https://clbin.com/F2LzF Here's an strace: https://clbin.com/EuesM Thanks, GuyHi, Can you try running svcstatus.sh by hand as the web user, with the appropriate QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http", and see what happens? From the stack trace it seems to get as far as the bash fork, but there's not much on the return to go on. -jc
list Guy
Yea, I'm lost. www-data at pihole:~$ export QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http" www-data at pihole:~$ echo $QUERY_STRING HOST=pihole.xx.lan&SERVICE=http www-data at pihole:~$ /usr/lib/xymon/cgi-bin/svcstatus.sh $QUERY_STRING
▸
Status: 403
Refresh: 30
Content-type: text/html
<html><head><title>Invalid request</title></head>
<body>Invalid request</body></html>
www-data at pihole:~$
I couldn't find anything in the man pages or on-line documentation
about invoking svcstatus.sh on the command line.
▸
On Thu, Aug 29, 2019 at 2:28 PM Japheth Cleaver <user-87556346d4af@xymon.invalid> wrote:On 8/29/2019 11:15 AM, Guy wrote:Hello, The Xymon index page is loading; however, when I attempt to drill down into an individual service check (invoking svcstatus.sh), lighttpd returns a 500 internal server error. OS: Devuan Ascii 2.0 Kernel: Linux pihole 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux Lighttpd: Version: 1.4.45-1 Xymon: Version: 4.3.28-2 Xymon and lighttpd were both installed from the official Devuan repos using apt-get. Here's my lighttpd config: https://clbin.com/F2LzF Here's an strace: https://clbin.com/EuesM Thanks, GuyHi, Can you try running svcstatus.sh by hand as the web user, with the appropriate QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http", and see what happens? From the stack trace it seems to get as far as the bash fork, but there's not much on the return to go on. -jc
list Japheth Cleaver
Sorry, there was more needed to simulate the CGI call. This should return something for you: su -s /bin/sh www-data QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http" REQUEST_METHOD=GET SCRIPT_NAME=svcstatus /usr/share/xymon/cgi-bin/svcstatus.sh -jc
▸
On 8/29/2019 11:57 AM, Guy wrote:Yea, I'm lost. www-data at pihole:~$ export QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http" www-data at pihole:~$ echo $QUERY_STRING HOST=pihole.xx.lan&SERVICE=http www-data at pihole:~$ /usr/lib/xymon/cgi-bin/svcstatus.sh $QUERY_STRING Status: 403 Refresh: 30 Content-type: text/html <html><head><title>Invalid request</title></head> <body>Invalid request</body></html> www-data at pihole:~$ I couldn't find anything in the man pages or on-line documentation about invoking svcstatus.sh on the command line. On Thu, Aug 29, 2019 at 2:28 PM Japheth Cleaver <user-87556346d4af@xymon.invalid> wrote:On 8/29/2019 11:15 AM, Guy wrote:Hello, The Xymon index page is loading; however, when I attempt to drill down into an individual service check (invoking svcstatus.sh), lighttpd returns a 500 internal server error. OS: Devuan Ascii 2.0 Kernel: Linux pihole 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux Lighttpd: Version: 1.4.45-1 Xymon: Version: 4.3.28-2 Xymon and lighttpd were both installed from the official Devuan repos using apt-get. Here's my lighttpd config: https://clbin.com/F2LzF Here's an strace: https://clbin.com/EuesM Thanks, GuyHi, Can you try running svcstatus.sh by hand as the web user, with the appropriate QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http", and see what happens? From the stack trace it seems to get as far as the bash fork, but there's not much on the return to go on. -jc
list Guy
▸
On Thu, Aug 29, 2019 at 4:16 PM Japheth Cleaver <user-87556346d4af@xymon.invalid> wrote:
Sorry, there was more needed to simulate the CGI call. This should return something for you: su -s /bin/sh www-data QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http" REQUEST_METHOD=GET SCRIPT_NAME=svcstatus /usr/share/xymon/cgi-bin/svcstatus.sh
pihole:~# su -s /bin/sh www-data \h:\w$ QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http" REQUEST_METHOD=GET SCRIPT_NAME=svcstatus /usr/lib/xymon/cgi-bin/svcstatus.sh Refresh: 60 Content-Security-Policy: script-src 'self'; connect-src 'self'; form-action 'self'; sandbox allow-forms allow-same-origin; X-Content-Security-Policy: script-src 'self'; connect-src 'self'; form-action 'self'; sandbox allow-forms allow-same-origin; X-Webkit-CSP: script-src 'self'; connect-src 'self'; form-action 'self'; sandbox allow-forms allow-same-origin; 2019-08-29 16:25:44.991272 No such host Status: 403 Refresh: 30 Content-type: text/html <html><head><title>Invalid request</title></head> <body>No such host</body></html> \h:\w$ The host (pihole.xx.lan) is defined in /etc/xymon/hosts.cfg
list Japheth Cleaver
▸
On 8/29/2019 1:30 PM, Guy wrote:
On Thu, Aug 29, 2019 at 4:16 PM Japheth Cleaver <user-87556346d4af@xymon.invalid> wrote:Sorry, there was more needed to simulate the CGI call. This should return something for you: su -s /bin/sh www-data QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http" REQUEST_METHOD=GET SCRIPT_NAME=svcstatus /usr/share/xymon/cgi-bin/svcstatus.shpihole:~# su -s /bin/sh www-data \h:\w$ QUERY_STRING="HOST=pihole.xx.lan&SERVICE=http" REQUEST_METHOD=GET SCRIPT_NAME=svcstatus /usr/lib/xymon/cgi-bin/svcstatus.sh Refresh: 60 Content-Security-Policy: script-src 'self'; connect-src 'self'; form-action 'self'; sandbox allow-forms allow-same-origin; X-Content-Security-Policy: script-src 'self'; connect-src 'self'; form-action 'self'; sandbox allow-forms allow-same-origin; X-Webkit-CSP: script-src 'self'; connect-src 'self'; form-action 'self'; sandbox allow-forms allow-same-origin; 2019-08-29 16:25:44.991272 No such host Status: 403 Refresh: 30 Content-type: text/html <html><head><title>Invalid request</title></head> <body>No such host</body></html> \h:\w$ The host (pihole.xx.lan) is defined in /etc/xymon/hosts.cfg
Hmm. Is the 'xx' here a literal or a replacement domain? If the latter, does the original domain have a "-" in it? There was a recent bug regarding display issues for this and the fix might not have made it into the package. More broadly, are you seeing similar behavior for all svcstatus pages you're hitting? And do other CGI pages come up OK? -jc
list Guy
▸
Hmm. Is the 'xx' here a literal or a replacement domain? If the latter, does the original domain have a "-" in it? There was a recent bug regarding display issues for this and the fix might not have made it into the package.
Literal, I lack creativity when it comes to naming schema, (xx.lan is used for internal hosts). There are no hyphens.
▸
More broadly, are you seeing similar behavior for all svcstatus pages you're hitting? And do other CGI pages come up OK? -jc
Until you mentioned it I hadn't tested the other CGI scripts. Turns out they all return, "500 Internal Server Error." SELinux isn't enabled and there aren't any errors in the lighttpd/error.log. Thank you for the suggestions. I've opened a thread on the lighttpd support forum but have yet to receive any feedback. -Guy
list Guy
▸
On Thu, Aug 29, 2019 at 2:15 PM Guy <user-6f21ad656c65@xymon.invalid> wrote:
Hello, The Xymon index page is loading; however, when I attempt to drill down into an individual service check (invoking svcstatus.sh), lighttpd returns a 500 internal server error.
Someone from the Lighttpd support forums responded with the following: "It appears that the CGI is not returning any output, which is an invalid CGI/1.1 response. lighttpd is receiving POLLHUP on the pipe and cleaning up the CGI. Perhaps you forgot to flush output in the script?" Here's the thread: https://redmine.lighttpd.net/boards/2/topics/8703?r=8705#message-8705 If I create /var/www/html/hello-world.sh and run it, it works as expected.
▸
OS: Devuan Ascii 2.0 Kernel: Linux pihole 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux Lighttpd: Version: 1.4.45-1 Xymon: Version: 4.3.28-2 Xymon and lighttpd were both installed from the official Devuan repos using apt-get. Here's my lighttpd config: https://clbin.com/F2LzF Here's an strace: https://clbin.com/EuesM Thanks, Guy
list Brian Scott
▸
On 4/9/19 9:15 pm, Guy wrote:
On Thu, Aug 29, 2019 at 2:15 PM Guy <user-6f21ad656c65@xymon.invalid> wrote:Hello, The Xymon index page is loading; however, when I attempt to drill down into an individual service check (invoking svcstatus.sh), lighttpd returns a 500 internal server error.Someone from the Lighttpd support forums responded with the following: "It appears that the CGI is not returning any output, which is an invalid CGI/1.1 response. lighttpd is receiving POLLHUP on the pipe and cleaning up the CGI. Perhaps you forgot to flush output in the script?" Here's the thread: https://redmine.lighttpd.net/boards/2/topics/8703?r=8705#message-8705 If I create /var/www/html/hello-world.sh and run it, it works as expected.OS: Devuan Ascii 2.0 Kernel: Linux pihole 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux Lighttpd: Version: 1.4.45-1 Xymon: Version: 4.3.28-2 Xymon and lighttpd were both installed from the official Devuan repos using apt-get. Here's my lighttpd config: https://clbin.com/F2LzF Here's an strace: https://clbin.com/EuesM Thanks, Guy
Hi,
The .sh files aren't actually shell scripts any more, they are compiled
programs. Try getting rid of the cgi.assign clause for .sh completely.
I've just had a look at an old lighttpd config that I used for xymon and
found this in my vhosts.d directory:
server.modules? += ( "mod_cgi",
??????? "mod_auth",
??????? "mod_alias"
)
alias.url += (
??????? "/xymon-cgi/" => "/usr/local/xymon/cgi-bin/",
??????? "/xymon-seccgi/" => "/usr/local/xymon/cgi-secure/",
??????? "/xymon/" => "/usr/local/xymon/server/www/",
??????? "/xymon" => "/usr/local/xymon/server/www/"
)
$HTTP["url"] =~ "^/xymon-cgi/|^/xymon-seccgi/" {
??????? dir-listing.activate = "disable"
??????? cgi.assign = ( "" => "" )
}
auth.backend = "htpasswd"
auth.backend.htpasswd.userfile = "/usr/local/xymon/server/etc/xymonpasswd"
auth.require = ( "/xymon-seccgi/" => (
??????? "method"? => "basic",
??????? "realm"?? => "Xymon Administration",
??????? "require" => "user=admin"
??????? )
)There is no mention of '.sh' mapping anywhere. While I'm a little hazy about all of this, I suspect that the line 'cgi.assign = ( "" => "" )' for the cgi directories is where the magic happens. Cheers, Brian Scott
list Guy
Brian, Thank you, this resolved the issue. I did a fresh install of lighttpd, and after adjusting some file-system permissions, your config works great. Thanks again, Guy
▸
On Wed, Sep 4, 2019 at 10:23 PM Brian Scott <user-b09d5329b577@xymon.invalid> wrote:On 4/9/19 9:15 pm, Guy wrote:On Thu, Aug 29, 2019 at 2:15 PM Guy <user-6f21ad656c65@xymon.invalid> wrote:Hello, The Xymon index page is loading; however, when I attempt to drill down into an individual service check (invoking svcstatus.sh), lighttpd returns a 500 internal server error.Someone from the Lighttpd support forums responded with the following: "It appears that the CGI is not returning any output, which is an invalid CGI/1.1 response. lighttpd is receiving POLLHUP on the pipe and cleaning up the CGI. Perhaps you forgot to flush output in the script?" Here's the thread: https://redmine.lighttpd.net/boards/2/topics/8703?r=8705#message-8705 If I create /var/www/html/hello-world.sh and run it, it works as expected.OS: Devuan Ascii 2.0 Kernel: Linux pihole 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux Lighttpd: Version: 1.4.45-1 Xymon: Version: 4.3.28-2 Xymon and lighttpd were both installed from the official Devuan repos using apt-get. Here's my lighttpd config: https://clbin.com/F2LzF Here's an strace: https://clbin.com/EuesM Thanks, GuyHi, The .sh files aren't actually shell scripts any more, they are compiled programs. Try getting rid of the cgi.assign clause for .sh completely. I've just had a look at an old lighttpd config that I used for xymon and found this in my vhosts.d directory:server.modules += ( "mod_cgi", "mod_auth", "mod_alias" ) alias.url += ( "/xymon-cgi/" => "/usr/local/xymon/cgi-bin/", "/xymon-seccgi/" => "/usr/local/xymon/cgi-secure/", "/xymon/" => "/usr/local/xymon/server/www/", "/xymon" => "/usr/local/xymon/server/www/" ) $HTTP["url"] =~ "^/xymon-cgi/|^/xymon-seccgi/" { dir-listing.activate = "disable" cgi.assign = ( "" => "" ) } auth.backend = "htpasswd" auth.backend.htpasswd.userfile = "/usr/local/xymon/server/etc/xymonpasswd" auth.require = ( "/xymon-seccgi/" => ( "method" => "basic", "realm" => "Xymon Administration", "require" => "user=admin" ) )There is no mention of '.sh' mapping anywhere. While I'm a little hazy about all of this, I suspect that the line 'cgi.assign = ( "" => "" )' for the cgi directories is where the magic happens. Cheers, Brian Scott