error running report.sh
list Jason Brockdorf
I'm still working on setting up Xymon (from the terabithia RPMs) and my manager tells me today that there's an error message when trying to run reports, like the availability report, snapshot report, etc. I'm getting an error message: Cannot create output directory When examining the logs for more details I don't get a whole lot to go on. I checked with both httpd and xymon logs and I didn't see anything relevant to this problem. I'm really trying to help myself and not make my problems everyone else's problems, and I'd really like to contribute to the community in some way, so here's what I've done so far to try to fix it: 1. going to the /var/cache/xymon/ directory and creating a rep directory2. giving rwxrwxrwx permissions to that directory I then managed to download the srpms from terabithia, find the offending file and examine the source to see what happens. The XYMONREPDIR environment variable is used in creating the directory where the report will go, but I don't know: 1. How to check which process is running the file that's encountering the problem (httpd or some xymon process)2. How to check for the presence or value of this variable in the environment that process is using So I decided to see if I could modify the source and make xymon give me a little more information. The resultant patch file is attached, though be forewarned, my C experience is limited to a primer class I took as a freshman in highschool and on top of not having that much knowledge in the first place, I'm very rusty. From what I understand in the source there's not much error checking or information provided if it fails. I do see: envcheck(reqenv); But this by itself doesn't prevent continued execution if say my $XYMONREPDIR is not found. Well, I added a line to report which directory xymon is attempting to create, and another which should report the error message if the attempt was unsuccessful. I did manage to create the attached patch file, but I was unable to recompile using the srpm *sniffle* <-- I'm crying from frustration at this point So, I downloaded the source from sourceforge, replaced the report.c file with the one I made, compiled using that, copied the report.sh file to a working xymon server to try to test it... and I get internal server error :( I then went back to test it on xymon compiled from sourceforge source, and I get an error that I don't have permission to access that directory. *rips hair out* Thus, it would seem that I am unable to reproduce the problem with "stock" xymon, and I'm apparently not bright enough to fix it with terabithia srpms either. At this point, I feel like I've put in enough effort to warrant asking for help. Anyone, please help?
Attachments (1)
list Japheth Cleaver
▸
On Tue, March 15, 2016 6:09 pm, Jason Brockdorf wrote:
I'm still working on setting up Xymon (from the terabithia RPMs) and my manager tells me today that there's an error message when trying to run reports, like the availability report, snapshot report, etc.
I'm getting an error message:Â Cannot create output directory
When examining the logs for more details I don't get a whole lot to go on.
 I checked with both httpd and xymon logs and I didn't see anything
▸
relevant to this problem.
I'm really trying to help myself and not make my problems everyone else's
problems, and I'd really like to contribute to the community in some way,
so here's what I've done so far to try to fix it:
1. going to the /var/cache/xymon/ directory and creating a rep directory2.
giving rwxrwxrwx permissions to that directory
I then managed to download the srpms from terabithia, find the offending
file and examine the source to see what happens.
The XYMONREPDIR environment variable is used in creating the directory
▸
where the report will go, but I don't know:
1. How to check which process is running the file that's encountering the
problem (httpd or some xymon process)2. How to check for the presence or
value of this variable in the environment that process is using
So I decided to see if I could modify the source and make xymon give me a
little more information. Â The resultant patch file is attached, though be
▸
forewarned, my C experience is limited to a primer class I took as a
freshman in highschool and on top of not having that much knowledge in the
first place, I'm very rusty.
From what I understand in the source there's not much error checking orinformation provided if it fails. Â I do see:
▸
envcheck(reqenv); But this by itself doesn't prevent continued execution if say my $XYMONREPDIR is not found. Well, I added a line to report which directory xymon is attempting to create, and another which should report the error message if the attempt
was unsuccessful. Â I did manage to create the attached patch file, but I
▸
was unable to recompile using the srpm *sniffle* <-- I'm crying from
frustration at this point
So, I downloaded the source from sourceforge, replaced the report.c file
with the one I made, compiled using that, copied the report.sh file to a
working xymon server to try to test it... and I get internal server error
:(
▸
I then went back to test it on xymon compiled from sourceforge source, and
I get an error that I don't have permission to access that directory.
*rips hair out*
Thus, it would seem that I am unable to reproduce the problem with "stock"
xymon, and I'm apparently not bright enough to fix it with terabithia
srpms either. Â At this point, I feel like I've put in enough effort to
warrant asking for help. Â Anyone, please help?
Hi,
This appears, from my testing, to be an SELinux issue -- although I'm
still trying to debug precisely what is happening here on the EL7 side.
A quick workaround would be to set SELinux to permissive ('setenforce 0')
on the main server, which should let snapshoting/reporting work. It's
definitely not a normal unix permissions/ownership issue.
Can you check /var/log/audit/audit.log on your system and verify that
snapshot.cgi is being denied? Also, can you post the output of `semodule
-l | grep xymon`? If 'xymon' is not listed, can you try running:
/usr/sbin/semodule -s targeted -i /usr/share/selinux/targeted/xymon.pp
and post the result?
Regards,
-jc
list Jason Brockdorf
J.C. I really want to buy you beer. I disabled SELinux and now it's working.
[root at hhsiamxymon ~]# grep snapshot.cgi /var/log/audit/audit.logtype=AVC msg=audit(1458047063.222:1954): avc: denied { search } for pid=17895 comm="snapshot.cgi" name="xymon" dev="sda3" ino=2894713 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:object_r:httpd_cache_t:s0 tclass=dir
type=SYSCALL msg=audit(1458047063.222:1954): arch=c000003e syscall=83 success=no exit=-13 a0=7fff496ed810 a1=1ed a2=7fff496ed836 a3=7fff496ec3d0 items=0 ppid=17723 pid=17895 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="snapshot.cgi" exe="/usr/libexec/xymon/snapshot.cgi" subj=system_u:system_r:httpd_sys_script_t:s0 key=(null)
[root at hhsiamxymon ~]# semodule -l | grep xymon
xymon-client 4.3.26.1.el7
▸
From: J.C. Cleaver <user-87556346d4af@xymon.invalid>
To: Jason Brockdorf <user-fa0be9c5d46d@xymon.invalid>
Cc: "xymon at xymon.com" <xymon at xymon.com>
Sent: Tuesday, March 15, 2016 10:38 PM
Subject: Re: error running report.sh
On Tue, March 15, 2016 6:09 pm, Jason Brockdorf wrote:I'm still working on setting up Xymon (from the terabithia RPMs) and my manager tells me today that there's an error message when trying to run reports, like the availability report, snapshot report, etc. I'm getting an error message: Cannot create output directory When examining the logs for more details I don't get a whole lot to go on.  I checked with both httpd and xymon logs and I didn't see anything relevant to this problem. I'm really trying to help myself and not make my problems everyone else's problems, and I'd really like to contribute to the community in some way, so here's what I've done so far to try to fix it: 1. going to the /var/cache/xymon/ directory and creating a rep directory2. giving rwxrwxrwx permissions to that directory I then managed to download the srpms from terabithia, find the offending file and examine the source to see what happens. The XYMONREPDIR environment variable is used in creating the directory where the report will go, but I don't know: 1. How to check which process is running the file that's encountering the problem (httpd or some xymon process)2. How to check for the presence or value of this variable in the environment that process is using So I decided to see if I could modify the source and make xymon give me a little more information.  The resultant patch file is attached, though be forewarned, my C experience is limited to a primer class I took as a freshman in highschool and on top of not having that much knowledge in the first place, I'm very rusty. From what I understand in the source there's not much error checking orinformation provided if it fails.  I do see:envcheck(reqenv); But this by itself doesn't prevent continued execution if say my $XYMONREPDIR is not found. Well, I added a line to report which directory xymon is attempting to create, and another which should report the error message if the attempt was unsuccessful.  I did manage to create the attached patch file, but I was unable to recompile using the srpm *sniffle* <-- I'm crying from frustration at this point So, I downloaded the source from sourceforge, replaced the report.c file with the one I made, compiled using that, copied the report.sh file to a working xymon server to try to test it... and I get internal server error
:(
▸
I then went back to test it on xymon compiled from sourceforge source, and
I get an error that I don't have permission to access that directory.
*rips hair out*
Thus, it would seem that I am unable to reproduce the problem with "stock"
xymon, and I'm apparently not bright enough to fix it with terabithia
srpms either. Â At this point, I feel like I've put in enough effort to
warrant asking for help. Â Anyone, please help?
Hi,
This appears, from my testing, to be an SELinux issue -- although I'm
still trying to debug precisely what is happening here on the EL7 side.
A quick workaround would be to set SELinux to permissive ('setenforce 0')
on the main server, which should let snapshoting/reporting work. It's
definitely not a normal unix permissions/ownership issue.
Can you check /var/log/audit/audit.log on your system and verify that
snapshot.cgi is being denied? Also, can you post the output of `semodule
-l | grep xymon`? If 'xymon' is not listed, can you try running:
/usr/sbin/semodule -s targeted -i /usr/share/selinux/targeted/xymon.pp
and post the result?
Regards,
-jc
list Jeremy Laidman
Jason On Wed, Mar 16, 2016 at 12:09 PM Jason Brockdorf via Xymon <xymon at xymon.com>
▸
wrote:
I'm getting an error message: Cannot create output directory
here's what I've done so far to try to fix it:
1. going to the /var/cache/xymon/ directory and creating a rep directory
You haven't set what your XYMONREPDIR is set to. On my system it's /usr/lib/xymon/server/www/rep/. On yours, it's probably ~xymon/server/www/rep/. 2. giving rwxrwxrwx permissions to that directory
That's a common solution to a permissions problem, but is not recommended. I'm hoping you are planning on removing the permissions once you get the problem sorted. When making world-writeable directories, it's best to set the sticky bit to avoid some potential security risks. Do this with "chmod +t /path/to/dir". The permissions should look the same as /tmp: $ ls -ld /tmp drwxrwxrwt 30 root root 344064 2016-03-16 14:33 /tmp But when you think about it, there are only two possible users. One is the xymon user and the other is the apache user. On my system, there's a www group, to which the apache user belongs. My "rep" directory is writeable by both the xymon user and the apache group, so that it looks like this: $ ls -ld ~xymon/server/www/rep drwxrwxr-x 3 xymon www 4096 2016-02-23 14:17 /usr/lib/xymon/server/www/rep
▸
The XYMONREPDIR environment variable is used in creating the directorywhere the report will go, but I don't know: 1. How to check which process is running the file that's encountering the
problem (httpd or some xymon process)
It's the web server, so "apache" or "wwwrun" or whatever your web server is running as.
2. How to check for the presence or value of this variable in the environment that process is using
The relevant variables are set by CGI shell scripts. These scripts import variables from cgioptions.cfg, which in turn pulls in variables from xymonserver.cfg, and in particular, XYMONREPDIR. You can probably simulate the environment that the CGI process runs as, fairly accurately, by doing: $ sudo -u www ~xymon/server/bin/xymoncmd --env=`echo ~xymon/server/etc/cgioptions.cfg` --env=`echo ~xymon/server/etc/xymonserver.cfg` $ echo $XYMONREPDIR /usr/lib/xymon/server/www/rep
▸
Thus, it would seem that I am unable to reproduce the problem with "stock"xymon, and I'm apparently not bright enough to fix it with terabithia srpms either. At this point, I feel like I've put in enough effort to warrant asking for help. Anyone, please help?
I'm not aware of a fault with the Terabithia (S)RPMs. At this stage I'm not sure you've proven that it's a problem with the package and not a problem with a config file or directory permission. I don't think you mentioned the OS you're using, and so it's possible that the Terabithia packages are not quite in alignment with your OS. I had a look through your patch, and I can't see any reason why it would not work. What I would do is create a copy of the report.sh script, eg report-test.sh and update it to run "strace" against the binary, so it looks like this: exec strace -f /usr/lib/xymon/server/bin/report.cgi $CGI_REP_OPTS $XYMONGENREPOPTS 2>/tmp/report.stderr Then run a report in your browser by adjusting the URL, and check the output file /tmp/report.stderr. This will not only show you where it's trying to create a report file, but the ownership of the file will tell you what user needs to own the rep directory. Cheers Jeremy
list Jason Brockdorf
The patch I submitted before is no bueno. Looks like I forgot an include line and to replace a stub variable name with the actual name. Sorry :\ I think the include(s) needed are: #include <errno.h> And the line that says: + sprintf(msg, "Cannot create output directory:<br>\n%s<br>\n%s", somepath, strerror(errno));should read: + sprintf(msg, "Cannot create output directory:<br>\n%s<br>\n%s", outdir, strerror(errno)); I'll try again tonight after work to download, recompile, and submit a valid patch. I already deleted the test VM I was using.________________________________
▸
From: Jason Brockdorf via Xymon <xymon at xymon.com>
To: "xymon at xymon.com" <xymon at xymon.com>
Sent: Tuesday, March 15, 2016 8:09 PM
Subject: [Xymon] error running report.sh
I'm still working on setting up Xymon (from the terabithia RPMs) and my manager tells me today that there's an error message when trying to run reports, like the availability report, snapshot report, etc.
I'm getting an error message: Cannot create output directory
When examining the logs for more details I don't get a whole lot to go on. I checked with both httpd and xymon logs and I didn't see anything relevant to this problem.
I'm really trying to help myself and not make my problems everyone else's problems, and I'd really like to contribute to the community in some way, so here's what I've done so far to try to fix it:
1. going to the /var/cache/xymon/ directory and creating a rep directory
2. giving rwxrwxrwx permissions to that directory
I then managed to download the srpms from terabithia, find the offending file and examine the source to see what happens.
The XYMONREPDIR environment variable is used in creating the directory where the report will go, but I don't know:
1. How to check which process is running the file that's encountering the problem (httpd or some xymon process)
2. How to check for the presence or value of this variable in the environment that process is using
So I decided to see if I could modify the source and make xymon give me a little more information. The resultant patch file is attached, though be forewarned, my C experience is limited to a primer class I took as a freshman in highschool and on top of not having that much knowledge in the first place, I'm very rusty.
From what I understand in the source there's not much error checking or information provided if it fails. I do see:
envcheck(reqenv);
But this by itself doesn't prevent continued execution if say my $XYMONREPDIR is not found.
Well, I added a line to report which directory xymon is attempting to create, and another which should report the error message if the attempt was unsuccessful. I did manage to create the attached patch file, but I was unable to recompile using the srpm *sniffle* <-- I'm crying from frustration at this point
So, I downloaded the source from sourceforge, replaced the report.c file with the one I made, compiled using that, copied the report.sh file to a working xymon server to try to test it... and I get internal server error :(
I then went back to test it on xymon compiled from sourceforge source, and I get an error that I don't have permission to access that directory. *rips hair out*
Thus, it would seem that I am unable to reproduce the problem with "stock" xymon, and I'm apparently not bright enough to fix it with terabithia srpms either. At this point, I feel like I've put in enough effort to warrant asking for help. Anyone, please help?
list Francois Claire
▸
Le 16/03/2016 04:38, J.C. Cleaver a écrit :
Hi, This appears, from my testing, to be an SELinux issue -- although I'm still trying to debug precisely what is happening here on the EL7 side.
Hi JC,
Here's what I do on my Centos 7 box to keep xymon working with selinux enabled:
semanage fcontext -a -t httpd_sys_rw_content_t "/var/cache/xymon(/.*)?"
restorecon -Rv /var/cache/xymon
semanage fcontext -a -t httpd_sys_script_exec_t "/usr/libexec/xymon/showgraph.cgi"
restorecon -Rv /usr/share/xymon/cgi-bin/showgraph.sh
semanage fcontext -a -t httpd_sys_rw_content_t "/etc/xymon(/.*)?"
restorecon -Rv /etc/xymon
I hope this helps...
Cheers,
Francois.
list Japheth Cleaver
No worries :) This was an ugly one, but I think I've tracked down the issue. At some point in a Fedora branch prior to the fork for EL7 'unconfined_execmem_t' was removed. I believe I've fixed the SELinux policy for this one. Would you be able to try out the 4.3.26-3 set in http://terabithia.org/rpms/xymon/testing/el7/x86_64/ ? Please be sure SELinux is enabled (doesn't have to be Enforcing, Permissive is fine... it just can't be disabled) prior to the update, and then see if you still see the same errors with snapshoting when setenforce is set to 1. If so, can you see if both modules have been installed via 'semodule -l | grep xymon'? Regards, -jc On Tue, March 15, 2016 9:18 pm, Jason Brockdorf wrote:
J.C. I really want to buy you beer. I disabled SELinux and now it's
working.
[root at hhsiamxymon ~]# grep snapshot.cgi /var/log/audit/audit.logtype=AVC
msg=audit(1458047063.222:1954): avc: denied { search } for pid=17895
▸
comm="snapshot.cgi" name="xymon" dev="sda3" ino=2894713
scontext=system_u:system_r:httpd_sys_script_t:s0
tcontext=system_u:object_r:httpd_cache_t:s0 tclass=dir
type=SYSCALL msg=audit(1458047063.222:1954): arch=c000003e syscall=83
success=no exit=-13 a0=7fff496ed810 a1=1ed a2=7fff496ed836 a3=7fff496ec3d0
items=0 ppid=17723 pid=17895 auid=4294967295 uid=48 gid=48 euid=48 suid=48
fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295
comm="snapshot.cgi" exe="/usr/libexec/xymon/snapshot.cgi"
subj=system_u:system_r:httpd_sys_script_t:s0 key=(null)
[root at hhsiamxymon ~]# semodule -l | grep xymon
xymon-client   4.3.26.1.el7
▸
From: J.C. Cleaver <user-87556346d4af@xymon.invalid> To: Jason Brockdorf <user-fa0be9c5d46d@xymon.invalid> Cc: "xymon at xymon.com" <xymon at xymon.com> Sent: Tuesday, March 15, 2016 10:38 PM Subject: Re: error running report.sh On Tue, March 15, 2016 6:09 pm, Jason Brockdorf wrote:I'm still working on setting up Xymon (from the terabithia RPMs) and my manager tells me today that there's an error message when trying to run reports, like the availability report, snapshot report, etc.
I'm getting an error message:ÃÂ Cannot create output directory
▸
When examining the logs for more details I don't get a whole lot to go on.
ÃÂ I checked with both httpd and xymon logs and I didn't see anything
▸
relevant to this problem. I'm really trying to help myself and not make my problems everyone else's problems, and I'd really like to contribute to the community in some way, so here's what I've done so far to try to fix it: 1. going to the /var/cache/xymon/ directory and creating a rep directory2. giving rwxrwxrwx permissions to that directory I then managed to download the srpms from terabithia, find the offending file and examine the source to see what happens.
TheÃÂ XYMONREPDIR environment variable is used in creating the
▸
directory where the report will go, but I don't know: 1. How to check which process is running the file that's encountering the problem (httpd or some xymon process)2. How to check for the presence or value of this variable in the environment that process is using So I decided to see if I could modify the source and make xymon give me a
little more information. ÃÂ The resultant patch file is attached,
▸
though be forewarned, my C experience is limited to a primer class I took as a freshman in highschool and on top of not having that much knowledge in the first place, I'm very rusty. From what I understand in the source there's not much error checking or
information provided if it fails. ÃÂ I do see:
▸
envcheck(reqenv); But this by itself doesn't prevent continued execution if say my $XYMONREPDIR is not found. Well, I added a line to report which directory xymon is attempting to create, and another which should report the error message if the attempt
was unsuccessful. ÃÂ I did manage to create the attached patch file,
▸
but I was unable to recompile using the srpm *sniffle* <-- I'm crying from frustration at this point So, I downloaded the source from sourceforge, replaced the report.c file with the one I made, compiled using that, copied the report.sh file to a working xymon server to try to test it... and I get internal server error
:(
▸
I then went back to test it on xymon compiled from sourceforge source, and I get an error that I don't have permission to access that directory. *rips hair out* Thus, it would seem that I am unable to reproduce the problem with "stock" xymon, and I'm apparently not bright enough to fix it with terabithia
srpms either. ÃÂ At this point, I feel like I've put in enough effort to warrant asking for help. ÃÂ Anyone, please help?
▸
Hi,
This appears, from my testing, to be an SELinux issue -- although I'm
still trying to debug precisely what is happening here on the EL7 side.
A quick workaround would be to set SELinux to permissive ('setenforce 0')
on the main server, which should let snapshoting/reporting work. It's
definitely not a normal unix permissions/ownership issue.
Can you check /var/log/audit/audit.log on your system and verify that
snapshot.cgi is being denied? Also, can you post the output of `semodule
-l | grep xymon`? If 'xymon' is not listed, can you try running:
/usr/sbin/semodule -s targeted -i /usr/share/selinux/targeted/xymon.pp
and post the result?
Regards,
-jc
list Japheth Cleaver
On Wed, March 16, 2016 5:51 am, Francois Claire wrote:
Le 16/03/2016 04:38, J.C. Cleaver a écrit :
▸
Hi, This appears, from my testing, to be an SELinux issue -- although I'm still trying to debug precisely what is happening here on the EL7 side.Hi JC, Here's what I do on my Centos 7 box to keep xymon working with selinux enabled: semanage fcontext -a -t httpd_sys_rw_content_t "/var/cache/xymon(/.*)?" restorecon -Rv /var/cache/xymon semanage fcontext -a -t httpd_sys_script_exec_t "/usr/libexec/xymon/showgraph.cgi" restorecon -Rv /usr/share/xymon/cgi-bin/showgraph.sh semanage fcontext -a -t httpd_sys_rw_content_t "/etc/xymon(/.*)?" restorecon -Rv /etc/xymon
Thanks, This is roughly what happens inside the xymon RPM on install (although the package is using httpd_cache_t instead of httpd_sys_rw_content), but the bigger problem here I believe was that the xymon policy module wasn't being loaded properly, alas. If you're using the RPMs, in theory an upgrade to 4.3.26-3 followed by a complete restorecon (/sbin/restorecon -R /usr/libexec/xymon/cgiwrap /usr/share/xymon/cgi-* /var/cache/xymon /var/run/xymon /var/lib/xymon /var/lib/xymon/configs /var/lib/xymon/tmp) should let you use it without any further changes. If you might also be able to test that on a side box, I'd appreciate it. SELinux policy sync across releases, let alone distributions, is not particularly unfrustrating... Regards, -jc