SSL stuck on purple
list Jaime Kikpole
list Jeremy Laidman
▸
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.Any ideas on what I could check?
list Jaime Kikpole
Jaime Kikpole Director of Technology |
▸
Hi JaimeOn Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.Any ideas on what I could check?First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>Good luck with finding the problem.CheersJeremy
list Josh Luthman
▸
Thanks for the ideas. That gave me more to try and I made some progress, but not much.I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:Hi JaimeOn Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.Any ideas on what I could check?First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>Good luck with finding the problem.Cheers
list Jaime Kikpole
▸
Jaime Kikpole Director of Technology |
▸
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.Are you certain the name of the host matches? Are there any ghosts?Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:Thanks for the ideas. That gave me more to try and I made some progress, but not much.I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:Hi JaimeOn Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.Any ideas on what I could check?First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>Good luck with finding the problem.Cheers
list Josh Luthman
▸
From hosts.cfg, with redactions between < and >:<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://<hostname>.ichabodcrane.org noconnThe ghosts report shows no clients.The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <user-4c45a83f15cb@xymon.invalid> wrote:The IP in the first column is used in the event the name can't be resolved (or # testip is appended).The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.Are you certain the name of the host matches? Are there any ghosts?Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:Thanks for the ideas. That gave me more to try and I made some progress, but not much.I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:Hi JaimeOn Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.Any ideas on what I could check?First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>Good luck with finding the problem.Cheers
list Jeremy Laidman
To clear a faulty state, you'd need to stop the xymond process (probably by stopping all of Xymon) and then edit/delete the state file, and then start the xymond process back up again. Editing the host's entry in the file would be less disruptive than deleting the whole file, but the fields aren't well documented so the simplest thing would be to find and delete the line with the troublesome hostname.
▸
Why is the http test red?On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> wrote:From hosts.cfg, with redactions between < and >:<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://<hostname>.ichabodcrane.org noconnThe ghosts report shows no clients.The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <user-4c45a83f15cb@xymon.invalid> wrote:The IP in the first column is used in the event the name can't be resolved (or # testip is appended).The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.Are you certain the name of the host matches? Are there any ghosts?Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:Thanks for the ideas. That gave me more to try and I made some progress, but not much.I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:Hi JaimeOn Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.Any ideas on what I could check?First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>Good luck with finding the problem.Cheers
list Henrik Størner
Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
Henrik Juul Størner
Senior IT Arkitekt - IT-sikkerhed
Fra: Josh Luthman <user-4c45a83f15cb@xymon.invalid>
Sendt: 20. marts 2025 21:56
Til: Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
Cc: Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
Emne: [Xymon] Re: SSL stuck on purple
OBS: Denne e-mail er ikke fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
▸
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://<hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <user-4c45a83f15cb@xymon.invalid> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
xymon@xymon.com
To unsubscribe send an email to xymon-leave@xymon.com
list Nicola Canepa
▸
Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
Henrik Juul Størner
Senior IT Arkitekt - IT-sikkerhedFra: Josh Luthman <user-4c45a83f15cb@xymon.invalid>
Sendt: 20. marts 2025 21:56
Til: Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
Cc: Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
Emne: [Xymon] Re: SSL stuck on purpleOBS: Denne e-mail er ikke fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://<hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <user-4c45a83f15cb@xymon.invalid> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
list Ron Cohen
▸
The sslcert detail oage is green because it was last time it was retrieved.There should be a timestamp for the last updateNicolaIl giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" <xymon@xymon.com> ha scritto:Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
Henrik Juul Størner
Senior IT Arkitekt - IT-sikkerhedFra: Josh Luthman <user-4c45a83f15cb@xymon.invalid>
Sendt: 20. marts 2025 21:56
Til: Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
Cc: Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
Emne: [Xymon] Re: SSL stuck on purpleOBS: Denne e-mail er ikke fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://<hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <user-4c45a83f15cb@xymon.invalid> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
list Jaime Kikpole
https://server.fw.ichabodcrane.org/filewave - SSL error
Seconds: 0.045584000
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
HTTP/1.1 200 OK
Date: Mon, 24 Mar 2025 16:03:18 GMT
Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: SAMEORIGIN
Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT
Accept-Ranges: bytes
Content-Length: 3548
X-Content-Type-Options: nosniff
Pragma: no-cache
Expires: Thu, 1 Jan 1970 00:00:00 GMT
Cache-Control: max-age=0, no-store, no-cache, must-revalidate
Content-Type: text/html
▸
Jaime Kikpole Director of Technology |
▸
The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.On Fri, 21 Mar 2025, 10:18 Nicola, <user-2c63804b6921@xymon.invalid> wrote:The sslcert detail oage is green because it was last time it was retrieved.There should be a timestamp for the last updateNicolaIl giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" <xymon@xymon.com> ha scritto:Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
Henrik Juul Størner
Senior IT Arkitekt - IT-sikkerhedFra: Josh Luthman <user-4c45a83f15cb@xymon.invalid>
Sendt: 20. marts 2025 21:56
Til: Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
Cc: Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
Emne: [Xymon] Re: SSL stuck on purpleOBS: Denne e-mail er ikke fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://<hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <user-4c45a83f15cb@xymon.invalid> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
list Matthew Goebel
* "3", e.g. https3://www.sample.com/ : use only SSLv3
* "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
* "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
* "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
* "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
* "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
* "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
* "10", e.g. http10://www.sample.com/ : use HTTP 1.0
* "11", e.g. http11://www.sample.com/ : use HTTP 1.1
Matt
▸
Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct?I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave.)I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says:
https://server.fw.ichabodcrane.org/filewave - SSL error
Seconds: 0.045584000I ran a test using https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichabodcrane.org and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago.I tried to check it using curl from the system running Xymon.# curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
HTTP/1.1 200 OK
Date: Mon, 24 Mar 2025 16:03:18 GMT
Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: SAMEORIGIN
Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT
Accept-Ranges: bytes
Content-Length: 3548
X-Content-Type-Options: nosniff
Pragma: no-cache
Expires: Thu, 1 Jan 1970 00:00:00 GMT
Cache-Control: max-age=0, no-store, no-cache, must-revalidate
Content-Type: text/htmlI also tried to check that the VM with Xymon installer on it was resolving things correctly.I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above.So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <user-f26e06d1e992@xymon.invalid> wrote:The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.On Fri, 21 Mar 2025, 10:18 Nicola, <user-2c63804b6921@xymon.invalid> wrote:The sslcert detail oage is green because it was last time it was retrieved.There should be a timestamp for the last updateNicolaIl giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" <xymon@xymon.com> ha scritto:Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
Henrik Juul Størner
Senior IT Arkitekt - IT-sikkerhedFra: Josh Luthman <user-4c45a83f15cb@xymon.invalid>
Sendt: 20. marts 2025 21:56
Til: Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
Cc: Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
Emne: [Xymon] Re: SSL stuck on purpleOBS: Denne e-mail er ikke fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://<hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <user-4c45a83f15cb@xymon.invalid> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
list Jaime Kikpole
▸
Jaime Kikpole Director of Technology |
▸
Hello,You might need to try one of the other https methods listed for hosts.cfg? https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html* "2", e.g. https2://www.sample.com/ : use only SSLv2
* "3", e.g. https3://www.sample.com/ : use only SSLv3
* "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
* "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
* "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
* "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
* "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
* "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
* "10", e.g. http10://www.sample.com/ : use HTTP 1.0
* "11", e.g. http11://www.sample.com/ : use HTTP 1.1ie...Thanks,
MattOn Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct?I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave.)I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says:
https://server.fw.ichabodcrane.org/filewave - SSL error
Seconds: 0.045584000I ran a test using https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichabodcrane.org and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago.I tried to check it using curl from the system running Xymon.# curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
HTTP/1.1 200 OK
Date: Mon, 24 Mar 2025 16:03:18 GMT
Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: SAMEORIGIN
Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT
Accept-Ranges: bytes
Content-Length: 3548
X-Content-Type-Options: nosniff
Pragma: no-cache
Expires: Thu, 1 Jan 1970 00:00:00 GMT
Cache-Control: max-age=0, no-store, no-cache, must-revalidate
Content-Type: text/htmlI also tried to check that the VM with Xymon installer on it was resolving things correctly.I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above.So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <user-f26e06d1e992@xymon.invalid> wrote:The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.On Fri, 21 Mar 2025, 10:18 Nicola, <user-2c63804b6921@xymon.invalid> wrote:The sslcert detail oage is green because it was last time it was retrieved.There should be a timestamp for the last updateNicolaIl giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" <xymon@xymon.com> ha scritto:Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
Henrik Juul Størner
Senior IT Arkitekt - IT-sikkerhedFra: Josh Luthman <user-4c45a83f15cb@xymon.invalid>
Sendt: 20. marts 2025 21:56
Til: Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
Cc: Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
Emne: [Xymon] Re: SSL stuck on purpleOBS: Denne e-mail er ikke fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://<hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <user-4c45a83f15cb@xymon.invalid> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
list Stephane Bakhos
You have TLS 1.2 From your xymon server, what does running this show: openssl s_client -connect server.fw.ichabodcrane.org:443 On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote:
Date: Mon, 24 Mar 2025 14:55:02 -0400
From: Jaime Kikpole via Xymon <xymon@xymon.com>
To: Matthew Goebel <user-15cc4fabfae6@xymon.invalid>
Cc: Xymon mailinglist <xymon@xymon.com>,
Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
Subject: [Xymon] Re: SSL stuck on purple
▸
Thanks. How can you tell which version of TLS it is? Jaime Kikpole Director of Technology Ichabod Crane Central School District (XXX) XXX-XXXX, x5425 On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <user-15cc4fabfae6@xymon.invalid> wrote:Hello, You might need to try one of the other https methods listed for hosts.cfg? https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html * "2", e.g. https2://www.sample.com/ : use only SSLv2 * "3", e.g. https3://www.sample.com/ : use only SSLv3 * "t", e.g. httpst://www.sample.com/ : use only TLSv1.0 * "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0 * "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1 * "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2 * "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers * "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers * "10", e.g. http10://www.sample.com/ : use HTTP 1.0 * "11", e.g. http11://www.sample.com/ : use HTTP 1.1 ie... httpsc://server.fw.ichabodcrane.org/filewave
<https://server.fw.ichabodcrane.org/filewave>;
▸
Thanks, Matt On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct? I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave .) I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says: https://server.fw.ichabodcrane.org/filewave - SSL error Seconds: 0.045584000 I ran a test using https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichabodcrane.org and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago. I tried to check it using curl from the system running Xymon. # curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Date: Mon, 24 Mar 2025 16:03:18 GMT Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT Accept-Ranges: bytes Content-Length: 3548 X-Content-Type-Options: nosniff Pragma: no-cache Expires: Thu, 1 Jan 1970 00:00:00 GMT Cache-Control: max-age=0, no-store, no-cache, must-revalidate Content-Type: text/html I also tried to check that the VM with Xymon installer on it was resolving things correctly. # host server.fw.ichabodcrane.org server.fw.ichabodcrane.org has address 34.236.200.223 I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above. So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas? Jaime Kikpole Director of Technology Ichabod Crane Central School District (XXX) XXX-XXXX, x5425 On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <user-f26e06d1e992@xymon.invalid> wrote:The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa. On Fri, 21 Mar 2025, 10:18 Nicola, <user-2c63804b6921@xymon.invalid> wrote:The sslcert detail oage is green because it was last time it was retrieved. There should be a timestamp for the last update Nicola Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < xymon@xymon.com> ha scritto:Hi, If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated. Med venlig hilsen / Best regards *Henrik Juul Størner* Senior IT Arkitekt - IT-sikkerhed *Fra:* Josh Luthman <user-4c45a83f15cb@xymon.invalid> *Sendt:* 20. marts 2025 21:56 *Til:* Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < user-0608abae5e7c@xymon.invalid> *Emne:* [Xymon] Re: SSL stuck on purple OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK. Why is the http test red? On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole < user-fb200a07d7dc@xymon.invalid> wrote: From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// <hostname>.ichabodcrane.org noconn
▸
The ghosts report shows no clients. The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service. *Jaime Kikpole* *Director of Technology* *Ichabod Crane Central School District* (XXX) XXX-XXXX, x5425 On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman < user-4c45a83f15cb@xymon.invalid> wrote: The IP in the first column is used in the event the name can't be resolved (or # testip is appended). The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test. Are you certain the name of the host matches? Are there any ghosts? Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url? On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon < xymon@xymon.com> wrote: Thanks for the ideas. That gave me more to try and I made some progress, but not much. I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that? I see what I would expect when using "curl https://<FQDN>"; and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address. When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address. history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple) What do you think? Should I just deal with the loss of historical data and drop the host by FQDN? *Jaime Kikpole* *Director of Technology* *Ichabod Crane Central School District* (XXX) XXX-XXXX, x5425 On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid> wrote:
Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20> <Hi%20Jaime%0d%0dOn%20Thu,%20> <Hi%20Jaime%0d%0dOn%20Thu,%20> <Hi%20Jaime%0d%0dOn%20Thu,%20> On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime
▸
Kikpole via Xymon <xymon@xymon.com> wrote: I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL. I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why. Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains. Any ideas on what I could check? First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page. Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result. Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname> Good luck with finding the problem. Cheers Jeremy--
Matthew Goebel : m <user-74d13dabeb26@xymon.invalid>user-9607b53e8435@xymon.invalid : Unix Jockey @ EMU : Hail Eris Neo-Student, Net Lurker, Donut consumer, and procrastinating medher... "Always with the negative waves, Moriarty" - Oddball "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
list Jaime Kikpole
▸
You have TLS 1.2
From your xymon server, what does running this show:
openssl s_client -connect server.fw.ichabodcrane.org:443
On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote:
> Date: Mon, 24 Mar 2025 14:55:02 -0400
> From: Jaime Kikpole via Xymon <xymon@xymon.com>
> To: Matthew Goebel <user-15cc4fabfae6@xymon.invalid>
> Cc: Xymon mailinglist <xymon@xymon.com>,
> Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
> Subject: [Xymon] Re: SSL stuck on purple
>
> Thanks. How can you tell which version of TLS it is?
>
>
>
> Jaime Kikpole
>
> Director of Technology
> Ichabod Crane Central School District
> (XXX) XXX-XXXX, x5425
>
>
> On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <user-15cc4fabfae6@xymon.invalid> wrote:
>
>> Hello,
>>
>> You might need to try one of the other https methods listed for hosts.cfg?
>> https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
>>
>> * "2", e.g. https2://www.sample.com/ : use only SSLv2
>> * "3", e.g. https3://www.sample.com/ : use only SSLv3
>> * "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
>> * "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
>> * "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
>> * "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
>> * "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
>> * "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
>> * "10", e.g. http10://www.sample.com/ : use HTTP 1.0
>> * "11", e.g. http11://www.sample.com/ : use HTTP 1.1
>>
>> ie...
>> httpsc://server.fw.ichabodcrane.org/filewave
>> <https://server.fw.ichabodcrane.org/filewave>
>>
>> Thanks,
>> Matt
>>
>>
>> On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com>
>> wrote:
>>
>>> Thanks for the suggestions and info so far. If I understand, my real
>>> issue is that the https test is failing and the sslcert test just doesn't
>>> have data to evaluate. Is that correct?
>>>
>>> I think I'm going to have to mention the FQDN in order to nail this down,
>>> even though I was trying to avoid it. Maybe the issue will be obvious to
>>> one of you. I'm trying to test https://server.fw.ichabdocrane.org.
>>> (Another valid test would be https://server.fw.ichabodcrane.org/filewave
>>> .)
>>>
>>> I noticed it was giving a 302 redirect, so I tried monitoring
>>> https://server.fw.ichabodcrane.org/filewave. Still had the same
>>> results. The web page for the http test now says:
>>>
>>> https://server.fw.ichabodcrane.org/filewave - SSL error
>>> Seconds: 0.045584000
>>>
>>> I ran a test using
>>> https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichabodcrane.org
>>> and it said that the certificate was good and that the IP address resolved
>>> to the IP that I expected it to point to. Note that this is the new IP.
>>> This issue started when the IP address of the hosted service was changed by
>>> the vendor a few weeks ago.
>>>
>>> I tried to check it using curl from the system running Xymon.
>>>
>>> # curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
>>> % Total % Received % Xferd Average Speed Time Time Time
>>> Current
>>> Dload Upload Total Spent Left
>>> Speed
>>> 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
>>> 0
>>> HTTP/1.1 200 OK
>>> Date: Mon, 24 Mar 2025 16:03:18 GMT
>>> Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12
>>> Strict-Transport-Security: max-age=31536000; includeSubDomains
>>> X-Frame-Options: SAMEORIGIN
>>> Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT
>>> Accept-Ranges: bytes
>>> Content-Length: 3548
>>> X-Content-Type-Options: nosniff
>>> Pragma: no-cache
>>> Expires: Thu, 1 Jan 1970 00:00:00 GMT
>>> Cache-Control: max-age=0, no-store, no-cache, must-revalidate
>>> Content-Type: text/html
>>>
>>>
>>> I also tried to check that the VM with Xymon installer on it was
>>> resolving things correctly.
>>>
>>> # host server.fw.ichabodcrane.org
>>> server.fw.ichabodcrane.org has address 34.236.200.223
>>>
>>>
>>> I also checked /etc/resolv.conf for the DNS resolvers and checked the
>>> FQDN resolving against both of those resolvers. The answer was the same as
>>> above.
>>>
>>> So as far as I can tell, the targeted URL is accessible from the CLI on
>>> that host, but Xymon thinks it isn't. Any ideas?
>>>
>>>
>>>
>>> Jaime Kikpole
>>>
>>> Director of Technology
>>> Ichabod Crane Central School District
>>> (XXX) XXX-XXXX, x5425
>>>
>>>
>>> On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <user-f26e06d1e992@xymon.invalid> wrote:
>>>
>>>> The SSL and http tests are not directly connected. The http retrieve a
>>>> page, which will return red if not found, and the SSL retrieve the SSL
>>>> certificate. So the SSL can be green and the http red and vis versa.
>>>>
>>>> On Fri, 21 Mar 2025, 10:18 Nicola, <user-2c63804b6921@xymon.invalid> wrote:
>>>>
>>>>> The sslcert detail oage is green because it was last time it was
>>>>> retrieved.
>>>>> There should be a timestamp for the last update
>>>>>
>>>>> Nicola
>>>>> Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" <
>>>>> xymon@xymon.com> ha scritto:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> If I remember correctly: If the http test is red because Xymon cannot
>>>>>> connect to the server on port 443, then Xymon will never retrieve the SSL
>>>>>> certificate (no connection = no SSL data). This will cause the sslcert
>>>>>> status to be purple, because it cannot be updated.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Med venlig hilsen / Best regards
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Henrik Juul Størner*
>>>>>> Senior IT Arkitekt - IT-sikkerhed
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Fra:* Josh Luthman <user-4c45a83f15cb@xymon.invalid>
>>>>>> *Sendt:* 20. marts 2025 21:56
>>>>>> *Til:* Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
>>>>>> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <
>>>>>> user-0608abae5e7c@xymon.invalid>
>>>>>> *Emne:* [Xymon] Re: SSL stuck on purple
>>>>>>
>>>>>>
>>>>>>
>>>>>> OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun
>>>>>> links eller vedhæftede filer, hvis du mener, de er OK.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Why is the http test red?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <
>>>>>> user-fb200a07d7dc@xymon.invalid> wrote:
>>>>>>
>>>>>> From hosts.cfg, with redactions between < and >:
>>>>>>
>>>>>> <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://
>>>>>> <hostname>.ichabodcrane.org noconn
>>>>>>
>>>>>>
>>>>>>
>>>>>> The ghosts report shows no clients.
>>>>>>
>>>>>>
>>>>>>
>>>>>> The hostname definitely resolves (tested on the VM with the xymon
>>>>>> server) to the IP address. The purple alert is in a column labeled
>>>>>> "sslcert" and the red is in "http". The data in the page with the details
>>>>>> about sslcert shows the correct dates and a green alert. The data in the
>>>>>> page with the http details shows the message "SSL error" after the
>>>>>> https://.... URL. When viewing that URL, it redirects path on the
>>>>>> same server. This hasn't given me trouble over the last several years and
>>>>>> the only thing that changed on the monitored system is that its IP address
>>>>>> changed at the VM hosting service.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Jaime Kikpole*
>>>>>>
>>>>>> *Director of Technology*
>>>>>> *Ichabod Crane Central School District*
>>>>>> (XXX) XXX-XXXX, x5425
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <
>>>>>> user-4c45a83f15cb@xymon.invalid> wrote:
>>>>>>
>>>>>> The IP in the first column is used in the event the name can't be
>>>>>> resolved (or # testip is appended).
>>>>>>
>>>>>>
>>>>>>
>>>>>> The purple test means there is no report for the test (for half an
>>>>>> hour I think?) In this case you're testing from the Xymon Server so it is
>>>>>> responsible for doing the test.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you certain the name of the host matches? Are there any ghosts?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you looking at the "sslcert" column? What does your hosts.cfg
>>>>>> line look like - does it have an https url?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <
>>>>>> xymon@xymon.com> wrote:
>>>>>>
>>>>>> Thanks for the ideas. That gave me more to try and I made some
>>>>>> progress, but not much.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I discovered that hosts.cfg still had the old IP address. I updated
>>>>>> that, but there was no change. I restarted the xymon server process and
>>>>>> there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there
>>>>>> was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or
>>>>>> something like that?
>>>>>>
>>>>>>
>>>>>>
>>>>>> I see what I would expect when using "curl https://<FQDN>" and the
>>>>>> logs only show one interesting line within 2025, so I suspect that it
>>>>>> all comes down to that one hosts.cfg line with the old IP address.
>>>>>>
>>>>>>
>>>>>>
>>>>>> When I used "grep <server> * | grep 2025 | less", I see the following
>>>>>> line and then everything else was from notifications.log about emailing my
>>>>>> department's outages notification email address.
>>>>>>
>>>>>>
>>>>>>
>>>>>> history.log:2025-03-18 14:54:12.042214 Will not update
>>>>>> /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color
>>>>>> unchanged (purple)
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think? Should I just deal with the loss of historical
>>>>>> data and drop the host by FQDN?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Jaime Kikpole*
>>>>>>
>>>>>> *Director of Technology*
>>>>>> *Ichabod Crane Central School District*
>>>>>> (XXX) XXX-XXXX, x5425
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime
>>>>>> Kikpole via Xymon <xymon@xymon.com> wrote:
>>>>>>
>>>>>> I have a Xymon install that was monitoring (among other things) an
>>>>>> HTTPS site. That site moved to a new IP and while I did update the public
>>>>>> DNS records it was a few days (or maybe two weeks?) before I remembered to
>>>>>> update the internal DNS records. By then, Xymon was reporting red for
>>>>>> HTTPS and purple for SSL.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I've corrected the DNS records and the VM running Xymon is resolving
>>>>>> it correctly, but now it is still showing the same red and purple alerts.
>>>>>> Every now and then, they'll switch to green for a little while and then
>>>>>> back to red and purple. I'm not sure why.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Xymon's web interface shows that the certificate is valid for more
>>>>>> than 2 months. So that isn't it. I've restarted the Xymon server process
>>>>>> and even the whole OS, but this issue remains.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Any ideas on what I could check?
>>>>>>
>>>>>>
>>>>>>
>>>>>> First, review the content of the https status page. There may be
>>>>>> non-obvious clues here. There's a difference between "connection refused"
>>>>>> and "connection timed out" and this can lead you in different directions to
>>>>>> identify the cause. For example, if you have a content check configured
>>>>>> ("cont=..." in hosts.cfg), which is failing due to new content on the new
>>>>>> webserver, this will be explained on the status page.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Second, I would check that your Xymon server can actually connect to
>>>>>> the new IP address on port 443. I usually use "telnet <IP> <port>" and see
>>>>>> if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc
>>>>>> works too. For an HTTPS website (as distinct from HTTP) I sometimes use an
>>>>>> openssl command to see if the TLS/SSL interaction works, which tells me
>>>>>> that there's no issue on the webserver itself, something like "openssl
>>>>>> s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget
>>>>>> to fetch a webpage, but if you have a proxy set anywhere, this may give you
>>>>>> an invalid result.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thirdly, I would check the xymonnet.log file for any indications of
>>>>>> problems, which might suggest a cause. Having xymonnet in debug mode might
>>>>>> help (adding "--debug" into the CMD line of the [xymonnet] block in
>>>>>> tasks.cfg). Alternatively, you could run xymonnet directly yourself, and
>>>>>> look at the output. Something like this: xymoncmd xymonnet --no-update
>>>>>> --debug <hostname>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Good luck with finding the problem.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Jeremy
>>>>>>
>>>>>> xymon@xymon.com
>>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>>
>>>>>> xymon@xymon.com
>>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>>
>>>>>> xymon@xymon.com
>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>
>>>> xymon@xymon.com
>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>
>>
>>
>> --
>> Matthew Goebel : m <user-74d13dabeb26@xymon.invalid>user-9607b53e8435@xymon.invalid : Unix
>> Jockey @ EMU : Hail Eris
>> Neo-Student, Net Lurker, Donut consumer, and procrastinating medher...
>> "Always with the negative waves, Moriarty" - Oddball
>> "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
>>
>>
>>
>
list Jaime Kikpole
▸
Jaime Kikpole Director of Technology |
▸
Thanks. I see the protocol entry now that I look through the output of that command. I'll make that edit when I get to a better endpoint than my phone.JaimeOn Mon, Mar 24, 2025, 4:07 PM Stephane Bakhos <user-5d5747647683@xymon.invalid> wrote:You have TLS 1.2
From your xymon server, what does running this show:
openssl s_client -connect server.fw.ichabodcrane.org:443
On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote:
> Date: Mon, 24 Mar 2025 14:55:02 -0400
> From: Jaime Kikpole via Xymon <xymon@xymon.com>
> To: Matthew Goebel <user-15cc4fabfae6@xymon.invalid>
> Cc: Xymon mailinglist <xymon@xymon.com>,
> Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
> Subject: [Xymon] Re: SSL stuck on purple
>
> Thanks. How can you tell which version of TLS it is?
>
>
>
> Jaime Kikpole
>
> Director of Technology
> Ichabod Crane Central School District
> (XXX) XXX-XXXX, x5425
>
>
> On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <user-15cc4fabfae6@xymon.invalid> wrote:
>
>> Hello,
>>
>> You might need to try one of the other https methods listed for hosts.cfg?
>> https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
>>
>> * "2", e.g. https2://www.sample.com/ : use only SSLv2
>> * "3", e.g. https3://www.sample.com/ : use only SSLv3
>> * "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
>> * "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
>> * "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
>> * "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
>> * "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
>> * "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
>> * "10", e.g. http10://www.sample.com/ : use HTTP 1.0
>> * "11", e.g. http11://www.sample.com/ : use HTTP 1.1
>>
>> ie...
>> httpsc://server.fw.ichabodcrane.org/filewave
>> <https://server.fw.ichabodcrane.org/filewave>
>>
>> Thanks,
>> Matt
>>
>>
>> On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com>
>> wrote:
>>
>>> Thanks for the suggestions and info so far. If I understand, my real
>>> issue is that the https test is failing and the sslcert test just doesn't
>>> have data to evaluate. Is that correct?
>>>
>>> I think I'm going to have to mention the FQDN in order to nail this down,
>>> even though I was trying to avoid it. Maybe the issue will be obvious to
>>> one of you. I'm trying to test https://server.fw.ichabdocrane.org.
>>> (Another valid test would be https://server.fw.ichabodcrane.org/filewave
>>> .)
>>>
>>> I noticed it was giving a 302 redirect, so I tried monitoring
>>> https://server.fw.ichabodcrane.org/filewave. Still had the same
>>> results. The web page for the http test now says:
>>>
>>> https://server.fw.ichabodcrane.org/filewave - SSL error
>>> Seconds: 0.045584000
>>>
>>> I ran a test using
>>> https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichabodcrane.org
>>> and it said that the certificate was good and that the IP address resolved
>>> to the IP that I expected it to point to. Note that this is the new IP.
>>> This issue started when the IP address of the hosted service was changed by
>>> the vendor a few weeks ago.
>>>
>>> I tried to check it using curl from the system running Xymon.
>>>
>>> # curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
>>> % Total % Received % Xferd Average Speed Time Time Time
>>> Current
>>> Dload Upload Total Spent Left
>>> Speed
>>> 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
>>> 0
>>> HTTP/1.1 200 OK
>>> Date: Mon, 24 Mar 2025 16:03:18 GMT
>>> Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12
>>> Strict-Transport-Security: max-age=31536000; includeSubDomains
>>> X-Frame-Options: SAMEORIGIN
>>> Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT
>>> Accept-Ranges: bytes
>>> Content-Length: 3548
>>> X-Content-Type-Options: nosniff
>>> Pragma: no-cache
>>> Expires: Thu, 1 Jan 1970 00:00:00 GMT
>>> Cache-Control: max-age=0, no-store, no-cache, must-revalidate
>>> Content-Type: text/html
>>>
>>>
>>> I also tried to check that the VM with Xymon installer on it was
>>> resolving things correctly.
>>>
>>> # host server.fw.ichabodcrane.org
>>> server.fw.ichabodcrane.org has address 34.236.200.223
>>>
>>>
>>> I also checked /etc/resolv.conf for the DNS resolvers and checked the
>>> FQDN resolving against both of those resolvers. The answer was the same as
>>> above.
>>>
>>> So as far as I can tell, the targeted URL is accessible from the CLI on
>>> that host, but Xymon thinks it isn't. Any ideas?
>>>
>>>
>>>
>>> Jaime Kikpole
>>>
>>> Director of Technology
>>> Ichabod Crane Central School District
>>> (XXX) XXX-XXXX, x5425
>>>
>>>
>>> On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <user-f26e06d1e992@xymon.invalid> wrote:
>>>
>>>> The SSL and http tests are not directly connected. The http retrieve a
>>>> page, which will return red if not found, and the SSL retrieve the SSL
>>>> certificate. So the SSL can be green and the http red and vis versa.
>>>>
>>>> On Fri, 21 Mar 2025, 10:18 Nicola, <user-2c63804b6921@xymon.invalid> wrote:
>>>>
>>>>> The sslcert detail oage is green because it was last time it was
>>>>> retrieved.
>>>>> There should be a timestamp for the last update
>>>>>
>>>>> Nicola
>>>>> Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" <
>>>>> xymon@xymon.com> ha scritto:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> If I remember correctly: If the http test is red because Xymon cannot
>>>>>> connect to the server on port 443, then Xymon will never retrieve the SSL
>>>>>> certificate (no connection = no SSL data). This will cause the sslcert
>>>>>> status to be purple, because it cannot be updated.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Med venlig hilsen / Best regards
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Henrik Juul Størner*
>>>>>> Senior IT Arkitekt - IT-sikkerhed
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Fra:* Josh Luthman <user-4c45a83f15cb@xymon.invalid>
>>>>>> *Sendt:* 20. marts 2025 21:56
>>>>>> *Til:* Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
>>>>>> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <
>>>>>> user-0608abae5e7c@xymon.invalid>
>>>>>> *Emne:* [Xymon] Re: SSL stuck on purple
>>>>>>
>>>>>>
>>>>>>
>>>>>> OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun
>>>>>> links eller vedhæftede filer, hvis du mener, de er OK.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Why is the http test red?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <
>>>>>> user-fb200a07d7dc@xymon.invalid> wrote:
>>>>>>
>>>>>> From hosts.cfg, with redactions between < and >:
>>>>>>
>>>>>> <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://
>>>>>> <hostname>.ichabodcrane.org noconn
>>>>>>
>>>>>>
>>>>>>
>>>>>> The ghosts report shows no clients.
>>>>>>
>>>>>>
>>>>>>
>>>>>> The hostname definitely resolves (tested on the VM with the xymon
>>>>>> server) to the IP address. The purple alert is in a column labeled
>>>>>> "sslcert" and the red is in "http". The data in the page with the details
>>>>>> about sslcert shows the correct dates and a green alert. The data in the
>>>>>> page with the http details shows the message "SSL error" after the
>>>>>> https://.... URL. When viewing that URL, it redirects path on the
>>>>>> same server. This hasn't given me trouble over the last several years and
>>>>>> the only thing that changed on the monitored system is that its IP address
>>>>>> changed at the VM hosting service.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Jaime Kikpole*
>>>>>>
>>>>>> *Director of Technology*
>>>>>> *Ichabod Crane Central School District*
>>>>>> (XXX) XXX-XXXX, x5425
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <
>>>>>> user-4c45a83f15cb@xymon.invalid> wrote:
>>>>>>
>>>>>> The IP in the first column is used in the event the name can't be
>>>>>> resolved (or # testip is appended).
>>>>>>
>>>>>>
>>>>>>
>>>>>> The purple test means there is no report for the test (for half an
>>>>>> hour I think?) In this case you're testing from the Xymon Server so it is
>>>>>> responsible for doing the test.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you certain the name of the host matches? Are there any ghosts?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you looking at the "sslcert" column? What does your hosts.cfg
>>>>>> line look like - does it have an https url?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <
>>>>>> xymon@xymon.com> wrote:
>>>>>>
>>>>>> Thanks for the ideas. That gave me more to try and I made some
>>>>>> progress, but not much.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I discovered that hosts.cfg still had the old IP address. I updated
>>>>>> that, but there was no change. I restarted the xymon server process and
>>>>>> there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there
>>>>>> was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or
>>>>>> something like that?
>>>>>>
>>>>>>
>>>>>>
>>>>>> I see what I would expect when using "curl https://<FQDN>" and the
>>>>>> logs only show one interesting line within 2025, so I suspect that it
>>>>>> all comes down to that one hosts.cfg line with the old IP address.
>>>>>>
>>>>>>
>>>>>>
>>>>>> When I used "grep <server> * | grep 2025 | less", I see the following
>>>>>> line and then everything else was from notifications.log about emailing my
>>>>>> department's outages notification email address.
>>>>>>
>>>>>>
>>>>>>
>>>>>> history.log:2025-03-18 14:54:12.042214 Will not update
>>>>>> /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color
>>>>>> unchanged (purple)
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think? Should I just deal with the loss of historical
>>>>>> data and drop the host by FQDN?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Jaime Kikpole*
>>>>>>
>>>>>> *Director of Technology*
>>>>>> *Ichabod Crane Central School District*
>>>>>> (XXX) XXX-XXXX, x5425
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime
>>>>>> Kikpole via Xymon <xymon@xymon.com> wrote:
>>>>>>
>>>>>> I have a Xymon install that was monitoring (among other things) an
>>>>>> HTTPS site. That site moved to a new IP and while I did update the public
>>>>>> DNS records it was a few days (or maybe two weeks?) before I remembered to
>>>>>> update the internal DNS records. By then, Xymon was reporting red for
>>>>>> HTTPS and purple for SSL.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I've corrected the DNS records and the VM running Xymon is resolving
>>>>>> it correctly, but now it is still showing the same red and purple alerts.
>>>>>> Every now and then, they'll switch to green for a little while and then
>>>>>> back to red and purple. I'm not sure why.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Xymon's web interface shows that the certificate is valid for more
>>>>>> than 2 months. So that isn't it. I've restarted the Xymon server process
>>>>>> and even the whole OS, but this issue remains.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Any ideas on what I could check?
>>>>>>
>>>>>>
>>>>>>
>>>>>> First, review the content of the https status page. There may be
>>>>>> non-obvious clues here. There's a difference between "connection refused"
>>>>>> and "connection timed out" and this can lead you in different directions to
>>>>>> identify the cause. For example, if you have a content check configured
>>>>>> ("cont=..." in hosts.cfg), which is failing due to new content on the new
>>>>>> webserver, this will be explained on the status page.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Second, I would check that your Xymon server can actually connect to
>>>>>> the new IP address on port 443. I usually use "telnet <IP> <port>" and see
>>>>>> if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc
>>>>>> works too. For an HTTPS website (as distinct from HTTP) I sometimes use an
>>>>>> openssl command to see if the TLS/SSL interaction works, which tells me
>>>>>> that there's no issue on the webserver itself, something like "openssl
>>>>>> s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget
>>>>>> to fetch a webpage, but if you have a proxy set anywhere, this may give you
>>>>>> an invalid result.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thirdly, I would check the xymonnet.log file for any indications of
>>>>>> problems, which might suggest a cause. Having xymonnet in debug mode might
>>>>>> help (adding "--debug" into the CMD line of the [xymonnet] block in
>>>>>> tasks.cfg). Alternatively, you could run xymonnet directly yourself, and
>>>>>> look at the output. Something like this: xymoncmd xymonnet --no-update
>>>>>> --debug <hostname>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Good luck with finding the problem.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Jeremy
>>>>>>
>>>>>> xymon@xymon.com
>>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>>
>>>>>> xymon@xymon.com
>>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>>
>>>>>> xymon@xymon.com
>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>
>>>> xymon@xymon.com
>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>
>>
>>
>> --
>> Matthew Goebel : m <user-74d13dabeb26@xymon.invalid>user-9607b53e8435@xymon.invalid : Unix
>> Jockey @ EMU : Hail Eris
>> Neo-Student, Net Lurker, Donut consumer, and procrastinating medher...
>> "Always with the negative waves, Moriarty" - Oddball
>> "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
>>
>>
>>
>
list Kris Springer
https://server.fw.ichabodcrane.org sni
I've had issues with some ssl tests not working on some specific servers, and this sni tag solved my issues.
Search for 'sni' to read about it in the man page.
https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
Thank You, Kris Springer Systems Admin I/O Network Administration
▸
I just tried it and gave it 15+ minutes to change from red or at least give another status message. It didn't change the alert color or the text on the alert's page.
Any other suggestions?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425
On Mon, Mar 24, 2025 at 5:24 PM Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> wrote:
Thanks. I see the protocol entry now that I look through the output of that command. I'll make that edit when I get to a better endpoint than my phone.
Jaime
On Mon, Mar 24, 2025, 4:07 PM Stephane Bakhos <user-5d5747647683@xymon.invalid> wrote:
You have TLS 1.2
>From your xymon server, what does running this show:
openssl s_client -connect server.fw.ichabodcrane.org:443
On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote:
> Date: Mon, 24 Mar 2025 14:55:02 -0400
> From: Jaime Kikpole via Xymon <xymon@xymon.com>
> To: Matthew Goebel <user-15cc4fabfae6@xymon.invalid>
> Cc: Xymon mailinglist <xymon@xymon.com>,
> Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
> Subject: [Xymon] Re: SSL stuck on purple
>
> Thanks. How can you tell which version of TLS it is?
>
>
>
> Jaime Kikpole
>
> Director of Technology
> Ichabod Crane Central School District
> (XXX) XXX-XXXX, x5425
>
>
> On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <user-15cc4fabfae6@xymon.invalid> wrote:
>
>> Hello,
>>
>> You might need to try one of the other https methods listed for hosts.cfg?
>> https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
>>
>> * "2", e.g. https2://www.sample.com/ : use only SSLv2
>> * "3", e.g. https3://www.sample.com/ : use only SSLv3
>> * "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
>> * "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
>> * "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
>> * "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
>> * "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
>> * "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
>> * "10", e.g. http10://www.sample.com/ : use HTTP 1.0
>> * "11", e.g. http11://www.sample.com/ : use HTTP 1.1
>>
>> ie...
>> httpsc://server.fw.ichabodcrane.org/filewave
>> <https://server.fw.ichabodcrane.org/filewave>
>>
>> Thanks,
>> Matt
>>
>>
>> On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com>
>> wrote:
>>
>>> Thanks for the suggestions and info so far. If I understand, my real
>>> issue is that the https test is failing and the sslcert test just doesn't
>>> have data to evaluate. Is that correct?
>>>
>>> I think I'm going to have to mention the FQDN in order to nail this down,
>>> even though I was trying to avoid it. Maybe the issue will be obvious to
>>> one of you. I'm trying to test https://server.fw.ichabdocrane.org.
>>> (Another valid test would be https://server.fw.ichabodcrane.org/filewave
>>> .)
>>>
>>> I noticed it was giving a 302 redirect, so I tried monitoring
>>> https://server.fw.ichabodcrane.org/filewave. Still had the same
>>> results. The web page for the http test now says:
>>>
>>> https://server.fw.ichabodcrane.org/filewave - SSL error
>>> Seconds: 0.045584000
>>>
>>> I ran a test using
>>> https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichabodcrane.org
>>> and it said that the certificate was good and that the IP address resolved
>>> to the IP that I expected it to point to. Note that this is the new IP.
>>> This issue started when the IP address of the hosted service was changed by
>>> the vendor a few weeks ago.
>>>
>>> I tried to check it using curl from the system running Xymon.
>>>
>>> # curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
>>> % Total % Received % Xferd Average Speed Time Time Time
>>> Current
>>> Dload Upload Total Spent Left
>>> Speed
>>> 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
>>> 0
>>> HTTP/1.1 200 OK
>>> Date: Mon, 24 Mar 2025 16:03:18 GMT
>>> Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12
>>> Strict-Transport-Security: max-age=31536000; includeSubDomains
>>> X-Frame-Options: SAMEORIGIN
>>> Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT
>>> Accept-Ranges: bytes
>>> Content-Length: 3548
>>> X-Content-Type-Options: nosniff
>>> Pragma: no-cache
>>> Expires: Thu, 1 Jan 1970 00:00:00 GMT
>>> Cache-Control: max-age=0, no-store, no-cache, must-revalidate
>>> Content-Type: text/html
>>>
>>>
>>> I also tried to check that the VM with Xymon installer on it was
>>> resolving things correctly.
>>>
>>> # host server.fw.ichabodcrane.org
>>> server.fw.ichabodcrane.org has address 34.236.200.223
>>>
>>>
>>> I also checked /etc/resolv.conf for the DNS resolvers and checked the
>>> FQDN resolving against both of those resolvers. The answer was the same as
>>> above.
>>>
>>> So as far as I can tell, the targeted URL is accessible from the CLI on
>>> that host, but Xymon thinks it isn't. Any ideas?
>>>
>>>
>>>
>>> Jaime Kikpole
>>>
>>> Director of Technology
>>> Ichabod Crane Central School District
>>> (XXX) XXX-XXXX, x5425
>>>
>>>
>>> On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <user-f26e06d1e992@xymon.invalid> wrote:
>>>
>>>> The SSL and http tests are not directly connected. The http retrieve a
>>>> page, which will return red if not found, and the SSL retrieve the SSL
>>>> certificate. So the SSL can be green and the http red and vis versa.
>>>>
>>>> On Fri, 21 Mar 2025, 10:18 Nicola, <user-2c63804b6921@xymon.invalid> wrote:
>>>>
>>>>> The sslcert detail oage is green because it was last time it was
>>>>> retrieved.
>>>>> There should be a timestamp for the last update
>>>>>
>>>>> Nicola
>>>>> Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" <
>>>>> xymon@xymon.com> ha scritto:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> If I remember correctly: If the http test is red because Xymon cannot
>>>>>> connect to the server on port 443, then Xymon will never retrieve the SSL
>>>>>> certificate (no connection = no SSL data). This will cause the sslcert
>>>>>> status to be purple, because it cannot be updated.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Med venlig hilsen / Best regards
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Henrik Juul Størner*
>>>>>> Senior IT Arkitekt - IT-sikkerhed
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Fra:* Josh Luthman <user-4c45a83f15cb@xymon.invalid>
>>>>>> *Sendt:* 20. marts 2025 21:56
>>>>>> *Til:* Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
>>>>>> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <
>>>>>> user-0608abae5e7c@xymon.invalid>
>>>>>> *Emne:* [Xymon] Re: SSL stuck on purple
>>>>>>
>>>>>>
>>>>>>
>>>>>> OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun
>>>>>> links eller vedhæftede filer, hvis du mener, de er OK.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Why is the http test red?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <
>>>>>> user-fb200a07d7dc@xymon.invalid> wrote:
>>>>>>
>>>>>> From hosts.cfg, with redactions between < and >:
>>>>>>
>>>>>> <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://
>>>>>> <hostname>.ichabodcrane.org noconn
>>>>>>
>>>>>>
>>>>>>
>>>>>> The ghosts report shows no clients.
>>>>>>
>>>>>>
>>>>>>
>>>>>> The hostname definitely resolves (tested on the VM with the xymon
>>>>>> server) to the IP address. The purple alert is in a column labeled
>>>>>> "sslcert" and the red is in "http". The data in the page with the details
>>>>>> about sslcert shows the correct dates and a green alert. The data in the
>>>>>> page with the http details shows the message "SSL error" after the
>>>>>> https://.... URL. When viewing that URL, it redirects path on the
>>>>>> same server. This hasn't given me trouble over the last several years and
>>>>>> the only thing that changed on the monitored system is that its IP address
>>>>>> changed at the VM hosting service.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Jaime Kikpole*
>>>>>>
>>>>>> *Director of Technology*
>>>>>> *Ichabod Crane Central School District*
>>>>>> (XXX) XXX-XXXX, x5425
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <
>>>>>> user-4c45a83f15cb@xymon.invalid> wrote:
>>>>>>
>>>>>> The IP in the first column is used in the event the name can't be
>>>>>> resolved (or # testip is appended).
>>>>>>
>>>>>>
>>>>>>
>>>>>> The purple test means there is no report for the test (for half an
>>>>>> hour I think?) In this case you're testing from the Xymon Server so it is
>>>>>> responsible for doing the test.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you certain the name of the host matches? Are there any ghosts?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you looking at the "sslcert" column? What does your hosts.cfg
>>>>>> line look like - does it have an https url?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <
>>>>>> xymon@xymon.com> wrote:
>>>>>>
>>>>>> Thanks for the ideas. That gave me more to try and I made some
>>>>>> progress, but not much.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I discovered that hosts.cfg still had the old IP address. I updated
>>>>>> that, but there was no change. I restarted the xymon server process and
>>>>>> there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there
>>>>>> was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or
>>>>>> something like that?
>>>>>>
>>>>>>
>>>>>>
>>>>>> I see what I would expect when using "curl https://<FQDN>" and the
>>>>>> logs only show one interesting line within 2025, so I suspect that it
>>>>>> all comes down to that one hosts.cfg line with the old IP address.
>>>>>>
>>>>>>
>>>>>>
>>>>>> When I used "grep <server> * | grep 2025 | less", I see the following
>>>>>> line and then everything else was from notifications.log about emailing my
>>>>>> department's outages notification email address.
>>>>>>
>>>>>>
>>>>>>
>>>>>> history.log:2025-03-18 14:54:12.042214 Will not update
>>>>>> /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color
>>>>>> unchanged (purple)
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think? Should I just deal with the loss of historical
>>>>>> data and drop the host by FQDN?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Jaime Kikpole*
>>>>>>
>>>>>> *Director of Technology*
>>>>>> *Ichabod Crane Central School District*
>>>>>> (XXX) XXX-XXXX, x5425
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime
>>>>>> Kikpole via Xymon <xymon@xymon.com> wrote:
>>>>>>
>>>>>> I have a Xymon install that was monitoring (among other things) an
>>>>>> HTTPS site. That site moved to a new IP and while I did update the public
>>>>>> DNS records it was a few days (or maybe two weeks?) before I remembered to
>>>>>> update the internal DNS records. By then, Xymon was reporting red for
>>>>>> HTTPS and purple for SSL.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I've corrected the DNS records and the VM running Xymon is resolving
>>>>>> it correctly, but now it is still showing the same red and purple alerts.
>>>>>> Every now and then, they'll switch to green for a little while and then
>>>>>> back to red and purple. I'm not sure why.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Xymon's web interface shows that the certificate is valid for more
>>>>>> than 2 months. So that isn't it. I've restarted the Xymon server process
>>>>>> and even the whole OS, but this issue remains.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Any ideas on what I could check?
>>>>>>
>>>>>>
>>>>>>
>>>>>> First, review the content of the https status page. There may be
>>>>>> non-obvious clues here. There's a difference between "connection refused"
>>>>>> and "connection timed out" and this can lead you in different directions to
>>>>>> identify the cause. For example, if you have a content check configured
>>>>>> ("cont=..." in hosts.cfg), which is failing due to new content on the new
>>>>>> webserver, this will be explained on the status page.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Second, I would check that your Xymon server can actually connect to
>>>>>> the new IP address on port 443. I usually use "telnet <IP> <port>" and see
>>>>>> if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc
>>>>>> works too. For an HTTPS website (as distinct from HTTP) I sometimes use an
>>>>>> openssl command to see if the TLS/SSL interaction works, which tells me
>>>>>> that there's no issue on the webserver itself, something like "openssl
>>>>>> s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget
>>>>>> to fetch a webpage, but if you have a proxy set anywhere, this may give you
>>>>>> an invalid result.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thirdly, I would check the xymonnet.log file for any indications of
>>>>>> problems, which might suggest a cause. Having xymonnet in debug mode might
>>>>>> help (adding "--debug" into the CMD line of the [xymonnet] block in
>>>>>> tasks.cfg). Alternatively, you could run xymonnet directly yourself, and
>>>>>> look at the output. Something like this: xymoncmd xymonnet --no-update
>>>>>> --debug <hostname>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Good luck with finding the problem.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Jeremy
>>>>>>
>>>>>> xymon@xymon.com
>>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>>
>>>>>> xymon@xymon.com
>>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>>
>>>>>> xymon@xymon.com
>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>
>>>> xymon@xymon.com
>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>
>>
>>
>> --
>> Matthew Goebel : m <user-74d13dabeb26@xymon.invalid>user-9607b53e8435@xymon.invalid : Unix
>> Jockey @ EMU : Hail Eris
>> Neo-Student, Net Lurker, Donut consumer, and procrastinating medher...
>> "Always with the negative waves, Moriarty" - Oddball
>> "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
>>
>>
>>
>
xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
list Jaime Kikpole
▸
Jaime Kikpole Director of Technology |
▸
Just jumping into the middle of this thread, but have you tried adding sni at the end of your https test in your hosts file?
https://server.fw.ichabodcrane.org sni
I've had issues with some ssl tests not working on some specific servers, and this sni tag solved my issues.
Search for 'sni' to read about it in the man page.
https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.htmlThank You, Kris Springer Systems Admin I/O Network AdministrationOn 3/26/25 11:24, Jaime Kikpole via Xymon wrote:
I just tried it and gave it 15+ minutes to change from red or at least give another status message. It didn't change the alert color or the text on the alert's page.
Any other suggestions?
Jaime Kikpole
Director of Technology
Ichabod Crane Central School District
(XXX) XXX-XXXX, x5425
On Mon, Mar 24, 2025 at 5:24 PM Jaime Kikpole <user-fb200a07d7dc@xymon.invalid> wrote:
Thanks. I see the protocol entry now that I look through the output of that command. I'll make that edit when I get to a better endpoint than my phone.
Jaime
On Mon, Mar 24, 2025, 4:07 PM Stephane Bakhos <user-5d5747647683@xymon.invalid> wrote:
You have TLS 1.2
>From your xymon server, what does running this show:
openssl s_client -connect server.fw.ichabodcrane.org:443
On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote:
> Date: Mon, 24 Mar 2025 14:55:02 -0400
> From: Jaime Kikpole via Xymon <xymon@xymon.com>
> To: Matthew Goebel <user-15cc4fabfae6@xymon.invalid>
> Cc: Xymon mailinglist <xymon@xymon.com>,
> Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
> Subject: [Xymon] Re: SSL stuck on purple
>
> Thanks. How can you tell which version of TLS it is?
>
>
>
> Jaime Kikpole
>
> Director of Technology
> Ichabod Crane Central School District
> (XXX) XXX-XXXX, x5425
>
>
> On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <user-15cc4fabfae6@xymon.invalid> wrote:
>
>> Hello,
>>
>> You might need to try one of the other https methods listed for hosts.cfg?
>> https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
>>
>> * "2", e.g. https2://www.sample.com/ : use only SSLv2
>> * "3", e.g. https3://www.sample.com/ : use only SSLv3
>> * "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
>> * "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
>> * "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
>> * "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
>> * "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
>> * "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
>> * "10", e.g. http10://www.sample.com/ : use HTTP 1.0
>> * "11", e.g. http11://www.sample.com/ : use HTTP 1.1
>>
>> ie...
>> httpsc://server.fw.ichabodcrane.org/filewave
>> <https://server.fw.ichabodcrane.org/filewave>
>>
>> Thanks,
>> Matt
>>
>>
>> On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com>
>> wrote:
>>
>>> Thanks for the suggestions and info so far. If I understand, my real
>>> issue is that the https test is failing and the sslcert test just doesn't
>>> have data to evaluate. Is that correct?
>>>
>>> I think I'm going to have to mention the FQDN in order to nail this down,
>>> even though I was trying to avoid it. Maybe the issue will be obvious to
>>> one of you. I'm trying to test https://server.fw.ichabdocrane.org.
>>> (Another valid test would be https://server.fw.ichabodcrane.org/filewave
>>> .)
>>>
>>> I noticed it was giving a 302 redirect, so I tried monitoring
>>> https://server.fw.ichabodcrane.org/filewave. Still had the same
>>> results. The web page for the http test now says:
>>>
>>> https://server.fw.ichabodcrane.org/filewave - SSL error
>>> Seconds: 0.045584000
>>>
>>> I ran a test using
>>> https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichabodcrane.org
>>> and it said that the certificate was good and that the IP address resolved
>>> to the IP that I expected it to point to. Note that this is the new IP.
>>> This issue started when the IP address of the hosted service was changed by
>>> the vendor a few weeks ago.
>>>
>>> I tried to check it using curl from the system running Xymon.
>>>
>>> # curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
>>> % Total % Received % Xferd Average Speed Time Time Time
>>> Current
>>> Dload Upload Total Spent Left
>>> Speed
>>> 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
>>> 0
>>> HTTP/1.1 200 OK
>>> Date: Mon, 24 Mar 2025 16:03:18 GMT
>>> Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12
>>> Strict-Transport-Security: max-age=31536000; includeSubDomains
>>> X-Frame-Options: SAMEORIGIN
>>> Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT
>>> Accept-Ranges: bytes
>>> Content-Length: 3548
>>> X-Content-Type-Options: nosniff
>>> Pragma: no-cache
>>> Expires: Thu, 1 Jan 1970 00:00:00 GMT
>>> Cache-Control: max-age=0, no-store, no-cache, must-revalidate
>>> Content-Type: text/html
>>>
>>>
>>> I also tried to check that the VM with Xymon installer on it was
>>> resolving things correctly.
>>>
>>> # host server.fw.ichabodcrane.org
>>> server.fw.ichabodcrane.org has address 34.236.200.223
>>>
>>>
>>> I also checked /etc/resolv.conf for the DNS resolvers and checked the
>>> FQDN resolving against both of those resolvers. The answer was the same as
>>> above.
>>>
>>> So as far as I can tell, the targeted URL is accessible from the CLI on
>>> that host, but Xymon thinks it isn't. Any ideas?
>>>
>>>
>>>
>>> Jaime Kikpole
>>>
>>> Director of Technology
>>> Ichabod Crane Central School District
>>> (XXX) XXX-XXXX, x5425
>>>
>>>
>>> On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <user-f26e06d1e992@xymon.invalid> wrote:
>>>
>>>> The SSL and http tests are not directly connected. The http retrieve a
>>>> page, which will return red if not found, and the SSL retrieve the SSL
>>>> certificate. So the SSL can be green and the http red and vis versa.
>>>>
>>>> On Fri, 21 Mar 2025, 10:18 Nicola, <user-2c63804b6921@xymon.invalid> wrote:
>>>>
>>>>> The sslcert detail oage is green because it was last time it was
>>>>> retrieved.
>>>>> There should be a timestamp for the last update
>>>>>
>>>>> Nicola
>>>>> Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" <
>>>>> xymon@xymon.com> ha scritto:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> If I remember correctly: If the http test is red because Xymon cannot
>>>>>> connect to the server on port 443, then Xymon will never retrieve the SSL
>>>>>> certificate (no connection = no SSL data). This will cause the sslcert
>>>>>> status to be purple, because it cannot be updated.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Med venlig hilsen / Best regards
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Henrik Juul Størner*
>>>>>> Senior IT Arkitekt - IT-sikkerhed
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Fra:* Josh Luthman <user-4c45a83f15cb@xymon.invalid>
>>>>>> *Sendt:* 20. marts 2025 21:56
>>>>>> *Til:* Jaime Kikpole <user-fb200a07d7dc@xymon.invalid>
>>>>>> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <
>>>>>> user-0608abae5e7c@xymon.invalid>
>>>>>> *Emne:* [Xymon] Re: SSL stuck on purple
>>>>>>
>>>>>>
>>>>>>
>>>>>> OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun
>>>>>> links eller vedhæftede filer, hvis du mener, de er OK.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Why is the http test red?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <
>>>>>> user-fb200a07d7dc@xymon.invalid> wrote:
>>>>>>
>>>>>> From hosts.cfg, with redactions between < and >:
>>>>>>
>>>>>> <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https://
>>>>>> <hostname>.ichabodcrane.org noconn
>>>>>>
>>>>>>
>>>>>>
>>>>>> The ghosts report shows no clients.
>>>>>>
>>>>>>
>>>>>>
>>>>>> The hostname definitely resolves (tested on the VM with the xymon
>>>>>> server) to the IP address. The purple alert is in a column labeled
>>>>>> "sslcert" and the red is in "http". The data in the page with the details
>>>>>> about sslcert shows the correct dates and a green alert. The data in the
>>>>>> page with the http details shows the message "SSL error" after the
>>>>>> https://.... URL. When viewing that URL, it redirects path on the
>>>>>> same server. This hasn't given me trouble over the last several years and
>>>>>> the only thing that changed on the monitored system is that its IP address
>>>>>> changed at the VM hosting service.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Jaime Kikpole*
>>>>>>
>>>>>> *Director of Technology*
>>>>>> *Ichabod Crane Central School District*
>>>>>> (XXX) XXX-XXXX, x5425
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <
>>>>>> user-4c45a83f15cb@xymon.invalid> wrote:
>>>>>>
>>>>>> The IP in the first column is used in the event the name can't be
>>>>>> resolved (or # testip is appended).
>>>>>>
>>>>>>
>>>>>>
>>>>>> The purple test means there is no report for the test (for half an
>>>>>> hour I think?) In this case you're testing from the Xymon Server so it is
>>>>>> responsible for doing the test.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you certain the name of the host matches? Are there any ghosts?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you looking at the "sslcert" column? What does your hosts.cfg
>>>>>> line look like - does it have an https url?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <
>>>>>> xymon@xymon.com> wrote:
>>>>>>
>>>>>> Thanks for the ideas. That gave me more to try and I made some
>>>>>> progress, but not much.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I discovered that hosts.cfg still had the old IP address. I updated
>>>>>> that, but there was no change. I restarted the xymon server process and
>>>>>> there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there
>>>>>> was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or
>>>>>> something like that?
>>>>>>
>>>>>>
>>>>>>
>>>>>> I see what I would expect when using "curl https://<FQDN>" and the
>>>>>> logs only show one interesting line within 2025, so I suspect that it
>>>>>> all comes down to that one hosts.cfg line with the old IP address.
>>>>>>
>>>>>>
>>>>>>
>>>>>> When I used "grep <server> * | grep 2025 | less", I see the following
>>>>>> line and then everything else was from notifications.log about emailing my
>>>>>> department's outages notification email address.
>>>>>>
>>>>>>
>>>>>>
>>>>>> history.log:2025-03-18 14:54:12.042214 Will not update
>>>>>> /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color
>>>>>> unchanged (purple)
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think? Should I just deal with the loss of historical
>>>>>> data and drop the host by FQDN?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Jaime Kikpole*
>>>>>>
>>>>>> *Director of Technology*
>>>>>> *Ichabod Crane Central School District*
>>>>>> (XXX) XXX-XXXX, x5425
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <user-0608abae5e7c@xymon.invalid>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20>
>>>>>>
>>>>>> On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime
>>>>>> Kikpole via Xymon <xymon@xymon.com> wrote:
>>>>>>
>>>>>> I have a Xymon install that was monitoring (among other things) an
>>>>>> HTTPS site. That site moved to a new IP and while I did update the public
>>>>>> DNS records it was a few days (or maybe two weeks?) before I remembered to
>>>>>> update the internal DNS records. By then, Xymon was reporting red for
>>>>>> HTTPS and purple for SSL.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I've corrected the DNS records and the VM running Xymon is resolving
>>>>>> it correctly, but now it is still showing the same red and purple alerts.
>>>>>> Every now and then, they'll switch to green for a little while and then
>>>>>> back to red and purple. I'm not sure why.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Xymon's web interface shows that the certificate is valid for more
>>>>>> than 2 months. So that isn't it. I've restarted the Xymon server process
>>>>>> and even the whole OS, but this issue remains.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Any ideas on what I could check?
>>>>>>
>>>>>>
>>>>>>
>>>>>> First, review the content of the https status page. There may be
>>>>>> non-obvious clues here. There's a difference between "connection refused"
>>>>>> and "connection timed out" and this can lead you in different directions to
>>>>>> identify the cause. For example, if you have a content check configured
>>>>>> ("cont=..." in hosts.cfg), which is failing due to new content on the new
>>>>>> webserver, this will be explained on the status page.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Second, I would check that your Xymon server can actually connect to
>>>>>> the new IP address on port 443. I usually use "telnet <IP> <port>" and see
>>>>>> if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc
>>>>>> works too. For an HTTPS website (as distinct from HTTP) I sometimes use an
>>>>>> openssl command to see if the TLS/SSL interaction works, which tells me
>>>>>> that there's no issue on the webserver itself, something like "openssl
>>>>>> s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget
>>>>>> to fetch a webpage, but if you have a proxy set anywhere, this may give you
>>>>>> an invalid result.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thirdly, I would check the xymonnet.log file for any indications of
>>>>>> problems, which might suggest a cause. Having xymonnet in debug mode might
>>>>>> help (adding "--debug" into the CMD line of the [xymonnet] block in
>>>>>> tasks.cfg). Alternatively, you could run xymonnet directly yourself, and
>>>>>> look at the output. Something like this: xymoncmd xymonnet --no-update
>>>>>> --debug <hostname>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Good luck with finding the problem.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Jeremy
>>>>>>
>>>>>> xymon@xymon.com
>>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>>
>>>>>> xymon@xymon.com
>>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>>
>>>>>> xymon@xymon.com
>>>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>>>
>>>> xymon@xymon.com
>>> To unsubscribe send an email to xymon-leave@xymon.com
>>>
>>
>>
>> --
>> Matthew Goebel : m <user-74d13dabeb26@xymon.invalid>user-9607b53e8435@xymon.invalid : Unix
>> Jockey @ EMU : Hail Eris
>> Neo-Student, Net Lurker, Donut consumer, and procrastinating medher...
>> "Always with the negative waves, Moriarty" - Oddball
>> "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
>>
>>
>>
>
xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com