Xymon Mailing List Archive search

Xymon 4.3.13: HTTPS check issues

10 messages in this thread

list Mark Felder · Thu, 09 Jan 2014 08:09:20 -0600 ·
Hello,

I just started testing of Xymon 4.3.13 before putting it in the FreeBSD
ports tree and noticed an issue with a couple servers we are monitoring.
The HTTPS check is going red and the SSL is purple. Xymonnet shows this
error:

Error output:
Unspecified SSL error in SSL_connect to https (47873/tcp) on host
66.170.1.47: error:14077458:SSL
routines:SSL23_GET_SERVER_HELLO:reason(1112)
Unspecified SSL error in SSL_connect to https (47873/tcp) on host
66.170.1.28: error:14077458:SSL
routines:SSL23_GET_SERVER_HELLO:reason(1112)

Any clue what is causing it?


Xymon 4.3.13 on FreeBSD 9.1
OpenSSL 0.9.8q 2 Dec 2010


Thanks
list Mark Felder · Thu, 09 Jan 2014 08:23:50 -0600 ·
I may have located the issue. This is making sense -- Henrik enabled SNI
support between 4.3.12 and 4.3.13

https://rt.openssl.org/Ticket/Display.html?id=3038&user=guest&pass=guest

Anyone using OpenSSL 0.9.8 could hit this issue.
list Mark Felder · Thu, 09 Jan 2014 08:37:16 -0600 ·
I confirmed that building Xymon 4.3.13 against OpenSSL 1.0.1e 11 Feb
2013 fixes my previous issues. Those two servers are no longer showing
any issues.

However, I have different issue now that seems to be on a group of
similarly configured servers with self signed certificates:

Error output:
Unspecified SSL error in SSL_connect to 58148/tcp on host 66.170.1.42:
error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal
error
Unspecified SSL error in SSL_connect to 64288/tcp on host 66.170.1.43:
error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal
error
Unspecified SSL error in SSL_connect to 64288/tcp on host 66.170.1.44:
error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal
error
Unspecified SSL error in SSL_connect to 64288/tcp on host 66.170.1.46:
error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal
error
list Mark Felder · Thu, 09 Jan 2014 08:46:54 -0600 ·
quoted from Mark Felder
On Thu, Jan 9, 2014, at 8:37, Mark Felder wrote:
I confirmed that building Xymon 4.3.13 against OpenSSL 1.0.1e 11 Feb
2013 fixes my previous issues. Those two servers are no longer showing
any issues.

However, I have different issue now that seems to be on a group of
similarly configured servers with self signed certificates:

Error output:
Unspecified SSL error in SSL_connect to 58148/tcp on host 66.170.1.42:
error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal
error
Unspecified SSL error in SSL_connect to 64288/tcp on host 66.170.1.43:
error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal
error
Unspecified SSL error in SSL_connect to 64288/tcp on host 66.170.1.44:
error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal
error
Unspecified SSL error in SSL_connect to 64288/tcp on host 66.170.1.46:
error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal
error
From someone commenting on an Ubuntu bug report containing this error
message:

"The issue is actually with certain SSLv3 servers that don't understand  
the TLSv1.1 handshake and are closing the connection. This unfortunately
can't be fixed on the client without disabling TLSv1.1, or forcing an 
SSLv3 connection."

I may be able to fix the CipherSuites on all but one of the remaining
affected servers to work around this issue. However, this is not ideal;
these are basically "appliances" where I am unsure of the consequences
of changing away from the vendor defaults. (I definitely use modern
ciphers on my normal webservers) 

I'm not comfortable with pushing this update into the FreeBSD ports tree
at this time; there's too much potential for headaches. The SNI support
a great feature but it seems there are some very rough edges that have
not been discovered until now.
list Henrik Størner · Thu, 09 Jan 2014 22:20:28 +0100 ·
Den 09-01-2014 15:46, Mark Felder skrev:

[snip problem report]
quoted from Mark Felder
I'm not comfortable with pushing this update into the FreeBSD ports tree
at this time; there's too much potential for headaches. The SNI support
a great feature but it seems there are some very rough edges that have
not been discovered until now.
I am not quite sure what the best way forward is. As I understand your analysis (and I am very grateful that you've taken the time to investigate it!), then this really is a server-side problem that we need to provide a workaround for.

Making SNI configurable at compile-time is simple, but rather heavy-handed. I can certainly imagine situations where you want to do SNI on some servers, but have one or two where it breaks.

So it must be configurable at least per hosts.cfg-entry. Should we default SNI support to off (which it has been until now) and require everyone to explicitly enable it if they need it? Or should it default to ON and have people with the problem-servers turn it off themselves?

Defaulting to "off" would be the easiest for all users, since everything will work the way it did before 4.3.13. On the other hand, it doesn't seem right to have a perfectly valid protocol disabled, just because of some random crappiness.


I lean towards defaulting it to ON, and then asking admins to add a flag to explicitly disable SNI on those hosts that need it. As attached. Any comments?


Regards,
Henrik
Attachments (1)
list Ryan Novosielski · Thu, 9 Jan 2014 20:25:46 -0500 ·
quoted from Henrik Størner
On Jan 9, 2014, at 4:20 PM, Henrik Størner <user-ce4a2c883f75@xymon.invalid> wrote:
I lean towards defaulting it to ON, and then asking admins to add a flag 
to explicitly disable SNI on those hosts that need it. As attached. Any 
comments?
I think the answer probably depends to a certain extent on the prevalence of the problem. Will most servers that are not making use of SNI work properly or not? If it’s a corner case, it’s probably fine. But if it’s a very popular configuration, it might make sense for it to default to off.

____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
|| \\UTGERS      |---------------------*O*---------------------
||_// Biomedical | Ryan Novosielski - Sr. Systems Programmer
|| \\ and Health | user-46c89e614701@xymon.invalid - 973/972.0922 (2x0922)
||  \\  Sciences | OIT/EI-Academic Svcs. - ADMC 450, Newark
list Mark Felder · Sat, 11 Jan 2014 11:44:08 -0600 ·
quoted from Henrik Størner

On Thu, Jan 9, 2014, at 15:20, Henrik Størner wrote:
Den 09-01-2014 15:46, Mark Felder skrev:

[snip problem report]
I'm not comfortable with pushing this update into the FreeBSD ports tree
at this time; there's too much potential for headaches. The SNI support
a great feature but it seems there are some very rough edges that have
not been discovered until now.
I am not quite sure what the best way forward is. As I understand your analysis (and I am very grateful that you've taken the time to investigate it!), then this really is a server-side problem that we need to provide a workaround for.

Making SNI configurable at compile-time is simple, but rather heavy-handed. I can certainly imagine situations where you want to do SNI on some servers, but have one or two where it breaks.

So it must be configurable at least per hosts.cfg-entry. Should we default SNI support to off (which it has been until now) and require everyone to explicitly enable it if they need it? Or should it default to ON and have people with the problem-servers turn it off themselves?

Defaulting to "off" would be the easiest for all users, since everything will work the way it did before 4.3.13. On the other hand, it doesn't seem right to have a perfectly valid protocol disabled, just because of some random crappiness.


I lean towards defaulting it to ON, and then asking admins to add a flag to explicitly disable SNI on those hosts that need it. As attached. Any comments?
Well I think there are a few levels of problems here. Consumers of Xymon
may have wildly different experiences based on which version of OpenSSL
their OS is providing or the package maintainer has targeted.

My own experience:

- 0.9.8: SNI was only causing issues with two servers. I almost feel
like it would be better if SNI was off by default
-1.0.0+: SNI issue was fixed, but the HTTPS check was not gracefully
falling back to SSLv3. If that was fixed, it could certainly be on by
default.

The difference is that I can better control the experience of FreeBSD
Xymon users. I have the ability to dictate that the next update pulls in
OpenSSL 1.0.0+. I think you're going to find a lot of Linux
distributions still running OpenSSL 0.9.8, which may be a problem.

I think the safe solution everywhere is "off by default", and further
testing of the HTTPS checking code with OpenSSL 1.0+ against servers
that don't support the latest TLS, or maybe not even TLS at all -- just
SSLv3. You're going to have users with appliances that can't be upgraded
but they still should be able to get monitored.
list Henrik Størner · Mon, 13 Jan 2014 09:29:50 +0100 ·
quoted from Mark Felder
Den 11.01.2014 18:44, Mark Felder skrev:
I think the safe solution everywhere is "off by default", and further
testing of the HTTPS checking code with OpenSSL 1.0+ against servers
that don't support the latest TLS, or maybe not even TLS at all -- just
SSLv3. You're going to have users with appliances that can't be upgraded
but they still should be able to get monitored.
Just to finish this thread: In 4.3.14 I have implemented a global option for xymonnet "--sni=[on|off]" to globally enable/disable SNI for SSL tests. Default is OFF. In addition there are two now tags for hosts.cfg, "sni" and "nosni" so regardless of the global option you can override it per host.

I think that is the best way to avoid unnecessary surprises when upgrading, while still making SNI available for those who need it.


Regards,
Henrik
list Mark Felder · Tue, 14 Jan 2014 10:44:56 -0600 ·

On Mon, Jan 13, 2014, at 2:29, user-ce4a2c883f75@xymon.invalid wrote:
Den 11.01.2014 18:44, Mark Felder skrev:
I think the safe solution everywhere is "off by default", and further
testing of the HTTPS checking code with OpenSSL 1.0+ against servers
that don't support the latest TLS, or maybe not even TLS at all -- > just
SSLv3. You're going to have users with appliances that can't be > upgraded
but they still should be able to get monitored.
Just to finish this thread: In 4.3.14 I have implemented a global option for xymonnet "--sni=[on|off]" to globally enable/disable SNI for SSL tests. Default is OFF. In addition there are two now tags for hosts.cfg, "sni" and "nosni" so regardless of the global option you can override it per host.

I think that is the best way to avoid unnecessary surprises when upgrading, while still making SNI available for those who need it.

Thanks Henrik!
list Ryan Novosielski · Tue, 14 Jan 2014 11:45:32 -0500 ·
Agreed, sounds great. Thank you once again for this great software!
signature

--
____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
 || \\UTGERS      |---------------------*O*---------------------
 ||_// Biomedical | Ryan Novosielski - Sr. Systems Programmer
 || \\ and Health | user-46c89e614701@xymon.invalid - 973/972.0922 (2x0922)
 ||  \\  Sciences | OIT/EI-Academic Svcs. - ADMC 450, Newark
      `'

quoted from Mark Felder
From: Xymon [xymon-bounces at xymon.com] On Behalf Of Mark Felder [user-db141d317836@xymon.invalid]
Sent: Tuesday, January 14, 2014 11:44 AM
To: xymon at xymon.com
Subject: Re: [Xymon] Xymon 4.3.13: HTTPS check issues

On Mon, Jan 13, 2014, at 2:29, user-ce4a2c883f75@xymon.invalid wrote:
Den 11.01.2014 18:44, Mark Felder skrev:
I think the safe solution everywhere is "off by default", and further
testing of the HTTPS checking code with OpenSSL 1.0+ against servers
that don't support the latest TLS, or maybe not even TLS at all --
just
SSLv3. You're going to have users with appliances that can't be
upgraded
but they still should be able to get monitored.
Just to finish this thread: In 4.3.14 I have implemented a global
option for xymonnet "--sni=[on|off]" to globally enable/disable SNI for
SSL tests. Default is OFF. In addition there are two now tags for
hosts.cfg, "sni" and "nosni" so regardless of the global option you can
override it per host.

I think that is the best way to avoid unnecessary surprises when
upgrading, while still making SNI available for those who need it.

Thanks Henrik!