Xymon notifcations disappear due to base64 encoding 🔗 link

On Wed, Feb 4, 2015, at 19:05, Benjamin Smith wrote:

We've wasted time playing with a wrapper for the mail command that would
strip 
out the "\r" in the alerts but this has, so far, proven fruitless.
Messages 
sent via the Sprint SMS gateway go through just fine. Wondering if
anybody else 
has seen this issue, or has any idea what the best way to handle it might
be? 

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Benjamin Smith
Sent: Wednesday, February 04, 2015 19:06
To: xymon at xymon.com
Subject: [Xymon] Xymon notifcations disappear due to base64 encoding

This morning, one of our servers was down and we (the admins) didn't immediately know because of a long chain of events. After spending the morning drilling down step by step, the issue was that Verizon's SMS email gateway doesn't accept base64 encoded messages, and the message being sent was base64 encoded because of <CR> (aka "\r" or ASCII 13) in the message. 
We could reproduce this issue with the following PHP script used at the command line as a Verizon customer. I don't know if this is a recent change at Verizon, and don't really have a way to find out. 

// GENERAL ENVIRONMENT CentOS 6, recently updated. Postfix is the MTA PHP5 

// THIS WON'T WORK Note the "\r" that causes postfix to base64 encode the message. file.php's contents, 1 line: <?php  echo "Test of bad message\r\n"; ?>
$ php file.php | mail -s test user-7e577af8398b@xymon.invalid 

// THIS WORKS FINE, Note the lack of "\r", postfix sends this as text/plain, 7 bit encoding. file.php's contents, 1 line: <?php  echo "Test of bad message\n"; ?>
$ php file.php | mail -s test user-7e577af8398b@xymon.invalid 


We've wasted time playing with a wrapper for the mail command that would strip out the "\r" in the alerts but this has, so far, proven fruitless. Messages sent via the Sprint SMS gateway go through just fine. Wondering if anybody else has seen this issue, or has any idea what the best way to handle it might be? 
Thanks, 
Ben

g


-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Ribeiro, Glauber
Sent: Thursday, February 05, 2015 09:37
To: Benjamin Smith; xymon at xymon.com
Subject: Re: [Xymon] Xymon notifcations disappear due to base64 encoding

I've always had this problem. I ended up setting the mail client in xymon server (via the configuration file) to be a script that pipes the messages through tr to delete the CRs, then pipes then to mail.

g

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Benjamin Smith
Sent: Wednesday, February 04, 2015 19:06
To: xymon at xymon.com
Subject: [Xymon] Xymon notifcations disappear due to base64 encoding

This morning, one of our servers was down and we (the admins) didn't immediately know because of a long chain of events. After spending the morning drilling down step by step, the issue was that Verizon's SMS email gateway doesn't accept base64 encoded messages, and the message being sent was base64 encoded because of <CR> (aka "\r" or ASCII 13) in the message. 
We could reproduce this issue with the following PHP script used at the command line as a Verizon customer. I don't know if this is a recent change at Verizon, and don't really have a way to find out. 

// GENERAL ENVIRONMENT CentOS 6, recently updated. Postfix is the MTA PHP5 

// THIS WON'T WORK Note the "\r" that causes postfix to base64 encode the message. file.php's contents, 1 line: <?php  echo "Test of bad message\r\n"; ?>
$ php file.php | mail -s test user-7e577af8398b@xymon.invalid 

// THIS WORKS FINE, Note the lack of "\r", postfix sends this as text/plain, 7 bit encoding. file.php's contents, 1 line: <?php  echo "Test of bad message\n"; ?>
$ php file.php | mail -s test user-7e577af8398b@xymon.invalid 


We've wasted time playing with a wrapper for the mail command that would strip out the "\r" in the alerts but this has, so far, proven fruitless. Messages sent via the Sprint SMS gateway go through just fine. Wondering if anybody else has seen this issue, or has any idea what the best way to handle it might be? 
Thanks, 
Ben

On 02/05/2015 08:40 AM, Mark Felder wrote:

I warn you this (probably) isn't exactly what you're looking for, but
I've stopped using the SMS email gateways due to reliability issues and
when things are really going bad -- rate limiting. Instead I've moved to
sending my notifications through Pushover.net which sends (fast!!) push
notifications to phones.

You can send messages two ways: their email gateway, or write a
notification script that uses their REST api. It's very simple. I wrote
this in a few minutes and haven't looked back.

On Thursday, February 05, 2015 07:40:13 AM Mark Felder wrote:

On Wed, Feb 4, 2015, at 19:05, Benjamin Smith wrote:

We've wasted time playing with a wrapper for the mail command that would
strip
out the "\r" in the alerts but this has, so far, proven fruitless.
Messages
sent via the Sprint SMS gateway go through just fine. Wondering if
anybody else
has seen this issue, or has any idea what the best way to handle it might
be?

I warn you this (probably) isn't exactly what you're looking for, but
I've stopped using the SMS email gateways due to reliability issues and
when things are really going bad -- rate limiting. Instead I've moved to
sending my notifications through Pushover.net which sends (fast!!) push
notifications to phones.

You can send messages two ways: their email gateway, or write a
notification script that uses their REST api. It's very simple. I wrote
this in a few minutes and haven't looked back.

g

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Benjamin Smith
Sent: Thursday, February 05, 2015 11:59
To: xymon at xymon.com
Subject: Re: [Xymon] Xymon notifcations disappear due to base64 encoding

On Thursday, February 05, 2015 07:40:13 AM Mark Felder wrote:

On Wed, Feb 4, 2015, at 19:05, Benjamin Smith wrote:

We've wasted time playing with a wrapper for the mail command that would
strip
out the "\r" in the alerts but this has, so far, proven fruitless.
Messages
sent via the Sprint SMS gateway go through just fine. Wondering if
anybody else
has seen this issue, or has any idea what the best way to handle it might
be?

I warn you this (probably) isn't exactly what you're looking for, but
I've stopped using the SMS email gateways due to reliability issues and
when things are really going bad -- rate limiting. Instead I've moved to
sending my notifications through Pushover.net which sends (fast!!) push
notifications to phones.

You can send messages two ways: their email gateway, or write a
notification script that uses their REST api. It's very simple. I wrote
this in a few minutes and haven't looked back.

I will be evaluating this alongside the other fix in this thread. 

Glauber presented a simple solution that solves our immediate problem. 
Otherwise, we've had no issue with SMS messages for years. (Verizon) I'm 
testing both solutions. 

Thanks! 

Ben

On Thu, Feb 5, 2015, at 11:47, Bill Arlofski wrote:

On 02/05/2015 08:40 AM, Mark Felder wrote:

I warn you this (probably) isn't exactly what you're looking for, but
I've stopped using the SMS email gateways due to reliability issues and
when things are really going bad -- rate limiting. Instead I've moved to
sending my notifications through Pushover.net which sends (fast!!) push
notifications to phones.

You can send messages two ways: their email gateway, or write a
notification script that uses their REST api. It's very simple. I wrote
this in a few minutes and haven't looked back.

Hi Mark... Thanks for that recommendation re: pushover.net

I have suffered for _years_ with the AT&T (now Frontier) email-to-sms
gateway.
Using the AT&T email gateway, Xymon notifications get mixed in with other
txt
messages because each message sent via their gateway is

From: ${SomeEverIncreasingNumber}

So I can not set a filter on a phone number, or username and have a
different
notification tone for Xymon messages.

Pushover currently has a limited number of notification tones, but that
is OK
because I can have my xymon alert tell Pushover which to send so I can
always
know if I got a txt or a Xymon alert now.

Still waiting on my first Xymon->Pushover alert, but this looks promising
to
me and my sanity.  lol

On Thu, Feb 5, 2015, at 10:23, Ribeiro, Glauber wrote:

More details:

In xymonserver.cfg, I have

#MAILC="mail"   
MAILC="/usr/local/hobbit/server/mail_nocr.sh"    # Command used to send
an e-mail with no subject
MAIL="$MAILC -s"                                # Command used to send an
e-mail with a subject

The mail_nocr.sh script is like this:

#! /bin/bash
# 20130104 - glauber - get rid of CR in emails
exec /usr/bin/tr -d '\015' | /bin/mail "$@"

g

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Mark Felder
Sent: Friday, February 06, 2015 09:21
To: xymon at xymon.com
Subject: Re: [Xymon] Xymon notifcations disappear due to base64 encoding


On Thu, Feb 5, 2015, at 10:23, Ribeiro, Glauber wrote:

More details:

In xymonserver.cfg, I have

#MAILC="mail"   
MAILC="/usr/local/hobbit/server/mail_nocr.sh"    # Command used to send
an e-mail with no subject
MAIL="$MAILC -s"                                # Command used to send an
e-mail with a subject

The mail_nocr.sh script is like this:

#! /bin/bash
# 20130104 - glauber - get rid of CR in emails
exec /usr/bin/tr -d '\015' | /bin/mail "$@"

Let's kill this in the code instead of resorting to hacks like this.

One or multiple of these instances of \r is bound to be the cause:

~/svn/xymon-code/xymond > grep '\\r' *
xymond.c:       if (sourcename) cause = strtok(NULL, "\r\n");
xymond.c:                       txtstart += strspn(txtstart, " \t\r\n");
xymond.c:       tok = strtok(buf, " \t\r\n");
xymond.c:       if (tok) tok = strtok(NULL, " \t\r\n");
xymond.c:               tok = strtok(NULL, " \t\r\n");
xymond.c:               ehost = bhost + strcspn(bhost, " \t\r\n");
xymond.c:                       bid = msg->buf + strcspn(msg->buf, "
\t\r\n"); bid += strspn(bid, " \t");
xymond.c:                       eid = bid + strcspn(bid, " \t\r\n");
xymond.c:               p = strtok(p, " \t\r\n");
xymond.c:               p = strtok(p, " \t\r\n");
xymond.c:               p = msg->buf + strcspn(msg->buf, "\r\n");
xymond.c:               if ((*p == '\r') || (*p == '\n')) {
xymond_channel.c:               hostname = inbuf + strcspn(inbuf,
"/|\r\n");
xymond_channel.c:                       hostend = hostname +
strcspn(hostname, "|\r\n");
xymond_client.c:        p = s + strcspn(s, "\r\n");
xymond_locator.c:       const char *delims = "|\r\n\t ";


JC, Henrik -- any ideas?

user-59d088777028@xymon.invalid> wrote:

More details:

In xymonserver.cfg, I have

#MAILC="mail"
MAILC="/usr/local/hobbit/server/mail_nocr.sh"    # Command used to send an
e-mail with no subject
MAIL="$MAILC -s"                                # Command used to send an
e-mail with a subject

The mail_nocr.sh script is like this:

#! /bin/bash
# 20130104 - glauber - get rid of CR in emails
exec /usr/bin/tr -d '\015' | /bin/mail "$@"


That's all, folks!


The occasional CR in the emails not only mess up the SMS gateways, they
also make Exchange turn a simple message into an email with a binary
attachment.

g


-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Ribeiro, Glauber
Sent: Thursday, February 05, 2015 09:37
To: Benjamin Smith; xymon at xymon.com
Subject: Re: [Xymon] Xymon notifcations disappear due to base64 encoding

I've always had this problem. I ended up setting the mail client in xymon
server (via the configuration file) to be a script that pipes the messages
through tr to delete the CRs, then pipes then to mail.

g

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Benjamin Smith
Sent: Wednesday, February 04, 2015 19:06
To: xymon at xymon.com
Subject: [Xymon] Xymon notifcations disappear due to base64 encoding

This morning, one of our servers was down and we (the admins) didn't
immediately know because of a long chain of events. After spending the
morning
drilling down step by step, the issue was that Verizon's SMS email gateway
doesn't accept base64 encoded messages, and the message being sent was
base64
encoded because of <CR> (aka "\r" or ASCII 13) in the message.

We could reproduce this issue with the following PHP script used at the
command line as a Verizon customer. I don't know if this is a recent
change at
Verizon, and don't really have a way to find out.


// GENERAL ENVIRONMENT
CentOS 6, recently updated.
Postfix is the MTA
PHP5


// THIS WON'T WORK Note the "\r" that causes postfix to base64 encode the
message.
file.php's contents, 1 line:
<?php  echo "Test of bad message\r\n"; ?>
$ php file.php | mail -s test user-7e577af8398b@xymon.invalid


// THIS WORKS FINE, Note the lack of "\r", postfix sends this as
text/plain, 7
bit encoding.
file.php's contents, 1 line:
<?php  echo "Test of bad message\n"; ?>
$ php file.php | mail -s test user-7e577af8398b@xymon.invalid


We've wasted time playing with a wrapper for the mail command that would
strip
out the "\r" in the alerts but this has, so far, proven fruitless. Messages
sent via the Sprint SMS gateway go through just fine. Wondering if anybody
else
has seen this issue, or has any idea what the best way to handle it might
be?

Thanks,

Ben

On Fri, Feb 6, 2015, at 10:26, Ribeiro, Glauber wrote:

I'm not sure the problem is internally in xymon, but rather in the
diversity of sources of the text that ends up in email bodies.

I think xymon should do some simple sanitization of alert texts (maybe a
configuration item), to eliminate CRs (for benefit of email), but also
characters that are not safe for HTML (for benefit of the browser). I
wonder is a crafty person would be able to trick xymon into putting
Javascript in an alert, by manipulating error messages.

On Fri, February 6, 2015 9:08 am, Mark Felder wrote:

On Fri, Feb 6, 2015, at 10:26, Ribeiro, Glauber wrote:

I'm not sure the problem is internally in xymon, but rather in the
diversity of sources of the text that ends up in email bodies.

Internally Xymon is using \r (CR) in a lot of places. The info that is
put in alerts is fairly straightforward -- just pieced together from
various ENVs related to the event. As far as I can tell there is no data
sourced directly from the clients in the alerts unless you include
BBALPHAMSG in your alert message, which is going to be a huge amount of
information.

A simple sanitation routine in do_alert.c would probably take care of
this, but if we can remove the \r from being used internally the problem
should solve itself. I think a sanitation routine is probably more
treating the symptom and not the problem, as well as unlikely to solve
all possibilities of dangerous data which shouldn't be getting this far
in the first place.

I think xymon should do some simple sanitization of alert texts (maybe a
configuration item), to eliminate CRs (for benefit of email), but also
characters that are not safe for HTML (for benefit of the browser). I
wonder is a crafty person would be able to trick xymon into putting
Javascript in an alert, by manipulating error messages.

If you use the full BBALPHAMSG in your alert message and the client
sends some nasty data I am certain it could do some unsavory things. I
also bet you could -- from a compromised client -- send a crafted alert
message to a new or existing column and attack the admin when he views
the alert in his browser. Your saving grace might be the fact that alert
messages are limited to 4096 bytes:

#define MAX_ALERTMSG_SCRIPTS 4096

Hopefully Xymon 5 brings us encrypted and authenticated transport
between the client and server as that will help prevent this type of
attack, as well as protect your sensitive info in transit :-)

Hopefully Xymon 5 brings us encrypted and authenticated transport
between the client and server as that will help prevent this type of
attack, as well as protect your sensitive info in transit :-)

This is really the solution -- end-to-end encoding using key trust; right
now the most client security that you have is IP-based. But even if your
transport mechanism is over an stunnel, you're really still at the mercy
of the original source. A local user could execute a script placing a
specially crafted message in $0, which would show up in the 'ps' output
and might survive <PRE> wrapping in the 'procs' test to cause a browser
problem, for example.