Clientlog in UTF8

6 messages in this thread

list Luboš Pavlíček · Wed, 22 Mar 2017 09:56:26 +0100 ·

Hi,

We use powershell client on computers with localized windows (Czech
language). EventLog messages are in Czech with non ASCII characters. I
modified the winpsclient to send clientlog in UTF8. Messages are displayed
on xymon web correctly, I was able filter messages, e.g.

LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red

Exists some pitfalls, why not send clientlog in UTF8, but convert into US-
ASCII?

Luboš Pavlíček

list Zak Beck · Wed, 22 Mar 2017 10:03:58 +0000 ·

Hi Luboš

I think you’re talking about the ASCII encoding in function XymonSend.

I don’t know the reason for choosing ASCII encoding, that code was written before I was involved.

If the server can accept UTF-8 on all platforms then there is no reason why it should not be changed – maybe one of the server developers (Japheth ?) could comment?

Zak

▸ quoted from Luboš Pavlíček

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Luboš Pavlícek
Sent: 22 March 2017 08:56
To: xymon at xymon.com
Subject: [Xymon] Clientlog in UTF8

Hi,
We use powershell client on computers with localized windows (Czech language). EventLog messages are in Czech with non ASCII characters. I modified the winpsclient to send clientlog in UTF8. Messages are displayed on xymon web correctly, I was able filter messages, e.g.

LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red


Exists some pitfalls, why not send clientlog in UTF8, but convert into US-ASCII?
Luboš Pavlíček


This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.

www.accenture.com

list David Baldwin · Thu, 23 Mar 2017 09:11:22 +1100 ·

▸ quoted from Zak Beck

On 22/3/17 9:03 pm, user-aada0fa38bf8@xymon.invalid wrote:

Hi Luboš

 
I think you’re talking about the ASCII encoding in function XymonSend.

 
I don’t know the reason for choosing ASCII encoding, that code was
written before I was involved.

I wrote that code. From memory, the xymon server could only handle the
clientlog message in ASCII at the time, and forcing the translation to
ASCII was a key part to getting it working. If the constraint of ASCII
clientlog message has changed, then by all means feel free to convert to
UTF8, however some users may still be running old servers that can only
handle ASCII, so best to leave it configurable.

You'd need to test that all client-based tests (DISK, SVC, PORT, PROC,
etc) are still working not just eventlog message matching.

▸ quoted from Zak Beck

 
If the server can accept UTF-8 on all platforms then there is no
reason why it should not be changed – maybe one of the server
developers (Japheth ?) could comment?

As above, need to verify server can handle UTF8. I suspect the clientlog
message must be all one encoding - can't just do eventlog section in
UTF8 and the rest in ASCII.

David.

Zak

 
*From:*Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *Luboš

▸ quoted from Zak Beck

Pavlícek
*Sent:* 22 March 2017 08:56
*To:* xymon at xymon.com
*Subject:* [Xymon] Clientlog in UTF8

 
Hi,

We use powershell client on computers with localized windows (Czech
language). EventLog messages are in Czech with non ASCII characters. I
modified the winpsclient to send clientlog in UTF8. Messages are
displayed on xymon web correctly, I was able filter messages, e.g.

LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red

That generally happens when someone leaves a remote session logged on
and then changes their password elsewhere. ;)

▸ quoted from Zak Beck

Exists some pitfalls, why not send clientlog in UTF8, but convert into
US-ASCII?

Luboš Pavlíček


This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise confidential information. If you
have received it in error, please notify the sender immediately and
delete the original. Any other use of the e-mail by you is prohibited.
Where allowed by local law, electronic communications with Accenture
and its affiliates, including e-mail and instant messaging (including
content), may be scanned by our systems for the purposes of
information security and assessment of internal compliance with
Accenture policy.

www.accenture.com

--


David Baldwin - Senior Systems Administrator (Datacentres + Networks)
Digital Information Management and Technology
Australian Sports Commission          http://ausport.gov.au
Tel 02 62147830 Fax 02 62141830       PO Box 176 Belconnen ACT 2616
user-cbbf693f2c89@xymon.invalid          1 Leverrier Street Bruce ACT 2617
Our Values: RESPECT + INTEGRITY + TEAMWORK + EXCELLENCE


Keep up to date with what's happening in Australian sport visit http://www.ausport.gov.au

This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.

list Zak Beck · Thu, 23 Mar 2017 09:11:54 +0000 ·

Hi David


Ø  some users may still be running old servers that can only handle ASCII, so best to leave it configurable

Thanks for your input – I was thinking the same as the above.

Zak

▸ quoted from David Baldwin

From: David Baldwin [mailto:user-cbbf693f2c89@xymon.invalid]
Sent: 22 March 2017 22:11
To: Beck, Zak <user-aada0fa38bf8@xymon.invalid>; user-09f9d95aed63@xymon.invalid; xymon at xymon.com
Subject: Re: [Xymon] Clientlog in UTF8

On 22/3/17 9:03 pm, user-aada0fa38bf8@xymon.invalid<mailto:user-aada0fa38bf8@xymon.invalid> wrote:
Hi Luboš

I think you’re talking about the ASCII encoding in function XymonSend.

I don’t know the reason for choosing ASCII encoding, that code was written before I was involved.
I wrote that code. From memory, the xymon server could only handle the clientlog message in ASCII at the time, and forcing the translation to ASCII was a key part to getting it working. If the constraint of ASCII clientlog message has changed, then by all means feel free to convert to UTF8, however some users may still be running old servers that can only handle ASCII, so best to leave it configurable.

You'd need to test that all client-based tests (DISK, SVC, PORT, PROC, etc) are still working not just eventlog message matching.

If the server can accept UTF-8 on all platforms then there is no reason why it should not be changed – maybe one of the server developers (Japheth ?) could comment?

As above, need to verify server can handle UTF8. I suspect the clientlog message must be all one encoding - can't just do eventlog section in UTF8 and the rest in ASCII.

David.

Zak

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Luboš Pavlícek
Sent: 22 March 2017 08:56
To: xymon at xymon.com<mailto:xymon at xymon.com>
Subject: [Xymon] Clientlog in UTF8

Hi,
We use powershell client on computers with localized windows (Czech language). EventLog messages are in Czech with non ASCII characters. I modified the winpsclient to send clientlog in UTF8. Messages are displayed on xymon web correctly, I was able filter messages, e.g.

LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red
That generally happens when someone leaves a remote session logged on and then changes their password elsewhere. ;)

Exists some pitfalls, why not send clientlog in UTF8, but convert into US-ASCII?
Luboš Pavlíček

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.


www.accenture.com<http://www.accenture.com>;


Xymon at xymon.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.xymon.com_mailman_listinfo_xymon&d=DwMDaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=56bdErr67jxvsqigxSp9k6anqyhtUKraJdOpHMsJlUI&s=oKULmdiOfZqPSYRktcv-fYhNtexl4bhKk6jqDsvvrT0&e=>;

▸ quoted from David Baldwin



--

David Baldwin - Senior Systems Administrator (Datacentres + Networks)

Digital Information Management and Technology


Australian Sports Commission          http://ausport.gov.au<https://urldefense.proofpoint.com/v2/url?u=http-3A__ausport.gov.au&d=DwMDaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=56bdErr67jxvsqigxSp9k6anqyhtUKraJdOpHMsJlUI&s=GL4fN3VjIF0jbEp6AIu7FU7RCXu1inLlRw7TQPsuUP8&e=>;

Tel 02 62147830 Fax 02 62141830       PO Box 176 Belconnen ACT 2616

user-cbbf693f2c89@xymon.invalid<mailto:user-cbbf693f2c89@xymon.invalid>          1 Leverrier Street Bruce ACT 2617

Our Values: RESPECT + INTEGRITY + TEAMWORK + EXCELLENCE

Keep up to date with what's happening in Australian sport visit www.ausport.gov.au<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ausport.gov.au&d=DwMDaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=56bdErr67jxvsqigxSp9k6anqyhtUKraJdOpHMsJlUI&s=XVgugJurmNjsulR2NzV0Jh7d_xw-_bOiWAWtWA5DxYo&e=>;

▸ quoted from David Baldwin


This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.

list Luboš Pavlíček · Thu, 23 Mar 2017 17:59:52 +0100 ·

Hi,

Thank you for your answers.

2017-03-22 23:11 GMT+01:00 David Baldwin <user-cbbf693f2c89@xymon.invalid>:

▸ quoted from Zak Beck

On 22/3/17 9:03 pm, user-aada0fa38bf8@xymon.invalid wrote:

Hi Luboš


I think you’re talking about the ASCII encoding in function XymonSend.


I don’t know the reason for choosing ASCII encoding, that code was written
before I was involved.

I wrote that code. From memory, the xymon server could only handle the
clientlog message in ASCII at the time, and forcing the translation to
ASCII was a key part to getting it working. If the constraint of ASCII
clientlog message has changed, then by all means feel free to convert to
UTF8, however some users may still be running old servers that can only
handle ASCII, so best to leave it configurable.

You'd need to test that all client-based tests (DISK, SVC, PORT, PROC,
etc) are still working not just eventlog message matching.

I try multiple tests with client messages in UTF8. First, I will describe
our environment. We use two Xymon servers (versions 4.3.25, 4.3.27), both
are running on Linux boxes (CentOS, Debian) with LANG=en_US.UTF-8. Apache
web server has set the default encoding to UTF-8.


Client tests DISK, UP,MEMHYS,MEMSWAP and MEMACT are without problems,
because corresponding sections from clientlog are in English.
CLOCK test are working correctly, because first line "epoch: <number>" from
[clock] section isn't localized.


Tests LOG, PROC, SVC and PORT successfully filter non-ASCII strings.
Examples:

    LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red
    PROC "%Nový textový dokument"
    PORT "LOCAL=%(:445)$" state=NASLOUCHÁNÍ TEXT=sdílení TRACK
    PORT "REMOTE=%([.:]445)$" state=NAVÁZÁNO min=1 max=10 color=yellow
TRACK=smb_servers TEXT=SMB_servers
    SVC  "Xymončesky" status=started


FILE and DIR: I successfully test file and directory with non-ASCII chars
in filename. File client-local.cfg on xymon server contains next lines:

file:d:\temp\čeština.txt
dir:d:\temp\český

and file analysis.cfg contains tests:

FILE "D:\temp\čeština.txt"      TYPE=file SIZE>100 MTIME<180 yellow TRACK
DIR  "D:\temp\český"  SIZE<1000 TRACK

Bad filenames are in graph: "D:<tab>emp\čeština.txt" and
"D:<tab>temp\český". But this si the slash problem.


Xymon server don't create some graph, e.g. netstat, user. This is a result of
localized messages in output from windows programs netstat, ipconfig or
qwinsta. See http://lists.xymon.com/archive/2014-January/038946.html and
following discussion.


Attached patch adds local setting XymonAcceptUTF8. If value set to 1, then
all client messages are converted to UTF8 before sending to xymon servers.
If value set to 0 or not specified, then script removes diacritics
(accents) from message, converts non-break spaces to normal spaces
(question marks in eventlog summary) and converts to ASCII before sending
to xymon server. The same setting is applied to all specified xymon servers.


Luboš Pavlíček

Attachments (1)

attachment.obj application/octet-stream · 1.7 KB

list Zak Beck · Fri, 24 Mar 2017 08:54:31 +0000 ·

Hi Luboš

Thanks for your patch.

I will test it in my environment – I don’t expect any issues, it looks sensible – and then update the documentation and release – this will probably be next week.

The UTF8 option will be “at your own risk” as I don’t have any non-EN platforms to test on – as you have said, some graphs and maybe tests won’t work due to localisation, but I think these wouldn’t work even with ASCII encoding – e.g. netstat would return localised messages regardless of UTF8/ASCII.

Thanks

Zak

▸ quoted from Luboš Pavlíček


From: Luboš Pavlíček [mailto:user-09f9d95aed63@xymon.invalid]
Sent: 23 March 2017 17:00
To: David Baldwin <user-cbbf693f2c89@xymon.invalid>
Cc: Beck, Zak <user-aada0fa38bf8@xymon.invalid>; xymon at xymon.com
Subject: Re: [Xymon] Clientlog in UTF8

Hi,

Thank you for your answers.


2017-03-22 23:11 GMT+01:00 David Baldwin <user-cbbf693f2c89@xymon.invalid<mailto:user-cbbf693f2c89@xymon.invalid>>:

▸ quoted from Luboš Pavlíček

On 22/3/17 9:03 pm, user-aada0fa38bf8@xymon.invalid<mailto:user-aada0fa38bf8@xymon.invalid> wrote:
Hi Luboš

I think you’re talking about the ASCII encoding in function XymonSend.

I don’t know the reason for choosing ASCII encoding, that code was written before I was involved.
I wrote that code. From memory, the xymon server could only handle the clientlog message in ASCII at the time, and forcing the translation to ASCII was a key part to getting it working. If the constraint of ASCII clientlog message has changed, then by all means feel free to convert to UTF8, however some users may still be running old servers that can only handle ASCII, so best to leave it configurable.

You'd need to test that all client-based tests (DISK, SVC, PORT, PROC, etc) are still working not just eventlog message matching.

I try multiple tests with client messages in UTF8. First, I will describe our environment. We use two Xymon servers (versions 4.3.25, 4.3.27), both are running on Linux boxes (CentOS, Debian) with LANG=en_US.UTF-8. Apache web server has set the default encoding to UTF-8.

Client tests DISK, UP,MEMHYS,MEMSWAP and MEMACT are without problems, because corresponding sections from clientlog are in English.
CLOCK test are working correctly, because first line "epoch: <number>" from [clock] section isn't localized.

Tests LOG, PROC, SVC and PORT successfully filter non-ASCII strings. Examples:

    LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red
    PROC "%Nový textový dokument"
    PORT "LOCAL=%(:445)$" state=NASLOUCHÁNÍ TEXT=sdílení TRACK
    PORT "REMOTE=%([.:]445)$" state=NAVÁZÁNO min=1 max=10 color=yellow TRACK=smb_servers TEXT=SMB_servers
    SVC  "Xymončesky" status=started

FILE and DIR: I successfully test file and directory with non-ASCII chars in filename. File client-local.cfg on xymon server contains next lines:

file:d:\temp\čeština.txt
dir:d:\temp\český

and file analysis.cfg contains tests:

FILE "D:\temp\čeština.txt"      TYPE=file SIZE>100 MTIME<180 yellow TRACK
DIR  "D:\temp\český"  SIZE<1000 TRACK
Bad filenames are in graph: "D:<tab>emp\čeština.txt" and "D:<tab>temp\český". But this si the slash problem.


Xymon server don't create some graph, e.g. netstat, user. This is a result of localized messages in output from windows programs netstat, ipconfig or qwinsta. See http://lists.xymon.com/archive/2014-January/038946.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.xymon.com_archive_2014-2DJanuary_038946.html&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=Sa_ZLbF2_4uGXImwd3VJ8TZDXxPc282dRhwIiR9qnZE&s=_TGPeWA6RPMQx1TMlXKM-GfN6ER62LiEG0VfTAILFqM&e=>; and following discussion.

▸ quoted from Zak Beck


Attached patch adds local setting XymonAcceptUTF8. If value set to 1, then all client messages are converted to UTF8 before sending to xymon servers. If value set to 0 or not specified, then script removes diacritics (accents) from message, converts non-break spaces to normal spaces (question marks in eventlog summary) and converts to ASCII before sending to xymon server. The same setting is applied to all specified xymon servers.

Luboš Pavlíček


This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.

www.accenture.com

Clientlog in UTF8 🔗 link

Clientlog in UTF8