Yes, I remember our conversation – very difficult for me to replicate.
I can definitely make this optional, and will most likely look to make
this default to off.
Zak
*From:* SebA <user-4631430d620a@xymon.invalid>
*Sent:* Wednesday, 30 January 2019 14:57
*To:* Beck, Zak <user-aada0fa38bf8@xymon.invalid>
*Cc:* Xymon Mailing List <xymon at xymon.com>
*Subject:* Re: [External] [Xymon] Xymon WinPSclient performance
Good news and bad news. The good news is that enabling XymonAcceptUTF8
makes the cycle run a lot faster and seems to work fine for us, without
doubling the number of bytes sent in the report. I think the default
though should probably be not to change the behaviour compared with the
previous version (v2.19). Failing that, the default should probably be to
enable XymonAcceptUTF8 - the default should not be something that slows
down the cycle and uses lots of CPU time IMHO.
The bad news is I have found another thing to takes time: XymonDiskPart.
2019-01-30 13:39:13 XymonDiskPart start
2019-01-30 13:39:17 XymonDiskPart finished
I see this only runs every 'slow scan', but:
(a) this also takes 4 seconds to complete;
(b) I don't know what the use of it is, at least for most people, and it's
not documented in XymonPSClient.doc - this is a possible example of feature
bloat;
(c) it causes a bad memory leak in the Virtual Disk service (vds.exe) on
our failover cluster servers that have clustered disk.
Commenting out line 3875 in v2.28 r8017 has no negative consequences for
us and removes this vds.exe memory leak. I have compared patched and
unpatched processes running for several months on otherwise identical
servers. We have had a conversation about this, see here:
https://lists.xymon.com/archive/2018-March/045341.html
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.xymon.com_archive_2018-2DMarch_045341.html&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=Yihzb9N4fl-Dl1BgOxMCrJsBMmLNdwI2ItiW5rRVcV8&s=-52Qhrm1-bvmWiTrhmSxyNmQdwM04ZP_KbPkNkWSIiA&e=>;
But you didn't respond to that last message. Since the Virtual Disk
service was last restarted on the unpatched server on 30/11/18 (so in 2
months), it has leaked up to today nearly 900 MB RAM and 9 GB of virtual
memory.
Please can XymonDiskPart be made optional in the config?
We have another server (2012 R2, no failover cluster, no clustered disks)
that is using 1 GB RAM and 2 GB virtual memory in the Xymon powershell
process also (unpatched) v2.28 after 43 hours running. I haven't yet
investigated the cause, but I'll try replacing it with the patched version
without XymonDiskPart and see if that cures the memory leak.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 15:28, SebA <user-4631430d620a@xymon.invalid> wrote:
Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it
out, or change my first paragraph to put the emphasis on reducing CPU time
spent, but I forgot! I noticed that powershell (on some versions of the
PSclient) was using more CPU time than the process the server exists to
run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages
sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I
was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <user-aada0fa38bf8@xymon.invalid> wrote:
Hi
Yes, there is no option for no encoding – I think the closest option to
this is actually UTF-8 because it does not attempt to adjust the message
body (remove diacritics and 0xa0 spaces) before sending. That’s what is
happening between your two log messages. Personally I don’t see that
behaviour albeit with a 10x smaller data packet (I checked on several
servers):
2019-01-29 11:43:06 Using ASCII encoding
2019-01-29 11:43:06 Connecting to host x.x.x.x
2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class
Win32_Processor. Yes, it does take time. I have tried over the years to
reduce the amount of WMI calls (I think we’re down to 4) because they do
seem to take a lot of time and on occasion are unreliable. Unfortunately I
have not been able to find an alternative to this call. And I have 1000 VMs
which can have CPUs hot added without reboot, so there is use case for
checking every time. The result of the WMI call is already cached so we
could amend the script to optionally run only on slow scans (every 6 hours
by default).
It may be simpler to add regex processing for your filename issue, I can
take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re
suffering from feature bloat and then requesting more enhancements /
features 😉
Zak
*From:* Xymon <xymon-bounces at xymon.com> *On Behalf Of *SebA
*Sent:* Monday, 28 January 2019 20:16
*To:* Xymon Mailing List <xymon at xymon.com>
*Subject:* [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I
think it's started suffering from feature bloat now, illustrating the
importance of making each feature optional like BBWin did very effectively.
We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started
another thread about memory leaks a while back - I'll come back to this
issue another time) and more and more CPU time being used up during each
cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was
added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 user-aada0fa38bf8@xymon.invalid
2019-01-28 19:19:26 Sending to server
2019-01-28 19:19:26 Using ASCII encoding
2019-01-28 19:*20*:25 Connecting to host x.x.x.x <-- Yes, that
took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*.
2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's
rather a lot of data, but still it's not good.
2019-01-28 19:20:26 Received 130 bytes from server
Compare with:
myServer1 - : xymonclient.ps1 2.19 2016-12-28 user-aada0fa38bf8@xymon.invalid
2019-01-28 19:15:59 Sending to server
2019-01-28 19:15:59 Connecting to host x.x.x.x
2019-01-28 19:15:59 Sent 2006765 bytes to server
2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made
optional? In the latest version it is possible to send via UTF8, which was
new in v2.21, or convert to ASCII, but it's not possible via configuration
alone, I believe, to have the old behaviour (from v2.19) that worked fine
for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info
2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation
2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI)
2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores
2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI)
2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
*8 seconds* for that. Why does it need to do this every time? The
number of CPUs does not normally change without at least rebooting. It
could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able
to dynamically specify the filenames using special characters or something
for the date, e.g. &DD &MM &YY
We have files that rotate with new filesnames and if we could narrow down
the filenames to process it would speed up processing by several more
seconds.
E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg:
Instead of:
log:C:\Temp\myServerLog*.log:10240
If we could to this, it would mean we process a small fraction of the
number of files:
log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise confidential information. If you have
received it in error, please notify the sender immediately and delete the
original. Any other use of the e-mail by you is prohibited. Where allowed
by local law, electronic communications with Accenture and its affiliates,
including e-mail and instant messaging (including content), may be scanned
by our systems for the purposes of information security and assessment of
internal compliance with Accenture policy. Your privacy is important to us.
Accenture uses your personal data only in compliance with data protection
laws. For further information on how Accenture processes your personal
data, please see our privacy statement at
https://www.accenture.com/us-en/privacy-policy.
www.accenture.com