So the memory usage on the machine is fairly high. This system is a VM,
and was built with only 2GB of memory, of which about 1.8GB is in use. I
have a maintenance window coming up this week where I am going to increase
the available memory to the server, but I will also try to inject the 2
MALLOC debugging that you suggested into things as well to watch for
additional issues as time goes on. Hopefully that can help to identify
where the problem lies and I can be a testbed to help determine a
resolution.
Greg.
On Mon, Oct 24, 2016 at 11:57 PM, Japheth Cleaver <user-87556346d4af@xymon.invalid>
wrote:
On 10/24/2016 10:59 AM, Greg Krpan wrote:
I haven't noticed any errors through xymond_client or from debug mode.
After running the xymoncmd line above I get the following:
./xymoncmd xymon 127.0.0.1 "xymondlog wstc-lr0dhcp1.svcs"
wstc-lr0dhcp1|svcs|red||1477331698|1477331698|1477333498|0|
0|15.1.160.11|460159|||Y|
red Mon Oct 24 11:55:55 2016 - Services NOT ok
&red BBWin: No matching service - want started/automatic
&green DHCPServer is started/automatic - want started/automatic
&green McAfeeFramework is started/automatic - want started/automatic
&green McShield is started/automatic - want started/automatic
&green McTaskManager is started/automatic - want started/automatic
&green VMTools is started/automatic - want started/automatic
Name StartupType Status DisplayName
AeLookupSvc manual stopped
Application Experience
ALG manual stopped
Application Layer Gateway Service
AppIDSvc manual stopped
Application Identity
Appinfo manual
sta]ted Application Information
AppMgmt manual stopped
Application Management
AppReadiness manual stopped App
Readiness
AppXSvc manual stopped AppX
Deployment Service (AppXSVC)
AudioEndpointBuilder manual stopped Windows
Audio Endpoint Builder
Audiosrv ]
manual st]
ped Windows Audio
]BWin automatic started Big
Brother Xymon Client
BFE automatic started Base
Filtering Engine
*snip*
Thanks; that confirms that the issue involved xymond_client or xymond, and
isn't related to the web display.
Looking through the changes from 4.3.25 to 4.3.27, it's hard to see what
might be causing this issue.
Is there any chance you're under a significant memory pressure on this
machine? Would you be able to add some glibc debugging at all?
If so, would you be able to add an:
(export) MALLOC_CHECK_=3
(export) MALLOC_PERTURB_=1
... into the environment? This might help trigger a memory issue that
could otherwise go unnoticed.
Alternatively, the next step might be to downgrade to 4.3.25 and see if
that fixes the problem (if so, that really indicated there's a specific
hidden issue here). Also, it might be interesting to see if the el7
Terabithia RPMs show the same problem for you. There was a significant
increase in lookup/buffer debugging in xymond_client in there that's also
in the 4.x-master branch but isn't in 4.3.x when compiled from source.
Regards,
-jc
--
In honor of those who lost their lives exploring the final frontier:
Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins White
II, Roger Bruce Chaffee
Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R.
Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald E.
McNair, Gregory B. Jarvis, Sharon Christa McAuliffe
Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D. Husband,
William C. McCool, Michael P. Anderson, Kalpana Chawla, David M. Brown,
Laurel Blair Salton Clark, Ilan Ramon