Formatting errors on log files

Hmm. Does the data after the corrupted lines appear to match the remaining data for the server in question? From the sample below it seems not (as I believe this is reported in alphabetical order), which might indicate indicate a broader memory corruption issue going on within xymond_client, where it's somehow losing track of the end or garbling the data in the buffer being used for holding status output. If it's causing a false positive, then it's not merely the final output that's the problem, but something occurring earlier in processing.

What OS+distro is the server running on?
Any chance you might be able to run xymond_client in debug mode for a bit while this is occurring?

-jc


On 10/17/2016 7:56 AM, Greg Krpan wrote:

Hi JC-

Thanks for the response.

I am using Xymon 4.3.27 currently.  The raw client data looks fine- there are no corrupted lines and no added brackets or special characters that I can see.  This only occurs on the status pages.

The server has been running since May, and this particular problem started at the end of Sept., after running Windows Update on my servers, but as both Windows and Linux clients are showing the behavior, I have ruled out the updates as the issue.

I have tried restarting the service with no effect on behavior and there is nothing in the log files that show a problem that I can see.  The level of false positives due to formatting errors has remained relatively consistent, and tends to be limited to the PROCS (Win, Linux) and SVCS (Win only) tests, but occasionally will see the same error occurring on the DISK and CPU tests, although that is significantly less frequent, and is not across all configured machines.  The PROCS/SVCS tests are showing random errors on one machine or another approximately every 5 minutes.

Thanks
Greg.

On Fri, Oct 14, 2016 at 6:52 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid <mailto:user-87556346d4af@xymon.invalid>> wrote:


    On Fri, October 14, 2016 3:52 pm, Greg Krpan wrote:

Recently, my monitoring has been generating frequent errors that are
false,
due to improper formatting,  It is happening on both Windows and

    Linux

clients.  I've included an example of how the tests are sending
    data back
to the xymon server.  I have not made any changes to my client
    or server
configurations.  Has anyone else been experiencing this
    behavior, or know
of a fix?

Greg.

Name                                StartupType Status
DisplayName
AeLookupSvc                         manual  stopped
Application Experience
ALG                                 manual  stopped
Application Layer Gateway Service
AppIDSvc                            manual  stopped
Application Identity
Appinfo                             manual  stopped
Application Information
AppMgmt                             manual  stopped
Application Management
AppReadiness                        manual  stopped        App
Readiness
AppXSvc                             manual  stopped        AppX
Deployment Service (AppXSVC)
AudioEndpointBuilder                manual
toppe]        Windows Audio Endpoint Builder
Audiosrv                            manual  stopped        Windows
Audio
BBWin                               automatic started        Big
Brother Xymon Client
BFE                                 automatic started        Base
Filtering Engine
BITS                                automatic started
Background Intelligent Transfer Serv
ce
BrokerInfrastructure   ]            automatic started
Background Tasks Infrastructure Service
Browser                             disabled  stopped           Computer
Browser
CcmExec                             automatic started        SMS

    Agent

Host
CertPropSvc                         manual  started
Certificate Propagation
CmRcService                         disabled  stopped
Configuration Manager Remote Control
COMSysApp                           manual
started        COM+ Sys]
m Application
CryptSvc]
                         ]
utomatic    started        Cr]
tographic Services
DcomLaunch   ]
                     automatic    sta]
ed        DCOM Serv]
 Process Launcher
defra]svc                           manual  stopped        Optimize
drives
DeviceAssociationService            manual  stopped        Device
Association Service

    Hi Greg,

    Is there anything unusual about the process names on the lines
    immediately
    before the corruption? There's a known issue in that lines
    starting with a
    bracket will cause missing data, and this can happen more
    frequently on
    Windows servers just by virtue of some of the data that's coming
    across,
    but that doesn't appear to be causing this specific issue.


    Can you confirm which version of Xymon server you're using? Do you
    see the
    same corruption in the "raw" Client Data for the affected servers,
    or is
    it only occurring on the status pages?

    Also -- anything unusual in the log files? Has this problem been
    constant
    since it started, or is it getting worse? Does restarting the xymon
    service fix it (temporarily)?


    Regards,
    -jc


-- 
In honor of those who lost their lives exploring the final frontier:
Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins White II, Roger Bruce Chaffee
Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R. Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald E. McNair, Gregory B. Jarvis, Sharon Christa McAuliffe
Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D. Husband, William C. McCool, Michael P. Anderson, Kalpana Chawla, David M. Brown, Laurel Blair Salton Clark, Ilan Ramon