I've included an entire "SVCS" status below on a failed status screen.
As
you can see, it is random as to how and where the output corrupts,
On the Windows systems, I run BBWin for the client (version 0.13). I
should be able to put the xymond_client process into debug mode to
monitor
for a while as well.. The problem is less predominant on Linux clients
than on Windows, but it is occurring on both.
The server is running on CentOS 7 with current patches.
# uname -a
Linux ************************* 3.10.0-327.36.1.el7.x86_64 #1 SMP Sun
Sep
18 13:04:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linu
# cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)
[# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/";
BUG_REPORT_URL="https://bugs.centos.org/";
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Name StartupType Status
DisplayName
AeLookupSvc manual stopped
Application Experience
ALG manual stopped
Application Layer Gateway Service
AppHostSvc automatic started
Application Host Helper Service
AppIDSvc manual stopped
Application Identity
Appinfo manual stopped
Application I
forma]ion
AppMgmt manual stopped
Application Management
AppReadiness manual stopped App
Readiness
AppXSvc manual stopped AppX
Deployment Service (AppXSVC)
aspnet_state manual stopped ASP.NET
State Service
AudioEndpointBuilder man
al stopped Wi]dows Audio Endpoint Builder
Audiosrv manual stopped Windows
Audio
BBWin automatic started Big
Brother Xymon Client
BFE automatic started Base
Filtering Engine
BITS automatic started Back
round Intelligent Trans]
r Service
BrokerInfrastru]
ure automatic ]
started Background Tasks]
nfrastructure Service
Browser ]
]isabled stopped Computer Browser
CcmExec automatic started SMS
Agent Host
CertPropSvc manual started
Certificate Propagation
CmRcService disabled stopped
Configuration Manager Remote Control
COMSysApp manual started COM+
System Application
CryptSvc automatic started
Cryptographic Services
DcomLaunch automatic started DCOM
Server Process Launcher
defragsvc manual stopped Optimize
drives
DeviceAssociationService manual stopped Device
Association Service
DeviceInstall manual stopped Device
Install Service
Dhcp automatic started DHCP
Client
DiagTrack automatic started
Diagnostics Tracking Service
Dnscache automatic started DNS
Client
dot3svc manual stopped Wired
AutoConfig
DPS automatic started
Diagnostic Policy Service
DsmSvc manual started Device
Setup Manager
Eaphost manual stopped
Extensible Authentication Protocol
EFS manual stopped
Encrypting File System (EFS)
EventLog automatic started Windows
Event Log
EventSystem automatic started COM+
Event System
fdPHost manual stopped Function
Discovery Provider Host
FDResPub manual stopped Function
Discovery Resource Publication
FontCache automatic started Windows
Font Cache Service
gpsvc automatic started Group
Policy Client
hidserv manual stopped Human
Interface Device Service
hkmsvc manual stopped Health
Key and Certificate Management
IEEtwCollectorService manual stopped Internet
Explorer ETW Collector Service
IISADMIN automatic started IIS
Admin Service
IKEEXT automatic started IKE and
AuthIP IPsec Keying Modules
iphlpsvc automatic started IP
Helper
KeyIso manual started CNG Key
Isolation
KPSSVC manual stopped KDC
Proxy Server service (KPS)
KtmRm manual stopped KtmRm
for Distributed Transaction Coordinator
LanmanServer automatic started Server
LanmanWorkstation automatic started
Workstation
lltdsvc manual stopped
Link-Layer Topology Discovery Mapper
lmhosts automatic started TCP/IP
NetBIOS Helper
lpasvc manual stopped
Microsoft Policy Platform Local Authority
lppsvc manual stopped
Microsoft Policy Platform Processor
LSM automatic started Local
Session Manager
McAfeeFramework automatic started McAfee
Framework Service
McShield automatic started McAfee
McShield
McTaskManager automatic started McAfee
Task Manager
MMCSS manual stopped
Multimedia Class Scheduler
MpsSvc
automatic started ] Windows Firewall
MSDTC automatic started
Distributed Transaction Coordinator
MSiSCSI manual stopped
Microsoft iSCSI Initiator Service
msiserver manual stopped Windows
Installer
napagent manual stopped Network
Access Protection Agent
NcaSvc manual stopped Network
Connectivity Assistant
Netlogon automatic started Netlogon
Netman manual stopped Network
Connections
netprofm manual started Network
List Service
NetTcpPortSharing disabled stopped Net.Tcp
Port Sharing Service
NlaSvc automatic started Network
Location Awareness
nsi automatic started Network
Store Interface Service
PerfHost manual stopped
Performance Counter DLL Host
pla manual stopped
Performance Logs & Alerts
PlugPlay manual started Plug and
Play
PolicyAgent manual started IPsec
Policy Agent
Power automatic started Power
PrintNotify manual stopped Printer
Extensions and Notifications
ProfSvc automatic started User
Profile Service
QBCFMonitorService automatic started
QuickBooks Database Manager Service
QBFCService manual stopped Intuit
QuickBooks FCS
QuickBooksDB17 automatic started
QuickBooksDB17
RasAuto manual stopped Remote
Access Auto Connection Manager
RasMan manual stopped Remote
Access Connection Manager
RemoteAccess disabled stopped Routing
and Remote Access
RemoteRegistry automatic stopped Remote
Registry
RpcEptMapper automatic started RPC
Endpoint Mapper
RpcLocator manual stopped Remote
Procedure Call (RPC) Locator
RpcSs automatic started Remote
Procedure Call (RPC)
RSoPProv manual stopped
Resultant Set of Policy Provider
sacsvr manual stopped Special
Administration Console Helper
SamSs automatic started Security
Accounts Manager
SCardSvr disabled stopped Smart
Card
ScDeviceEnum manual stopped Smart
Card Device Enumeration Service
Schedule automatic started Task
Scheduler
SCPolicySvc manual stopped Smart
Card Removal Policy
seclogon manual stopped
Secondary Logon
SENS automatic started System
Event Notification Service
SessionEnv manual started Remote
Desktop Configuration
SharedAccess disabled stopped Internet
Connection Sharing (ICS)
ShellHWDetection automatic stopped Shell
Hardware Detection
smphost manual stopped
Microsoft Storage Spaces SMP
smstsmgr manual stopped
ConfigMgr Task Sequence Agent
SNMP automatic started SNMP
Service
SNMPTRAP automatic started SNMP
Trap
Spooler automatic started Print
Spooler
sppsvc automatic stopped Software
Protection
SSDPSRV disabled stopped SSDP
Discovery
SstpSvc manual stopped Secure
Socket Tunneling Protocol Service
svsvc manual stopped Spot
Verifier
swprv manual stopped
Microsoft Software Shadow Copy Provider
SysMain manual stopped
Superfetch
SystemEventsBroker automatic started System
Events Broker
TapiSrv manual stopped
Telephony
TermService manual started Remote
Desktop Services
Themes automatic started Themes
THREADORDER manual stopped Thread
Ordering Server
TieringEngineService manual stopped Storage
Tiers Management
TrkWks automatic started
Distributed Link Tracking Client
TrustedInstaller manual stopped Windows
Modules Installer
UALSVC automatic started User
Access Logging Service
UI0Detect manual stopped
Interactive Services Detection
UmRdpService manual started Remote
Desktop Services UserMode Port Redirector
upnphost disabled stopped UPnP
Device Host
VaultSvc manual stopped
Credential Manager
vds manual stopped Virtual
Disk
VGAuthService automatic started VMware
Alias Manager and Ticket Service
vmicguestinterface manual stopped Hyper-V
Guest Service Interface
vmicheartbeat manual stopped Hyper-V
Heartbeat Service
vmickvpexchange manual stopped Hyper-V
Data Exchange Service
vmicrdv manual stopped Hyper-V
Remote Desktop Virtualization Service
vmicshutdown manual stopped Hyper-V
Guest Shutdown Service
vmictimesync manual stopped Hyper-V
Time Synchronization Service
vmicvss manual stopped Hyper-V
Volume Shadow Copy Requestor
VMTools automatic started VMware
Tools
vmvss manual stopped VMware
Snapshot Provider
VSS manual stopped Volume
Shadow Copy
W32Time manual started Windows
Time
w3logsvc manual stopped W3C
Logging Service
W3SVC automatic started World
Wide Web Publishing Service
WAS manual started Windows
Process Activation Service
Wcmsvc automatic started Windows
Connection Manager
WcsPlugInService manual stopped Windows
Color System
WdiServiceHost manual stopped
Diagnostic Service Host
WdiSystemHost manual stopped
Diagnostic System Host
Wecsvc manual stopped Windows
Event Collector
WEPHOSTSVC manual stopped Windows
Encryption Provider Host Service
wercplsupport manual stopped Problem
Reports and Solutions Control Panel Support
WerSvc manual stopped Windows
Error Reporting Service
WinHttpAutoProxySvc manual started WinHTTP
Web Proxy Auto-Discovery Service
Winmgmt automatic started Windows
Management Instrumentation
WinRM automatic started Windows
Remote Management (WS-Management)
wmiApSrv manual stopped WMI
Performance Adapter
WPDBusEnum manual stopped Portable
Device Enumerator Service
WSService manual stopped Windows
Store Service (WSService)
wuauserv automatic started Windows
Update
wudfsvc manual stopped Windows
Driver Foundation - User-mode Driver Framework
On Mon, Oct 17, 2016 at 4:10 PM, Japheth Cleaver
<user-87556346d4af@xymon.invalid>
wrote:
Hmm. Does the data after the corrupted lines appear to match the
remaining data for the server in question? From the sample below it
seems
not (as I believe this is reported in alphabetical order), which might
indicate indicate a broader memory corruption issue going on within
xymond_client, where it's somehow losing track of the end or garbling
the
data in the buffer being used for holding status output. If it's
causing a
false positive, then it's not merely the final output that's the
problem,
but something occurring earlier in processing.
What OS+distro is the server running on?
Any chance you might be able to run xymond_client in debug mode for a
bit
while this is occurring?
-jc
On 10/17/2016 7:56 AM, Greg Krpan wrote:
Hi JC-
Thanks for the response.
I am using Xymon 4.3.27 currently. The raw client data looks fine-
there
are no corrupted lines and no added brackets or special characters that
I
can see. This only occurs on the status pages.
The server has been running since May, and this particular problem
started at the end of Sept., after running Windows Update on my
servers,
but as both Windows and Linux clients are showing the behavior, I have
ruled out the updates as the issue.
I have tried restarting the service with no effect on behavior and
there
is nothing in the log files that show a problem that I can see. The
level
of false positives due to formatting errors has remained relatively
consistent, and tends to be limited to the PROCS (Win, Linux) and SVCS
(Win
only) tests, but occasionally will see the same error occurring on the
DISK
and CPU tests, although that is significantly less frequent, and is not
across all configured machines. The PROCS/SVCS tests are showing
random
errors on one machine or another approximately every 5 minutes.
Thanks
Greg.
On Fri, Oct 14, 2016 at 6:52 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid>
wrote:
On Fri, October 14, 2016 3:52 pm, Greg Krpan wrote:
Recently, my monitoring has been generating frequent errors that are
false,
due to improper formatting, It is happening on both Windows and
Linux
clients. I've included an example of how the tests are sending data
back
to the xymon server. I have not made any changes to my client or
server
configurations. Has anyone else been experiencing this behavior, or
know
of a fix?
Greg.
Name StartupType Status
DisplayName
AeLookupSvc manual stopped
Application Experience
ALG manual stopped
Application Layer Gateway Service
AppIDSvc manual stopped
Application Identity
Appinfo manual stopped
Application Information
AppMgmt manual stopped
Application Management
AppReadiness manual stopped App
Readiness
AppXSvc manual stopped AppX
Deployment Service (AppXSVC)
AudioEndpointBuilder manual
toppe] Windows Audio Endpoint Builder
Audiosrv manual stopped
Windows
Audio
BBWin automatic started Big
Brother Xymon Client
BFE automatic started Base
Filtering Engine
BITS automatic started
Background Intelligent Transfer Serv
ce
BrokerInfrastructure ] automatic started
Background Tasks Infrastructure Service
Browser disabled stopped
Computer
Browser
CcmExec automatic started SMS
Agent
Host
CertPropSvc manual started
Certificate Propagation
CmRcService disabled stopped
Configuration Manager Remote Control
COMSysApp manual
started COM+ Sys]
m Application
CryptSvc]
]
utomatic started Cr]
tographic Services
DcomLaunch ]
automatic sta]
ed DCOM Serv]
Process Launcher
defra]svc manual stopped
Optimize
drives
DeviceAssociationService manual stopped
Device
Association Service
Hi Greg,
Is there anything unusual about the process names on the lines
immediately
before the corruption? There's a known issue in that lines starting
with
a
bracket will cause missing data, and this can happen more frequently
on
Windows servers just by virtue of some of the data that's coming
across,
but that doesn't appear to be causing this specific issue.
Can you confirm which version of Xymon server you're using? Do you see
the
same corruption in the "raw" Client Data for the affected servers, or
is
it only occurring on the status pages?
Also -- anything unusual in the log files? Has this problem been
constant
since it started, or is it getting worse? Does restarting the xymon
service fix it (temporarily)?
Regards,
-jc
--
In honor of those who lost their lives exploring the final frontier:
Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins
White II, Roger Bruce Chaffee
Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R.
Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald
E.
McNair, Gregory B. Jarvis, Sharon Christa McAuliffe
Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D.
Husband, William C. McCool, Michael P. Anderson, Kalpana Chawla, David
M.
Brown, Laurel Blair Salton Clark, Ilan Ramon
--
In honor of those who lost their lives exploring the final frontier:
Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins
White
II, Roger Bruce Chaffee
Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R.
Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald
E.
McNair, Gregory B. Jarvis, Sharon Christa McAuliffe
Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D.
Husband,
William C. McCool, Michael P. Anderson, Kalpana Chawla, David M. Brown,
Laurel Blair Salton Clark, Ilan Ramon