problem with receiving bbwin data

Phil,

Thanks again. Yes, snare / winevtmsgs.pl looks like a good work around for this issue....

Let me know if you try it out in anger. I'm not aware of anyone who
actually uses it, and I've probably made some updates. I've also got a
mass of localised rules for event log filtering since the severity
levels, etc can be pretty bogus - errors that can be safely ignored and
"informational" messages that you really need to be alerted on...

David.

cheers, P


From: David Baldwin
Sent: Thursday, 11 July 2013 2:45 PM
To: Phil Crooker
Cc: xymon at xymon.com
Subject: Re: [Xymon] problem with receiving bbwin data

Phil,

Thanks, David.... Yes, I don't have clientversion.cfg either - I tried
putting in "0.13" in a file of that name in \etc, the client says it
reads it and no more error, but who knows what this does or doesn't do...

Anyway, on to the rest. As I described in the previous email, I did
let the clientlocal.cfg update successfully. But as soon as I add the
msgs.dll agent bbwin still ignores the maxdata parameter, tries to
send everything and xymond drops the connection before completion
(this latest time at 31MB).

Do you have this working? Does it actually reduce the size of the msgs
sent?

I have it working, but I don't actually use BBWin for event log
reporting - I forward all my event logs using Snare to a central syslog
server and have my own checking against that server-side - see
http://xymonton.org/monitors:winevtmsgs.pl

My client-local.cfg for Windows clients includes the following to filter
out everything client-side. You may be able to play with the ignore
rules to tune it a bit more to your needs while still reducing the size
of the report. By checking the BBWin\tmp\msg.<clientname>.txt file you
should see what is taking up all the space.

log:eventlog_security:10240
ignore .*
ignore .
eventlog:security:10240
ignore handle
ignore .*
ignore .
eventlog:System:10240
ignore .*
ignore .
eventlog:application:10240
ignore .*
ignore .
eventlog:directory service:10240
ignore .*
ignore .

David.

thanks, Phil


*From:* David Baldwin
*Sent:* Thursday, 11 July 2013 11:28 AM
*To:* Phil Crooker
*Cc:* xymon at xymon.com
*Subject:* Re: [Xymon] problem with receiving bbwin data

Phil,

Thanks for the reply.

Before continuing, what is supposed to be in logfetch.status? and in
clientversion.cfg?

logfetch.status will possibly only exist if you are checking any
regular logfiles - e.g. if client-local.cfg contains something like:

[win32]
log:c:\WINDOWS\WindowsUpdate.log:20480

clientlocal.cfg will contain the relevant section similar to your 4
line example below. clientversion.cfg doesn't exist on my systems

As BBWin runs, check the size of clientlocal.cfg - if it gets wiped,
try saving the contents back again. Also, updating BBWin.cfg should
cause BBWin to run immediately without restarting the service. I
haven't played around with commenting out msgs.dll

David.

So, I have one server with the msgs.dll out of the config. It has
been running for many days with the client updating fine. I redid the
xymon client-local.cfg, stopped bbwin, deleted \tmp\clientlocal.cfg
and restarted it (as it wouldn't update), the \tmp\clientlocal.cfg
now reads what the client-local has:

    eventlog:application:1024
    ignore BigBrotherXymonClient
    eventlog:system:1025
    eventlog:security:10240

I let that run a cycle, it still updates OK. I then put in the
msgs.dll statement in BBWin.cfg; that registers and BBWin.log shows:

2013/07/11 10:42:55 [DEBUG]: [bbwinupdate]: can't open C:\Program
Files (x86)\BBWin\etc\clientversion.cfg : The system cannot find the
file specified.
2013/07/11 10:42:55 [DEBUG]: [cpu]: cpu review started
2013/07/11 10:42:55 [DEBUG]: [cpu]: cpu review ended
2013/07/11 10:42:55 [DEBUG]: [disk]: drive string A:\
2013/07/11 10:42:55 [DEBUG]: [disk]: drive string C:\
2013/07/11 10:42:55 [DEBUG]: [disk]: drive string D:\
2013/07/11 10:42:55 [DEBUG]: [disk]: drive string E:\
2013/07/11 10:42:55 [DEBUG]: [filesystem]: start
AgentFileSystem::InitCentralMode
2013/07/11 10:42:55 [DEBUG]: [filesystem]: checking file C:\Program
Files (x86)\BBWin\tmp\clientlocal.cfg
2013/07/11 10:42:55 [DEBUG]: [filesystem]: reading file C:\Program
Files (x86)\BBWin\tmp\clientlocal.cfg
2013/07/11 10:42:55 [DEBUG]: [msgs]: start AgentMsgs::InitCentralMode
2013/07/11 10:42:55 [DEBUG]: [msgs]: checking file C:\Program Files
(x86)\BBWin\tmp\clientlocal.cfg
2013/07/11 10:42:55 [DEBUG]: [msgs]: reading file C:\Program Files
(x86)\BBWin\tmp\clientlocal.cfg
2013/07/11 10:42:55 [DEBUG]: [msgs]: will use maxdata size : 1024
2013/07/11 10:42:55 [DEBUG]: [msgs]: will ignore : BigBrotherXymonClient
2013/07/11 10:42:55 [DEBUG]: [msgs]: will use maxdata size : 1025
2013/07/11 10:42:55 [DEBUG]: [msgs]: will use maxdata size : 10240

and no more updates. The .\tmp\msgs.hostname.txt file is 88MB.

thanks, Phil

*From:* Scot Kreienkamp
*Sent:* Wednesday, 10 July 2013 10:19 PM
*To:* David Baldwin; Phil Crooker
*Cc:* xymon at xymon.com
*Subject:* RE: [Xymon] problem with receiving bbwin data


If the eventlog data being transmitted is too large to transmit until
the clientlocal.cfg is sent back and populated in BBWin, then why not
disable the eventlog client in BBWin until it successfully sends 2-3
status messages?  By that time the clientlocal.cfg will be populated
on the client and you can enable the eventlog reader again and
everything should work fine.


*Scot Kreienkamp *


*From:*Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *David
Baldwin
*Sent:* Wednesday, July 10, 2013 4:19 AM
*To:* Phil Crooker
*Cc:* xymon at xymon.com
*Subject:* Re: [Xymon] problem with receiving bbwin data


Phil,

Just found a fix that works for me :)

    I've been having difficulties setting up the bbwin client (ver
    0.13 going to xymon 4.3.10 server) running on win7 sp1, 2008r2
    and win2012 boxes in central mode. I've had zero response from
    the bbwin-help forum for this, so please bear with me.


    For background: I can't get the bbwin client to stop sending all
    the logs, it ignores any maxdata parameters I use. Eg:


         eventlog:system:1024


    It sends everything anyway. If I use log:system:1024, the bbwin
    client throws an error that it can't find the system log file. It
    does find the file without the maxdata parameter.


One problem with the settings in client-local.cfg on the xymon server
is that they only propagate to the client after a successful sending
of a client message. The mechanics is that after the client message
has been sent, BBWin does a shutdown on the write socket and then
reads from the read socket until EOF. If the write socket is never
shutdown because the server breaks the connection due to flooding,
then the client-local.cfg section will never be sent to the client.

I've previously attempted to manually create C:\Program
Files(x86)\BBWin\tmp\clientlocal.cfg to trick it, but with limited
success. Just played around with it again - with client reporting to
my primary server I just got it going - previously I was reporting to
2 servers. The trick seems to be in having logfetch.status also
created. I then modified my BBWin.cfg back to 2 servers and it broke
again. Maybe the 2nd server is responding differently. It is
undefined which server's clientlocal.cfg to believe anyway :) Swapped
the order of the servers around and it seems stable.

If the server just discarded all the flooding data beyond MAXMSG_*
and then returned the client config section anyway maybe it could be
made to work, but there may be good reasons why that would not be
sensible.

This aside, I do need to get this working as I have to start
monitoring 10 new windows servers and have to monitor the event logs,
so I can't just stop sending them as has been suggested. Yes I could
do some powershell scripts and send them via bbwincmd but the bbwin
client is made for this task.....


So, looking at this from the other side, the xymon server appears to
be resetting the session after about 22MB of data has been sent (I
know, this is ludicrous, but it is windows). Nothing in the xymond
logs (except for the occasional data flooding error ("1st line
client", always)); on the client side it reports it can't send the
data to the xymon server.


Depending on what auditting you have enabled, 22MB is very easy to
achieve! Turn on Security event logging including success and failure
and it's almost guaranteed :)

David.

I've set the MAXMSG_* to quite silly levels:


# ipcs

------ Shared Memory Segments --------

key               shmid         owner      perms      bytes
 nattch     status

0x01034be7 16908288   xymon      600        102400000  2


0x02034be7 16941057   xymon      600        102400000  2


0x03034be7 16973826   xymon      600        102400000  2


0x04034be7 17006595   xymon      600        102400000  2


0x05034be7 17039364   xymon      600        262144     1


0x06034be7 17072133   xymon      600        32768      1


0x07034be7 17104902   xymon      600        102400000  2


0x08034be7 17137671   xymon      600        102400000  2


0x09034be7 17170440   xymon      600        131072     1


Anything up to and including this size has no effect on the problem.


Looking at the tcpdump stream, the bbwin client sends data normally
with regular ACKs from xymond till around that 22MB mark then xymond
responds with a FIN packet, then with RST packets and the session
shuts down. Nothing in the packets themselves indicate what the
problem is.


If anyone can help with this, please, it would be great.


thanks, Phil

--
David Baldwin - Senior Systems Administrator (Datacentres + Networks)
Information and Communication Technology Services
Australian Sports Commission          http://ausport.gov.au
Tel 02 62147830 Fax 02 62141830       PO Box 176 Belconnen ACT 2616
user-cbbf693f2c89@xymon.invalid          Leverrier Street Bruce ACT 2617


Keep up to date with what's happening in Australian sport visit http://www.ausport.gov.au

This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.

Please consider the environment before printing this e-mail

This message from ORIX Australia may contain confidential and/or privileged information. If you are not the intended recipient, any use, disclosure or copying of this message (or of any attachments to it) is not authorised. If you have received this message in error, please notify the sender immediately and delete the message and any attachments from your system. Please inform the sender if you do not wish to receive further communications by email. ORIX handles personal information according to a Privacy Policy that is consistent with the National Privacy Principles. Please let us know if you would like a copy.

It is also available at www.orix.com.au<http://www.orix.com.au>;

-- 
David Baldwin - Senior Systems Administrator (Datacentres + Networks)
Information and Communication Technology Services
Australian Sports Commission          http://ausport.gov.au
Tel 02 62147830 Fax 02 62141830       PO Box 176 Belconnen ACT 2616
user-cbbf693f2c89@xymon.invalid          Leverrier Street Bruce ACT 2617