Purple storm

On 12-04-2012 05:28, Poppy, Ben wrote:

To be honest, I'm not sure what the cause is 100%.

Me neither.

The setup we have, should not have any dependencies on our DR site.
Our xymon servers at our primary site use DNS servers in our primary
site. They monitor a bunch of servers at our DR site, but the
dependency ends there (and all that should mean is the servers show
RED when DR site is down).

Aha - but you DO have tests in each setup that checks systems on the other site ? Would that happen to include any DNS or NTP checks ?

I suspect that you have each of your Xymon's setup to test availability of the DNS servers on both the primary and the DR site. That could be a problem.

Another bit of information, during this 5 hour outage, both of our
xymon servers went from showing properly (where DR servers were
showing RED conn as they weren't reachable, but the servers we
monitor in our primary site were up), to everything going purple in
conn (and other tests).. It would alternate back and forth over the
course of the outage (I didn't detect a regular timeframe of when it
switched from RED to PURPLE)..

The interesting thing is that they switch to purple, indicating that something is stalled.

I have seen something like this happen when we had a number of DNS checks in the Xymon servers, and network access to these failed (broken switch to a customer network). This caused xymon to stall on these DNS checks, and all of the network tests went purple.

I know that this is difficult to test, because obviously you cannot just cut the connection between the two sites to try it out. But you could try applying this patch which changes the DNS lookup code to use the same kind of timeout settings as the development version - the 4.3.x versions suffer from a common misunderstanding about how the C-ARES library handles timeout that make DNS timeouts take much too long.

One possible way of testing it would be if you can firewall access from e.g. your DR site Xymon server to the primary site's DNS server. If you are running Xymon on a Linux server, then "iptables" can do that for you. If your primary site DNS server is 10.1.2.3, then

   iptables -I OUTPUT 1 -d 10.1.2.3 -j DROP
   iptables -I INPUT 1 -s 10.1.2.3 -j DROP

will cause all traffic to/from this server to be dropped.


Regards,
Henrik