On Sun, Dec 03, 2006 at 12:10:03AM +0100, Henrik Stoerner wrote:
Besides, "fail over" means lot of different things. For a true fail
over setup, you'll need some hardware support on top of Hobbit -
providing a virtual IP for your resilient hosts, and probably some sort
of shared storage. Most of that is handled outside Hobbit.
So what exactly do you have in mind ?
I'm replying to my own mail to pick up all the responses that have come
about this.
Trever Noggle:
I would like to do like you can with BB.. The master and the backups
will be on completely different networks [...] I want to have a main
monitor server at location 1 monitoring devices at both locations. I
then want location 2 to take over if location 1 goes down.
Anton Burkhalter:
I use two independent Hobbit servers; each client reports to both
servers. The question is how to synchronize the two servers after an
outage of a server.
Ralph Mitchell:
The thing that concerns me is that I can't be running the same checks
from two servers at the same time. People around here get irritated when
their webserver stats are artificially inflated
Daniel J McDonald:
I'd like hobbit-alerts to only run on one box at a time. Displays and
tests can all run independently
For myself, I might add that having access to the historical data - both
graphs and history logs - is also a requirement.
The simple "do it like BB does" is inadequate - it cannot handle keeping
the historical data up-to-date on both servers, and it also fails to carry
over the current alerts that are active: If the master server sent out an
alert for something before it crashed, and the next alert should go out 12
hours later, then this repeat-setting isn't transferred to the slave
server. So when the master server drops off the net and the slave server
takes over, it will immediately start by sending out alerts for everything
that is down. Not good.
The current state of a Hobbit server can easily be shared among servers.
The checkpoint files that go into ~hobbit/server/tmp/ can be copied
across to another server, and if you do that often enough then starting up
Hobbit on the other server will pick up all of the current status.
So that part is easy - for convenience I might want to implement some
sort of internal Hobbit protocol for distributing the checkpoint files, but
you can already today just use scp, rsync or similar to copy those files
over.
The downside of this of course is that something has to recognize when
the primary site is gone, and start up Hobbit on the secondary server. That
is not very attractive; I would rather have Hobbit running on both servers
all the time - this would require some work. But let's assume for now that
this is possible.
Then there are the on-disk files: History logs and graphs. Something has
happened here recently, since it is now possible to distribute these over
multiple servers - and also to have more than one site perform all of the
updates of those files. So instead of periodically copying the files from
a master server to a slave, you just copy them once and then mirror all of
the updates to the relevant servers. The code for this is in the current
snapshots; it isn't documented yet, and hasn't had much testing. I use it
currently for another purpose: Load-sharing of the updates.
Finally there are the various Hobbit tasks: The display, the network
tests, the alerts.
The display tasks are very easily distributed to multiple servers - it
is somewhat inconvenient that there are static webpages built for the
overview webpages, I want to eliminate those and have all of the webpages
generated dynamically - but the web display does not have to be on the
same physical server as the rest of Hobbit, so doing failover for the web
interface is relatively simple.
Alerts - the code is *almost* ready. It is based on the same principle as
what is used for distributing the history- and RRD-files across multiple
servers; the hobbitd_alert module runs on all of the servers - so it
keeps track of the repeat times etc - but it only actually sends the
alerts from one of the servers at any time.
Network tests - I've heard arguments going both ways as to whether one
should run network tests on all servers ("it is interesting to see if the
site is down when tested from all of our locations, or only from the
primary location"), or on just a single server ("we want to minimize
traffic from the monitoring systems towards the webservers"). I'm still
thinking about how to handle this - if they run on all Hobbit servers,
there has to be some way of choosing which test result should be used; if
they run only on a single server I will probably use the same method for
choosing which server runs the tests as I use to decide who gets to send
out alerts.
Regards,
Henrik