On Monday, 26 October 2009 20:55:15 user-762ee872a5a4@xymon.invalid wrote:
Hi all,
First off, sorry for the long post, I'm trying to supply as much data as
possible for analysis.
I have a single Hobbit server with approximately 3500 hosts, a mixture
of windows and unix, some DB tests,
some BEA tests and a few custom tests. I have over 70000 RRD files
which seems to be causing Hobbit performance
problems, most specifcally clock offset. I have a cron job that
restarts Hobbit every 30 minutes otherwise the offset
grows so large it eats all memory and OOM kill starts. NTP is fine, it
seems to be the time it takes for Hobbit to process
the client data. OS resides on RAID1 146GB drives SAS 15K RPM, second
drive for RRDs is a single 300GB SAS 15K RPM.
At the end is a graph showing the clock offset. What else can I try?
Add more spindles.
70 000 RRD files will result in a minimum of 233 IOPS (assuming they are all
being updated at 5-minute intervals). The EMC people I've spoken to say a 15k
FC disk shouldn't really be averaging much more than 180 IOPS, 15k SAS or 15k
SCSI wouldn't be any better. The 311 you seem to be doing isn't significant
overhead for the minumum of 233, so it is unlikely that any tuning will help.
If you can't add spindles, you could look at the 4.3 branch, which has some
features that allow scaling out to more hosts, or streamlining RRD writes
(which may allow you to lose the clock offset, but will likely not reduce the
load average much).
Regards,
Buchan