Most number of hosts
list Michael Beatty
Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running. My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds. I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are. -- Michael Beatty Sherwin-Williams IT Analyst/Developer user-4aea7c115850@xymon.invalid XXX-XXX-XXXX
list Greg Shea
Hi Michael, At one point, I had 6115 hosts running from 1 physical Xymon server. Some tests, specifically WebLogic, I run from a separate VM so if the java process hangs, it doesn't affect the main Xymon server. Physical HW Dell PE 2950 2 Xeon 5110 CPUs 8GB RAM 2 146GB, 10K RPM SAS (RAID1 root disk) 2 146GB. 15K RPM SAS (apps & data) Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Xymon 4.3.4 That's all changing now because of the push to virtualize everything, so I used the opportunity to build in some redundancy. Currently we have 2750 hosts running in this environment and will be migrating the rest of the old server over before EOY. Primary and standby VMs 2 Virtual Xeon E7549 CPUs 8GB RAM 120GB root disk and 200GB apps disk Red Hat Enterprise Linux Server release 5.5 (Tikanga) Xymon 4.3.7 Primary and standby external test VMs (WebLogic, custom tests) (same as above)
▸
-----Original Message-----
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty
Sent: Tuesday, May 22, 2012 1:01 PM
To: xymon at xymon.com
Subject: [Xymon] Most number of hosts
Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered"
setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running.
My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds.
I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are.
--
Michael Beatty
Sherwin-Williams
IT Analyst/Developer
user-4aea7c115850@xymon.invalid
XXX-XXX-XXXX
list Nicolas Lienard
Hi Michael Here we are monitoring > 10 000 hosts over 18 hobbit clusters (1 cluster per datacenter) but we have one cluster with > 3600 hosts: Hardware is HP DL380 G7 2 x Xeon X5650 RAID 1 for OS RAID 5 for data with 4 SSD hard drives (165Gb) for xymon data. 8 Gb RAM. xymongen: Statistics: Hosts : 3692 Pages : 497 Status messages : 46298 - Red : 318 ( 0.69 %) - Red (non-propagating) : 0 ( 0.00 %) - Yellow : 347 ( 0.75 %) - Yellow (non-propagating) : 0 ( 0.00 %) - Clear : 902 ( 1.95 %) - Green : 43537 (94.04 %) - Purple : 82 ( 0.18 %) - Blue : 1112 ( 2.40 %) xymonnet: Statistics: Hosts total : 3660 Hosts with no tests : 5 Total test count : 9645 Status messages : 10034 Alert status msgs : 0 Transmissions : 332 DNS statistics: # hostnames resolved : 1763 # succesful : 628 # failed : 6 # calls to dnsresolve : 1727 TCP test statistics: # TCP tests total : 5993 # HTTP tests : 2222 # Simple TCP tests : 3771 # Connection attempts : 5993 # bytes written : 376789 # bytes read : 25004690 [root at xxxxxxxx:/opt/xymon/server/etc/hosts.d/customer ] 21 grep -c proc /proc/cpuinfo 24 load average: 2.03, 1.97, 2.00 CPU %idle average: 90% !!! you can have many hosts; but i think the most important things is the number of check per hosts and the external scripts running on your xymon server. (oracle check; backup check, reporting, etc) which increases the load. Having SSD is a good option because in the past, the main load was I/O wait. Cheers Nico
▸
Le 22 mai 2012 à 20:11, <user-028bf2df2227@xymon.invalid> a écrit :
Hi Michael, At one point, I had 6115 hosts running from 1 physical Xymon server. Some tests, specifically WebLogic, I run from a separate VM so if the java process hangs, it doesn't affect the main Xymon server. Physical HW Dell PE 2950 2 Xeon 5110 CPUs 8GB RAM 2 146GB, 10K RPM SAS (RAID1 root disk) 2 146GB. 15K RPM SAS (apps & data) Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Xymon 4.3.4 That's all changing now because of the push to virtualize everything, so I used the opportunity to build in some redundancy. Currently we have 2750 hosts running in this environment and will be migrating the rest of the old server over before EOY. Primary and standby VMs 2 Virtual Xeon E7549 CPUs 8GB RAM 120GB root disk and 200GB apps disk Red Hat Enterprise Linux Server release 5.5 (Tikanga) Xymon 4.3.7 Primary and standby external test VMs (WebLogic, custom tests) (same as above) -----Original Message----- From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Tuesday, May 22, 2012 1:01 PM To: xymon at xymon.com Subject: [Xymon] Most number of hosts Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running. My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds. I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are. -- Michael Beatty Sherwin-Williams IT Analyst/Developer user-4aea7c115850@xymon.invalid XXX-XXX-XXXX
list Henrik Størner
▸
On 22-05-2012 19:00, Michael Beatty wrote:
Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running. My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds.
I don't have much experience with high-latency connections - you'll obviously have to tweak some network timeout settings.
The main performance issue in Xymon these days are the disk I/O from updating all of the RRD files. Here are some numbers fresh off my main server:
$ iostat
Linux 2.6.32-5-amd64 (myserver) 05/22/2012 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.95 0.06 0.79 0.65 0.00 96.56
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
cciss/c0d0 5.81 40.83 142.54 23238617 81132930
cciss/c0d1 79.20 31.19 3220.84 17755352 1833330720
It's been up for close to a week. Both of these disks are RAID-1 devices; c0d0 are normal 10K RPM disks, c0d1 are SSD disks and holds ONLY the xymon/rrd and xymon/tmp directories. As you can see, the write activity on the SSD disks is huge - they are about a year old now, and I guess I'll have to replace them soon.
CPU- and memory-wise, this box is doing nothing at all. It has 2 quad-core Xeon's and 24 GB RAM, but it only uses about 6 GB - the rest is used for disk cache (the way Linux does automatically).
Since the box has so much RAM and the RRD files only take up about 8 GB right now, I have been thinking about moving them to a RAM disk, and just rsync'ing them to the SSD disks once an hour or so.
It handles about 4000 full clients, and another 2500 for which we only do network tests.
Regards,
Henrik
list Wilfrid Beauchaine
hi nico how did you setup your hobbit cluster ? hearbeat/pacemaker + drdb , corosync/pacemaker + drdb, keapalived/opensvc + lvm ? Regards, Wilfrid :)
▸
Le 2012-05-22 20:46, Nico a écrit :Hi Michael Here we are monitoring > 10 000 hosts over 18 hobbit clusters (1 cluster per datacenter) but we have one cluster with > 3600 hosts: Hardware is HP DL380 G7 2 x Xeon X5650 RAID 1 for OS RAID 5 for data with 4 SSD hard drives (165Gb) for xymon data. 8 Gb RAM. xymongen: Statistics: Hosts : 3692 Pages : 497 Status messages : 46298 - Red : 318 ( 0.69 %) - Red (non-propagating) : 0 ( 0.00 %) - Yellow : 347 ( 0.75 %) - Yellow (non-propagating) : 0 ( 0.00 %) - Clear : 902 ( 1.95 %) - Green : 43537 (94.04 %) - Purple : 82 ( 0.18 %) - Blue : 1112 ( 2.40 %) xymonnet: Statistics: Hosts total : 3660 Hosts with no tests : 5 Total test count : 9645 Status messages : 10034 Alert status msgs : 0 Transmissions : 332 DNS statistics: # hostnames resolved : 1763 # succesful : 628 # failed : 6 # calls to dnsresolve : 1727 TCP test statistics: # TCP tests total : 5993 # HTTP tests : 2222 # Simple TCP tests : 3771 # Connection attempts : 5993 # bytes written : 376789 # bytes read : 25004690 [root at xxxxxxxx:/opt/xymon/server/etc/hosts.d/customer ] 21 grep -c proc /proc/cpuinfo 24 load average: 2.03, 1.97, 2.00 CPU %idle average: 90% !!! you can have many hosts; but i think the most important things is the number of check per hosts and the external scripts running on your xymon server. (oracle check; backup check, reporting, etc) which increases the load. Having SSD is a good option because in the past, the main load was I/O wait. Cheers Nico Le 22 mai 2012 à 20:11, <user-028bf2df2227@xymon.invalid [5]> a écrit :Hi Michael, At one point, I had 6115 hosts running from 1 physical Xymon server. Some tests, specifically WebLogic, I run from a separate VM so if the java process hangs, it doesn't affect the main Xymon server. Physical HW Dell PE 2950 2 Xeon 5110 CPUs 8GB RAM 2 146GB, 10K RPM SAS (RAID1 root disk) 2 146GB. 15K RPM SAS (apps & data) Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Xymon 4.3.4 That's all changing now because of the push to virtualize everything, so I used the opportunity to build in some redundancy. Currently we have 2750 hosts running in this environment and will be migrating the rest of the old server over before EOY. Primary and standby VMs 2 Virtual Xeon E7549 CPUs 8GB RAM 120GB root disk and 200GB apps disk Red Hat Enterprise Linux Server release 5.5 (Tikanga) Xymon 4.3.7 Primary and standby external test VMs (WebLogic, custom tests) (same as above) -----Original Message-----
From: xymon-bounces at xymon.com [1] [mailto:xymon-bounces at xymon.com]
▸
On Behalf Of Michael Beatty Sent: Tuesday, May 22, 2012 1:01 PM
To: xymon at xymon.com [2]
▸
Subject: [Xymon] Most number of hosts Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running. My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds. I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are. -- Michael Beatty Sherwin-Williams IT Analyst/Developer
user-4aea7c115850@xymon.invalid [3] XXX-XXX-XXXXLinks: [3] mailto:user-4aea7c115850@xymon.invalid [5] mailto:user-028bf2df2227@xymon.invalid
list Nicolas Lienard
Heartbeat/LVM/DRBD :) Envoyé de mon iPhone
▸
Le 25 mai 2012 à 22:05, wbeauchaine <user-ee69b0db7015@xymon.invalid> a écrit :
hi nico how did you setup your hobbit cluster ? hearbeat/pacemaker + drdb , corosync/pacemaker + drdb, keapalived/opensvc + lvm ? Regards, Wilfrid :) Le 2012-05-22 20:46, Nico a écrit :Hi Michael Here we are monitoring > 10 000 hosts over 18 hobbit clusters (1 cluster per datacenter) but we have one cluster with > 3600 hosts: Hardware is HP DL380 G7 2 x Xeon X5650 RAID 1 for OS RAID 5 for data with 4 SSD hard drives (165Gb) for xymon data. 8 Gb RAM. xymongen: Statistics: Hosts : 3692 Pages : 497 Status messages : 46298 - Red : 318 ( 0.69 %) - Red (non-propagating) : 0 ( 0.00 %) - Yellow : 347 ( 0.75 %) - Yellow (non-propagating) : 0 ( 0.00 %) - Clear : 902 ( 1.95 %) - Green : 43537 (94.04 %) - Purple : 82 ( 0.18 %) - Blue : 1112 ( 2.40 %) xymonnet: Statistics: Hosts total : 3660 Hosts with no tests : 5 Total test count : 9645 Status messages : 10034 Alert status msgs : 0 Transmissions : 332 DNS statistics: # hostnames resolved : 1763 # succesful : 628 # failed : 6 # calls to dnsresolve : 1727 TCP test statistics: # TCP tests total : 5993 # HTTP tests : 2222 # Simple TCP tests : 3771 # Connection attempts : 5993 # bytes written : 376789 # bytes read : 25004690 [root at xxxxxxxx:/opt/xymon/server/etc/hosts.d/customer ] 21 grep -c proc /proc/cpuinfo 24 load average: 2.03, 1.97, 2.00 CPU %idle average: 90% !!! you can have many hosts; but i think the most important things is the number of check per hosts and the external scripts running on your xymon server. (oracle check; backup check, reporting, etc) which increases the load. Having SSD is a good option because in the past, the main load was I/O wait. Cheers Nico Le 22 mai 2012 à 20:11, <user-028bf2df2227@xymon.invalid [5]> a écrit :Hi Michael, At one point, I had 6115 hosts running from 1 physical Xymon server. Some tests, specifically WebLogic, I run from a separate VM so if the java process hangs, it doesn't affect the main Xymon server. Physical HW Dell PE 2950 2 Xeon 5110 CPUs 8GB RAM 2 146GB, 10K RPM SAS (RAID1 root disk) 2 146GB. 15K RPM SAS (apps & data) Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Xymon 4.3.4 That's all changing now because of the push to virtualize everything, so I used the opportunity to build in some redundancy. Currently we have 2750 hosts running in this environment and will be migrating the rest of the old server over before EOY. Primary and standby VMs 2 Virtual Xeon E7549 CPUs 8GB RAM 120GB root disk and 200GB apps disk Red Hat Enterprise Linux Server release 5.5 (Tikanga) Xymon 4.3.7 Primary and standby external test VMs (WebLogic, custom tests) (same as above) -----Original Message----- From: xymon-bounces at xymon.com [1] [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Tuesday, May 22, 2012 1:01 PM To: xymon at xymon.com [2] Subject: [Xymon] Most number of hosts Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running. My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds. I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are. -- Michael Beatty Sherwin-Williams IT Analyst/Developer user-4aea7c115850@xymon.invalid [3] XXX-XXX-XXXXLinks: [3] mailto:user-4aea7c115850@xymon.invalid [5] mailto:user-028bf2df2227@xymon.invalid