Determining yellow and red level warnings for cpu/load
list Josh Luthman
I have a server that does does a lot of work all the time - a vmware server. Each virtual machine does its own things, but the host OS has load that gets to 20 at times. Everything is just as responsive during these times, so the default load warnings aren't valid for me. I changed my alerts based on a test with stress ( http://freshmeat.net/projects/stress). Does everyone else just accept the defaults for yellow and red? If not, how do you increase it? Josh Luthman Office: XXX-XXX-XXXX Direct: XXX-XXX-XXXX XXXX Wayne St Suite XXXX Troy, OH XXXXX "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle
list Kim Johansen
▸
On Sunday 17 May 2009 03:22:43 Josh Luthman wrote:
I have a server that does does a lot of work all the time - a vmware server. Each virtual machine does its own things, but the host OS has load that gets to 20 at times. Everything is just as responsive during these times, so the default load warnings aren't valid for me. I changed my alerts based on a test with stress ( http://freshmeat.net/projects/stress). Does everyone else just accept the defaults for yellow and red? If not, how do you increase it?
In the help -> Configuring Monitoring is there a good description on how to change default. It is the hobbit-clients.cfg you need to change in, the file it self also have a good description -- Kim Johansen
list Josh Luthman
I understand how and I even explained that I have changed the values. The default for yellow and red is 5 and 10. The server that introduced the question was increased to 10 and 25. My question is what do the other users on this mailing list change them to and based on what information?
▸
On 5/17/09, Kim Johansen <user-e03b25f5b161@xymon.invalid> wrote:On Sunday 17 May 2009 03:22:43 Josh Luthman wrote:I have a server that does does a lot of work all the time - a vmware server. Each virtual machine does its own things, but the host OS has load that gets to 20 at times. Everything is just as responsive during these times, so the default load warnings aren't valid for me. I changed my alerts based on a test with stress ( http://freshmeat.net/projects/stress). Does everyone else just accept the defaults for yellow and red? If not, how do you increase it?In the help -> Configuring Monitoring is there a good description on how to change default. It is the hobbit-clients.cfg you need to change in, the file it self also have a good description -- Kim Johansen
-- Josh Luthman Office: XXX-XXX-XXXX Direct: XXX-XXX-XXXX XXXX Wayne St Suite XXXX Troy, OH XXXXX "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle
list Xymon User in Richmond
Josh, I read it the same way Kim did--that you were literally asking _how_ to change the values. I thought it strange that you wouldn't know given that you're one of the most active contributors on this list WRT solving config problems, but it seemed you were saying you made changes and they didn't work. 5 and 10 are way too low IMO for most scenarios on current multi-core/multi-processor hardware. I typically use 20 and 30 for servers where online response time is important, much higher for background-type servers like sendmail.
▸
On Sun, May 17, 2009 11:32, Josh Luthman wrote:I understand how and I even explained that I have changed the values. The default for yellow and red is 5 and 10. The server that introduced the question was increased to 10 and 25. My question is what do the other users on this mailing list change them to and based on what information? On 5/17/09, Kim Johansen <user-e03b25f5b161@xymon.invalid> wrote:On Sunday 17 May 2009 03:22:43 Josh Luthman wrote:I have a server that does does a lot of work all the time - a vmware server. Each virtual machine does its own things, but the host OS has load that gets to 20 at times. Everything is just as responsive during these times, so the default load warnings aren't valid for me. I changed my alerts based on a test with stress ( http://freshmeat.net/projects/stress). Does everyone else just accept the defaults for yellow and red? If not, how do you increase it?In the help -> Configuring Monitoring is there a good description on how to change default. It is the hobbit-clients.cfg you need to change in, the file it self also have a good description -- Kim Johansen-- Josh Luthman Office: XXX-XXX-XXXX Direct: XXX-XXX-XXXX XXXX Wayne St Suite XXXX Troy, OH XXXXX "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle
list Josh Luthman
Can I ask why you picked 20 and 30? Could you share what hardware you're running?
▸
On 5/17/09, Xymon User in Richmond <user-24d6f8323faa@xymon.invalid> wrote:Josh, I read it the same way Kim did--that you were literally asking _how_ to change the values. I thought it strange that you wouldn't know given that you're one of the most active contributors on this list WRT solving config problems, but it seemed you were saying you made changes and they didn't work. 5 and 10 are way too low IMO for most scenarios on current multi-core/multi-processor hardware. I typically use 20 and 30 for servers where online response time is important, much higher for background-type servers like sendmail. On Sun, May 17, 2009 11:32, Josh Luthman wrote:I understand how and I even explained that I have changed the values. The default for yellow and red is 5 and 10. The server that introduced the question was increased to 10 and 25. My question is what do the other users on this mailing list change them to and based on what information? On 5/17/09, Kim Johansen <user-e03b25f5b161@xymon.invalid> wrote:On Sunday 17 May 2009 03:22:43 Josh Luthman wrote:I have a server that does does a lot of work all the time - a vmware server. Each virtual machine does its own things, but the host OS has load that gets to 20 at times. Everything is just as responsive during these times, so the default load warnings aren't valid for me. I changed my alerts based on a test with stress ( http://freshmeat.net/projects/stress). Does everyone else just accept the defaults for yellow and red? If not, how do you increase it?In the help -> Configuring Monitoring is there a good description on how to change default. It is the hobbit-clients.cfg you need to change in, the file it self also have a good description -- Kim Johansen-- Josh Luthman Office: XXX-XXX-XXXX Direct: XXX-XXX-XXXX XXXX Wayne St Suite XXXX Troy, OH XXXXX "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle
-- Josh Luthman Office: XXX-XXX-XXXX Direct: XXX-XXX-XXXX XXXX Wayne St Suite XXXX Troy, OH XXXXX "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle
list Xymon User in Richmond
The servers those thresholds are applied to are mostly Dell PE2850 machines with dual 3.4 single cores, and the values are based on experiences over time with where we'll start to get problem tickets based on response times.
▸
On Sun, May 17, 2009 12:08, Josh Luthman wrote:Can I ask why you picked 20 and 30? Could you share what hardware you're running? On 5/17/09, Xymon User in Richmond <user-24d6f8323faa@xymon.invalid> wrote:Josh, I read it the same way Kim did--that you were literally asking _how_ to change the values. I thought it strange that you wouldn't know given that you're one of the most active contributors on this list WRT solving config problems, but it seemed you were saying you made changes and they didn't work. 5 and 10 are way too low IMO for most scenarios on current multi-core/multi-processor hardware. I typically use 20 and 30 for servers where online response time is important, much higher for background-type servers like sendmail. On Sun, May 17, 2009 11:32, Josh Luthman wrote:I understand how and I even explained that I have changed the values. The default for yellow and red is 5 and 10. The server that introduced the question was increased to 10 and 25. My question is what do the other users on this mailing list change them to and based on what information? On 5/17/09, Kim Johansen <user-e03b25f5b161@xymon.invalid> wrote:On Sunday 17 May 2009 03:22:43 Josh Luthman wrote:I have a server that does does a lot of work all the time - a vmware server. Each virtual machine does its own things, but the host OS has load that gets to 20 at times. Everything is just as responsive during these times, so the default load warnings aren't valid for me. I changed my alerts based on a test with stress ( http://freshmeat.net/projects/stress). Does everyone else just accept the defaults for yellow and red? If not, how do you increase it?In the help -> Configuring Monitoring is there a good description on how to change default. It is the hobbit-clients.cfg you need to change in, the file it self also have a good description -- Kim Johansen
list Josh Luthman
Awesome! Thanks for the information!
▸
On 5/17/09, Xymon User in Richmond <user-24d6f8323faa@xymon.invalid> wrote:The servers those thresholds are applied to are mostly Dell PE2850 machines with dual 3.4 single cores, and the values are based on experiences over time with where we'll start to get problem tickets based on response times. On Sun, May 17, 2009 12:08, Josh Luthman wrote:Can I ask why you picked 20 and 30? Could you share what hardware you're running? On 5/17/09, Xymon User in Richmond <user-24d6f8323faa@xymon.invalid> wrote:Josh, I read it the same way Kim did--that you were literally asking _how_ to change the values. I thought it strange that you wouldn't know given that you're one of the most active contributors on this list WRT solving config problems, but it seemed you were saying you made changes and they didn't work. 5 and 10 are way too low IMO for most scenarios on current multi-core/multi-processor hardware. I typically use 20 and 30 for servers where online response time is important, much higher for background-type servers like sendmail. On Sun, May 17, 2009 11:32, Josh Luthman wrote:I understand how and I even explained that I have changed the values. The default for yellow and red is 5 and 10. The server that introduced the question was increased to 10 and 25. My question is what do the other users on this mailing list change them to and based on what information? On 5/17/09, Kim Johansen <user-e03b25f5b161@xymon.invalid> wrote:On Sunday 17 May 2009 03:22:43 Josh Luthman wrote:I have a server that does does a lot of work all the time - a vmware server. Each virtual machine does its own things, but the host OS has load that gets to 20 at times. Everything is just as responsive during these times, so the default load warnings aren't valid for me. I changed my alerts based on a test with stress ( http://freshmeat.net/projects/stress). Does everyone else just accept the defaults for yellow and red? If not, how do you increase it?In the help -> Configuring Monitoring is there a good description on how to change default. It is the hobbit-clients.cfg you need to change in, the file it self also have a good description -- Kim Johansen
-- Josh Luthman Office: XXX-XXX-XXXX Direct: XXX-XXX-XXXX XXXX Wayne St Suite XXXX Troy, OH XXXXX "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle
list Buchan Milne
▸
On Sunday 17 May 2009 03:22:43 Josh Luthman wrote:
Does everyone else just accept the defaults for yellow and red?
No, they are only really applicable to hosts with fewer than 4 cores.
If not, how do you increase it?
We set yellow to about 2-4 times the number of CPU cores, and red to about 4-8 times number of cores (depending on the role). Unfortunately, load average is not a great measure of how busy the CPU really is, it would be much better to be able to alarm on vmstat data ... Regards, Buchan
list Tom Georgoulias
▸
Josh Luthman wrote:
I have a server that does does a lot of work all the time - a vmware server. Each virtual machine does its own things, but the host OS has load that gets to 20 at times. Everything is just as responsive during these times, so the default load warnings aren't valid for me. I changed my alerts based on a test with stress (http://freshmeat.net/projects/stress). Does everyone else just accept the defaults for yellow and red? If not, how do you increase it?
I agree with the others, the default values are pretty low and may not work for your situation. The absolute best way to determine what works for you is to simply run the xymon client for a while (days/weeks/etc.) and see where your server(s) tend to operate normally and what happens when things aren't going well. After you gather that data, you will have a good idea of what you can use for yellow and red thresholds. Tom