no vmstat data for 8 hours on weekdays?
list Mike Burger
Hello, all. I've got something odd going on...I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior. I've installed the same Hobbit client on all of my AIX instances. The vmstat processes are running as they should, and all of the other graphs in the trends area get data all day long. I've already looked and I have no "TIME" options set for this specific for this server in either the localclient.cfg file on the server nor the analysis.cfg on the Xymon server...in fact, other than this system's entry in the hosts.cfg, there is no mention of this system in any config file EXCEPT hosts.cfg. Any thoughts would be appreciated. -- Mike Burger AIX Administrator <http://freedomhome.fhmc.local/intranet/main.jsp>; Phone (XXX) XXX-XXXX, Fax (XXX) XXX-XXXX, Cell (XXX) XXX-XXXX E-mail: user-c26873f0522a@xymon.invalid "Once word leaks out that a pirate's gone soft, people begin to disobey you and then it's nothing but work, work, work, all the time." --Westley/The Dread Pirate Roberts CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
list Jeremy Laidman
▸
On 13 November 2012 08:33, Mike Burger <user-c26873f0522a@xymon.invalid>wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate? Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different. Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps). J
list Mike Burger
▸
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <user-71895fb2e44c@xymon.invalid> wrote:
On 13 November 2012 08:33, Mike Burger <user-c26873f0522a@xymon.invalid> wrote: I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior. Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate? Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different. Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).
Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM). Over the weekend, there were no breaks in data graphing. I'll have to also look at a continuous vmstat output, just in case. Sent from my iPhone...please excuse any typos or short answers. It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
▸
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday. Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters? In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
▸
From: Mike Burger
Sent: Monday, November 12, 2012 8:42 PM
To: Jeremy Laidman
Cc: Mike Burger; xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <user-71895fb2e44c@xymon.invalid> wrote:
On 13 November 2012 08:33, Mike Burger <user-c26873f0522a@xymon.invalid> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).
Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
▸
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:23 AM
To: Mike Burger; Jeremy Laidman
Cc: xymon at xymon.com
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger
Sent: Monday, November 12, 2012 8:42 PM
To: Jeremy Laidman
Cc: Mike Burger; xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <user-71895fb2e44c@xymon.invalid> wrote:
On 13 November 2012 08:33, Mike Burger <user-c26873f0522a@xymon.invalid> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).
Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
▸
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:24 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:23 AM
To: Mike Burger; Jeremy Laidman
Cc: xymon at xymon.com
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger
Sent: Monday, November 12, 2012 8:42 PM
To: Jeremy Laidman
Cc: Mike Burger; xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <user-71895fb2e44c@xymon.invalid> wrote:
On 13 November 2012 08:33, Mike Burger <user-c26873f0522a@xymon.invalid> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).
Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
Ok…I may have spoken too soon…9:05 data…waiting for 9:10 data. (hangs head)
▸
From: Mike Burger
Sent: Tuesday, November 13, 2012 9:09 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:24 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:23 AM
To: Mike Burger; Jeremy Laidman
Cc: xymon at xymon.com
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger
Sent: Monday, November 12, 2012 8:42 PM
To: Jeremy Laidman
Cc: Mike Burger; xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <user-71895fb2e44c@xymon.invalid> wrote:
On 13 November 2012 08:33, Mike Burger <user-c26873f0522a@xymon.invalid> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).
Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
Last data recorded, now, is from 9:10. I’ve got vmstat output that I’d be happy to share if it helps anyone understand why this is happening.
▸
From: Mike Burger
Sent: Tuesday, November 13, 2012 9:11 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I may have spoken too soon…9:05 data…waiting for 9:10 data. (hangs head)
From: Mike Burger
Sent: Tuesday, November 13, 2012 9:09 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:24 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:23 AM
To: Mike Burger; Jeremy Laidman
Cc: xymon at xymon.com
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger
Sent: Monday, November 12, 2012 8:42 PM
To: Jeremy Laidman
Cc: Mike Burger; xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <user-71895fb2e44c@xymon.invalid> wrote:
On 13 November 2012 08:33, Mike Burger <user-c26873f0522a@xymon.invalid> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).
Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
FWIW, I’m attaching vmstat output that ends pre-dropoff and another that starts pre-dropoff. With any luck, something will stand out. Xymon server is 4.3.10, Hobbit client on AIX is 4.2.0, if that makes any difference.
▸
From: Mike Burger
Sent: Tuesday, November 13, 2012 9:40 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Last data recorded, now, is from 9:10.
I’ve got vmstat output that I’d be happy to share if it helps anyone understand why this is happening.
From: Mike Burger
Sent: Tuesday, November 13, 2012 9:11 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I may have spoken too soon…9:05 data…waiting for 9:10 data. (hangs head)
From: Mike Burger
Sent: Tuesday, November 13, 2012 9:09 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:24 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:23 AM
To: Mike Burger; Jeremy Laidman
Cc: xymon at xymon.com
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger
Sent: Monday, November 12, 2012 8:42 PM
To: Jeremy Laidman
Cc: Mike Burger; xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <user-71895fb2e44c@xymon.invalid> wrote:
On 13 November 2012 08:33, Mike Burger <user-c26873f0522a@xymon.invalid> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).
Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
It would likely help if I would have attached the file. *facepalm*
▸
From: Mike Burger
Sent: Tuesday, November 13, 2012 11:13 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
FWIW, I’m attaching vmstat output that ends pre-dropoff and another that starts pre-dropoff. With any luck, something will stand out.
Xymon server is 4.3.10, Hobbit client on AIX is 4.2.0, if that makes any difference.
From: Mike Burger
Sent: Tuesday, November 13, 2012 9:40 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Last data recorded, now, is from 9:10.
I’ve got vmstat output that I’d be happy to share if it helps anyone understand why this is happening.
From: Mike Burger
Sent: Tuesday, November 13, 2012 9:11 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I may have spoken too soon…9:05 data…waiting for 9:10 data. (hangs head)
From: Mike Burger
Sent: Tuesday, November 13, 2012 9:09 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:24 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger
Sent: Tuesday, November 13, 2012 8:23 AM
To: Mike Burger; Jeremy Laidman
Cc: xymon at xymon.com
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger
Sent: Monday, November 12, 2012 8:42 PM
To: Jeremy Laidman
Cc: Mike Burger; xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <user-71895fb2e44c@xymon.invalid> wrote:
On 13 November 2012 08:33, Mike Burger <user-c26873f0522a@xymon.invalid> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).
Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Jeremy Laidman
▸
On 14 November 2012 00:22, Mike Burger <user-c26873f0522a@xymon.invalid>wrote:
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
Some parsers don't use whitespace delimeters, but instead use column position. But I don't know if the vmstat parser does the same.
▸
vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2second intervals (10 minutes worth of captures).
No, I think you'll find that it's two captures in 300 second intervals. The first sample is the total/average since last boot, and the second sample is the total/average since the first one. The first sample is discarded.
▸
As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved?
Yes, perhaps. I only have one vmstat command running at any one time. You might try killing both off and see what happens. J
list Jeremy Laidman
▸
On 14 November 2012 03:14, Mike Burger <user-c26873f0522a@xymon.invalid>wrote:
It would likely help if I would have attached the file. **facepalm**
I'm not entirely sure what I'm looking at. I expected only two lines of output. Nevertheless, the columns do seem to be a little irregular. Can you attach the [vmstat] section from the client data for the host? If you have xymond running with "--store-clientlogs" then you can find historical data snapshots in $XYMONVAR/hostdata/name-of-host/. If not, you can get the [vmstat] section of the client data, at the appropriate times, with this command run on the Xymon server: xymoncmd xymon localhost 'clientlog name-of-host section=vmstat' J
list Jeremy Laidman
▸
On 14 November 2012 10:09, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:
Some parsers don't use whitespace delimeters, but instead use column position. But I don't know if the vmstat parser does the same.
I just checked the code, and I think the vmstat parser (in do_vmstat.c function do_vmstat_rrd()) splits on whitespace, thus uses column number rather than column position. So this shouldn't be a problem for you.
list Mike Burger
Sorry...that was the output from my own vmstat collections, rather than the file(s) being collected by the Hobbit Client.
▸
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Tuesday, November 13, 2012 6:29 PM
To: Mike Burger
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 14 November 2012 03:14, Mike Burger <user-c26873f0522a@xymon.invalid>
▸
wrote:
It would likely help if I would have attached the file.
*facepalm*
I'm not entirely sure what I'm looking at. I expected only two lines of
output. Nevertheless, the columns do seem to be a little irregular.
Can you attach the [vmstat] section from the client data for the host?
If you have xymond running with "--store-clientlogs" then you can find
historical data snapshots in $XYMONVAR/hostdata/name-of-host/.
If not, you can get the [vmstat] section of the client data, at the
appropriate times, with this command run on the Xymon server:
xymoncmd xymon localhost 'clientlog name-of-host section=vmstat'
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
Ok...that's good news on one front...I'd love to know what the heck is going on then. I'm going to stop the hobbit client, kill any lingering vmstat processes and then restart the hobbit client and see what happens.
▸
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Tuesday, November 13, 2012 6:38 PM
To: Mike Burger
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 14 November 2012 10:09, Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
wrote:
Some parsers don't use whitespace delimeters, but instead use
column position. But I don't know if the vmstat parser does the same.
I just checked the code, and I think the vmstat parser (in do_vmstat.c
function do_vmstat_rrd()) splits on whitespace, thus uses column number
rather than column position. So this shouldn't be a problem for you.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
Silly question time...is that code in the server or client?
▸
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Tuesday, November 13, 2012 6:38 PM
To: Mike Burger
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 14 November 2012 10:09, Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
wrote:
Some parsers don't use whitespace delimeters, but instead use
column position. But I don't know if the vmstat parser does the same.
I just checked the code, and I think the vmstat parser (in do_vmstat.c
function do_vmstat_rrd()) splits on whitespace, thus uses column number
rather than column position. So this shouldn't be a problem for you.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Jeremy Laidman
On 15 November 2012 02:04, Mike Burger <user-c26873f0522a@xymon.invalid>wrote:
Silly question time…is that code in the server or client?
There are no silly questions: all questions lead to enlightenment! The parser runs on the server. J
list Mike Burger
Good to know...I was trying (unsuccessfully) to compile the 4.3.10 client for AIX, in case it was the client side. On the other side...I did stop the client, killed any remaining vmstat processes and started the client back up. Initially, only one vmstat process was running, and then eventually there were two at any given time, again. They kick off every 5 minutes, outputting to a vmstat file tied to that process' PID (hobbit_vmstat.<hostname>.PID), and at the end of the run, the PID specific vmstat file is supposed to be renamed to hobbit_vmstat.<hostname>. The one thing I notice is that I never see a file hobbit_vmstat.<hostname>...even now, while there is still data actually being reported in the graph (it's only 7:44AM at present), there is no vmstat file without the PID as part of the file name present. Not that I think that is the problem. Is there anything I can do to increase verbosity in the client and/or server logging to determine if there's some sort of communication issue (or lack thereof)?
▸
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Wednesday, November 14, 2012 6:20 PM
To: Mike Burger
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 15 November 2012 02:04, Mike Burger <user-c26873f0522a@xymon.invalid>
▸
wrote:
Silly question time...is that code in the server or client?
There are no silly questions: all questions lead to enlightenment!
The parser runs on the server.
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
I think I've found it in the server's xymond.log: Oversize status msg from <client IP> for <hostname>:procs truncated (n=<some #>, limit=262144) Oversize data/client msg from <client IP <truncated (n=<some #>, limit=524288). Now, I need to figure out how to change those limits. From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Mike Burger Sent: Thursday, November 15, 2012 7:46 AM To: Jeremy Laidman
▸
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
Good to know...I was trying (unsuccessfully) to compile the 4.3.10
client for AIX, in case it was the client side.
On the other side...I did stop the client, killed any remaining vmstat
processes and started the client back up. Initially, only one vmstat
process was running, and then eventually there were two at any given
time, again. They kick off every 5 minutes, outputting to a vmstat file
tied to that process' PID (hobbit_vmstat.<hostname>.PID), and at the end
of the run, the PID specific vmstat file is supposed to be renamed to
hobbit_vmstat.<hostname>. The one thing I notice is that I never see a
file hobbit_vmstat.<hostname>...even now, while there is still data
actually being reported in the graph (it's only 7:44AM at present),
there is no vmstat file without the PID as part of the file name
present. Not that I think that is the problem.
Is there anything I can do to increase verbosity in the client and/or
server logging to determine if there's some sort of communication issue
(or lack thereof)?
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Wednesday, November 14, 2012 6:20 PM
To: Mike Burger
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 15 November 2012 02:04, Mike Burger <user-c26873f0522a@xymon.invalid>
wrote:
Silly question time...is that code in the server or client?
There are no silly questions: all questions lead to enlightenment!
The parser runs on the server.
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments,
is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal
law. If you
are not the intended recipient, please contact the sender by reply
e-mail,
delete this email and destroy all copies of the message.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
Ok...I've modified the following options in the xymonserver.cfg file: MAXMSG_CLIENT changed from 512 to 1024 MAXMSG_STATUS changed from 256 to 512 MAXMSG_DATA changed from 256 to 512 I'll be watching to see how things look as we get to/past the 9AM hour.
▸
From: Mike Burger
Sent: Thursday, November 15, 2012 8:03 AM
To: Mike Burger; Jeremy Laidman
Cc: xymon at xymon.com
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I think I've found it in the server's xymond.log:
Oversize status msg from <client IP> for <hostname>:procs truncated
(n=<some #>, limit=262144)
Oversize data/client msg from <client IP <truncated (n=<some #>,
limit=524288).
Now, I need to figure out how to change those limits.
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf
Of Mike Burger
Sent: Thursday, November 15, 2012 7:46 AM
To: Jeremy Laidman
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
Good to know...I was trying (unsuccessfully) to compile the 4.3.10
client for AIX, in case it was the client side.
On the other side...I did stop the client, killed any remaining vmstat
processes and started the client back up. Initially, only one vmstat
process was running, and then eventually there were two at any given
time, again. They kick off every 5 minutes, outputting to a vmstat file
tied to that process' PID (hobbit_vmstat.<hostname>.PID), and at the end
of the run, the PID specific vmstat file is supposed to be renamed to
hobbit_vmstat.<hostname>. The one thing I notice is that I never see a
file hobbit_vmstat.<hostname>...even now, while there is still data
actually being reported in the graph (it's only 7:44AM at present),
there is no vmstat file without the PID as part of the file name
present. Not that I think that is the problem.
Is there anything I can do to increase verbosity in the client and/or
server logging to determine if there's some sort of communication issue
(or lack thereof)?
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Wednesday, November 14, 2012 6:20 PM
To: Mike Burger
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 15 November 2012 02:04, Mike Burger <user-c26873f0522a@xymon.invalid>
wrote:
Silly question time...is that code in the server or client?
There are no silly questions: all questions lead to enlightenment!
The parser runs on the server.
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments,
is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal
law. If you
are not the intended recipient, please contact the sender by reply
e-mail,
delete this email and destroy all copies of the message.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Mike Burger
That did it. I'm not sure if MAXMSG_STATUS or MAXMSG_DATA is the culprit, though I suspect it's _DATA.
▸
From: Mike Burger
Sent: Thursday, November 15, 2012 8:08 AM
To: Mike Burger; 'Jeremy Laidman'
Cc: 'xymon at xymon.com'
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok...I've modified the following options in the xymonserver.cfg file:
MAXMSG_CLIENT changed from 512 to 1024
MAXMSG_STATUS changed from 256 to 512
MAXMSG_DATA changed from 256 to 512
I'll be watching to see how things look as we get to/past the 9AM hour.
From: Mike Burger
Sent: Thursday, November 15, 2012 8:03 AM
To: Mike Burger; Jeremy Laidman
Cc: xymon at xymon.com
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I think I've found it in the server's xymond.log:
Oversize status msg from <client IP> for <hostname>:procs truncated
(n=<some #>, limit=262144)
Oversize data/client msg from <client IP <truncated (n=<some #>,
limit=524288).
Now, I need to figure out how to change those limits.
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf
Of Mike Burger
Sent: Thursday, November 15, 2012 7:46 AM
To: Jeremy Laidman
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
Good to know...I was trying (unsuccessfully) to compile the 4.3.10
client for AIX, in case it was the client side.
On the other side...I did stop the client, killed any remaining vmstat
processes and started the client back up. Initially, only one vmstat
process was running, and then eventually there were two at any given
time, again. They kick off every 5 minutes, outputting to a vmstat file
tied to that process' PID (hobbit_vmstat.<hostname>.PID), and at the end
of the run, the PID specific vmstat file is supposed to be renamed to
hobbit_vmstat.<hostname>. The one thing I notice is that I never see a
file hobbit_vmstat.<hostname>...even now, while there is still data
actually being reported in the graph (it's only 7:44AM at present),
there is no vmstat file without the PID as part of the file name
present. Not that I think that is the problem.
Is there anything I can do to increase verbosity in the client and/or
server logging to determine if there's some sort of communication issue
(or lack thereof)?
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Wednesday, November 14, 2012 6:20 PM
To: Mike Burger
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 15 November 2012 02:04, Mike Burger <user-c26873f0522a@xymon.invalid>
wrote:
Silly question time...is that code in the server or client?
There are no silly questions: all questions lead to enlightenment!
The parser runs on the server.
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments,
is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal
law. If you
are not the intended recipient, please contact the sender by reply
e-mail,
delete this email and destroy all copies of the message.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Jeremy Laidman
▸
On 16 November 2012 02:48, Mike Burger <user-c26873f0522a@xymon.invalid>wrote:
That did it. I’m not sure if MAXMSG_STATUS or MAXMSG_DATA is the culprit, though I suspect it’s _DATA.
Nice one. Pretty sure it would be MAXMSG_CLIENT, which relates to the client messages. I have this set to 2048 on my server because I have some particularly fast-moving log files I want to monitor on a couple of my servers. Rather than increasing the limit, you might want to find out why the client message is so large and try to reduce the size if you can. Perhaps the server is logging heeeeaps of messages into a logfile that's being monitored. Have a browse through the client data and see if there's one section that's really big and can probably be adjusted or eliminated. $ xymoncmd xymon localhost 'clientlog name-of-client' | less If it's log data, and you don't actually report on log messages, you might consider removing the "log:" entry from client-local.cfg on the Xymon server - either from the [aix] section, or (if it exist) from the [name-of-host] section. Or you can have the client put a cap on the size of log data being sent back in its client message by adjusting the "log:" entry in the file client-local.cfg on the server. This limit is fetched by the client from the server after it reports its client data. So an adjustment on the server can take up to 10 minutes to be honoured by the client. The default for AIX syslog.log is 10k, so it's unlikely that this alone could push your client messages beyond 512k. J
list Chris Morris
I think Mike said yesterday that is was his "procs" section that was causing the client to generate oversized client message. This test precedes the vmstat test in xymonclient-aix.sh From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Jeremy Laidman Sent: 15 November 2012 23:17
▸
To: Mike Burger
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 16 November 2012 02:48, Mike Burger
<user-c26873f0522a@xymon.invalid> wrote:
That did it. I'm not sure if MAXMSG_STATUS or
MAXMSG_DATA is the culprit, though I suspect it's _DATA.
Nice one. Pretty sure it would be MAXMSG_CLIENT, which relates
to the client messages. I have this set to 2048 on my server because I
have some particularly fast-moving log files I want to monitor on a
couple of my servers.
Rather than increasing the limit, you might want to find out why
the client message is so large and try to reduce the size if you can.
Perhaps the server is logging heeeeaps of messages into a logfile that's
being monitored. Have a browse through the client data and see if
there's one section that's really big and can probably be adjusted or
eliminated.
$ xymoncmd xymon localhost 'clientlog name-of-client' | less
If it's log data, and you don't actually report on log messages,
you might consider removing the "log:" entry from client-local.cfg on
the Xymon server - either from the [aix] section, or (if it exist) from
the [name-of-host] section.
Or you can have the client put a cap on the size of log data
being sent back in its client message by adjusting the "log:" entry in
the file client-local.cfg on the server. This limit is fetched by the
client from the server after it reports its client data. So an
adjustment on the server can take up to 10 minutes to be honoured by the
client. The default for AIX syslog.log is 10k, so it's unlikely that
this alone could push your client messages beyond 512k.
J
****************************************************************************
The information contained in this email is intended only for the use of the intended recipient at the email address to which it has been addressed. If the reader of this message is not an intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination or copying of the message or associated attachments is strictly prohibited. If you have received this email in error, please contact the sender by return email or call 01793 877777 and ask for the sender and then delete it immediately from your system.Please note that neither the RWE Group of Companies nor the sender accepts any responsibility for viruses and it is your responsibility to scan attachments (if any).
*****************************************************************************
list Mike Burger
Right...it wasn't logging, but CPU utilization data that was missing in the graphs. Logging was fine, and all other graphs were fine...just the CPU utilization graph was having issues. J
▸
From: user-e510f6c03e57@xymon.invalid [mailto:user-e510f6c03e57@xymon.invalid]
Sent: Friday, November 16, 2012 4:02 AM
To: user-71895fb2e44c@xymon.invalid; Mike Burger
Cc: xymon at xymon.com
Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I think Mike said yesterday that is was his "procs" section that was
causing the client to generate oversized client message. This test
precedes the vmstat test in xymonclient-aix.sh
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com]
On Behalf Of Jeremy Laidman
Sent: 15 November 2012 23:17
To: Mike Burger
Cc: xymon at xymon.com
Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 16 November 2012 02:48, Mike Burger
<user-c26873f0522a@xymon.invalid> wrote:
That did it. I'm not sure if MAXMSG_STATUS or
MAXMSG_DATA is the culprit, though I suspect it's _DATA.
Nice one. Pretty sure it would be MAXMSG_CLIENT, which relates
to the client messages. I have this set to 2048 on my server because I
have some particularly fast-moving log files I want to monitor on a
couple of my servers.
Rather than increasing the limit, you might want to find out why
the client message is so large and try to reduce the size if you can.
Perhaps the server is logging heeeeaps of messages into a logfile that's
being monitored. Have a browse through the client data and see if
there's one section that's really big and can probably be adjusted or
eliminated.
$ xymoncmd xymon localhost 'clientlog name-of-client' | less
If it's log data, and you don't actually report on log messages,
you might consider removing the "log:" entry from client-local.cfg on
the Xymon server - either from the [aix] section, or (if it exist) from
the [name-of-host] section.
Or you can have the client put a cap on the size of log data
being sent back in its client message by adjusting the "log:" entry in
the file client-local.cfg on the server. This limit is fetched by the
client from the server after it reports its client data. So an
adjustment on the server can take up to 10 minutes to be honoured by the
client. The default for AIX syslog.log is 10k, so it's unlikely that
this alone could push your client messages beyond 512k.
J
************************************************************************
****
The information contained in this email is intended only for the use of
the intended recipient at the email address to which it has been
addressed. If the reader of this message is not an intended recipient,
you are hereby notified that you have received this document in error
and that any review, dissemination or copying of the message or
associated attachments is strictly prohibited. If you have received this
email in error, please contact the sender by return email or call 01793
877777 and ask for the sender and then delete it immediately from your
system.Please note that neither the RWE Group of Companies nor the
sender accepts any responsibility for viruses and it is your
responsibility to scan attachments (if any).
************************************************************************
*****
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for
the sole use of the intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure, alteration or
distribution is strictly prohibited and may violate state or federal law. If you
are not the intended recipient, please contact the sender by reply e-mail,
delete this email and destroy all copies of the message.
list Jeremy Laidman
▸
On 16 November 2012 20:01, <user-e510f6c03e57@xymon.invalid> wrote:
** I think Mike said yesterday that is was his "procs" section that was causing the client to generate oversized client message. This test precedes the vmstat test in xymonclient-aix.sh
Yes you are correct. I was confusing "client messages" with "client data". J