Solaris vmstat and sar
list James Wade
All, Solaris vmstat is crummy for CPU Utilization. If I do a vmstat on a Solaris box that's having high load, I get 67% idle, yet sar shows 3% idle. Has anyone seen this before? sar -u 1 1 12:08:40 %usr %sys %wio %idle 12:08:41 46 44 6 4 vmstat 1 1 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s3 s3 s9 s1 in sy cs us sy id 0 4 0 68897248 24039040 673 3180 1550 15 14 0 0 12 12 0 0 12901 44078 19363 18 15 67 So, what's the work around because Hobbit is using vmstat? James
list Lars Ebeling
Hello, I don't think you should compare like that. They are snapshots and only reflecting the status at that moment. I did vmstat -n 5 5 on HP-UX and got: VM memory page faults avm free re at pi po fr de sr in sy cs 65466 9823 22 8 2 0 0 0 1 113 3 139 CPU cpu procs us sy id r b w 37 4 60 2 0 0 65466 9675 6 4 1 0 0 0 0 113 377 87 2 1 96 2 0 0 71121 8536 174 43 5 0 0 0 0 117 6766 142 23 13 65 2 4 0 71121 8991 201 98 3 0 0 0 0 98 8594 238 65 24 12 2 4 0 84056 8990 86 34 1 0 0 0 0 103 3979 120 5 3 92 5 1 0 # Here you can see that idle varies 60% 96% 65% 12% 92% Regards Lars ----- Original Message ----- From: James Wade To: user-ae9b8668bcde@xymon.invalid Sent: Monday, December 03, 2007 7:10 PM Subject: [hobbit] Solaris vmstat and sar
▸
All,
Solaris vmstat is crummy for CPU Utilization.
If I do a vmstat on a Solaris box that's having high load,
I get 67% idle, yet sar shows 3% idle.
Has anyone seen this before?
sar -u 1 1
12:08:40 %usr %sys %wio %idle
12:08:41 46 44 6 4
vmstat 1 1
kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s3 s3 s9 s1 in sy cs us sy id
0 4 0 68897248 24039040 673 3180 1550 15 14 0 0 12 12 0 0 12901 44078 19363 18 15 67
So, what's the work around because Hobbit is using vmstat?
James
list James Wade
I agree with you, but the snapshots that I emailed were taken at the same time. I'm now graphing both Sar CPU Utilization, and Hobbit CPU Utilization which uses vmstat. SAR shows the more correct CPU Utilization as compared to CPU Load (uptime). Also, "TOP" confirms that CPU is loaded. I have a multi-cpu version of top I use and all the CPU's are max'd with 1% or 2% idle. During this timeframe, cpu utilization using vmstat shows 67% idle.. I've seen this problem for a while now on multiple systems, but I just recently wrote my own sar script to put sar cpu into hobbit graphs to confirm this. Here's where I just did a 5 minute average in two different windows on the same box at the same time: vmstat shows 67% idle, sar, 26% idle: vmstat 300 2
▸
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s3 s3 s9 s1 in sy cs us sy
id
0 4 0 68896560 24040168 673 3181 1548 15 14 0 0 12 12 0 0 12938 44125 19506
18 15 67
0 4 0 68896480 24040352 673 3181 1548 15 14 0 0 12 12 0 0 12942 44130 19525
18 15 67
$ sar 300 1
12:44:17 %usr %sys %wio %idle
12:49:17 31 32 11 26
TOP shows sar is correct:
load averages: 18.56, 17.41, 17.71
12:54:14
866 processes: 836 sleeping, 1 running, 7 stopped, 22 on cpu
CPU states: 14.6% idle, 36.6% user, 32.4% kernel, 16.5% iowait, 0.0% swap
Memory: 48G real, 24G free, 18G swap in use, 65G swap free
James
▸
From: Lars Ebeling [mailto:user-1fecd3eafd52@xymon.invalid]
Sent: Monday, December 03, 2007 12:28 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Solaris vmstat and sar
Hello,
I don't think you should compare like that. They are snapshots and only
reflecting the status at that moment.
I did vmstat -n 5 5 on HP-UX and got:
VM
memory page faults
avm free re at pi po fr de sr in sy cs
65466 9823 22 8 2 0 0 0 1 113 3 139
CPU
cpu procs
us sy id r b w
37 4 60 2 0 0
65466 9675 6 4 1 0 0 0 0 113 377 87
2 1 96 2 0 0
71121 8536 174 43 5 0 0 0 0 117 6766 142
23 13 65 2 4 0
71121 8991 201 98 3 0 0 0 0 98 8594 238
65 24 12 2 4 0
84056 8990 86 34 1 0 0 0 0 103 3979 120
5 3 92 5 1 0
#
Here you can see that idle varies 60% 96% 65% 12% 92%
Regards
Lars
----- Original Message -----
From: James <mailto:user-659655b2ea05@xymon.invalid> Wade
To: user-ae9b8668bcde@xymon.invalid
Sent: Monday, December 03, 2007 7:10 PM
Subject: [hobbit] Solaris vmstat and sar
All,
Solaris vmstat is crummy for CPU Utilization.
If I do a vmstat on a Solaris box that's having high load,
I get 67% idle, yet sar shows 3% idle.
Has anyone seen this before?
sar -u 1 1
12:08:40 %usr %sys %wio %idle
12:08:41 46 44 6 4
vmstat 1 1
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s3 s3 s9 s1 in sy cs us sy
id
0 4 0 68897248 24039040 673 3180 1550 15 14 0 0 12 12 0 0 12901 44078 19363
18 15 67
So, what's the work around because Hobbit is using vmstat?
James
list Holly Lund
vmstat first output is invalid. doing vmstat 5 5 the other 4 should change to reflect sar Holly Lund XXX-XXX-XXXX XXX-XXX-XXXX
▸
-----Original Message-----
From: James Wade [mailto:user-659655b2ea05@xymon.invalid]
Sent: Monday, December 03, 2007 1:55 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Solaris vmstat and sar
I agree with you, but the snapshots that I emailed were taken
at the same time.
I'm now graphing both Sar CPU Utilization, and Hobbit CPU Utilization
which uses vmstat. SAR shows the more correct CPU Utilization as
compared
to CPU Load (uptime). Also, "TOP" confirms that CPU is loaded. I have
a multi-cpu version of top I use and all the CPU's are max'd with 1% or
2% idle.
During this timeframe, cpu utilization using vmstat shows 67% idle..
I've seen this problem for a while now on multiple systems, but I just
recently
wrote my own sar script to put sar cpu into hobbit graphs to confirm
this.
Here's where I just did a 5 minute average in two different windows on
the same box
at the same time: vmstat shows 67% idle, sar, 26% idle:
vmstat 300 2
kthr memory page disk faults
cpu
r b w swap free re mf pi po fr de sr s3 s3 s9 s1 in sy cs us
sy
id
0 4 0 68896560 24040168 673 3181 1548 15 14 0 0 12 12 0 0 12938 44125
19506
18 15 67
0 4 0 68896480 24040352 673 3181 1548 15 14 0 0 12 12 0 0 12942 44130
19525
18 15 67
$ sar 300 1
12:44:17 %usr %sys %wio %idle
12:49:17 31 32 11 26
TOP shows sar is correct:
load averages: 18.56, 17.41, 17.71
12:54:14
866 processes: 836 sleeping, 1 running, 7 stopped, 22 on cpu
CPU states: 14.6% idle, 36.6% user, 32.4% kernel, 16.5% iowait, 0.0%
swap
Memory: 48G real, 24G free, 18G swap in use, 65G swap free
James
From: Lars Ebeling [mailto:user-1fecd3eafd52@xymon.invalid]
Sent: Monday, December 03, 2007 12:28 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Solaris vmstat and sar
Hello,
I don't think you should compare like that. They are snapshots and only
reflecting the status at that moment.
I did vmstat -n 5 5 on HP-UX and got:
VM
memory page faults
avm free re at pi po fr de sr in sy
cs
65466 9823 22 8 2 0 0 0 1 113 3
139
CPU
cpu procs
us sy id r b w
37 4 60 2 0 0
65466 9675 6 4 1 0 0 0 0 113 377
87
2 1 96 2 0 0
71121 8536 174 43 5 0 0 0 0 117 6766
142
23 13 65 2 4 0
71121 8991 201 98 3 0 0 0 0 98 8594
238
65 24 12 2 4 0
84056 8990 86 34 1 0 0 0 0 103 3979
120
5 3 92 5 1 0
#
Here you can see that idle varies 60% 96% 65% 12% 92%
Regards
Lars
----- Original Message -----
From: James <mailto:user-659655b2ea05@xymon.invalid> Wade
To: user-ae9b8668bcde@xymon.invalid
Sent: Monday, December 03, 2007 7:10 PM
Subject: [hobbit] Solaris vmstat and sar
All,
Solaris vmstat is crummy for CPU Utilization.
If I do a vmstat on a Solaris box that's having high load,
I get 67% idle, yet sar shows 3% idle.
Has anyone seen this before?
sar -u 1 1
12:08:40 %usr %sys %wio %idle
12:08:41 46 44 6 4
vmstat 1 1
kthr memory page disk faults
cpu
r b w swap free re mf pi po fr de sr s3 s3 s9 s1 in sy cs us
sy
id
0 4 0 68897248 24039040 673 3180 1550 15 14 0 0 12 12 0 0 12901 44078
19363
18 15 67
So, what's the work around because Hobbit is using vmstat?
James
list James Wade
Tried that, It didn't work. You still get some lines that have incorrect data, like 67% idle. The sar data showed cpu idle at 1% or 0% all 5 lines. James
▸
-----Original Message-----
From: Lund, Holly [mailto:user-15ee9e30aac3@xymon.invalid]
Sent: Monday, December 03, 2007 1:12 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Solaris vmstat and sar
vmstat first output is invalid. doing vmstat 5 5 the other 4 should
change to reflect sar
Holly Lund
list Kenneth Langford
The vmstat first line shows the system average stats for systems since the last time the system was booted. Vmstat shows the average use over the 5 second interval in you example below of vmstat 5 5. Sar shows the stats at the time the check was taken. So you are not comparing apples to apples. So they are going to give you different results. Ken L.
▸
-----Original Message-----
From: James Wade [mailto:user-659655b2ea05@xymon.invalid]
Sent: Monday, December 03, 2007 2:34 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Solaris vmstat and sar
Tried that, It didn't work. You still get some lines that
have incorrect data, like 67% idle. The sar data showed cpu
idle at 1% or 0% all 5 lines.
James
-----Original Message-----
From: Lund, Holly [mailto:user-15ee9e30aac3@xymon.invalid]
Sent: Monday, December 03, 2007 1:12 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Solaris vmstat and sar
vmstat first output is invalid. doing vmstat 5 5 the other 4 should
change to reflect sar
Holly Lund
list James Wade
But you should not see a major difference like 67% idle to 1% idle.... I've tried using two screens on the same system comparing sar to vmstat and using different intervals. vmstat significantly understates cpu usage in comparison to top, uptime, and sar. Does any have any ideas? James
▸
-----Original Message-----
From: Langford, Kenneth [mailto:user-e8981d70691e@xymon.invalid]
Sent: Monday, December 03, 2007 2:13 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Solaris vmstat and sar
The vmstat first line shows the system average stats for systems since
the last time the system was booted. Vmstat shows the average use over
the 5 second interval in you example below of vmstat 5 5. Sar shows the
stats at the time the check was taken. So you are not comparing apples
to apples. So they are going to give you different results.
Ken L.
-----Original Message-----
From: James Wade [mailto:user-659655b2ea05@xymon.invalid]
Sent: Monday, December 03, 2007 2:34 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Solaris vmstat and sar
Tried that, It didn't work. You still get some lines that
have incorrect data, like 67% idle. The sar data showed cpu
idle at 1% or 0% all 5 lines.
James
-----Original Message-----
From: Lund, Holly [mailto:user-15ee9e30aac3@xymon.invalid]
Sent: Monday, December 03, 2007 1:12 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Solaris vmstat and sar
vmstat first output is invalid. doing vmstat 5 5 the other 4 should
change to reflect sar
Holly Lund