Solaris 10 swap not showing on memory alert correctly?
list David W David Gore
Wed Jun 26 16:26:48 GMT 2013 - Memory low Memory Used Total Percentage yellow Physical 23355M 24576M 95% green Swap 106M 35913M 0% From the client data: [swap] total: 18459352k bytes allocated + 3036456k reserved = 21495808k used, 32308920k available I would expect to see the 21495808k used number? Where is it getting the 106M number? Anyone else seen this on Solaris? ~David
list Jeremy Laidman
▸
On 27 June 2013 02:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid> wrote:
Where is it getting the 106M number? Anyone else seen this on Solaris? * ***
Yes, same here. It gets the number from the [swaplist] section (from `swap -l`) of your client message, in preference to the [swap] section (from `swap -s`). [Reference: xymond/client/solaris.c, in function handle_solaris_client()] So the real question is, why does "swap -s" and "swap -l" give different results? The man page for swap indicates that "swap -s" includes swap space in the form of physical memory (in addition to swap partitions and files). Physical memory used for swap?! Huh!? I really don't know how the Solaris memory management works! J
list Vernon Everett
Solaris and swap is tricky to nail down, specifically because different commands will report different values, by calculating the values in different ways. You need to choose your commands carefully, depending on what value you need to know. This might help explain things. http://www.f3partners.com/blog/bid/49584/Solaris-Swap-Q-A It all becomes rather fuzzy when you consider what ZFS is doing in your memory. Also, in a zone, all bets are off. Caps in particular, on memory or swap, can completely muddy the water. All the checks within the zone, will show the max available as being the capped value. However when you interrogate the kernel for amount free, it returns what the kernel can see. And the kernel can see everything, because it exists in the global zone. Regards Vernon P.S. WARNING: Thinking too hard about this can cause your brain to melt. :)
▸
On 27 June 2013 00:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid> wrote:
Wed Jun 26 16:26:48 GMT 2013 - Memory low**** Memory Used Total Percentage**** yellow Physical 23355M 24576M 95%**** green Swap 106M 35913M 0%**** ** ** From the client data:**** ** ** [swap]**** total: 18459352k bytes allocated + 3036456k reserved = 21495808k used, 32308920k available**** ** ** I would expect to see the 21495808k used number?**** ** ** Where is it getting the 106M number? Anyone else seen this on Solaris? * *** ** ** ~David**** ** **
--
"Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton
list David W David Gore
Yeah it is not a zone there are all sorts of annoyances when using a zone and you do not have access to root. I will take a look at the article but it may not answer my question of why Xymon is doing what it is doing. 'top' also reports the same thing as swap -s, not that is any guarantee of accuracy for such an old tool. 'top' is still one of my favorite tools although I wouldn't mind having the multi-cpu version that I see running on my hp-ux boxes running on Solaris. ~David
▸
From: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] Sent: Wednesday, June 26, 2013 11:00 PM To: Gore, David W (David) Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Solaris and swap is tricky to nail down, specifically because different commands will report different values, by calculating the values in different ways. You need to choose your commands carefully, depending on what value you need to know. This might help explain things. http://www.f3partners.com/blog/bid/49584/Solaris-Swap-Q-A It all becomes rather fuzzy when you consider what ZFS is doing in your memory. Also, in a zone, all bets are off. Caps in particular, on memory or swap, can completely muddy the water. All the checks within the zone, will show the max available as being the capped value. However when you interrogate the kernel for amount free, it returns what the kernel can see. And the kernel can see everything, because it exists in the global zone. Regards Vernon P.S. WARNING: Thinking too hard about this can cause your brain to melt. :) On 27 June 2013 00:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid<mailto:user-368fd67cc6bd@xymon.invalid>> wrote: Wed Jun 26 16:26:48 GMT 2013 - Memory low Memory Used Total Percentage yellow Physical 23355M 24576M 95% green Swap 106M 35913M 0% From the client data: [swap] total: 18459352k bytes allocated + 3036456k reserved = 21495808k used, 32308920k available I would expect to see the 21495808k used number? Where is it getting the 106M number? Anyone else seen this on Solaris? ~David -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list David W David Gore
You made me revisit the man pages. Without reading Vernon's link I think Xymon is correct and 'top' is incorrect. Both swap -s and swap -l are correct. I did already know Solaris does some magic foot work with real memory when calculating swap which may be in Vernon's article but I think I can explain the numbers in the AM. Time for sleep before the babies wake up. ~David
▸
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Wednesday, June 26, 2013 10:58 PM
To: Gore, David W (David)
Cc: xymon at xymon.com
Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly?
On 27 June 2013 02:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid<mailto:user-368fd67cc6bd@xymon.invalid>> wrote:
Where is it getting the 106M number? Anyone else seen this on Solaris?
Yes, same here. It gets the number from the [swaplist] section (from `swap -l`) of your client message, in preference to the [swap] section (from `swap -s`). [Reference: xymond/client/solaris.c, in function handle_solaris_client()]
So the real question is, why does "swap -s" and "swap -l" give different results? The man page for swap indicates that "swap -s" includes swap space in the form of physical memory (in addition to swap partitions and files). Physical memory used for swap?! Huh!? I really don't know how the Solaris memory management works!
J
list Jeremy Laidman
▸
On 27 June 2013 13:00, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:
P.S. WARNING: Thinking too hard about this can cause your brain to melt. :)
Awesome, thanks Vernon. I have just increased my awareness of the things I know nothing about! J
list David W David Gore
That link appears to only apply to Zones as I can see the numbers roughly adding up similarly on at least one of my Solaris machines allocated as a zone. Here is how I think the numbers add up on my non-zone traditional machine. prtconf|grep Memory Memory size: 24576 Megabytes Physical memory: 25,769,803,776 bytes swap -s total: 18615728k bytes allocated + 3112800k reserved = 21728528k used, 32078664k available Converted to bytes: (x1000)total: 18,615,728,000 bytes allocated + 3,112,800,000 reserved = 21,728,528,000 used, 32,078,664,000 available (x1024) 19,062,505,472 + 3,187,507,200 = 22,250,012,672 used, 32,848,551,936 available swap -l swapfile dev swaplo blocks free /dev/vx/dsk/bootdg/swapvol 278,60000 16 73552112 72369616 Let's look at the last two numbers 73,552,112*512 = 37658681344 bytes 72,369,616*512= 37,053,243,392 37,658,681,344 -37,053,243,392 = 605,437,952 == 577M swap used So to answer my own question. I think the first 2 numbers in swap -s represent physical memory. I think the last number is some amount from swap and some amount from physical memory. Some of what's in Vernon's article hints at where it all gets allocated. It would appear that Xymon just subtracts the last two numbers in swap -l to get swap used in megabytes.
▸
~David From: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] Sent: Wednesday, June 26, 2013 11:00 PM To: Gore, David W (David) Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Solaris and swap is tricky to nail down, specifically because different commands will report different values, by calculating the values in different ways. You need to choose your commands carefully, depending on what value you need to know. This might help explain things. http://www.f3partners.com/blog/bid/49584/Solaris-Swap-Q-A It all becomes rather fuzzy when you consider what ZFS is doing in your memory. Also, in a zone, all bets are off. Caps in particular, on memory or swap, can completely muddy the water. All the checks within the zone, will show the max available as being the capped value. However when you interrogate the kernel for amount free, it returns what the kernel can see. And the kernel can see everything, because it exists in the global zone. Regards Vernon P.S. WARNING: Thinking too hard about this can cause your brain to melt. :) On 27 June 2013 00:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid<mailto:user-368fd67cc6bd@xymon.invalid>> wrote: Wed Jun 26 16:26:48 GMT 2013 - Memory low Memory Used Total Percentage yellow Physical 23355M 24576M 95% green Swap 106M 35913M 0% From the client data: [swap] total: 18459352k bytes allocated + 3036456k reserved = 21495808k used, 32308920k available I would expect to see the 21495808k used number? Where is it getting the 106M number? Anyone else seen this on Solaris? ~David -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list David W David Gore
Just a heads up. So unfortunately, I am going to have to try something different. One of my xymon scripts got hung and launched every 5 minutes and filled /tmp and swap was consumed as reported by swap -s but not swap -l. Production processes could not allocate memory and Xymon was blissfully unaware reporting 0 swap used and memory Ok. Here is a message from the log (/var/adm/messages). Of course, we added this message to our alert list. Aug 24 15:10:24 ourHost genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 19464 (ksh) swap as shown by Xymon at 0 used memory less than 50% of 128G used Xymon client 4.3.10 Xymon server 4.3.12 Server monitored is a SUN T5240 running Solaris 10 Ok, I just realized this is a bit odd isn't it? Out of swap but plenty of physical memory left. Time to Google. David W Gore
▸
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Gore, David W (David) Sent: Thursday, June 27, 2013 9:06 AM To: Vernon Everett Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? That link appears to only apply to Zones as I can see the numbers roughly adding up similarly on at least one of my Solaris machines allocated as a zone. Here is how I think the numbers add up on my non-zone traditional machine. prtconf|grep Memory Memory size: 24576 Megabytes Physical memory: 25,769,803,776 bytes swap -s total: 18615728k bytes allocated + 3112800k reserved = 21728528k used, 32078664k available Converted to bytes: (x1000)total: 18,615,728,000 bytes allocated + 3,112,800,000 reserved = 21,728,528,000 used, 32,078,664,000 available (x1024) 19,062,505,472 + 3,187,507,200 = 22,250,012,672 used, 32,848,551,936 available swap -l swapfile dev swaplo blocks free /dev/vx/dsk/bootdg/swapvol 278,60000 16 73552112 72369616 Let's look at the last two numbers 73,552,112*512 = 37658681344 bytes 72,369,616*512= 37,053,243,392 37,658,681,344 -37,053,243,392 = 605,437,952 == 577M swap used So to answer my own question. I think the first 2 numbers in swap -s represent physical memory. I think the last number is some amount from swap and some amount from physical memory. Some of what's in Vernon's article hints at where it all gets allocated. It would appear that Xymon just subtracts the last two numbers in swap -l to get swap used in megabytes. ~David From: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] Sent: Wednesday, June 26, 2013 11:00 PM To: Gore, David W (David) Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Solaris and swap is tricky to nail down, specifically because different commands will report different values, by calculating the values in different ways. You need to choose your commands carefully, depending on what value you need to know. This might help explain things. http://www.f3partners.com/blog/bid/49584/Solaris-Swap-Q-A It all becomes rather fuzzy when you consider what ZFS is doing in your memory. Also, in a zone, all bets are off. Caps in particular, on memory or swap, can completely muddy the water. All the checks within the zone, will show the max available as being the capped value. However when you interrogate the kernel for amount free, it returns what the kernel can see. And the kernel can see everything, because it exists in the global zone. Regards Vernon P.S. WARNING: Thinking too hard about this can cause your brain to melt. :) On 27 June 2013 00:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid> wrote: Wed Jun 26 16:26:48 GMT 2013 - Memory low Memory Used Total Percentage yellow Physical 23355M 24576M 95% green Swap 106M 35913M 0% From the client data: [swap] total: 18459352k bytes allocated + 3036456k reserved = 21495808k used, 32308920k available I would expect to see the 21495808k used number? Where is it getting the 106M number? Anyone else seen this on Solaris? ~David -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
list Jim Smith
David...I just emailed you a script that I use on Solaris 8. Older than dirt, but you might be able to make use of it. Explanation in the email. If you don't receive it (i.e. your email server rejects the file), holler back and I'll paste the entire thing into an email. On second thought, this server is older than Solaris 8. Here is what 'uname' says: "SunOS selitsolaris01 5.5.1 Generic_103640-40 sun4u sparc SUNW,Ultra-2". My fellow IT folks are hounding me to let them throw it in the dumpster. It is running "Big Brother". I have "Hobbit" on an AIX server. I've yet to install "Xymon". Enjoy! Jim Smith
▸
-----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Gore, David W (David) Sent: Monday, August 26, 2013 4:50 PM To: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Just a heads up. So unfortunately, I am going to have to try something different. One of my xymon scripts got hung and launched every 5 minutes and filled /tmp and swap was consumed as reported by swap -s but not swap -l. Production processes could not allocate memory and Xymon was blissfully unaware reporting 0 swap used and memory Ok. Here is a message from the log (/var/adm/messages). Of course, we added this message to our alert list. Aug 24 15:10:24 ourHost genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 19464 (ksh) swap as shown by Xymon at 0 used memory less than 50% of 128G used Xymon client 4.3.10 Xymon server 4.3.12 Server monitored is a SUN T5240 running Solaris 10 Ok, I just realized this is a bit odd isn't it? Out of swap but plenty of physical memory left. Time to Google. David W Gore From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Gore, David W (David) Sent: Thursday, June 27, 2013 9:06 AM To: Vernon Everett Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? That link appears to only apply to Zones as I can see the numbers roughly adding up similarly on at least one of my Solaris machines allocated as a zone. Here is how I think the numbers add up on my non-zone traditional machine. prtconf|grep Memory Memory size: 24576 Megabytes Physical memory: 25,769,803,776 bytes swap -s total: 18615728k bytes allocated + 3112800k reserved = 21728528k used, 32078664k available Converted to bytes: (x1000)total: 18,615,728,000 bytes allocated + 3,112,800,000 reserved = 21,728,528,000 used, 32,078,664,000 available (x1024) 19,062,505,472 + 3,187,507,200 = 22,250,012,672 used, 32,848,551,936 available swap -l swapfile dev swaplo blocks free /dev/vx/dsk/bootdg/swapvol 278,60000 16 73552112 72369616 Let's look at the last two numbers 73,552,112*512 = 37658681344 bytes 72,369,616*512= 37,053,243,392 37,658,681,344 -37,053,243,392 = 605,437,952 == 577M swap used So to answer my own question. I think the first 2 numbers in swap -s represent physical memory. I think the last number is some amount from swap and some amount from physical memory. Some of what's in Vernon's article hints at where it all gets allocated. It would appear that Xymon just subtracts the last two numbers in swap -l to get swap used in megabytes. ~David From: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] Sent: Wednesday, June 26, 2013 11:00 PM To: Gore, David W (David) Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Solaris and swap is tricky to nail down, specifically because different commands will report different values, by calculating the values in different ways. You need to choose your commands carefully, depending on what value you need to know. This might help explain things. http://www.f3partners.com/blog/bid/49584/Solaris-Swap-Q-A It all becomes rather fuzzy when you consider what ZFS is doing in your memory. Also, in a zone, all bets are off. Caps in particular, on memory or swap, can completely muddy the water. All the checks within the zone, will show the max available as being the capped value. However when you interrogate the kernel for amount free, it returns what the kernel can see. And the kernel can see everything, because it exists in the global zone. Regards Vernon P.S. WARNING: Thinking too hard about this can cause your brain to melt. :) On 27 June 2013 00:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid> wrote: Wed Jun 26 16:26:48 GMT 2013 - Memory low Memory Used Total Percentage yellow Physical 23355M 24576M 95% green Swap 106M 35913M 0% From the client data: [swap] total: 18459352k bytes allocated + 3036456k reserved = 21495808k used, 32308920k available I would expect to see the 21495808k used number? Where is it getting the 106M number? Anyone else seen this on Solaris? ~David -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton
This electronic mail and any attached documents are intended solely for the named addressee(s) and contain confidential information. If you are not an addressee, or responsible for delivering this email to an addressee, you have received this email in error and are notified that reading, copying, or disclosing this email is prohibited. If you received this email in error, immediately reply to the sender and delete the message completely from your computer system.
list David W David Gore
Thanks Jim. I am not sure it will work properly on Solaris 10 since it calls some commands that do not appear to exist on Solaris 10. I wrote a PERL script anyway using the swap -s statistics. Thanks for taking the time to send it to me regardless. David W Gore
▸
-----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Smith, Jim Sent: Tuesday, August 27, 2013 3:17 PM To: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? David...I just emailed you a script that I use on Solaris 8. Older than dirt, but you might be able to make use of it. Explanation in the email. If you don't receive it (i.e. your email server rejects the file), holler back and I'll paste the entire thing into an email. On second thought, this server is older than Solaris 8. Here is what 'uname' says: "SunOS selitsolaris01 5.5.1 Generic_103640-40 sun4u sparc SUNW,Ultra-2". My fellow IT folks are hounding me to let them throw it in the dumpster. It is running "Big Brother". I have "Hobbit" on an AIX server. I've yet to install "Xymon". Enjoy! Jim Smith -----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Gore, David W (David) Sent: Monday, August 26, 2013 4:50 PM To: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Just a heads up. So unfortunately, I am going to have to try something different. One of my xymon scripts got hung and launched every 5 minutes and filled /tmp and swap was consumed as reported by swap -s but not swap -l. Production processes could not allocate memory and Xymon was blissfully unaware reporting 0 swap used and memory Ok. Here is a message from the log (/var/adm/messages). Of course, we added this message to our alert list. Aug 24 15:10:24 ourHost genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 19464 (ksh) swap as shown by Xymon at 0 used memory less than 50% of 128G used Xymon client 4.3.10 Xymon server 4.3.12 Server monitored is a SUN T5240 running Solaris 10 Ok, I just realized this is a bit odd isn't it? Out of swap but plenty of physical memory left. Time to Google. David W Gore From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Gore, David W (David) Sent: Thursday, June 27, 2013 9:06 AM To: Vernon Everett Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? That link appears to only apply to Zones as I can see the numbers roughly adding up similarly on at least one of my Solaris machines allocated as a zone. Here is how I think the numbers add up on my non-zone traditional machine. prtconf|grep Memory Memory size: 24576 Megabytes Physical memory: 25,769,803,776 bytes swap -s total: 18615728k bytes allocated + 3112800k reserved = 21728528k used, 32078664k available Converted to bytes: (x1000)total: 18,615,728,000 bytes allocated + 3,112,800,000 reserved = 21,728,528,000 used, 32,078,664,000 available (x1024) 19,062,505,472 + 3,187,507,200 = 22,250,012,672 used, 32,848,551,936 available swap -l swapfile dev swaplo blocks free /dev/vx/dsk/bootdg/swapvol 278,60000 16 73552112 72369616 Let's look at the last two numbers 73,552,112*512 = 37658681344 bytes 72,369,616*512= 37,053,243,392 37,658,681,344 -37,053,243,392 = 605,437,952 == 577M swap used So to answer my own question. I think the first 2 numbers in swap -s represent physical memory. I think the last number is some amount from swap and some amount from physical memory. Some of what's in Vernon's article hints at where it all gets allocated. It would appear that Xymon just subtracts the last two numbers in swap -l to get swap used in megabytes. ~David From: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] Sent: Wednesday, June 26, 2013 11:00 PM To: Gore, David W (David) Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Solaris and swap is tricky to nail down, specifically because different commands will report different values, by calculating the values in different ways. You need to choose your commands carefully, depending on what value you need to know. This might help explain things. http://www.f3partners.com/blog/bid/49584/Solaris-Swap-Q-A It all becomes rather fuzzy when you consider what ZFS is doing in your memory. Also, in a zone, all bets are off. Caps in particular, on memory or swap, can completely muddy the water. All the checks within the zone, will show the max available as being the capped value. However when you interrogate the kernel for amount free, it returns what the kernel can see. And the kernel can see everything, because it exists in the global zone. Regards Vernon P.S. WARNING: Thinking too hard about this can cause your brain to melt. :) On 27 June 2013 00:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid> wrote: Wed Jun 26 16:26:48 GMT 2013 - Memory low Memory Used Total Percentage yellow Physical 23355M 24576M 95% green Swap 106M 35913M 0% From the client data: [swap] total: 18459352k bytes allocated + 3036456k reserved = 21495808k used, 32308920k available I would expect to see the 21495808k used number? Where is it getting the 106M number? Anyone else seen this on Solaris? ~David -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton This electronic mail and any attached documents are intended solely for the named addressee(s) and contain confidential information. If you are not an addressee, or responsible for delivering this email to an addressee, you have received this email in error and are notified that reading, copying, or disclosing this email is prohibited. If you received this email in error, immediately reply to the sender and delete the message completely from your computer system.
list Ryan Novosielski
I am nearly certain I've seen servers alert on Solaris 10 over swap. I'll have to get back to you. We ran 4.2.3 and 4.3.11.
▸
----- Original Message ----- From: Gore, David W (David) [mailto:user-368fd67cc6bd@xymon.invalid] Sent: Tuesday, August 27, 2013 10:21 PM To: Smith, Jim <user-dc30f243a817@xymon.invalid>; xymon at xymon.com <xymon at xymon.com> Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Thanks Jim. I am not sure it will work properly on Solaris 10 since it calls some commands that do not appear to exist on Solaris 10. I wrote a PERL script anyway using the swap -s statistics. Thanks for taking the time to send it to me regardless. David W Gore -----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Smith, Jim Sent: Tuesday, August 27, 2013 3:17 PM To: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? David...I just emailed you a script that I use on Solaris 8. Older than dirt, but you might be able to make use of it. Explanation in the email. If you don't receive it (i.e. your email server rejects the file), holler back and I'll paste the entire thing into an email. On second thought, this server is older than Solaris 8. Here is what 'uname' says: "SunOS selitsolaris01 5.5.1 Generic_103640-40 sun4u sparc SUNW,Ultra-2". My fellow IT folks are hounding me to let them throw it in the dumpster. It is running "Big Brother". I have "Hobbit" on an AIX server. I've yet to install "Xymon". Enjoy! Jim Smith -----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Gore, David W (David) Sent: Monday, August 26, 2013 4:50 PM To: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Just a heads up. So unfortunately, I am going to have to try something different. One of my xymon scripts got hung and launched every 5 minutes and filled /tmp and swap was consumed as reported by swap -s but not swap -l. Production processes could not allocate memory and Xymon was blissfully unaware reporting 0 swap used and memory Ok. Here is a message from the log (/var/adm/messages). Of course, we added this message to our alert list. Aug 24 15:10:24 ourHost genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 19464 (ksh) swap as shown by Xymon at 0 used memory less than 50% of 128G used Xymon client 4.3.10 Xymon server 4.3.12 Server monitored is a SUN T5240 running Solaris 10 Ok, I just realized this is a bit odd isn't it? Out of swap but plenty of physical memory left. Time to Google. David W Gore From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Gore, David W (David) Sent: Thursday, June 27, 2013 9:06 AM To: Vernon Everett Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? That link appears to only apply to Zones as I can see the numbers roughly adding up similarly on at least one of my Solaris machines allocated as a zone. Here is how I think the numbers add up on my non-zone traditional machine. prtconf|grep Memory Memory size: 24576 Megabytes Physical memory: 25,769,803,776 bytes swap -s total: 18615728k bytes allocated + 3112800k reserved = 21728528k used, 32078664k available Converted to bytes: (x1000)total: 18,615,728,000 bytes allocated + 3,112,800,000 reserved = 21,728,528,000 used, 32,078,664,000 available (x1024) 19,062,505,472 + 3,187,507,200 = 22,250,012,672 used, 32,848,551,936 available swap -l swapfile dev swaplo blocks free /dev/vx/dsk/bootdg/swapvol 278,60000 16 73552112 72369616 Let's look at the last two numbers 73,552,112*512 = 37658681344 bytes 72,369,616*512= 37,053,243,392 37,658,681,344 -37,053,243,392 = 605,437,952 == 577M swap used So to answer my own question. I think the first 2 numbers in swap -s represent physical memory. I think the last number is some amount from swap and some amount from physical memory. Some of what's in Vernon's article hints at where it all gets allocated. It would appear that Xymon just subtracts the last two numbers in swap -l to get swap used in megabytes. ~David From: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] Sent: Wednesday, June 26, 2013 11:00 PM To: Gore, David W (David) Cc: xymon at xymon.com Subject: Re: [Xymon] Solaris 10 swap not showing on memory alert correctly? Solaris and swap is tricky to nail down, specifically because different commands will report different values, by calculating the values in different ways. You need to choose your commands carefully, depending on what value you need to know. This might help explain things. http://www.f3partners.com/blog/bid/49584/Solaris-Swap-Q-A It all becomes rather fuzzy when you consider what ZFS is doing in your memory. Also, in a zone, all bets are off. Caps in particular, on memory or swap, can completely muddy the water. All the checks within the zone, will show the max available as being the capped value. However when you interrogate the kernel for amount free, it returns what the kernel can see. And the kernel can see everything, because it exists in the global zone. Regards Vernon P.S. WARNING: Thinking too hard about this can cause your brain to melt. :) On 27 June 2013 00:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid> wrote: Wed Jun 26 16:26:48 GMT 2013 - Memory low Memory Used Total Percentage yellow Physical 23355M 24576M 95% green Swap 106M 35913M 0% From the client data: [swap] total: 18459352k bytes allocated + 3036456k reserved = 21495808k used, 32308920k available I would expect to see the 21495808k used number? Where is it getting the 106M number? Anyone else seen this on Solaris? ~David -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton This electronic mail and any attached documents are intended solely for the named addressee(s) and contain confidential information. If you are not an addressee, or responsible for delivering this email to an addressee, you have received this email in error and are notified that reading, copying, or disclosing this email is prohibited. If you received this email in error, immediately reply to the sender and delete the message completely from your computer system.
list Ryan Novosielski
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
▸
On 06/26/2013 10:58 PM, Jeremy Laidman wrote:On 27 June 2013 02:38, Gore, David W (David) <user-368fd67cc6bd@xymon.invalid <mailto:user-368fd67cc6bd@xymon.invalid>> wrote: Where is it getting the 106M number? Anyone else seen this on Solaris? ____ Yes, same here. It gets the number from the [swaplist] section (from `swap -l`) of your client message, in preference to the [swap] section (from `swap -s`). [Reference: xymond/client/solaris.c, in function handle_solaris_client()] So the real question is, why does "swap -s" and "swap -l" give different results? The man page for swap indicates that "swap -s" includes swap space in the form of physical memory (in addition to swap partitions and files). Physical memory used for swap?! Huh!? I really don't know how the Solaris memory management works!
So I was about to write: "Another great question: why does one of (and
probably more, but not all) my machines NOT have a [swaplist] section
and therefore, possibly as a result, show the correct info? I'm
suspecting this is why I thought this worked fine, because it does, at
least on some of my equipment. Two machines are easy to compare
because they're both on the same page in Xymon. Both are Solaris 10u10
SPARC."
...and then I discovered that the working machine is running the Xymon
client v4.2.3 and the broken one is running Xymon 4.3.10. Hmmmmm.
- --
____*Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
|| \\UTGERS |---------------------*O*---------------------
||_// Biomedical | Ryan Novosielski - Sr. Systems Programmer
|| \\ and Health | user-46c89e614701@xymon.invalid - 973/972.0922 (2x0922)
|| \\ Sciences | OIT/EI-Academic Svcs. - ADMC 450, Newark
`'
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlIfw3oACgkQmb+gadEcsb5jyACfYbTlfCmdFcbfRXt/mgskFE5k
V4wAoMLHxrHtQVWOpuwN+VW4Y6PSx2Qd
=AQBO
-----END PGP SIGNATURE-----