Memory check
list David Gilmore
My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When the server is backed, up using Retrospect client, REAL memory usage spikes from 34% to 97% and stays at that level until a reboot. When I check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional processes between 1% and 2%. Everything else is well under 1%. What exactly is hobbit reporting on when it says that Physical/Real memory is at 97%, Actual memory is at 17%, and Swap is at 0%? Thanks, David Gilmore Consultant Stenhouse Consulting, LLC. X Traverse St Providence, RI XXXXX XXX.XXX.XXXX XXX.XXX.XXXX (fax)
list Ralph Mitchell
I'm not sure on the details, but I believe the Linux kernel loads up a large percentage of available memory with disk buffers as files are read. Those buffers don't really count against any process and will be evicted as soon as an actual process asks for real memory. The idea is that it speeds up disk access for those files that are in memory, and doesn't cost much (in machine time) to keep them around until the space is needed for something else. Memory usage spikes during a backup because it's reading *all* the files on disk... Ralph Mitchell
▸
On 2/7/06, David Gilmore <user-70507ff7198d@xymon.invalid> wrote:My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When the server is backed, up using Retrospect client, REAL memory usage spikes from 34% to 97% and stays at that level until a reboot. When I check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional processes between 1% and 2%. Everything else is well under 1%. What exactly is hobbit reporting on when it says that Physical/Real memory is at 97%, Actual memory is at 17%, and Swap is at 0%? Thanks, David Gilmore Consultant Stenhouse Consulting, LLC. X Traverse St Providence, RI XXXXX XXX.XXX.XXXX XXX.XXX.XXXX (fax)
list Henrik Størner
▸
On Tue, Feb 07, 2006 at 07:41:23PM -0500, David Gilmore wrote:
My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When the server is backed, up using Retrospect client, REAL memory usage spikes from 34% to 97% and stays at that level until a reboot. When I check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional processes between 1% and 2%. Everything else is well under 1%. What exactly is hobbit reporting on when it says that Physical/Real memory is at 97%, Actual memory is at 17%, and Swap is at 0%?
Hobbit reports the output from the "free" command. It probably looks
somewhat like this after you've run a backup:
total used free shared buffers cached
Mem: 646432 642172 4260 0 167676 136068
-/+ buffers/cache: 338428 308004
Swap: 511992 4 511988
The "Mem" line here tells you that there is 640 MB RAM installed, and
all except 4 MB is being "used". However, a lot of that is used for
"buffers" and "cache", which is the Linux kernel's dynamically resized
disk cache; if an application needs more RAM that is "free", the
disk cache/buffers are discarded and the memory made available to the
application.
So that's why the "-/+ buffers/cache" line is interesting: This shows
the used/free memory count if the buffers/cached is counted as "free"
memory. Hobbit report this as the "actual" memory count.
So a Linux system will practically always have a REAL memory usage
close to 100% (Linus Torvalds once said that "unused RAM is *wasted*
RAM, and there's no reason to spend lots of money on something that
isn't used" - quoting from memory). The ACTUAL memory usage (should)
be a lot less, and is what you'll want to keep an eye on.
Regards,
Henrik
list Rolf Schrittenlocher
Hi,
▸
On Tue, Feb 07, 2006 at 07:41:23PM -0500, David Gilmore wrote:My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When the server is backed, up using Retrospect client, REAL memory usage spikes from 34% to 97% and stays at that level until a reboot. When I check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional processes between 1% and 2%. Everything else is well under 1%. What exactly is hobbit reporting on when it says that Physical/Real memory is at 97%, Actual memory is at 17%, and Swap is at 0%?Hobbit reports the output from the "free" command. It probably looks somewhat like this after you've run a backup: total used free shared buffers cached Mem: 646432 642172 4260 0 167676 136068 -/+ buffers/cache: 338428 308004 Swap: 511992 4 511988 The "Mem" line here tells you that there is 640 MB RAM installed, and all except 4 MB is being "used". However, a lot of that is used for "buffers" and "cache", which is the Linux kernel's dynamically resized disk cache; if an application needs more RAM that is "free", the disk cache/buffers are discarded and the memory made available to the application. So that's why the "-/+ buffers/cache" line is interesting: This shows the used/free memory count if the buffers/cached is counted as "free" memory. Hobbit report this as the "actual" memory count. So a Linux system will practically always have a REAL memory usage close to 100% (Linus Torvalds once said that "unused RAM is *wasted* RAM, and there's no reason to spend lots of money on something that isn't used" - quoting from memory). The ACTUAL memory usage (should) be a lot less, and is what you'll want to keep an eye on.
We are working with Solaris 9 and for them unfortunately there is no ACTUAL MEMory presented. But PHYSical MEMory is always high on some machines runing a database in warm standby for the same reasons as Henrik explained. In other words: The way it is this test is useless for us. I'd prefer other columns of vmstat checked as io, etc. Did anyone work in this direction or where have I to dig in trying these changes? regards Rolf
-- Mit freundlichen Gruessen Rolf Schrittenlocher HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (XX) XX - XXX XXXXX Fax: (XX) XX XXX XXXX LBS: user-1e39a1813094@xymon.invalid Persoenlich: user-6ea8e907e200@xymon.invalid
list Stef Coene
▸
On Wednesday 08 February 2006 08:34, Rolf Schrittenlocher wrote:
We are working with Solaris 9 and for them unfortunately there is no ACTUAL MEMory presented. But PHYSical MEMory is always high on some machines runing a database in warm standby for the same reasons as Henrik explained. In other words: The way it is this test is useless for us. I'd prefer other columns of vmstat checked as io, etc. Did anyone work in this direction or where have I to dig in trying these changes?
Swap is usage is _not_ important. Most systems are a) using all available memory for disk cache, b) swapping not recently used programs to swap in favour for disk cache and c) doing prea-llocation of swap space (reserving disk cache even when it's not yet needed). So, memory usage is 100% and swap usage can be high even if you have enough memory and not using any swap. What you have to monitor is a) swap-in / swap-out and b) free list (I know this exist for AIX, but I don't know for other os's). -> The swap-in / swap-out statistics determine how much of swap is activly used by the system as memory (in best scenario this is most of the time 0 or at least sometimes 0). -> The free list determines how much memory is free and can be used for programs who really need it very fast (in best scenario this never turns 0). There are other numbers that are important like page scans and stuff, but page-in and page-out are the most important. That's why I mailed some days ago to this list to add extra graphs to the memory overview page. The answer was that this will be not so easy to implement. Stef
list Henrik Størner
▸
On Wed, Feb 08, 2006 at 08:34:21AM +0100, Rolf Schrittenlocher wrote:
We are working with Solaris 9 and for them unfortunately there is no ACTUAL MEMory presented. But PHYSical MEMory is always high on some machines runing a database in warm standby for the same reasons as Henrik explained. In other words: The way it is this test is useless for us. I'd prefer other columns of vmstat checked as io, etc. Did anyone work in this direction or where have I to dig in trying these changes?
Most of the data you want to look at is available in the vmstat data which Hobbit also collects. And it is highly OS dependant - there is no "one-size-fits-all" solution to memory monitoring or memory management. My plan is to make it possible for any test (cpu, memory ...) to have its status modified by some of the data collected by vmstat (currently, the vmstat data is only used to feed the graph data, not to change a color or trigger alerts). But that will require some serious re-arranging of how a status is handled internally by Hobbit (it needs a feed-back mechanism from the hobbitd modules back into hobbitd, and a mechanism for modifying the color of an existing status without causing huge amounts of history updates due to the color changes. Plus some way of configuring the whole thing. But then it would also let you do neat stuff like using the standard Hobbit configuration tools to configure alerts based on custom-built tests and things like that). Henrik
list David Gilmore
Ok I understand the concept. However, I don't want to continue to receive Alerts because Linux is doing exactly what it is designed to do. Does anyone have a script that can clear the buffers and stop hobbit from paging me? Can I modify the script to only alert when REAL memory is at 100% or higher? Or do I have to reboot my server ever morning to resolve this alert? I currently have the alerts disabled, but I am concerned that I could miss a critical error Dave
-----Original Message----- From: hobbit-return-5584-david=user-9e293dd11111@xymon.invalid [mailto:hobbit-return-5584-david=stenhouseconsulting.com at hswn. dk] On Behalf Of Henrik Stoerner Sent: Wednesday, February 08, 2006 1:45 AM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Memory check
▸
On Tue, Feb 07, 2006 at 07:41:23PM -0500, David Gilmore wrote:My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When > the server is backed, up using Retrospect client, REAL memory usage > spikes from 34% to 97% and stays at that level until a reboot. When I > check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional > processes between 1% and 2%. Everything else is well under 1%. What > exactly is hobbit reporting on when it says that Physical/Real memory > is at 97%, Actual memory is at 17%, and Swap is at 0%?Hobbit reports the output from the "free" command. It probably looks somewhat like this after you've run a backup: total used free shared buffers cached Mem: 646432 642172 4260 0 167676 136068 -/+ buffers/cache: 338428 308004 Swap: 511992 4 511988 The "Mem" line here tells you that there is 640 MB RAM installed, and all except 4 MB is being "used". However, a lot of that is used for "buffers" and "cache", which is the Linux kernel's dynamically resized disk cache; if an application needs more RAM that is "free", the disk cache/buffers are discarded and the memory made available to the application. So that's why the "-/+ buffers/cache" line is interesting: This shows the used/free memory count if the buffers/cached is counted as "free" memory. Hobbit report this as the "actual" memory count. So a Linux system will practically always have a REAL memory usage close to 100% (Linus Torvalds once said that "unused RAM is *wasted* RAM, and there's no reason to spend lots of money on something that isn't used" - quoting from memory). The ACTUAL memory usage (should) be a lot less, and is what you'll want to keep an eye on. Regards, Henrik
list Stef Coene
▸
On Wednesday 08 February 2006 15:26, David Gilmore wrote:
Ok I understand the concept. However, I don't want to continue to receive Alerts because Linux is doing exactly what it is designed to do. Does anyone have a script that can clear the buffers and stop hobbit from paging me? Can I modify the script to only alert when REAL memory is at 100% or higher? Or do I have to reboot my server ever morning to resolve this alert? I currently have the alerts disabled, but I am concerned that I could miss a critical error
I'm not sure if it can, but can't you alert on the swap usage? Stef
list Henrik Størner
▸
On Wed, Feb 08, 2006 at 09:26:31AM -0500, David Gilmore wrote:
Ok I understand the concept. However, I don't want to continue to receive Alerts because Linux is doing exactly what it is designed to do. Does anyone have a script that can clear the buffers and stop hobbit from paging me? Can I modify the script to only alert when REAL memory is at 100% or higher? Or do I have to reboot my server ever morning to resolve this alert? I currently have the alerts disabled, but I am concerned that I could miss a critical error
Assuming that you're using the Hobbit client on the Linux box, you
can just configure the client to only go red if the actual memory
usage goes above a certain threshold. In your hobbit-clients.cfg,
you would have
HOST=linux.foo.com
MEMPHYS 100 101
MEMACT 80 95
MEMSWAP 40 70
Then it will stay green as long as the "actual" memory usage is
below 80%, go yellow when it's between 80-95%, and go red when
it is at 95% or higher.
Regards,
Henrik
list Michael Lowery
Ok, I don't have a clear understanding of the memory check. I do
understand what has been discussed, but I have a Redhat 9 where the
"actual" memory utilized is 96%. However, when I run: ps vax --sort
-rss, I get the following. It looks like my actual memory usage
shouldn't be at 96%, is this correct?
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
2916 ? SL 0:00 470 298 2093 2388 0.9 [ntpd]
2948 pts/1 R 0:00 187 66 2561 684 0.2 ps vax --sort
-rss
2690 ? S 0:00 11252 265 6594 612 0.2 /usr/sbin/sshd
2692 pts/1 S 0:00 2666 588 3775 604 0.2 -bash
891 ? S 3:15 30426 24 1415 100 0.0 syslogd -m 0
31227 ? S 0:00 50692 33 1374 60 0.0
/home/hobbit/client/bin/hobbitlaunch
--config=/home/hobbit/client/etc/client
1 ? S 0:04 28806 23 1352 36 0.0 init
2878 ? S 0:00 7714 131 4480 36 0.0 [nqmgr]
2885 ? S 0:00 4231 190 4481 16 0.0 [smtpd]
2902 ? S 0:00 2566 130 4485 16 0.0 [cleanup]
895 ? S 0:00 124 18 1349 4 0.0 klogd -x
1029 ? S 0:01 520 265 3238 4 0.0 /usr/sbin/sshd
1043 ? S 0:00 97 129 1890 4 0.0 xinetd
-stayalive -reuse -pidfile /var/run/xinetd.pid
1142 ? S 0:00 340 56 1347 4 0.0 gpm -t ps/2 -m
/dev/psaux
1151 ? S 0:00 42450 19 1404 4 0.0 crond
1185 tty1 S 0:00 98 6 1345 4 0.0 /sbin/mingetty
tty1
1186 tty2 S 0:00 98 6 1345 4 0.0 /sbin/mingetty
tty2
1187 tty3 S 0:00 98 6 1345 4 0.0 /sbin/mingetty
tty3
1188 tty4 S 0:00 98 6 1345 4 0.0 /sbin/mingetty
tty4
1189 tty5 S 0:00 98 6 1345 4 0.0 /sbin/mingetty
tty5
1190 tty6 S 0:00 98 6 1345 4 0.0 /sbin/mingetty
tty6
31309 ? S 0:10 185817 16 9287 4 0.0 /usr/sbin/snmpd
-s -l /dev/null -P /var/run/snmpd -a
2941 ? S 0:00 473 588 1447 4 0.0 sh -c vmstat 300
2 1>/home/hobbit/client/tmp/hobbit_vmstat.2930 2>&1; mv /ho
2943 ? S 0:00 118 7 1408 4 0.0 vmstat 300 2
2 ? SW 0:00 0 0 0 0 0.0 [keventd]
Thanks,
Michael
▸
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Wednesday, February 08, 2006 9:47 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Memory check
On Wed, Feb 08, 2006 at 09:26:31AM -0500, David Gilmore wrote:Ok I understand the concept. However, I don't want to continue to receive Alerts because Linux is doing exactly what it is designed to do. Does anyone have a script that can clear the buffers and stop hobbit from paging me? Can I modify the script to only alert when REAL memory is at 100% or higher? Or do I have to reboot my server ever morning to resolve this alert? I currently have the alerts disabled, but I am concerned that
I
could miss a critical error
Assuming that you're using the Hobbit client on the Linux box, you
can just configure the client to only go red if the actual memory
usage goes above a certain threshold. In your hobbit-clients.cfg,
you would have
HOST=linux.foo.com
MEMPHYS 100 101
MEMACT 80 95
MEMSWAP 40 70
Then it will stay green as long as the "actual" memory usage is
below 80%, go yellow when it's between 80-95%, and go red when
it is at 95% or higher.
Regards,
Henrik
list Jeff Newman
I have seen in the past from various companies products graphs which plot memory usage by process, so you can see which process is using the most memory. In addition, there have been CPU usage per process graphs as well. Seeing the below made me think about the memory/per process. I tried the ps vax command on both an AIX and Redhat box and both worked. Would it be possible to implement this sort of a graph in hobbit? -Jeff
▸
On 5/1/06, Michael Lowery <user-89d72f0c2e3d@xymon.invalid> wrote:Ok, I don't have a clear understanding of the memory check. I do understand what has been discussed, but I have a Redhat 9 where the "actual" memory utilized is 96%. However, when I run: ps vax --sort -rss, I get the following. It looks like my actual memory usage shouldn't be at 96%, is this correct? PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 2916 ? SL 0:00 470 298 2093 2388 0.9 [ntpd] 2948 pts/1 R 0:00 187 66 2561 684 0.2 ps vax --sort -rss 2690 ? S 0:00 11252 265 6594 612 0.2 /usr/sbin/sshd 2692 pts/1 S 0:00 2666 588 3775 604 0.2 -bash 891 ? S 3:15 30426 24 1415 100 0.0 syslogd -m 0 31227 ? S 0:00 50692 33 1374 60 0.0 /home/hobbit/client/bin/hobbitlaunch --config=/home/hobbit/client/etc/client 1 ? S 0:04 28806 23 1352 36 0.0 init 2878 ? S 0:00 7714 131 4480 36 0.0 [nqmgr] 2885 ? S 0:00 4231 190 4481 16 0.0 [smtpd] 2902 ? S 0:00 2566 130 4485 16 0.0 [cleanup] 895 ? S 0:00 124 18 1349 4 0.0 klogd -x 1029 ? S 0:01 520 265 3238 4 0.0 /usr/sbin/sshd 1043 ? S 0:00 97 129 1890 4 0.0 xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid 1142 ? S 0:00 340 56 1347 4 0.0 gpm -t ps/2 -m /dev/psaux 1151 ? S 0:00 42450 19 1404 4 0.0 crond 1185 tty1 S 0:00 98 6 1345 4 0.0 /sbin/mingetty tty1 1186 tty2 S 0:00 98 6 1345 4 0.0 /sbin/mingetty tty2 1187 tty3 S 0:00 98 6 1345 4 0.0 /sbin/mingetty tty3 1188 tty4 S 0:00 98 6 1345 4 0.0 /sbin/mingetty tty4 1189 tty5 S 0:00 98 6 1345 4 0.0 /sbin/mingetty tty5 1190 tty6 S 0:00 98 6 1345 4 0.0 /sbin/mingetty tty6 31309 ? S 0:10 185817 16 9287 4 0.0 /usr/sbin/snmpd -s -l /dev/null -P /var/run/snmpd -a 2941 ? S 0:00 473 588 1447 4 0.0 sh -c vmstat 300 2 1>/home/hobbit/client/tmp/hobbit_vmstat.2930 2>&1; mv /ho 2943 ? S 0:00 118 7 1408 4 0.0 vmstat 300 2 2 ? SW 0:00 0 0 0 0 0.0 [keventd] Thanks, Michael -----Original Message----- From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] Sent: Wednesday, February 08, 2006 9:47 AM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Memory check On Wed, Feb 08, 2006 at 09:26:31AM -0500, David Gilmore wrote:Ok I understand the concept. However, I don't want to continue to receive Alerts because Linux is doing exactly what it is designed to do. Does anyone have a script that can clear the buffers and stop hobbit from paging me? Can I modify the script to only alert when REAL memory is at 100% or higher? Or do I have to reboot my server ever morning to resolve this alert? I currently have the alerts disabled, but I am concerned thatIcould miss a critical errorAssuming that you're using the Hobbit client on the Linux box, you can just configure the client to only go red if the actual memory usage goes above a certain threshold. In your hobbit-clients.cfg, you would have HOST=linux.foo.com MEMPHYS 100 101 MEMACT 80 95 MEMSWAP 40 70 Then it will stay green as long as the "actual" memory usage is below 80%, go yellow when it's between 80-95%, and go red when it is at 95% or higher. Regards, Henrik
list Henrik Størner
▸
On Thu, May 04, 2006 at 10:48:51AM -0500, Jeff Newman wrote:
I have seen in the past from various companies products graphs which plot memory usage by process, so you can see which process is using the most memory. In addition, there have been CPU usage per process graphs as well. Seeing the below made me think about the memory/per process. I tried the ps vax command on both an AIX and Redhat box and both worked. Would it be possible to implement this sort of a graph in hobbit?
What I'll do for now (the 4.2 release) is to use a "ps" command in the client that provides this information. That will allow us to experiment with the best way of tracking memory usage for individual processes. E.g. I do anticipate some problems with how you determine which processes should be tracked, and how to group them (e.g. you don't want to track individual "httpd" processes - you want a total for all of them). Regards, Henrik