Memstat for Solaris (10)

3 messages in this thread

list Vernon Everett · Thu, 20 May 2010 14:17:56 +0800 ·

Hi all

Just cobbled together a test to keep tabs on where memory is going on my
Solaris 10 servers, in particular, how much is going to kernel and ZFS
cache.
(It might work for other Solaris versions, but I haven't tried. YMMV)

Please let me know if you spot any bugs.

It's 2 scripts.
An mod to hobbitserver.cfg and hobbitgraph.cfg, and an update to
clientlaunch.cfg

Here goes.

On the client, add this to clientlaunch.cfg
[memstat]
        ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
        CMD $HOBBITCLIENTHOME/ext/memstat.ksh
        LOGFILE $HOBBITCLIENTHOME/logs/memstat.log
        INTERVAL 5m

Client side scripts.
# cat memstat.ksh
#!/bin/ksh
TEMPFILE=$BBTMP/memstat.tmp
COLOUR=green
ALERT=1  # 1= alert on high usage
YELLOW=75
RED=90
date > $TEMPFILE
echo >> $TEMPFILE
$HOBBITCLIENTHOME/ext/getmemstat.ksh > $TEMPFILE.data
cat $TEMPFILE.data  >> $TEMPFILE
if [ $ALERT -eq 1 ]
then
   cat $TEMPFILE.data | egrep "^Kernel|^Anon|^Exec" | cut -c52-56 | while
read a
   do
      if [ $a -gt $YELLOW -a "$COLOUR" = "green" ]
      then
          COLOUR=yellow
      else
         [ $a -gt $RED ] && COLOUR=red
      fi
   done
fi
echo '<p><font size="1" color="black">' >> $TEMPFILE
cat $TEMPFILE.data | egrep -v -- "-----|Summary|^$|^Total|^Physical" \
                   | sed "s/^Free //g" \
                   | sed "s/[(|)|%]//g" \
                   | awk '{ print $1" : "$NF }' >> $TEMPFILE
echo '</font></p>' >> $TEMPFILE
rm $TEMPFILE.data
$BB $BBDISP "status $MACHINE.memstat $COLOUR $(cat $TEMPFILE)"
rm $TEMPFILE

# cat getmemstat.ksh
#!/bin/ksh
echo "::memstat" | mdb -k

Hobbit user cannot run mdb, so we need to set permissions to 4755 and owner
root on getmemstat.ksh
chown root getmemstat.ksh
chmod 4755 getmemstat.ksh


On the server
Add this to hobbitserver
TEST2RRD="blah,blah,blah,memstat=ncv"
GRAPHS="whatever was there, memstat"
NCV_memstat="*:GAUGE"

Add this to hobbitgraph.cfg
[memstat]
        TITLE Kernel Memory Allocation
        YAXIS %
        -u 100
        -l 0
        DEF:a=memstat.rrd:Kernel:AVERAGE
        DEF:b=memstat.rrd:ZFS:AVERAGE
        DEF:c=memstat.rrd:Anon:AVERAGE
        DEF:d=memstat.rrd:Exec:AVERAGE
        DEF:e=memstat.rrd:Page:AVERAGE
        DEF:f=memstat.rrd:cachelist:AVERAGE
        DEF:g=memstat.rrd:freelist:AVERAGE
        AREA:a#00FF00:Kernel
        STACK:b#000099:ZFS
        STACK:c#009999:Anon
        STACK:d#FF00FF:Exec
        STACK:e#FFFF00:Page
        STACK:f#00FFFF:cachelist
        STACK:g#FF0000:freelist
        COMMENT:\n
        GPRINT:a:LAST:Kernel    \: %5.1lf%s (cur)
        GPRINT:a:MAX: \: %5.1lf%s (max)
        GPRINT:a:MIN: \: %5.1lf%s (min)
        GPRINT:a:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:b:LAST:ZFS       \: %5.1lf%s (cur)
        GPRINT:b:MAX: \: %5.1lf%s (max)
        GPRINT:b:MIN: \: %5.1lf%s (min)
        GPRINT:b:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:c:LAST:Anon      \: %5.1lf%s (cur)
        GPRINT:c:MAX: \: %5.1lf%s (max)
        GPRINT:c:MIN: \: %5.1lf%s (min)
        GPRINT:c:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:d:LAST:Exec      \: %5.1lf%s (cur)
        GPRINT:d:MAX: \: %5.1lf%s (max)
        GPRINT:d:MIN: \: %5.1lf%s (min)
        GPRINT:d:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:e:LAST:Page      \: %5.1lf%s (cur)
        GPRINT:e:MAX: \: %5.1lf%s (max)
        GPRINT:e:MIN: \: %5.1lf%s (min)
        GPRINT:e:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:f:LAST:cachelist \: %5.1lf%s (cur)
        GPRINT:f:MAX: \: %5.1lf%s (max)
        GPRINT:f:MIN: \: %5.1lf%s (min)
        GPRINT:f:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:g:LAST:freelist  \: %5.1lf%s (cur)
        GPRINT:g:MAX: \: %5.1lf%s (max)
        GPRINT:g:MIN: \: %5.1lf%s (min)
        GPRINT:g:AVERAGE: \: %5.1lf%s (avg)\n

Regards
     Vernon

list TJ Yang · Fri, 21 May 2010 07:38:53 -0500 ·

Thanks for sharing this code, Vernon.

I will find some time to test this script out.


tj

On Thu, May 20, 2010 at 1:17 AM, Vernon Everett

▸ quoted from Vernon Everett

<user-b3f8dacb72c8@xymon.invalid> wrote:

Hi all

Just cobbled together a test to keep tabs on where memory is going on my
Solaris 10 servers, in particular, how much is going to kernel and ZFS
cache.
(It might work for other Solaris versions, but I haven't tried. YMMV)

Please let me know if you spot any bugs.

It's 2 scripts.
An mod to hobbitserver.cfg and hobbitgraph.cfg, and an update to
clientlaunch.cfg

Here goes.

On the client, add this to clientlaunch.cfg
[memstat]
        ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
        CMD $HOBBITCLIENTHOME/ext/memstat.ksh
        LOGFILE $HOBBITCLIENTHOME/logs/memstat.log
        INTERVAL 5m

Client side scripts.
# cat memstat.ksh
#!/bin/ksh
TEMPFILE=$BBTMP/memstat.tmp
COLOUR=green
ALERT=1  # 1= alert on high usage
YELLOW=75
RED=90
date > $TEMPFILE
echo >> $TEMPFILE
$HOBBITCLIENTHOME/ext/getmemstat.ksh > $TEMPFILE.data
cat $TEMPFILE.data  >> $TEMPFILE
if [ $ALERT -eq 1 ]
then
   cat $TEMPFILE.data | egrep "^Kernel|^Anon|^Exec" | cut -c52-56 | while
read a
   do
      if [ $a -gt $YELLOW -a "$COLOUR" = "green" ]
      then
          COLOUR=yellow
      else
         [ $a -gt $RED ] && COLOUR=red
      fi
   done
fi
echo '<p><font size="1" color="black">' >> $TEMPFILE
cat $TEMPFILE.data | egrep -v -- "-----|Summary|^$|^Total|^Physical" \
                   | sed "s/^Free //g" \
                   | sed "s/[(|)|%]//g" \
                   | awk '{ print $1" : "$NF }' >> $TEMPFILE
echo '</font></p>' >> $TEMPFILE
rm $TEMPFILE.data
$BB $BBDISP "status $MACHINE.memstat $COLOUR $(cat $TEMPFILE)"
rm $TEMPFILE

# cat getmemstat.ksh
#!/bin/ksh
echo "::memstat" | mdb -k

Hobbit user cannot run mdb, so we need to set permissions to 4755 and owner
root on getmemstat.ksh
chown root getmemstat.ksh
chmod 4755 getmemstat.ksh


On the server
Add this to hobbitserver
TEST2RRD="blah,blah,blah,memstat=ncv"
GRAPHS="whatever was there, memstat"
NCV_memstat="*:GAUGE"

Add this to hobbitgraph.cfg
[memstat]
        TITLE Kernel Memory Allocation
        YAXIS %
        -u 100
        -l 0
        DEF:a=memstat.rrd:Kernel:AVERAGE
        DEF:b=memstat.rrd:ZFS:AVERAGE
        DEF:c=memstat.rrd:Anon:AVERAGE
        DEF:d=memstat.rrd:Exec:AVERAGE
        DEF:e=memstat.rrd:Page:AVERAGE
        DEF:f=memstat.rrd:cachelist:AVERAGE
        DEF:g=memstat.rrd:freelist:AVERAGE
        AREA:a#00FF00:Kernel
        STACK:b#000099:ZFS
        STACK:c#009999:Anon
        STACK:d#FF00FF:Exec
        STACK:e#FFFF00:Page
        STACK:f#00FFFF:cachelist
        STACK:g#FF0000:freelist
        COMMENT:\n
        GPRINT:a:LAST:Kernel    \: %5.1lf%s (cur)
        GPRINT:a:MAX: \: %5.1lf%s (max)
        GPRINT:a:MIN: \: %5.1lf%s (min)
        GPRINT:a:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:b:LAST:ZFS       \: %5.1lf%s (cur)
        GPRINT:b:MAX: \: %5.1lf%s (max)
        GPRINT:b:MIN: \: %5.1lf%s (min)
        GPRINT:b:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:c:LAST:Anon      \: %5.1lf%s (cur)
        GPRINT:c:MAX: \: %5.1lf%s (max)
        GPRINT:c:MIN: \: %5.1lf%s (min)
        GPRINT:c:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:d:LAST:Exec      \: %5.1lf%s (cur)
        GPRINT:d:MAX: \: %5.1lf%s (max)
        GPRINT:d:MIN: \: %5.1lf%s (min)
        GPRINT:d:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:e:LAST:Page      \: %5.1lf%s (cur)
        GPRINT:e:MAX: \: %5.1lf%s (max)
        GPRINT:e:MIN: \: %5.1lf%s (min)
        GPRINT:e:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:f:LAST:cachelist \: %5.1lf%s (cur)
        GPRINT:f:MAX: \: %5.1lf%s (max)
        GPRINT:f:MIN: \: %5.1lf%s (min)
        GPRINT:f:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:g:LAST:freelist  \: %5.1lf%s (cur)
        GPRINT:g:MAX: \: %5.1lf%s (max)
        GPRINT:g:MIN: \: %5.1lf%s (min)
        GPRINT:g:AVERAGE: \: %5.1lf%s (avg)\n

Regards
     Vernon

--


T.J. Yang

list Vernon Everett · Mon, 24 May 2010 14:17:15 +0800 ·

Hi all

Just put this on a Solaris 10 box not using ZFS, and the graph is borked.
As a work-around, I manually added a ZFS column to my rrd file for that
service.

If that's too difficult, try this. It generates a working rrd file.
Take a host with a working graph, let's say HostA.
cd to the appropriate rrd directory.
Backup, and delete the existing memstat.rrd file.
    mv memstat.rrd memstat.rrd.bak
wait for a few minutes, and Zymon will create a new memstat.rrd.
Save it.
    mv memstat.rrd memstat.rrd.new
Then copy back the old one.
    mv memstat.rrd.bak memstat.rrd

This new rrd file will have the correct format and columnts, but no actual
data.
You can now copy it over the rrd file for the host that doesn't have ZFS,
and will not graph.
mv memstat.rrd.new ../HostB/memstat.rrd

The graph for HostA will be missing a single entry, but that's not a major
issue.
The graph for HostB will have all values set to N/A for ZFS.
Also not a train smash.

I could also change the graphing mechanism to SPLIT_NCV, but I am not sure
it is implemented in my version.

Enjoy.

Regards
    Vernon

▸ quoted from TJ Yang



On Fri, May 21, 2010 at 8:38 PM, TJ Yang <user-61afc885aa73@xymon.invalid> wrote:

Thanks for sharing this code, Vernon.

I will find some time to test this script out.


tj

On Thu, May 20, 2010 at 1:17 AM, Vernon Everett
<user-b3f8dacb72c8@xymon.invalid> wrote:

Hi all

Just cobbled together a test to keep tabs on where memory is going on my
Solaris 10 servers, in particular, how much is going to kernel and ZFS
cache.
(It might work for other Solaris versions, but I haven't tried. YMMV)

Please let me know if you spot any bugs.

It's 2 scripts.
An mod to hobbitserver.cfg and hobbitgraph.cfg, and an update to
clientlaunch.cfg

Here goes.

On the client, add this to clientlaunch.cfg
[memstat]
        ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
        CMD $HOBBITCLIENTHOME/ext/memstat.ksh
        LOGFILE $HOBBITCLIENTHOME/logs/memstat.log
        INTERVAL 5m

Client side scripts.
# cat memstat.ksh
#!/bin/ksh
TEMPFILE=$BBTMP/memstat.tmp
COLOUR=green
ALERT=1  # 1= alert on high usage
YELLOW=75
RED=90
date > $TEMPFILE
echo >> $TEMPFILE
$HOBBITCLIENTHOME/ext/getmemstat.ksh > $TEMPFILE.data
cat $TEMPFILE.data  >> $TEMPFILE
if [ $ALERT -eq 1 ]
then
   cat $TEMPFILE.data | egrep "^Kernel|^Anon|^Exec" | cut -c52-56 | while
read a
   do
      if [ $a -gt $YELLOW -a "$COLOUR" = "green" ]
      then
          COLOUR=yellow
      else
         [ $a -gt $RED ] && COLOUR=red
      fi
   done
fi
echo '<p><font size="1" color="black">' >> $TEMPFILE
cat $TEMPFILE.data | egrep -v -- "-----|Summary|^$|^Total|^Physical" \
                   | sed "s/^Free //g" \
                   | sed "s/[(|)|%]//g" \
                   | awk '{ print $1" : "$NF }' >> $TEMPFILE
echo '</font></p>' >> $TEMPFILE
rm $TEMPFILE.data
$BB $BBDISP "status $MACHINE.memstat $COLOUR $(cat $TEMPFILE)"
rm $TEMPFILE

# cat getmemstat.ksh
#!/bin/ksh
echo "::memstat" | mdb -k

Hobbit user cannot run mdb, so we need to set permissions to 4755 and
owner
root on getmemstat.ksh
chown root getmemstat.ksh
chmod 4755 getmemstat.ksh


On the server
Add this to hobbitserver
TEST2RRD="blah,blah,blah,memstat=ncv"
GRAPHS="whatever was there, memstat"
NCV_memstat="*:GAUGE"

Add this to hobbitgraph.cfg
[memstat]
        TITLE Kernel Memory Allocation
        YAXIS %
        -u 100
        -l 0
        DEF:a=memstat.rrd:Kernel:AVERAGE
        DEF:b=memstat.rrd:ZFS:AVERAGE
        DEF:c=memstat.rrd:Anon:AVERAGE
        DEF:d=memstat.rrd:Exec:AVERAGE
        DEF:e=memstat.rrd:Page:AVERAGE
        DEF:f=memstat.rrd:cachelist:AVERAGE
        DEF:g=memstat.rrd:freelist:AVERAGE
        AREA:a#00FF00:Kernel
        STACK:b#000099:ZFS
        STACK:c#009999:Anon
        STACK:d#FF00FF:Exec
        STACK:e#FFFF00:Page
        STACK:f#00FFFF:cachelist
        STACK:g#FF0000:freelist
        COMMENT:\n
        GPRINT:a:LAST:Kernel    \: %5.1lf%s (cur)
        GPRINT:a:MAX: \: %5.1lf%s (max)
        GPRINT:a:MIN: \: %5.1lf%s (min)
        GPRINT:a:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:b:LAST:ZFS       \: %5.1lf%s (cur)
        GPRINT:b:MAX: \: %5.1lf%s (max)
        GPRINT:b:MIN: \: %5.1lf%s (min)
        GPRINT:b:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:c:LAST:Anon      \: %5.1lf%s (cur)
        GPRINT:c:MAX: \: %5.1lf%s (max)
        GPRINT:c:MIN: \: %5.1lf%s (min)
        GPRINT:c:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:d:LAST:Exec      \: %5.1lf%s (cur)
        GPRINT:d:MAX: \: %5.1lf%s (max)
        GPRINT:d:MIN: \: %5.1lf%s (min)
        GPRINT:d:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:e:LAST:Page      \: %5.1lf%s (cur)
        GPRINT:e:MAX: \: %5.1lf%s (max)
        GPRINT:e:MIN: \: %5.1lf%s (min)
        GPRINT:e:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:f:LAST:cachelist \: %5.1lf%s (cur)
        GPRINT:f:MAX: \: %5.1lf%s (max)
        GPRINT:f:MIN: \: %5.1lf%s (min)
        GPRINT:f:AVERAGE: \: %5.1lf%s (avg)\n
        GPRINT:g:LAST:freelist  \: %5.1lf%s (cur)
        GPRINT:g:MAX: \: %5.1lf%s (max)
        GPRINT:g:MIN: \: %5.1lf%s (min)
        GPRINT:g:AVERAGE: \: %5.1lf%s (avg)\n

Regards
     Vernon

--
T.J. Yang

Memstat for Solaris (10) 🔗 link

Memstat for Solaris (10)