Netstat rrd error updating 🔗 link

Searched the archives and found other mentions of this error but no
solution. I have a Solaris 8 server with the Hobbit client reporting to the
Hobbit server, and everything works except the netstat trend graph. I made
sure Hobbit generates the netstat.rrd file itself, and verified that it has
the correct fields:

$ rrdtool dump netstat.rrd |grep "<name>"

                <name> udpInDatagrams </name>
                <name> udpOutDatagrams </name>
                <name> udpInErrors </name>
                <name> tcpActiveOpens </name>
                <name> tcpPassiveOpens </name>
                <name> tcpAttemptFails </name>
                <name> tcpEstabResets </name>
                <name> tcpCurrEstab </name>
                <name> tcpOutDataBytes </name>
                <name> tcpInInorderBytes </name>
                <name> tcpInUnorderBytes </name>
                <name> tcpRetransBytes </name>
                <name> tcpOutDataPackets </name>
                <name> tcpInInorderPackets </name>
                <name> tcpInUnorderPackets </name>
                <name> tcpRetransPackets </name>

But viewing a debug output of rrd-data.log shows that something is making it
think there should only be 11 fields:

2006-10-13 19:18:18 Want msg 667, startpos 66490, fillpos 66490, endpos -1,
usedbytes=0, bufleft 461893
2006-10-13 19:18:22 hobbitd_rrd: Got message 667
@@data#667|1160767102.598295|10.x.x.x||x.x.x.x|netstat
2006-10-13 19:18:22 startpos 72703, fillpos 72703, endpos -1
2006-10-13 19:18:22 RRD update param 00: 'rrdupdate'
2006-10-13 19:18:22 RRD update param 01:
'/cust/bbh/data/rrd/x.x.x.x/netstat.rrd'
2006-10-13 19:18:22 RRD update param 02: '-t'
2006-10-13 19:18:22 RRD update param 03:
'udpInDatagrams:udpOutDatagrams:udpInErrors:tcpActiveOpens:tcpPassiveOpens:t
cpAttemptFails:tcpEstabResets:tcpCurrEstab:tcpOutDataBytes:tcpInInorderBytes
:tcpInUnorderBytes'
2006-10-13 19:18:22 RRD update param 04:
'1160767102:1580411:5594723:0:113952:19256340:17330:36082:7:653169856:440119
035:9398449:159635980:26380336:13186666:30559:612744'
2006-10-13 19:18:22 RRD error updating
/cust/bbh/data/rrd/x.x.x.x/netstat.rrd from x.x.x.x: expected 11 data source
readings (got 16) from
1160767102:1580411:5594723:0:113952:19256340:17330:36082:7:653169856:4401190
35:9398449:159635980:26380336:13186666:30559:612744:...


Where is "param 03" coming from? From some configuration on the client side?

On Fri, Oct 13, 2006 at 04:09:59PM -0400, Epp, Matthew Contractor PEO EIS AKO wrote:

11 items here, yes.

2006-10-13 19:18:22 RRD update param 04:
'1160767102:1580411:5594723:0:113952:19256340:17330:36082:7:653169856:440119
035:9398449:159635980:26380336:13186666:30559:612744'

And 16 values there. Most odd.

Where is "param 03" coming from? From some configuration on the client side?

Param 03 is the "-t" option, which tells rrd_update() that param 04
defines the sequence of data set values.


Hmm ... could you try attaching gdb to the running hobbitd_rrd process.
There are two of them - one is handling "status" messages, the other
"data" messages. You'll need to find the process-ID of the one handling
"data" messages. Do something like

  $ ps -u hobbit -f|egrep "hobbitd_rrd|hobbitd_channel"

and look for the lines with

hobbit   25196 25150  0 Oct04 ?        00:00:02 hobbitd_channel --channel=data --log=/var/log/hobbit/rrd-data.log hobbitd_rrd --rrddir=/var/lib/hobbit/rrd
hobbit   25219 25196  0 Oct04 ?        00:00:53 hobbitd_rrd --rrddir=/var/lib/hobbit/rrd

where the PID of the hobbitd_channel process (25196 here) matches the 
parent-PID of the hobbitd_rrd process (25219). The "25219" is what you
need. Now run gdb like this (as the hobbit user):

  $ gdb ~/server/bin/hobbitd_rrd 
  (gdb) attach 25219
  [... lots of gdb printout ...]
  (gdb) p netstat_tpl

I'm interested in what the netstat_tpl output shows, since that is what
is used to build that "param 04" value. Here it looks like

(gdb) p netstat_tpl
$1 = 0x8076a68
"udpInDatagrams:udpOutDatagrams:udpInErrors:tcpActiveOpens:tcpPassiveOpens:tcpAttemptFails:tcpEstabResets:tcpCurrEstab:tcpOutDataBytes:tcpInInorderBytes:tcpInUnorderBytes:tcpRetransBytes:tcpOutDataPack"...
(gdb) p netstat_tpl+200
$2 = 0x8076b30
"ets:tcpInInorderPackets:tcpInUnorderPackets:tcpRetransPackets"

After that, just ctrl-d to exit gdb. If it asks if you want to "detach",
answer "y".


Also, could you run this command (as the hobbit user)

  $ bbcmd hobbitd_channel --channel=data hobbitd_capture --hosts=localhost

(replace "localhost" with the hostname of your solaris 8 client) ?
Let it run for 10 minutes and send me the output.


Thanks,
Henrik

'udpInDatagrams:udpOutDatagrams:udpInErrors:tcpActiveOpens:tcp
PassiveOpens:t

cpAttemptFails:tcpEstabResets:tcpCurrEstab:tcpOutDataBytes:tcp
InInorderBytes

:tcpInUnorderBytes'

11 items here, yes.

2006-10-13 19:18:22 RRD update param 04:

'1160767102:1580411:5594723:0:113952:19256340:17330:36082:7:65
3169856:440119

035:9398449:159635980:26380336:13186666:30559:612744'

And 16 values there. Most odd.

I had a chance to revisit this problem I was having with the netstat rrd
data. I noticed that our Solaris 8 servers are not reporting any of the
following fields "tcpOutDataPackets tcpInInorderPackets tcpInUnorderPackets
tcpRetransPackets". They just don't exist on a "netstat -s" output. Do those
come from some GNU version of netstat?

I only really needed the established TCP connections data anyway, so I ended
up writing a small perl script to report whatever fields I needed and
disabled the built-in netstat tool on the client.

On Wed, Nov 08, 2006 at 05:41:59PM -0500, Epp, Matthew Contractor PEO EIS AKO wrote:

No, it appears to depend on the Solaris version if those fields are
present or not.


Regards,
Henrik

Which Solaris version is supposed to contain those fields then? I've tried
Solaris 8 and Solaris 10 and neither version produce those fields with
netstat. Can I recompile part of the Hobbit server so that it will just not
look for the fields and process what it does get?