monitoring aggregated network traffic 🔗 link

On 13-10-2011 16:44, netz-haut - stephan seitz wrote:

is there any "usual" way, to monitor and graph the traffic of a bulk
of network interfaces? The reason is, some of our hosts are connected
to iSCSI storages via multipath over two or four independent
interfaces. Since the traffic is balanced equally, every interface
shows (more or less) the same amount of traffic. This leads to very
useless rrdtool graphics as there is only one multicolored thick line
visible. It would be really helpful to see a graph which shows the
SUM of the particular interfaces.

-----Original Message-----
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Henrik Størner
Sent: Thursday, October 13, 2011 1:42 PM
To: xymon at xymon.com
Subject: Re: [Xymon] monitoring aggregated network traffic

On 13-10-2011 16:44, netz-haut - stephan seitz wrote:

is there any "usual" way, to monitor and graph the traffic of a bulk
of network interfaces? The reason is, some of our hosts are connected
to iSCSI storages via multipath over two or four independent
interfaces. Since the traffic is balanced equally, every interface
shows (more or less) the same amount of traffic. This leads to very
useless rrdtool graphics as there is only one multicolored thick line
visible. It would be really helpful to see a graph which shows the
SUM of the particular interfaces.

This can be done by creating an RRD graph definition to do the summation
for you.

In RRDtool, you first use one or more "DEF" lines to grab specific data
items from one or more RRD files, and put them into some "variables".
You can then either use these variables directly to make a line on the
graph, or you can use them for calculating other variables via a "CDEF"
line.

The default ifstat graph has these lines:

         DEF:inbytes at RRDIDX@=@RRDFN@:bytesReceived:AVERAGE
         CDEF:in at RRDIDX@=inbytes at RRDIDX@,8,*

(The @RRDIDX@ and @RRDFN@ are because this graph repeats over all the
interfaces - Xymon replaces these with a number and the RRD filename).

What this does is to first define a variable "inbytes at RRDIDX@" which has
the "bytesReceived" value from the RRD. The it computes another
variable, "in at RRDIDX@" as 8*inbytes - to convert from bytes to bits.
The notation here is Reverse Polish (RPN).

So if you have 4 RRD files, one for each interface, you can have RRDtool
do the math for you like this:

[ifsum]
         TITLE Network Traffic
         YAXIS Bits/second
         DEF:inbytes0=ifstat.eth0.rrd:bytesReceived:AVERAGE
         DEF:inbytes1=ifstat.eth1.rrd:bytesReceived:AVERAGE
         DEF:inbytes2=ifstat.eth2.rrd:bytesReceived:AVERAGE
         DEF:inbytes3=ifstat.eth3.rrd:bytesReceived:AVERAGE
         CDEF:intotal=inbytes0,inbytes1,inbytes2,inbytes3,+,+,+
     CDEF:inbits=intotal,8,*
     LINE:inbits#FF0000:Aggregate inbound

The "LINE" definition then uses the "inbits" value to draw a (red) line
with this data.

Have a look at graphs.cfg, and I'm sure you can figure out to add a line
for the outgoing data also.

Add "ifsum" to the GRAPHS setting in xymonserver.cfg, and the graph
should show up on the "trends" page.


Regards,
Henrik

PS: I haven't actually tested any of this, so feel free to provide
corrections :-)

On 13-10-2011 19:52, Scot Kreienkamp wrote:

How would that work when there are varying numbers of interfaces?  I
have 14 on one, and 10 on another, each interface in its own RRD
being collected by MRTG.  If I put all 14 in, won't it break on the
10?

user-462cf0b6d846@xymon.invalid


-----Original Message-----
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Henrik Størner
Sent: Thursday, October 13, 2011 4:48 PM
To: xymon at xymon.com
Subject: Re: [Xymon] monitoring aggregated network traffic

On 13-10-2011 19:52, Scot Kreienkamp wrote:

How would that work when there are varying numbers of interfaces?  I
have 14 on one, and 10 on another, each interface in its own RRD
being collected by MRTG.  If I put all 14 in, won't it break on the
10?

I was afraid You were going to ask that question. Yes, it will break because some of those files just do not exist, and RRDtool cannot handle that.

I don't have an automatic solution for that. You can define multiple graph definitions - one for each number of interfaces - and then use the "TRENDS" setting for each host to determine which one will be shown. Not pretty, but it works.


Regards,
Henrik

On Friday 14 October 2011, Scot Kreienkamp wrote:

Yeah, when you wrote that solution like that I was afraid that's what you'd
say.  :)

Scot Kreienkamp
user-462cf0b6d846@xymon.invalid


-----Original Message-----
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Stef Coene
Sent: Friday, October 14, 2011 1:01 PM
To: xymon at xymon.com
Subject: Re: [Xymon] monitoring aggregated network traffic

On Friday 14 October 2011, Scot Kreienkamp wrote:

Yeah, when you wrote that solution like that I was afraid that's what you'd
say.  :)

What you can do is create a subdirectory 'overview1' in the data/rrd directory.  In that directory you can then create symlinks for the data you want to graph.

I use this to graph the memory usage of AIX lpars running on the same hardware box.  So I create a subdirectory 'data/rrd/pSerie1' and in the subdirectory I create symlinks for each server running on that pSerie:
vmstat-server1.rrd -> ../server1/vmstat.rrd
vmstat-server2.rrd -> ../server2/vmstat.rrd
vmstat-server3.rrd -> ../server3/vmstat.rrd

I then added a section to hobbit-graph.cfg to process all 'vmstat-(.*).rrd' files and stack the data on top of each other:

[vmstat-avm-total]
   FNPATTERN vmstat-(.+).rrd
   TITLE Accessed Virtual Memory (Stacked)
   YAXIS Size kB
   -b 1024
   DEF:avmt at RRDIDX@=@RRDFN@:mem_avm:AVERAGE
   CDEF:avm at RRDIDX@=avmt at RRDIDX@,4048,*
   LINE1:avm at RRDIDX@#@COLOR@:@RRDPARAM@:STACK
   GPRINT:avm at RRDIDX@:LAST:AVM  \: %5.1lf%s (cur)
   GPRINT:avm at RRDIDX@:MAX: \: %5.1lf%s (max)
   GPRINT:avm at RRDIDX@:MIN: \: %5.1lf%s (min)
   GPRINT:avm at RRDIDX@:AVERAGE: \: %5.1lf%s (avg)\n

The result is a stacked graph with 1 line per server.

For the graph, the url is something like
hobbitgraph.sh?host=pSerie1&service=vmstat-avm-total&action=menu

I hope this info can help you.


Stef

On Friday 14 October 2011, Stef Coene wrote:

I use this to graph the memory usage of AIX lpars running on the same
hardware
box.  So I create a subdirectory 'data/rrd/pSerie1' and in the
subdirectory I
create symlinks for each server running on that pSerie:
vmstat-server1.rrd -> ../server1/vmstat.rrd
vmstat-server2.rrd -> ../server2/vmstat.rrd
vmstat-server3.rrd -> ../server3/vmstat.rrd

On Sunday 16 October 2011 10:34:32 Scott Pfister wrote:

On Friday 14 October 2011, Stef Coene wrote:

I use this to graph the memory usage of AIX lpars running on the same

hardware

box.  So I create a subdirectory 'data/rrd/pSerie1' and in the

subdirectory I

create symlinks for each server running on that pSerie:
vmstat-server1.rrd -> ../server1/vmstat.rrd
vmstat-server2.rrd -> ../server2/vmstat.rrd
vmstat-server3.rrd -> ../server3/vmstat.rrd

That's a good idea.  Do you do similar aggregation with other graph points
on the lpars running on same hardware?  like Physical CPU?
If so would you be willing to share those graph definitions?

	
	On 13-10-2011 16:44, netz-haut - stephan seitz wrote:
	

is there any "usual" way, to monitor and graph the traffic of a bulk
of network interfaces? The reason is, some of our hosts are connected
to iSCSI storages via multipath over two or four independent
interfaces. Since the traffic is balanced equally, every interface
shows (more or less) the same amount of traffic. This leads to very
useless rrdtool graphics as there is only one multicolored thick line
visible. It would be really helpful to see a graph which shows the
SUM of the particular interfaces.

	
	This can be done by creating an RRD graph definition to do the summation
	for you.
	
	In RRDtool, you first use one or more "DEF" lines to grab specific data
	items from one or more RRD files, and put them into some "variables".
	You can then either use these variables directly to make a line on the
	graph, or you can use them for calculating other variables via a "CDEF"
	line.
	
	The default ifstat graph has these lines:
	
	         DEF:inbytes at RRDIDX@=@RRDFN@:bytesReceived:AVERAGE
	         CDEF:in at RRDIDX@=inbytes at RRDIDX@,8,*
	
	(The @RRDIDX@ and @RRDFN@ are because this graph repeats over all the
	interfaces - Xymon replaces these with a number and the RRD filename).
	
	What this does is to first define a variable "inbytes at RRDIDX@" which has
	the "bytesReceived" value from the RRD. The it computes another
	variable, "in at RRDIDX@" as 8*inbytes - to convert from bytes to bits.
	The notation here is Reverse Polish (RPN).
	
	So if you have 4 RRD files, one for each interface, you can have RRDtool
	do the math for you like this:
	
	[ifsum]
	         TITLE Network Traffic
	         YAXIS Bits/second
	         DEF:inbytes0=ifstat.eth0.rrd:bytesReceived:AVERAGE
	         DEF:inbytes1=ifstat.eth1.rrd:bytesReceived:AVERAGE
	         DEF:inbytes2=ifstat.eth2.rrd:bytesReceived:AVERAGE
	         DEF:inbytes3=ifstat.eth3.rrd:bytesReceived:AVERAGE
	         CDEF:intotal=inbytes0,inbytes1,inbytes2,inbytes3,+,+,+
	     CDEF:inbits=intotal,8,*
	     LINE:inbits#FF0000:Aggregate inbound
	
	The "LINE" definition then uses the "inbits" value to draw a (red) line
	with this data.
	
	Have a look at graphs.cfg, and I'm sure you can figure out to add a line
	for the outgoing data also.
	
	Add "ifsum" to the GRAPHS setting in xymonserver.cfg, and the graph
	should show up on the "trends" page.
	
	
	Regards,
	Henrik
	
	PS: I haven't actually tested any of this, so feel free to provide
	corrections :-)
	

On Sun, 30 Oct 2011, netz-haut - stephan seitz wrote:

Hi,

thanks for your very helpful answer.
It took me a few days to get back to that issue, so sorry for that late responose.

I've added additional lines for outgoing, and it works as expected.

I'm absolutely fine with hardcoed interfaces, cause I need the aggregated view of
particular interfaces.


Thanks again,

Stephan


Am Donnerstag, den 13.10.2011, 19:42 +0200 schrieb Henrik Størner:


On 13-10-2011 16:44, netz-haut - stephan seitz wrote:

is there any "usual" way, to monitor and graph the traffic of a bulk
of network interfaces? The reason is, some of our hosts are connected
to iSCSI storages via multipath over two or four independent
interfaces. Since the traffic is balanced equally, every interface
shows (more or less) the same amount of traffic. This leads to very
useless rrdtool graphics as there is only one multicolored thick line
visible. It would be really helpful to see a graph which shows the
SUM of the particular interfaces.

This can be done by creating an RRD graph definition to do the summation
for you.

In RRDtool, you first use one or more "DEF" lines to grab specific data
items from one or more RRD files, and put them into some "variables".
You can then either use these variables directly to make a line on the
graph, or you can use them for calculating other variables via a "CDEF"
line.

The default ifstat graph has these lines:

         DEF:inbytes at RRDIDX@=@RRDFN@:bytesReceived:AVERAGE
         CDEF:in at RRDIDX@=inbytes at RRDIDX@,8,*

(The @RRDIDX@ and @RRDFN@ are because this graph repeats over all the
interfaces - Xymon replaces these with a number and the RRD filename).

What this does is to first define a variable "inbytes at RRDIDX@" which has
the "bytesReceived" value from the RRD. The it computes another
variable, "in at RRDIDX@" as 8*inbytes - to convert from bytes to bits.
The notation here is Reverse Polish (RPN).

So if you have 4 RRD files, one for each interface, you can have RRDtool
do the math for you like this:

[ifsum]
         TITLE Network Traffic
         YAXIS Bits/second
         DEF:inbytes0=ifstat.eth0.rrd:bytesReceived:AVERAGE
         DEF:inbytes1=ifstat.eth1.rrd:bytesReceived:AVERAGE
         DEF:inbytes2=ifstat.eth2.rrd:bytesReceived:AVERAGE
         DEF:inbytes3=ifstat.eth3.rrd:bytesReceived:AVERAGE
         CDEF:intotal=inbytes0,inbytes1,inbytes2,inbytes3,+,+,+
     CDEF:inbits=intotal,8,*
     LINE:inbits#FF0000:Aggregate inbound

The "LINE" definition then uses the "inbits" value to draw a (red) line
with this data.

Have a look at graphs.cfg, and I'm sure you can figure out to add a line
for the outgoing data also.

Add "ifsum" to the GRAPHS setting in xymonserver.cfg, and the graph
should show up on the "trends" page.


Regards,
Henrik

PS: I haven't actually tested any of this, so feel free to provide
corrections :-)

On Sun, 30 Oct 2011, netz-haut - stephan seitz wrote:

Hi,

thanks for your very helpful answer.
It took me a few days to get back to that issue, so sorry for that late 
responose.

I've added additional lines for outgoing, and it works as expected.

I'm absolutely fine with hardcoed interfaces, cause I need the aggregated 
view of
particular interfaces.


Thanks again,

Stephan


Am Donnerstag, den 13.10.2011, 19:42 +0200 schrieb Henrik Størner:


On 13-10-2011 16:44, netz-haut - stephan seitz wrote:

is there any "usual" way, to monitor and graph the traffic of a bulk
of network interfaces? The reason is, some of our hosts are connected
to iSCSI storages via multipath over two or four independent
interfaces. Since the traffic is balanced equally, every interface
shows (more or less) the same amount of traffic. This leads to very
useless rrdtool graphics as there is only one multicolored thick line
visible. It would be really helpful to see a graph which shows the
SUM of the particular interfaces.

This can be done by creating an RRD graph definition to do the summation
for you.

In RRDtool, you first use one or more "DEF" lines to grab specific data
items from one or more RRD files, and put them into some "variables".
You can then either use these variables directly to make a line on the
graph, or you can use them for calculating other variables via a "CDEF"
line.

The default ifstat graph has these lines:

         DEF:inbytes at RRDIDX@=@RRDFN@:bytesReceived:AVERAGE
         CDEF:in at RRDIDX@=inbytes at RRDIDX@,8,*

(The @RRDIDX@ and @RRDFN@ are because this graph repeats over all the
interfaces - Xymon replaces these with a number and the RRD filename).

What this does is to first define a variable "inbytes at RRDIDX@" which has
the "bytesReceived" value from the RRD. The it computes another
variable, "in at RRDIDX@" as 8*inbytes - to convert from bytes to bits.
The notation here is Reverse Polish (RPN).

So if you have 4 RRD files, one for each interface, you can have RRDtool
do the math for you like this:

[ifsum]
         TITLE Network Traffic
         YAXIS Bits/second
         DEF:inbytes0=ifstat.eth0.rrd:bytesReceived:AVERAGE
         DEF:inbytes1=ifstat.eth1.rrd:bytesReceived:AVERAGE
         DEF:inbytes2=ifstat.eth2.rrd:bytesReceived:AVERAGE
         DEF:inbytes3=ifstat.eth3.rrd:bytesReceived:AVERAGE
         CDEF:intotal=inbytes0,inbytes1,inbytes2,inbytes3,+,+,+
     CDEF:inbits=intotal,8,*
     LINE:inbits#FF0000:Aggregate inbound

The "LINE" definition then uses the "inbits" value to draw a (red) line
with this data.

Have a look at graphs.cfg, and I'm sure you can figure out to add a line
for the outgoing data also.

Add "ifsum" to the GRAPHS setting in xymonserver.cfg, and the graph
should show up on the "trends" page.


Regards,
Henrik

PS: I haven't actually tested any of this, so feel free to provide
corrections :-)

Hi !

I knew, this thread is very old,
but i want this feature very much :)

..  unfortunately something is missing in my enviroment,
because i couldn't see the ifsum-graph on the trend-page ... :-(

Can somebody give me an advice how i debug this missing graph ?

My xmon-server is running the 4.3.7-version ..

thanks & cheers,

      Martin

<user-f286aaa49a76@xymon.invalid> wrote:

Ok, correction ... I can't see the ifsum-graph
on the default trend-page, but if i take on graph of the trend-page
and change the service=ports to service=ifsum
then i got the graph :-)

But how can i display this on the default trend-page ?

On Thu, 16 Feb 2012, Jeremy Laidman wrote:

On Thu, Feb 16, 2012 at 6:53 AM, Martin Flemming
<user-f286aaa49a76@xymon.invalid> wrote:

Ok, correction ... I can't see the ifsum-graph
on the default trend-page, but if i take on graph of the trend-page
and change the service=ports to service=ifsum
then i got the graph :-)

But how can i display this on the default trend-page ?

From the "how to setup custom graphs" page:

"If you want the graph included with the other graphs on the trends
column, you must add it to the GRAPHS setting in the
~xymon/server/etc/xymonserver.cfg file."

So in your case, add ",ifsum" to your GRAPHS setting and then reload
the trends page.

I've had to make an touch  ifsum.rrd and voila, the graph will be display

Is that an bug ?

On Fri, 17 Feb 2012, Jeremy Laidman wrote:

On Thu, Feb 16, 2012 at 6:26 PM, Martin Flemming
<user-f286aaa49a76@xymon.invalid> wrote:

Now i've found this thread and the solution :-)

http://lists.xymon.com/pipermail/xymon/2011-December/033323.html

I remember that thread now, but I didn't understand quite what was
going on.  I've recently been trying to solve a different problem
related to the same bit of code.

I've had to make an touch  ifsum.rrd and voila, the graph will be display

Is that an bug ?

Probably not really a bug.  It's just that the code that generates the
trends page was not (it seems) designed to handle graphs where the
graph name doesn't match the RRD filename.  It probably needs
enhancements to incorporate this as a feature.

The code also looks at FNPATTERN in the graph definition to tell the
code which filename to look at, and while it is probably not intended
for use with fixed filenames like you're using, it might still work,
without having to create a dummy file.

[ifsum]
         TITLE Network Traffic
         YAXIS Bits/second
         DEF:inbytes0=ifstat.eth0.rrd:bytesReceived:AVERAGE
         DEF:inbytes1=ifstat.eth1.rrd:bytesReceived:AVERAGE
         DEF:inbytes2=ifstat.eth2.rrd:bytesReceived:AVERAGE
         DEF:inbytes3=ifstat.eth3.rrd:bytesReceived:AVERAGE
         CDEF:intotal=inbytes0,inbytes1,inbytes2,inbytes3,+,+,+
     CDEF:inbits=intotal,8,*
     LINE:inbits#FF0000:Aggregate inbound

<user-f286aaa49a76@xymon.invalid> wrote:

Hmmmm, ok before i start another thread to the similar problem,
i want to ask how i could solve this problem whith FNPATTERN ?