Devmon causing core dumps 🔗 link

On Friday 31 October 2008 05:51:42 Everett, Vernon wrote:

Hi all

Devmon was causing the hobbitd_rrd module to crash and burn.
Now this could be a bug, but it could also be a PEBKAC. I am hoping
somebody can assist either way.

I added a Cisco 2851 to Hobbit, using devmon.
Now here is the possible PEBKAC
Since Devmon doesn't have templates for the 2851, I used the template for
the Cisco 2811. (Network guru told me they are pretty much the same, except
for a few extra bells and whistles on the 2851.)

The data for the device started appearing in Hobbit, and all looked good.
Devmon even created the rrd files for the new Cisco device.

However, the hobbitd_rrd module started core dumping, and the Hobbit server
page started displaying red for hobbitd_rrd with the crash detected
message. See core data below.
Took the new Cisco device out of Hobbit, and cores stopped, and life was
good again.

Is there a significant enough difference between the 2851 and the 2811 to
cause this, or are we looking at a genuine bug?

I am leaning towards a bug,
because even if the collected data was complete rubbish, should it cause
the module to core?

Regards
     Vernon

My Linux guy reckons this is the important stuff from the core.
uname -a
Linux las006 2.6.18-92.1.1.el5 #1 SMP Thu May 22 09:01:47 EDT 2008 x86_64
x86_64 x86_64 GNU/Linux cat /etc/redhat-release Red Hat Enterprise Linux
Client release 5.2 (Tikanga)

gdb -c core.8550 /usr/lib/hobbit/server/bin/hobbitd_rrd
GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software
Foundation, Inc. GDB is free software, covered by the GNU General Public
License, and you are welcome to change it and/or distribute copies of it
under certain conditions. Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
libthread_db library "/lib64/libthread_db.so.1".

Reading symbols from /usr/lib64/librrd.so.2...done.
Loaded symbols for /usr/lib64/librrd.so.2 Reading symbols from
/usr/lib64/libpng12.so.0...done. Loaded symbols for
/usr/lib64/libpng12.so.0 Reading symbols from /lib64/libpcre.so.0...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libfreetype.so.6...done.
Loaded symbols for /usr/lib64/libfreetype.so.6 Reading symbols from
/usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libart_lgpl_2.so.2...done.
Loaded symbols for /usr/lib64/libart_lgpl_2.so.2 Reading symbols from
/lib64/libm.so.6...done. Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2 Core was generated by
`hobbitd_rrd --rrddir=/var/lib/hobbit/rrd --debug'. Program terminated with
signal 6, Aborted.
#0  0x0000003db7a30155 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003db7a30155 in raise () from /lib64/libc.so.6
#1  0x0000003db7a31bf0 in abort () from /lib64/libc.so.6
#2  0x00000000004119f3 in sigsegv_handler (signum=<value optimized out>) at
sig.c:57 #3  <signal handler called>
#4  0x0000003db7a77ac0 in strcat () from /lib64/libc.so.6
#5  0x000000000040462a in do_devmon_rrd (hostname=0x2ada311e2806
"PERIR205", testname=0x2ada311e280f "if_load", msg=<value optimized out>,
tstamp=<value optimized out>) at rrd/do_devmon.c:87
#6  0x000000000040b656 in update_rrd (hostname=0x2ada311e2806 "PERIR205",
testname=0x2ada311e280f "if_load", msg=0x2ada311e2842 "status
PERIR205.if_load green Fri Oct 31 10:31:39 2008", tstamp=1225416699,
sender=<value optimized out>, ldef=0xfeffffffffffff00) at do_rrd.c:372 #7 
0x000000000040261d in main (argc=<value optimized out>,
argv=0x7fff7a088318) at hobbitd_rrd.c:153 (gdb)

On Fri, Oct 31, 2008 at 3:15 AM, Buchan Milne <user-9b139aff4dec@xymon.invalid>wrote:

On Friday 31 October 2008 05:51:42 Everett, Vernon wrote:

Hi all

Devmon was causing the hobbitd_rrd module to crash and burn.
Now this could be a bug, but it could also be a PEBKAC. I am hoping
somebody can assist either way.

I added a Cisco 2851 to Hobbit, using devmon.
Now here is the possible PEBKAC
Since Devmon doesn't have templates for the 2851, I used the template for
the Cisco 2811. (Network guru told me they are pretty much the same,
except
for a few extra bells and whistles on the 2851.)

The data for the device started appearing in Hobbit, and all looked good.
Devmon even created the rrd files for the new Cisco device.

However, the hobbitd_rrd module started core dumping, and the Hobbit
server
page started displaying red for hobbitd_rrd with the crash detected
message. See core data below.
Took the new Cisco device out of Hobbit, and cores stopped, and life was
good again.

Is there a significant enough difference between the 2851 and the 2811 to
cause this, or are we looking at a genuine bug?

Real bug. I see it on the temperature tests on a new IOS.

I am leaning towards a bug,
because even if the collected data was complete rubbish, should it cause
the module to core?

Regards
     Vernon

My Linux guy reckons this is the important stuff from the core.
uname -a
Linux las006 2.6.18-92.1.1.el5 #1 SMP Thu May 22 09:01:47 EDT 2008 x86_64
x86_64 x86_64 GNU/Linux cat /etc/redhat-release Red Hat Enterprise Linux
Client release 5.2 (Tikanga)

gdb -c core.8550 /usr/lib/hobbit/server/bin/hobbitd_rrd
GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software
Foundation, Inc. GDB is free software, covered by the GNU General Public
License, and you are welcome to change it and/or distribute copies of it
under certain conditions. Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
libthread_db library "/lib64/libthread_db.so.1".

Reading symbols from /usr/lib64/librrd.so.2...done.
Loaded symbols for /usr/lib64/librrd.so.2 Reading symbols from
/usr/lib64/libpng12.so.0...done. Loaded symbols for
/usr/lib64/libpng12.so.0 Reading symbols from /lib64/libpcre.so.0...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libfreetype.so.6...done.
Loaded symbols for /usr/lib64/libfreetype.so.6 Reading symbols from
/usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libart_lgpl_2.so.2...done.
Loaded symbols for /usr/lib64/libart_lgpl_2.so.2 Reading symbols from
/lib64/libm.so.6...done. Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2 Core was generated by
`hobbitd_rrd --rrddir=/var/lib/hobbit/rrd --debug'. Program terminated
with
signal 6, Aborted.
#0  0x0000003db7a30155 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003db7a30155 in raise () from /lib64/libc.so.6
#1  0x0000003db7a31bf0 in abort () from /lib64/libc.so.6
#2  0x00000000004119f3 in sigsegv_handler (signum=<value optimized out>)
at
sig.c:57 #3  <signal handler called>
#4  0x0000003db7a77ac0 in strcat () from /lib64/libc.so.6
#5  0x000000000040462a in do_devmon_rrd (hostname=0x2ada311e2806
"PERIR205", testname=0x2ada311e280f "if_load", msg=<value optimized out>,
tstamp=<value optimized out>) at rrd/do_devmon.c:87
#6  0x000000000040b656 in update_rrd (hostname=0x2ada311e2806 "PERIR205",
testname=0x2ada311e280f "if_load", msg=0x2ada311e2842 "status
PERIR205.if_load green Fri Oct 31 10:31:39 2008", tstamp=1225416699,
sender=<value optimized out>, ldef=0xfeffffffffffff00) at do_rrd.c:372 #7
0x000000000040261d in main (argc=<value optimized out>,
argv=0x7fff7a088318) at hobbitd_rrd.c:153 (gdb)

Could you show the Devmon RRD section of the message for the if_load test
on
the PERIR205 host? I can confirm the cause, and maybe offer a workaround.

I am actually (constantly) reproducing the issue on my workstation against
the
new IOS that can trigger this, I have a workaround in place in production,
and
was hoping to get around to fixing this next week.

Regards,
Buchan

On Friday 31 October 2008 17:35:44 Robert Holden wrote:

I have seen this as well.  I finally determined it was caused by ATM
interfaces.  Devmon does not give different components of an ATM circuit
(the physical interface, the -atm layer, .0 sub interface, -aal5 layer)
unique names.  So rrd was receiving data for 5 interfaces all with the same
name.  As a temporary interface, I stopped monitoring the atm interfaces,
but this is a bug.

Interface names:
ATM5/0/0
ATM5/0/0-atm layer
ATM5/0/0.0-atm subif
ATM5/0/0-aal5 layer
ATM5/0/0.0-aal5 layer

Devmon sees these all as: ATM5/0/0  because devmon templates (atleast for
6509's) are looking at ifName as the main identifier, which is not always
unique.

Robert

On Fri, Oct 31, 2008 at 9:19 AM, Buchan Milne <user-9b139aff4dec@xymon.invalid>wrote:

On Friday 31 October 2008 17:35:44 Robert Holden wrote:

I have seen this as well.  I finally determined it was caused by ATM
interfaces.  Devmon does not give different components of an ATM circuit
(the physical interface, the -atm layer, .0 sub interface, -aal5 layer)
unique names.  So rrd was receiving data for 5 interfaces all with the
same
name.  As a temporary interface, I stopped monitoring the atm interfaces,
but this is a bug.

Interface names:
ATM5/0/0
ATM5/0/0-atm layer
ATM5/0/0.0-atm subif
ATM5/0/0-aal5 layer
ATM5/0/0.0-aal5 layer

It's the spaces in the interface names.

Devmon sees these all as: ATM5/0/0  because devmon templates (atleast for
6509's) are looking at ifName as the main identifier, which is not always
unique.

No, on earlier IOSs, there will still many interface names for ATM
interfaces,
but there were no spaces in the names. I have adjusted my template in this
case as follows to avoid the problem with spaces:

in transforms:

ifNameFixed     : REGSUB        : {ifName} /(\S+)(\s*)?(\S+)$/$1$3/

then replace ifName with ifNameFixed in the message file.

Not sure on a solution yet.  MRTG uses ifIndex as it's unique key.

Well, this is not the issue, but using ifIndex is not a solution either.

(BTW, there isn't a bug filed on this issue, so so far I've just been
wanting
to fix this for my own setup ...)

Regards,
Buchan

On Friday 31 October 2008 19:55:35 Robert Holden wrote:

When I run snmpwalk on my interfaces I get the following:
(I believe that the devmon template is using this ... notice that they are
all the same.

A transform won't help)
IF-MIB::ifName.70 = STRING: AT5/0/0
IF-MIB::ifName.71 = STRING: AT5/0/0
IF-MIB::ifName.72 = STRING: AT5/0/0
IF-MIB::ifName.73 = STRING: AT5/0/0
IF-MIB::ifName.74 = STRING: AT5/0/0

(This could be used, but you would end up w/ very long names.  G0/0 would
become GigabitEthernet0/0)
IF-MIB::ifDescr.70 = STRING: ATM5/0/0
IF-MIB::ifDescr.71 = STRING: ATM5/0/0-atm layer
IF-MIB::ifDescr.72 = STRING: ATM5/0/0.0-atm subif
IF-MIB::ifDescr.73 = STRING: ATM5/0/0-aal5 layer
IF-MIB::ifDescr.74 = STRING: ATM5/0/0.0-aal5 layer

(This may be helpful ... somehow ignore all type 37 and type 49, and sub
interface 0 ??)
IF-MIB::ifType.70 = INTEGER: sonet(39)
IF-MIB::ifType.71 = INTEGER: atm(37)
IF-MIB::ifType.72 = INTEGER: atmSubInterface(134)
IF-MIB::ifType.73 = INTEGER: aal5(49)
IF-MIB::ifType.74 = INTEGER: aal5(49)

I will open a bug if you like.  Is the issue that originated this thread
the same as the one I am experiencing?

On Monday 03 November 2008 09:46:42 Buchan Milne wrote:

On Friday 31 October 2008 19:55:35 Robert Holden wrote:

When I run snmpwalk on my interfaces I get the following:
(I believe that the devmon template is using this ... notice that they
are all the same.

Depending on the SNMP implementation (and may differ on different IOS
versions).

A transform won't help)
IF-MIB::ifName.70 = STRING: AT5/0/0
IF-MIB::ifName.71 = STRING: AT5/0/0
IF-MIB::ifName.72 = STRING: AT5/0/0
IF-MIB::ifName.73 = STRING: AT5/0/0
IF-MIB::ifName.74 = STRING: AT5/0/0

One of our 7613's has:

IF-MIB::ifName.1 = STRING: Gi3/1
[...]
IF-MIB::ifName.128 = STRING: ATM10/1/0.0-atm subif
IF-MIB::ifName.129 = STRING: ATM10/1/0-aal5 layer
IF-MIB::ifName.130 = STRING: ATM10/1/0.0-aal5 layer
IF-MIB::ifName.131 = STRING: VLAN-1

(This could be used, but you would end up w/ very long names.  G0/0 would
become GigabitEthernet0/0)
IF-MIB::ifDescr.70 = STRING: ATM5/0/0
IF-MIB::ifDescr.71 = STRING: ATM5/0/0-atm layer
IF-MIB::ifDescr.72 = STRING: ATM5/0/0.0-atm subif
IF-MIB::ifDescr.73 = STRING: ATM5/0/0-aal5 layer
IF-MIB::ifDescr.74 = STRING: ATM5/0/0.0-aal5 layer

On the same device as above:

IF-MIB::ifDescr.1 = STRING: GigabitEthernet3/1
[...]
IF-MIB::ifDescr.128 = STRING: ATM10/1/0.0-atm subif
IF-MIB::ifDescr.129 = STRING: ATM10/1/0-aal5 layer
IF-MIB::ifDescr.130 = STRING: ATM10/1/0.0-aal5 layer
IF-MIB::ifDescr.131 = STRING: unrouted VLAN 1

(This may be helpful ... somehow ignore all type 37 and type 49, and sub
interface 0 ??)
IF-MIB::ifType.70 = INTEGER: sonet(39)
IF-MIB::ifType.71 = INTEGER: atm(37)
IF-MIB::ifType.72 = INTEGER: atmSubInterface(134)
IF-MIB::ifType.73 = INTEGER: aal5(49)
IF-MIB::ifType.74 = INTEGER: aal5(49)

But, in my case, I want to graph:
IF-MIB::ifName.103 = STRING: ATM10/1/0.1-aal5 layer

(it is the only interface which shows the correct traffic)

I will open a bug if you like.  Is the issue that originated this thread
the same as the one I am experiencing?

Well, there are two aspects to the bug:

1)Devmon should strip invalid characters out of interface names for the RRD
data sent with the status message.

2)The Hobbit RRD collector module for devmon should not segfault if data is
not in the format:

<name_without_spaces> value[:value[:value]]

On Fri, Nov 7, 2008 at 7:28 AM, Buchan Milne <user-9b139aff4dec@xymon.invalid>wrote:

On Monday 03 November 2008 09:46:42 Buchan Milne wrote:

On Friday 31 October 2008 19:55:35 Robert Holden wrote:

When I run snmpwalk on my interfaces I get the following:
(I believe that the devmon template is using this ... notice that they
are all the same.

Depending on the SNMP implementation (and may differ on different IOS
versions).

A transform won't help)
IF-MIB::ifName.70 = STRING: AT5/0/0
IF-MIB::ifName.71 = STRING: AT5/0/0
IF-MIB::ifName.72 = STRING: AT5/0/0
IF-MIB::ifName.73 = STRING: AT5/0/0
IF-MIB::ifName.74 = STRING: AT5/0/0

One of our 7613's has:

IF-MIB::ifName.1 = STRING: Gi3/1
[...]
IF-MIB::ifName.128 = STRING: ATM10/1/0.0-atm subif
IF-MIB::ifName.129 = STRING: ATM10/1/0-aal5 layer
IF-MIB::ifName.130 = STRING: ATM10/1/0.0-aal5 layer
IF-MIB::ifName.131 = STRING: VLAN-1

(This could be used, but you would end up w/ very long names.  G0/0
would
become GigabitEthernet0/0)
IF-MIB::ifDescr.70 = STRING: ATM5/0/0
IF-MIB::ifDescr.71 = STRING: ATM5/0/0-atm layer
IF-MIB::ifDescr.72 = STRING: ATM5/0/0.0-atm subif
IF-MIB::ifDescr.73 = STRING: ATM5/0/0-aal5 layer
IF-MIB::ifDescr.74 = STRING: ATM5/0/0.0-aal5 layer

On the same device as above:

IF-MIB::ifDescr.1 = STRING: GigabitEthernet3/1
[...]
IF-MIB::ifDescr.128 = STRING: ATM10/1/0.0-atm subif
IF-MIB::ifDescr.129 = STRING: ATM10/1/0-aal5 layer
IF-MIB::ifDescr.130 = STRING: ATM10/1/0.0-aal5 layer
IF-MIB::ifDescr.131 = STRING: unrouted VLAN 1

(This may be helpful ... somehow ignore all type 37 and type 49, and
sub
interface 0 ??)
IF-MIB::ifType.70 = INTEGER: sonet(39)
IF-MIB::ifType.71 = INTEGER: atm(37)
IF-MIB::ifType.72 = INTEGER: atmSubInterface(134)
IF-MIB::ifType.73 = INTEGER: aal5(49)
IF-MIB::ifType.74 = INTEGER: aal5(49)

But, in my case, I want to graph:
IF-MIB::ifName.103 = STRING: ATM10/1/0.1-aal5 layer

(it is the only interface which shows the correct traffic)

I will open a bug if you like.  Is the issue that originated this
thread
the same as the one I am experiencing?

Well, there are two aspects to the bug:

1)Devmon should strip invalid characters out of interface names for the

RRD

data sent with the status message.

2)The Hobbit RRD collector module for devmon should not segfault if data
is
not in the format:

<name_without_spaces> value[:value[:value]]

I fixed this one last night. You can either grab the new do_devmon.c, and
run
'make' again, and copy the hobbitd_rrd over the previous binary, or you can
grab the new complete patch for a build from scratch:

http://devmon.svn.sourceforge.net/viewvc/devmon?view=rev&revision=90

(at present, viewvc seems to be a bit bust on sourceforge, but these should
be
the URLs to the current versions of the files when it is working:
http://devmon.svn.sourceforge.net/viewvc/devmon/trunk/extras/do_devmon.c
http://devmon.svn.sourceforge.net/viewvc/devmon/trunk/extras/hobbit-4.2.0-

)

I will update my Hobbit packages with this once I've upgraded my production
Hobbit box (but the SRPM is in Mandriva cooker already).

Regards,
Buchan

with your do_devmon.c
revision 154.
I figured that the core file are repeating for the same equipments. cisco
6513 and cisco 4507 for if_load.

Thanks in advance,

Mario.

Cisco 4507:

Alarming on
(Gi1/1,Gi1/2,Gi2/1,Gi2/2,Gi3/1,Gi3/2,Gi3/3,Gi3/4,Gi3/5,Gi3/6,Gi3/7)
Alarming on (Gi3/8,Gi3/9,Gi3/10,Gi3/11,Gi3/12,Gi3/13,Gi3/14,Gi3/15)
Alarming on (Gi3/16,Gi3/17,Gi3/18,Gi3/19,Gi3/20,Gi3/21,Gi3/22)
Alarming on (Gi3/23,Gi3/24,Gi3/25,Gi3/26,Gi3/27,Gi3/28,Gi3/29)
Alarming on (Gi3/30,Gi3/31,Gi3/32,Gi3/33,Gi3/34,Gi3/35,Gi3/36)
Alarming on (Gi3/37,Gi3/38,Gi3/39,Gi3/40,Gi3/41,Gi3/42,Gi3/43)
Alarming on (Gi3/44,Gi3/45,Gi3/46,Gi3/47,Gi3/48,Gi4/1,Gi4/2,Gi4/3)
Alarming on (Gi4/4,Gi4/5,Gi4/6,Gi4/7,Gi4/8,Gi4/9,Gi4/10,Gi4/11)
Alarming on (Gi4/12,Gi4/13,Gi4/14,Gi4/15,Gi4/16,Gi4/17,Gi4/18)
Alarming on (Gi4/19,Gi4/20,Gi4/21,Gi4/22,Gi4/23,Gi4/24,Gi4/25)
Alarming on (Gi4/26,Gi4/27,Gi4/28,Gi4/29,Gi4/30,Gi4/31,Gi4/32)
Alarming on (Gi4/33,Gi4/34,Gi4/35,Gi4/36,Gi4/37,Gi4/38,Gi4/39)
Alarming on (Gi4/40,Gi4/41,Gi4/42,Gi4/43,Gi4/44,Gi4/45,Gi4/46)
Alarming on (Gi4/47,Gi4/48,Gi5/1,Gi5/2,Gi5/3,Gi5/4,Gi5/5,Gi5/6)
Alarming on (Gi5/7,Gi5/8,Gi5/9,Gi5/10,Gi5/11,Gi5/12,Gi5/13,Gi5/14)
Alarming on (Gi5/15,Gi5/16,Gi5/17,Gi5/18,Gi5/19,Gi5/20,Gi5/21)
Alarming on (Gi5/22,Gi5/23,Gi5/24,Gi5/25,Gi5/26,Gi5/27,Gi5/28)
Alarming on (Gi5/29,Gi5/30,Gi5/31,Gi5/32,Gi5/33,Gi5/34,Gi5/35)
Alarming on (Gi5/36,Gi5/37,Gi5/38,Gi5/39,Gi5/40,Gi5/41,Gi5/42)
Alarming on (Gi5/43,Gi5/44,Gi5/45,Gi5/46,Gi5/47,Gi5/48,Gi6/1)
Alarming on (Gi6/2,Gi6/3,Gi6/4,Gi6/5,Gi6/6,Gi6/7,Gi6/8,Gi6/9)
Alarming on (Gi6/10,Gi6/11,Gi6/12,Gi6/13,Gi6/14,Gi6/15,Gi6/16)
Alarming on (Gi6/17,Gi6/18,Gi6/19,Gi6/20,Gi6/21,Gi6/22,Gi6/23)
Alarming on (Gi6/24,Gi6/25,Gi6/26,Gi6/27,Gi6/28,Gi6/29,Gi6/30)
Alarming on (Gi6/31,Gi6/32,Gi6/33,Gi6/34,Gi6/35,Gi6/36,Gi6/37)
Alarming on (Gi6/38,Gi6/39,Gi6/40,Gi6/41,Gi6/42,Gi6/43,Gi6/44)
Alarming on (Gi6/45,Gi6/46,Gi6/47,Gi6/48)


And Cisco 6513:

Input load:  yellow=75%, red=95%
Output load: yellow=75%, red=95%
Alarming on
(Gi1/1,Gi1/2,Gi2/1,Gi2/2,Gi3/1,Gi3/2,Gi3/3,Gi3/4,Gi3/5,Gi3/6,Gi3/7)
Alarming on (Gi3/8,Gi3/9,Gi3/10,Gi3/11,Gi3/12,Gi3/13,Gi3/14,Gi3/15)
Alarming on (Gi3/16,Gi3/17,Gi3/18,Gi3/19,Gi3/20,Gi3/21,Gi3/22)
Alarming on (Gi3/23,Gi3/24,Gi3/25,Gi3/26,Gi3/27,Gi3/28,Gi3/29)
Alarming on (Gi3/30,Gi3/31,Gi3/32,Gi3/33,Gi3/34,Gi3/35,Gi3/36)
Alarming on (Gi3/37,Gi3/38,Gi3/39,Gi3/40,Gi3/41,Gi3/42,Gi3/43)
Alarming on (Gi3/44,Gi3/45,Gi3/46,Gi3/47,Gi3/48,Fa6/1,Fa6/2,Fa6/3)
Alarming on (Fa6/4,Fa6/5,Fa6/6,Fa6/7,Fa6/8,Fa6/9,Fa6/10,Fa6/11)
Alarming on (Fa6/12,Fa6/13,Fa6/14,Fa6/15,Fa6/16,Fa6/17,Fa6/18)
Alarming on (Fa6/19,Fa6/20,Fa6/21,Fa6/22,Fa6/23,Fa6/24,Fa6/25)
Alarming on (Fa6/26,Fa6/27,Fa6/28,Fa6/29,Fa6/30,Fa6/31,Fa6/32)
Alarming on (Fa6/33,Fa6/34,Fa6/35,Fa6/36,Fa6/37,Fa6/38,Fa6/39)
Alarming on (Fa6/40,Fa6/41,Fa6/42,Fa6/43,Fa6/44,Fa6/45,Fa6/46)
Alarming on (Fa6/47,Fa6/48,Fa9/1,Fa9/2,Fa9/3,Fa9/4,Fa9/5,Fa9/6)
Alarming on (Fa9/7,Fa9/8,Fa9/9,Fa9/10,Fa9/11,Fa9/12,Fa9/13,Fa9/14)
Alarming on (Fa9/15,Fa9/16,Fa9/17,Fa9/18,Fa9/19,Fa9/20,Fa9/21)
Alarming on (Fa9/22,Fa9/23,Fa9/24,Fa9/25,Fa9/26,Fa9/27,Fa9/28)
Alarming on (Fa9/29,Fa9/30,Fa9/31,Fa9/32,Fa9/33,Fa9/34,Fa9/35)
Alarming on (Fa9/36,Fa9/37,Fa9/38,Fa9/39,Fa9/40,Fa9/41,Fa9/42)
Alarming on (Fa9/43,Fa9/44,Fa9/45,Fa9/46,Fa9/47,Fa9/48,Fa10/1)
Alarming on (Fa10/2,Fa10/3,Fa10/4,Fa10/5,Fa10/6,Fa10/7,Fa10/8)
Alarming on (Fa10/9,Fa10/10,Fa10/11,Fa10/12,Fa10/13,Fa10/14,Fa10/15)
Alarming on (Fa10/16,Fa10/17,Fa10/18,Fa10/19,Fa10/20,Fa10/21)
Alarming on (Fa10/22,Fa10/23,Fa10/24,Fa10/25,Fa10/26,Fa10/27)
Alarming on (Fa10/28,Fa10/29,Fa10/30,Fa10/31,Fa10/32,Fa10/33)
Alarming on (Fa10/34,Fa10/35,Fa10/36,Fa10/37,Fa10/38,Fa10/39)
Alarming on (Fa10/40,Fa10/41,Fa10/42,Fa10/43,Fa10/44,Fa10/45)
Alarming on (Fa10/46,Fa10/47,Fa10/48,Gi11/1,Gi11/2,Gi11/3,Gi11/4)
Alarming on (Gi11/5,Gi11/6,Gi11/7,Gi11/8,Gi11/9,Gi11/10,Gi11/11)
Alarming on (Gi11/12,Gi11/13,Gi11/14,Gi11/15,Gi11/16,Gi12/1,Gi12/2)
Alarming on (Gi12/3,Gi12/4,Gi12/5,Gi12/6,Gi12/7,Gi12/8,Gi12/9)
Alarming on (Gi12/10,Gi12/11,Gi12/12,Gi12/13,Gi12/14,Gi12/15)
Alarming on (Gi12/16,Gi13/1,Gi13/2,Gi13/3,Gi13/4,Gi13/5,Gi13/6)
Alarming on (Gi13/7,Gi13/8,Gi13/9,Gi13/10,Gi13/11,Gi13/12,Gi13/13)
Alarming on (Gi13/14,Gi13/15,Gi13/16,EO0/0,Po1,Po2,Po7,Po8,Gi5/1)
Alarming on (Gi5/2,Gi5/3,Gi5/4,Gi5/5,Gi5/6,Gi5/7,Gi5/8,Gi5/9)
Alarming on (Gi5/10,Gi5/11,Gi5/12,Gi5/13,Gi5/14,Gi5/15,Gi5/16)
Alarming on (Gi5/17,Gi5/18,Gi5/19,Gi5/20,Gi5/21,Gi5/22,Gi5/23)
Alarming on (Gi5/24,Gi5/25,Gi5/26,Gi5/27,Gi5/28,Gi5/29,Gi5/30)
Alarming on (Gi5/31,Gi5/32,Gi5/33,Gi5/34,Gi5/35,Gi5/36,Gi5/37)
Alarming on (Gi5/38,Gi5/39,Gi5/40,Gi5/41,Gi5/42,Gi5/43,Gi5/44)
Alarming on (Gi5/45,Gi5/46,Gi5/47,Gi5/48,Po10,Po30,Po50)