help setting up SNMP traps to Xymon
list Nicole Beck
Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1- The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal 2- The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server? Example snmptrapd.log entry (host name changed): 2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00 .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993" Example snmptt.log entry: Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993 Thanks in advance for any help. I'm sorry for the lengthy question. Nicole Beck
list Andy Farrior
1 - clearing the trap. Unfortunately, the only way the SNMP trap status changes on Xymon is when it receives a new SNMP trap from the device. If the device would send an "all clear" type of trap when it was happy, that'd change the status back to green. If there's a way to create an event on the device that generates a "Normal" event, you could use that as a mechanism to change the SNMP status on Xymon. (Example, if saving the configuration generates a "Normal" SNMP trap....) Ugly, but it'd work. The other way to address it would be to come up with a way to manually acknowledge or clear a trap similar to the alert acknowledgment in Xymon. I've never had time to pursue that. 2 - origination of the trap The only way I can think of addressing this is if each database server can be configured to send SNMP traps on their own. The SNMPTRAPD daemon uses the sending device's hostname and that's what is used to send to Xymon. Hope this helps, Andy
▸
From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Friday, October 15, 2010 9:42 AM To: 'xymon at xymon.com' Subject: [xymon] help setting up SNMP traps to Xymon Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1. The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal
1. The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server?
▸
Example snmptrapd.log entry (host name changed):
2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public
.1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00
.1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993"
Example snmptt.log entry:
Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993
Thanks in advance for any help. I'm sorry for the lengthy question.
Nicole Beck
list Nicole Beck
Thanks Andy. I thought that the severity of the trap came from what is defined in the /etc/snmp/snmptt.conf.oracle file? The cerebro.victoriacollege.edu website has a section saying: I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file. Does the snmptt.conf.oracle file have to be changed in order to deal with the different severity codes that it receives from the snmptrap? It seems like the way my snmptt.conf.oracle file is configured right now, it would always have a severity of "Normal", so Xymon would never change colors. I'm just learning SNMP and don't completely understand it yet, so maybe I'm missing something. I'll have to check with my DBA's again to find out if they can send their traps directly from the database server, rather than thru the Grid server. Thanks, Nicole
▸
From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Friday, October 15, 2010 12:48 PM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon 1 - clearing the trap. Unfortunately, the only way the SNMP trap status changes on Xymon is when it receives a new SNMP trap from the device. If the device would send an "all clear" type of trap when it was happy, that'd change the status back to green. If there's a way to create an event on the device that generates a "Normal" event, you could use that as a mechanism to change the SNMP status on Xymon. (Example, if saving the configuration generates a "Normal" SNMP trap....) Ugly, but it'd work. The other way to address it would be to come up with a way to manually acknowledge or clear a trap similar to the alert acknowledgment in Xymon. I've never had time to pursue that. 2 - origination of the trap The only way I can think of addressing this is if each database server can be configured to send SNMP traps on their own. The SNMPTRAPD daemon uses the sending device's hostname and that's what is used to send to Xymon. Hope this helps, Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Friday, October 15, 2010 9:42 AM To: 'xymon at xymon.com' Subject: [xymon] help setting up SNMP traps to Xymon Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1. The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal 1. The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server? Example snmptrapd.log entry (host name changed): 2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00 .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993" Example snmptt.log entry: Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993 Thanks in advance for any help. I'm sorry for the lengthy question. Nicole Beck
list Doug Williams
Check the SEC startup script (in my case /sbin/init.d/rc.sec) for the paths: OPTIONS="-detach -conf=/usr/local/etc/sec.conf -log=/var/adm/syslog/sec.log" OPTIONS2="-input=/var/adm/syslog/snmptt.log -input=/var/adm/syslog/snmpttunknown.log" Look in the snmpttunknown.log for any traps it does not have a pattern match for, in case the CLEAR trap may be formatted differently: Look in sec.conf and ensure there is pattern that matches the CLEAR trap if format different
▸
From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Wednesday, November 03, 2010 10:02 AM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon Thanks Andy. I thought that the severity of the trap came from what is defined in the /etc/snmp/snmptt.conf.oracle file? The cerebro.victoriacollege.edu website has a section saying: I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file. Does the snmptt.conf.oracle file have to be changed in order to deal with the different severity codes that it receives from the snmptrap? It seems like the way my snmptt.conf.oracle file is configured right now, it would always have a severity of "Normal", so Xymon would never change colors. I'm just learning SNMP and don't completely understand it yet, so maybe I'm missing something. I'll have to check with my DBA's again to find out if they can send their traps directly from the database server, rather than thru the Grid server. Thanks, Nicole From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Friday, October 15, 2010 12:48 PM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon 1 - clearing the trap. Unfortunately, the only way the SNMP trap status changes on Xymon is when it receives a new SNMP trap from the device. If the device would send an "all clear" type of trap when it was happy, that'd change the status back to green. If there's a way to create an event on the device that generates a "Normal" event, you could use that as a mechanism to change the SNMP status on Xymon. (Example, if saving the configuration generates a "Normal" SNMP trap....) Ugly, but it'd work. The other way to address it would be to come up with a way to manually acknowledge or clear a trap similar to the alert acknowledgment in Xymon. I've never had time to pursue that. 2 - origination of the trap The only way I can think of addressing this is if each database server can be configured to send SNMP traps on their own. The SNMPTRAPD daemon uses the sending device's hostname and that's what is used to send to Xymon. Hope this helps, Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Friday, October 15, 2010 9:42 AM To: 'xymon at xymon.com' Subject: [xymon] help setting up SNMP traps to Xymon Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1. The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal 2. The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server? Example snmptrapd.log entry (host name changed): 2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00 .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993" Example snmptt.log entry: Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993 Thanks in advance for any help. I'm sorry for the lengthy question. Nicole Beck
list Andy Farrior
The severity of the trap is defined in the corresponding snmptt.conf file. That's what makes the determination if the trap is Normal, SEVERE, or whatever. The only way to change the trap status on Xymon for a given device is for Xymon to receive a new trap (or the aforementioned fictional tool to acknowledge or set-to-normal the trap status) from the device that's different than the current status. To do that, something has to happen to create an event on that device for a "Normal" SNMP trap to be sent. That would replace whatever trap status Xymon shows for that device and change the status to "green". Example: 2010-10-30 15:56:11 .1.3.6.1.4.1.318.0.9 [cid:image001.gif at 01CB7B5A.68BCFC40]INFORMATIONAL APC UPS: Utility power restored: Returned from battery backup power; utility power restored. 2010-10-30 15:56:05 .1.3.6.1.4.1.318.0.5 [cid:image002.gif at 01CB7B5A.68BCFC40]WARNING APC UPS: On battery: The UPS has switched to battery backup power. In this instance the device sent an "WARNING" level event and it followed up with an "INFORMATIONAL" event that changed the trap status to "green". If your device doesn't send a "back to normal" type of trap, I'm suggesting you could generate an event on that device that would send a "Normal" or "INFORMATIONAL" level trap. You can look through the snmptt.conf file for your device and find these traps and try to figure out what you could do to generate that trap. My example of saving new copy of a configuration on that device may not generate a trap at all. Just have to find something that does. But like I said, it's an ugly workaround .... Sorry if I confused you. Hope this helps,
▸
Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Wednesday, November 03, 2010 9:02 AM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon Thanks Andy. I thought that the severity of the trap came from what is defined in the /etc/snmp/snmptt.conf.oracle file? The cerebro.victoriacollege.edu website has a section saying: I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file. Does the snmptt.conf.oracle file have to be changed in order to deal with the different severity codes that it receives from the snmptrap? It seems like the way my snmptt.conf.oracle file is configured right now, it would always have a severity of "Normal", so Xymon would never change colors. I'm just learning SNMP and don't completely understand it yet, so maybe I'm missing something. I'll have to check with my DBA's again to find out if they can send their traps directly from the database server, rather than thru the Grid server. Thanks, Nicole From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Friday, October 15, 2010 12:48 PM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon 1 - clearing the trap. Unfortunately, the only way the SNMP trap status changes on Xymon is when it receives a new SNMP trap from the device. If the device would send an "all clear" type of trap when it was happy, that'd change the status back to green. If there's a way to create an event on the device that generates a "Normal" event, you could use that as a mechanism to change the SNMP status on Xymon. (Example, if saving the configuration generates a "Normal" SNMP trap....) Ugly, but it'd work. The other way to address it would be to come up with a way to manually acknowledge or clear a trap similar to the alert acknowledgment in Xymon. I've never had time to pursue that. 2 - origination of the trap The only way I can think of addressing this is if each database server can be configured to send SNMP traps on their own. The SNMPTRAPD daemon uses the sending device's hostname and that's what is used to send to Xymon. Hope this helps, Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Friday, October 15, 2010 9:42 AM To: 'xymon at xymon.com' Subject: [xymon] help setting up SNMP traps to Xymon Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1. The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal 1. The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server? Example snmptrapd.log entry (host name changed): 2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00 .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993" Example snmptt.log entry: Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993 Thanks in advance for any help. I'm sorry for the lengthy question. Nicole Beck
list Nicole Beck
Hi Andy,
I think you may have answered part of my question. So I have to correctly define traps in my snmptt.conf.oracle file. The only thing in this file for the trap that's being sent is below. Given this definition, any trap I get, even if the severity is Critical, is still going to be "Normal" based on this file, thus Xymon won't ever change the status to red. (For tests, I changed the "Normal" in this file to "Severe", and every trap is red, even when something is cleared according to the logs).
EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal
FORMAT The variables included in the oraEM4Alert trap. $*
SDESC
The variables included in the oraEM4Alert trap.
Variables:
1: oraEM4AlertTargetName
Syntax="OCTETSTR"
Descr="The name of the target to which this alert applies."
... (lines removed for brevity)
8: oraEM4AlertSeverity
Syntax="OCTETSTR"
Descr="The severity of the alert e.g. Critical."
... (lines removed for brevity)
EDESC
I was rereading the snmptt documentation, and it looks like I might be able to use the "match' rule in the file. I can give that a try.
▸
Thanks,
Nicole
From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid]
Sent: Wednesday, November 03, 2010 2:32 PM
To: xymon at xymon.com
Subject: [xymon] RE: help setting up SNMP traps to Xymon
The severity of the trap is defined in the corresponding snmptt.conf file. That's what makes the determination if the trap is Normal, SEVERE, or whatever.
The only way to change the trap status on Xymon for a given device is for Xymon to receive a new trap (or the aforementioned fictional tool to acknowledge or set-to-normal the trap status) from the device that's different than the current status.
To do that, something has to happen to create an event on that device for a "Normal" SNMP trap to be sent. That would replace whatever trap status Xymon shows for that device and change the status to "green".
Example:
2010-10-30 15:56:11
.1.3.6.1.4.1.318.0.9
[cid:image001.gif at 01CB7C25.83D9E300]INFORMATIONAL
▸
APC UPS: Utility power restored: Returned from battery backup power; utility power restored.
2010-10-30 15:56:05
.1.3.6.1.4.1.318.0.5
[cid:image002.gif at 01CB7C25.83D9E300]WARNING
▸
APC UPS: On battery: The UPS has switched to battery backup power. In this instance the device sent an "WARNING" level event and it followed up with an "INFORMATIONAL" event that changed the trap status to "green". If your device doesn't send a "back to normal" type of trap, I'm suggesting you could generate an event on that device that would send a "Normal" or "INFORMATIONAL" level trap. You can look through the snmptt.conf file for your device and find these traps and try to figure out what you could do to generate that trap. My example of saving new copy of a configuration on that device may not generate a trap at all. Just have to find something that does. But like I said, it's an ugly workaround .... Sorry if I confused you. Hope this helps, Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Wednesday, November 03, 2010 9:02 AM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon Thanks Andy. I thought that the severity of the trap came from what is defined in the /etc/snmp/snmptt.conf.oracle file? The cerebro.victoriacollege.edu website has a section saying: I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file. Does the snmptt.conf.oracle file have to be changed in order to deal with the different severity codes that it receives from the snmptrap? It seems like the way my snmptt.conf.oracle file is configured right now, it would always have a severity of "Normal", so Xymon would never change colors. I'm just learning SNMP and don't completely understand it yet, so maybe I'm missing something. I'll have to check with my DBA's again to find out if they can send their traps directly from the database server, rather than thru the Grid server. Thanks, Nicole From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Friday, October 15, 2010 12:48 PM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon 1 - clearing the trap. Unfortunately, the only way the SNMP trap status changes on Xymon is when it receives a new SNMP trap from the device. If the device would send an "all clear" type of trap when it was happy, that'd change the status back to green. If there's a way to create an event on the device that generates a "Normal" event, you could use that as a mechanism to change the SNMP status on Xymon. (Example, if saving the configuration generates a "Normal" SNMP trap....) Ugly, but it'd work. The other way to address it would be to come up with a way to manually acknowledge or clear a trap similar to the alert acknowledgment in Xymon. I've never had time to pursue that. 2 - origination of the trap The only way I can think of addressing this is if each database server can be configured to send SNMP traps on their own. The SNMPTRAPD daemon uses the sending device's hostname and that's what is used to send to Xymon. Hope this helps, Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Friday, October 15, 2010 9:42 AM To: 'xymon at xymon.com' Subject: [xymon] help setting up SNMP traps to Xymon Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1. The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal 1. The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server? Example snmptrapd.log entry (host name changed): 2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00 .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993" Example snmptt.log entry: Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993 Thanks in advance for any help. I'm sorry for the lengthy question. Nicole Beck
list Nicole Beck
Hi Doug, I found a CLEAR trap in my snmptrapd and snmptt logs, and it says "Unreachable Clear". There is nothing like that defined in my snmptt.conf.oracle file. If I understand this right, I have some things to change in the snmptt.conf.oracle file. Thanks, Nicole
▸
From: Williams, Doug (Consultant-RIC) [mailto:user-2bde7ec54a85@xymon.invalid] Sent: Wednesday, November 03, 2010 2:32 PM To: xymon at xymon.com Subject: RE: [xymon] RE: help setting up SNMP traps to Xymon Check the SEC startup script (in my case /sbin/init.d/rc.sec) for the paths: OPTIONS="-detach -conf=/usr/local/etc/sec.conf -log=/var/adm/syslog/sec.log" OPTIONS2="-input=/var/adm/syslog/snmptt.log -input=/var/adm/syslog/snmpttunknown.log" Look in the snmpttunknown.log for any traps it does not have a pattern match for, in case the CLEAR trap may be formatted differently: Look in sec.conf and ensure there is pattern that matches the CLEAR trap if format different From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Wednesday, November 03, 2010 10:02 AM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon Thanks Andy. I thought that the severity of the trap came from what is defined in the /etc/snmp/snmptt.conf.oracle file? The cerebro.victoriacollege.edu website has a section saying: I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file. Does the snmptt.conf.oracle file have to be changed in order to deal with the different severity codes that it receives from the snmptrap? It seems like the way my snmptt.conf.oracle file is configured right now, it would always have a severity of "Normal", so Xymon would never change colors. I'm just learning SNMP and don't completely understand it yet, so maybe I'm missing something. I'll have to check with my DBA's again to find out if they can send their traps directly from the database server, rather than thru the Grid server. Thanks, Nicole From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Friday, October 15, 2010 12:48 PM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon 1 - clearing the trap. Unfortunately, the only way the SNMP trap status changes on Xymon is when it receives a new SNMP trap from the device. If the device would send an "all clear" type of trap when it was happy, that'd change the status back to green. If there's a way to create an event on the device that generates a "Normal" event, you could use that as a mechanism to change the SNMP status on Xymon. (Example, if saving the configuration generates a "Normal" SNMP trap....) Ugly, but it'd work. The other way to address it would be to come up with a way to manually acknowledge or clear a trap similar to the alert acknowledgment in Xymon. I've never had time to pursue that. 2 - origination of the trap The only way I can think of addressing this is if each database server can be configured to send SNMP traps on their own. The SNMPTRAPD daemon uses the sending device's hostname and that's what is used to send to Xymon. Hope this helps, Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Friday, October 15, 2010 9:42 AM To: 'xymon at xymon.com' Subject: [xymon] help setting up SNMP traps to Xymon Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1. The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal 1. The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server? Example snmptrapd.log entry (host name changed): 2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00 .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993" Example snmptt.log entry: Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993 Thanks in advance for any help. I'm sorry for the lengthy question. Nicole Beck
list Andy Farrior
If snmptrapd/snmptt receives the SNMP trap .1.3.6.1.4.1.111.15.2.0.1, snmptt will say the condition is "Normal" which gets converted to "green". When you change "Normal" to "SEVERE" or "CRITICAL" it will get converted to "red". If some status variable is being sent along with this trap OID, I'm not sure how snmptt handles it. If the SNMP trap OID sent to snmptrapd is not defined in the snmptt.conf files, snmptt will consider that trap an "unknown" trap and log it in the "snmpttunknown.log". If traps of interest are showing up in the unknown log, you'll need to update the snmptt.conf file with those OIDs of intereset.
▸
Andy
From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid]
Sent: Thursday, November 04, 2010 12:46 PM
To: xymon at xymon.com
Subject: [xymon] RE: help setting up SNMP traps to Xymon
Hi Andy,
I think you may have answered part of my question. So I have to correctly define traps in my snmptt.conf.oracle file. The only thing in this file for the trap that's being sent is below. Given this definition, any trap I get, even if the severity is Critical, is still going to be "Normal" based on this file, thus Xymon won't ever change the status to red. (For tests, I changed the "Normal" in this file to "Severe", and every trap is red, even when something is cleared according to the logs).
EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal
FORMAT The variables included in the oraEM4Alert trap. $*
SDESC
The variables included in the oraEM4Alert trap.
Variables:
1: oraEM4AlertTargetName
Syntax="OCTETSTR"
Descr="The name of the target to which this alert applies."
... (lines removed for brevity)
8: oraEM4AlertSeverity
Syntax="OCTETSTR"
Descr="The severity of the alert e.g. Critical."
... (lines removed for brevity)
EDESC
I was rereading the snmptt documentation, and it looks like I might be able to use the "match' rule in the file. I can give that a try.
Thanks,
Nicole
From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid]
Sent: Wednesday, November 03, 2010 2:32 PM
To: xymon at xymon.com
Subject: [xymon] RE: help setting up SNMP traps to Xymon
The severity of the trap is defined in the corresponding snmptt.conf file. That's what makes the determination if the trap is Normal, SEVERE, or whatever.
The only way to change the trap status on Xymon for a given device is for Xymon to receive a new trap (or the aforementioned fictional tool to acknowledge or set-to-normal the trap status) from the device that's different than the current status.
To do that, something has to happen to create an event on that device for a "Normal" SNMP trap to be sent. That would replace whatever trap status Xymon shows for that device and change the status to "green".
Example:
2010-10-30 15:56:11
.1.3.6.1.4.1.318.0.9
[cid:image001.gif at 01CB7C2A.4CC32840]INFORMATIONAL
▸
APC UPS: Utility power restored: Returned from battery backup power; utility power restored.
2010-10-30 15:56:05
.1.3.6.1.4.1.318.0.5
[cid:image002.gif at 01CB7C2A.4CC32840]WARNING
▸
APC UPS: On battery: The UPS has switched to battery backup power.
In this instance the device sent an "WARNING" level event and it followed up with an "INFORMATIONAL" event that changed the trap status to "green".
If your device doesn't send a "back to normal" type of trap, I'm suggesting you could generate an event on that device that would send a "Normal" or "INFORMATIONAL" level trap. You can look through the snmptt.conf file for your device and find these traps and try to figure out what you could do to generate that trap. My example of saving new copy of a configuration on that device may not generate a trap at all. Just have to find something that does.
But like I said, it's an ugly workaround ....
Sorry if I confused you. Hope this helps,
Andy
▸
From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Wednesday, November 03, 2010 9:02 AM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon Thanks Andy. I thought that the severity of the trap came from what is defined in the /etc/snmp/snmptt.conf.oracle file? The cerebro.victoriacollege.edu website has a section saying: I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file. Does the snmptt.conf.oracle file have to be changed in order to deal with the different severity codes that it receives from the snmptrap? It seems like the way my snmptt.conf.oracle file is configured right now, it would always have a severity of "Normal", so Xymon would never change colors. I'm just learning SNMP and don't completely understand it yet, so maybe I'm missing something. I'll have to check with my DBA's again to find out if they can send their traps directly from the database server, rather than thru the Grid server. Thanks, Nicole From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Friday, October 15, 2010 12:48 PM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon 1 - clearing the trap. Unfortunately, the only way the SNMP trap status changes on Xymon is when it receives a new SNMP trap from the device. If the device would send an "all clear" type of trap when it was happy, that'd change the status back to green. If there's a way to create an event on the device that generates a "Normal" event, you could use that as a mechanism to change the SNMP status on Xymon. (Example, if saving the configuration generates a "Normal" SNMP trap....) Ugly, but it'd work. The other way to address it would be to come up with a way to manually acknowledge or clear a trap similar to the alert acknowledgment in Xymon. I've never had time to pursue that. 2 - origination of the trap The only way I can think of addressing this is if each database server can be configured to send SNMP traps on their own. The SNMPTRAPD daemon uses the sending device's hostname and that's what is used to send to Xymon. Hope this helps, Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Friday, October 15, 2010 9:42 AM To: 'xymon at xymon.com' Subject: [xymon] help setting up SNMP traps to Xymon Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1. The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal 1. The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server? Example snmptrapd.log entry (host name changed): 2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00 .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993" Example snmptt.log entry: Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993 Thanks in advance for any help. I'm sorry for the lengthy question. Nicole Beck
list Doug Williams
It depends on how you have it all configured. For example, I have our setup such that a single host receives a boat load of different traps and so a clear trap does not really make sense since it has no way of knowing which trap alert it is clearing. But if you have yours setup in bb-hosts such that oracle_<some_state> is defined as a host or entity with it's own IP (alias) for example and it only receives a specific type of trap, then a clear trap would make sense. In this case you could even define your own OIDs (or extend existing ones) (making sure all unique amongst all your snmptt.conf entries )
▸
From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Thursday, November 04, 2010 1:56 PM To: xymon at xymon.com Subject: RE: [xymon] RE: help setting up SNMP traps to Xymon Hi Doug, I found a CLEAR trap in my snmptrapd and snmptt logs, and it says "Unreachable Clear". There is nothing like that defined in my snmptt.conf.oracle file. If I understand this right, I have some things to change in the snmptt.conf.oracle file. Thanks, Nicole From: Williams, Doug (Consultant-RIC) [mailto:user-2bde7ec54a85@xymon.invalid] Sent: Wednesday, November 03, 2010 2:32 PM To: xymon at xymon.com Subject: RE: [xymon] RE: help setting up SNMP traps to Xymon Check the SEC startup script (in my case /sbin/init.d/rc.sec) for the paths: OPTIONS="-detach -conf=/usr/local/etc/sec.conf -log=/var/adm/syslog/sec.log" OPTIONS2="-input=/var/adm/syslog/snmptt.log -input=/var/adm/syslog/snmpttunknown.log" Look in the snmpttunknown.log for any traps it does not have a pattern match for, in case the CLEAR trap may be formatted differently: Look in sec.conf and ensure there is pattern that matches the CLEAR trap if format different From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Wednesday, November 03, 2010 10:02 AM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon Thanks Andy. I thought that the severity of the trap came from what is defined in the /etc/snmp/snmptt.conf.oracle file? The cerebro.victoriacollege.edu website has a section saying: I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file. Does the snmptt.conf.oracle file have to be changed in order to deal with the different severity codes that it receives from the snmptrap? It seems like the way my snmptt.conf.oracle file is configured right now, it would always have a severity of "Normal", so Xymon would never change colors. I'm just learning SNMP and don't completely understand it yet, so maybe I'm missing something. I'll have to check with my DBA's again to find out if they can send their traps directly from the database server, rather than thru the Grid server. Thanks, Nicole From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Friday, October 15, 2010 12:48 PM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon 1 - clearing the trap. Unfortunately, the only way the SNMP trap status changes on Xymon is when it receives a new SNMP trap from the device. If the device would send an "all clear" type of trap when it was happy, that'd change the status back to green. If there's a way to create an event on the device that generates a "Normal" event, you could use that as a mechanism to change the SNMP status on Xymon. (Example, if saving the configuration generates a "Normal" SNMP trap....) Ugly, but it'd work. The other way to address it would be to come up with a way to manually acknowledge or clear a trap similar to the alert acknowledgment in Xymon. I've never had time to pursue that. 2 - origination of the trap The only way I can think of addressing this is if each database server can be configured to send SNMP traps on their own. The SNMPTRAPD daemon uses the sending device's hostname and that's what is used to send to Xymon. Hope this helps, Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Friday, October 15, 2010 9:42 AM To: 'xymon at xymon.com' Subject: [xymon] help setting up SNMP traps to Xymon Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1. The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal 2. The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server? Example snmptrapd.log entry (host name changed): 2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00 .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993" Example snmptt.log entry: Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993 Thanks in advance for any help. I'm sorry for the lengthy question. Nicole Beck
list Doug Williams
And (sorry for after-thoughts, replying to my own posts), I should say it could be of no consequence not knowing when a specific trap clears, due to other faulty device or application condition(s) will keep continue sending an error condition alert trap, so even if say oracle sends clear trap and other red or yellow alert conditions exists, all would be well since the trap icon would remain red and if no other device/application trap alert conditions exist it would clear. That was likely what you wanted, I'm just long on chewing through it... J So, just define the desired unique trap OID in snmptt.conf and ensure it matches a pattern in sec.conf, and it should be golden.
▸
From: Williams, Doug (Consultant-RIC) [mailto:user-2bde7ec54a85@xymon.invalid] Sent: Friday, November 05, 2010 4:15 PM To: xymon at xymon.com Subject: RE: [xymon] RE: help setting up SNMP traps to Xymon It depends on how you have it all configured. For example, I have our setup such that a single host receives a boat load of different traps and so a clear trap does not really make sense since it has no way of knowing which trap alert it is clearing. But if you have yours setup in bb-hosts such that oracle_<some_state> is defined as a host or entity with it's own IP (alias) for example and it only receives a specific type of trap, then a clear trap would make sense. In this case you could even define your own OIDs (or extend existing ones) (making sure all unique amongst all your snmptt.conf entries ) From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Thursday, November 04, 2010 1:56 PM To: xymon at xymon.com Subject: RE: [xymon] RE: help setting up SNMP traps to Xymon Hi Doug, I found a CLEAR trap in my snmptrapd and snmptt logs, and it says "Unreachable Clear". There is nothing like that defined in my snmptt.conf.oracle file. If I understand this right, I have some things to change in the snmptt.conf.oracle file. Thanks, Nicole From: Williams, Doug (Consultant-RIC) [mailto:user-2bde7ec54a85@xymon.invalid] Sent: Wednesday, November 03, 2010 2:32 PM To: xymon at xymon.com Subject: RE: [xymon] RE: help setting up SNMP traps to Xymon Check the SEC startup script (in my case /sbin/init.d/rc.sec) for the paths: OPTIONS="-detach -conf=/usr/local/etc/sec.conf -log=/var/adm/syslog/sec.log" OPTIONS2="-input=/var/adm/syslog/snmptt.log -input=/var/adm/syslog/snmpttunknown.log" Look in the snmpttunknown.log for any traps it does not have a pattern match for, in case the CLEAR trap may be formatted differently: Look in sec.conf and ensure there is pattern that matches the CLEAR trap if format different From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Wednesday, November 03, 2010 10:02 AM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon Thanks Andy. I thought that the severity of the trap came from what is defined in the /etc/snmp/snmptt.conf.oracle file? The cerebro.victoriacollege.edu website has a section saying: I've noticed that the APC and Dell MIB files have a SEVERITY definition in them. SNMPTT uses that to establish the severity for each trap (Normal|INFORMATIONAL|SEVERE|WARNING|...). However, I've noticed that Cisco and Canoga Perkins don't have those definitions; so, every trap event is considered Normal. You'll need to change the severity for the various traps as desired in the snmptt.conf file. Does the snmptt.conf.oracle file have to be changed in order to deal with the different severity codes that it receives from the snmptrap? It seems like the way my snmptt.conf.oracle file is configured right now, it would always have a severity of "Normal", so Xymon would never change colors. I'm just learning SNMP and don't completely understand it yet, so maybe I'm missing something. I'll have to check with my DBA's again to find out if they can send their traps directly from the database server, rather than thru the Grid server. Thanks, Nicole From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Friday, October 15, 2010 12:48 PM To: xymon at xymon.com Subject: [xymon] RE: help setting up SNMP traps to Xymon 1 - clearing the trap. Unfortunately, the only way the SNMP trap status changes on Xymon is when it receives a new SNMP trap from the device. If the device would send an "all clear" type of trap when it was happy, that'd change the status back to green. If there's a way to create an event on the device that generates a "Normal" event, you could use that as a mechanism to change the SNMP status on Xymon. (Example, if saving the configuration generates a "Normal" SNMP trap....) Ugly, but it'd work. The other way to address it would be to come up with a way to manually acknowledge or clear a trap similar to the alert acknowledgment in Xymon. I've never had time to pursue that. 2 - origination of the trap The only way I can think of addressing this is if each database server can be configured to send SNMP traps on their own. The SNMPTRAPD daemon uses the sending device's hostname and that's what is used to send to Xymon. Hope this helps, Andy From: Nicole Beck [mailto:user-80034b0579c6@xymon.invalid] Sent: Friday, October 15, 2010 9:42 AM To: 'xymon at xymon.com' Subject: [xymon] help setting up SNMP traps to Xymon Hello, I'm running Xymon 4.2.3 on RHEL 5.4 and I'm trying to setup Xymon to receive SNMP traps from our Oracle database servers. We are also running Oracle Enterprise Manager on another server (we call it the "grid") that monitors the Oracle database servers too, but our DBA's want to use Xymon to alert them of Oracle problems instead of the grid server so they don't get alerts from two different places. The DBA's manage OEM(therefore I'm not familiar with it), and have configured it to recieve SNMP traps and send them to Xymon. I configured the Xymon server to receive traps using the documentation at http://cerebro.victoriacollege.edu/hobbit-trap.html. We do receive traps, but there are two problems: 1. The trap status diamond does not change color when it gets a trap or when the trap clears. My understanding is that this is set in the Oracle MIBS that are translated to the /etc/snmp/snmptt.conf.oracle file. I'm not sure how to change the file to get it to give different alerts. If I change the word "NORMAL" in the following line to "SEVERE" the trap status will go red, but will not return to green when the trap clears. Any suggestions on how to fix this? EVENT oraEM4Alert .1.3.6.1.4.1.111.15.2.0.1 "Status Events" Normal 2. The traps status changes for the grid server instead of the database server with the problem. Drilling down into the "trap" status, you will see an error for the database server though. The snmptrapd.log and snmptt.log files (examples below) show the traps coming from the grid server. Does anyone know how to get the traps to appear under the database server name rather than the grid server? Example snmptrapd.log entry (host name changed): 2010-10-14 19:04:49 grid-server [xxx.xxx.xxx.xxx] (via UDP: [xxx.xxx.xxx.xxx]:51617) TRAP, SNMP v1, community public .1.3.6.1.4.1.111.15.2 Enterprise Specific Trap (1) Uptime: 0:06:31.00 .1.3.6.1.4.1.111.15.1.1.1.2.1 = STRING: "DB00157.world" .1.3.6.1.4.1.111.15.1.1.1.3.1 = STRING: "Database Instance" .1.3.6.1.4.1.111.15.1.1.1.4.1 = STRING: "database-server " .1.3.6.1.4.1.111.15.1.1.1.5.1 = STRING: "Status".1.3.6.1.4.1.111.15.1.1.1.6.1 = "" .1.3.6.1.4.1.111.15.1.1.1.7.1 = "" .1.3.6.1.4.1.111.15.1.1.1.8.1 = STRING: "Oct 14, 2010 7:04:00 PM EDT" .1.3.6.1.4.1.111.15.1.1.1.9.1 = STRING: "Critical" .1.3.6.1.4.1.111.15.1.1.1.10.1 = STRING: "Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin)." .1.3.6.1.4.1.111.15.1.1.1.11.1 = STRING: "Test SNMP database-server DB Down " .1.3.6.1.4.1.111.15.1.1.1.12.1 = STRING: "SYSMAN" .1.3.6.1.4.1.111.15.1.1.1.13.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.14.1 = "" .1.3.6.1.4.1.111.15.1.1.1.15.1 = STRING: "923A730174F36B87E040E6801417642D" .1.3.6.1.4.1.111.15.1.1.1.16.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.17.1 = "" .1.3.6.1.4.1.111.15.1.1.1.18.1 = STRING: "0" .1.3.6.1.4.1.111.15.1.1.1.19.1 = "" .1.3.6.1.4.1.111.15.1.1.1.20.1 = STRING: "1" .1.3.6.1.4.1.111.15.1.1.1.21.1 = STRING: "923A421065A8A060E040E68015173993" Example snmptt.log entry: Oct 14 19:04:52 xymon-server snmptt[0]: .1.3.6.1.4.1.111.15.2.0.1 SEVERE "Status Events" grid-server - The variables included in the oraEM4Alert trap. DB00157.world Database Instance database-server Status Oct 14, 2010 7:04:00 PM EDT Critical Failed to connect to database instance: ORA-01033: ORACLE initialization or shutdown in progress (DBD ERROR: OCISessionBegin). Test SNMP database-server DB Down SYSMAN 0 923A730174F36B87E040E6801417642D 0 0 1 923A421065A8A060E040E68015173993 Thanks in advance for any help. I'm sorry for the lengthy question. Nicole Beck