Xymon Mailing List Archive search

Devmon tests clear but snmpwalk works

13 messages in this thread

list Colin Coe · Thu, 26 May 2016 20:20:53 +0800 ·
Hi all

I know this is the Xymon list not Devmon but I think I'll have most luck on
this list so apologies in advance.

I have a few of Cisco switches that I've just started monitoring at a
couple of remote sites. snmpwalk works from the Xymon server but the Devmon
tests are all clear (Devmon runs on the Xymon server).

When I look at the devmon.log I just see lots of
---
[16-05-26 at 20:19:09] No SNMP data found for ifName on swi02
[16-05-26 at 20:19:09] No SNMP data found for ifOutPktsSec on swi02
[16-05-26 at 20:19:09] No SNMP data found for ifOutCollisions on swi02
---

Any ideas on tracking this down?

We have many if these switches at other sites and they display correctly in
Xymon.

Thanks

CC
list Jeremy Laidman · Thu, 26 May 2016 12:42:50 +0000 ·
Colin

Might be the 32bit ID limit. See here:

http://lists.xymon.com/archive/2014-November/040624.html

J
quoted from Colin Coe

On Thu, 26 May 2016, 22:21 Colin Coe <user-5b250cd7a540@xymon.invalid> wrote:
Hi all

I know this is the Xymon list not Devmon but I think I'll have most luck
on this list so apologies in advance.

I have a few of Cisco switches that I've just started monitoring at a
couple of remote sites. snmpwalk works from the Xymon server but the Devmon
tests are all clear (Devmon runs on the Xymon server).

When I look at the devmon.log I just see lots of
---
[16-05-26 at 20:19:09] No SNMP data found for ifName on swi02
[16-05-26 at 20:19:09] No SNMP data found for ifOutPktsSec on swi02
[16-05-26 at 20:19:09] No SNMP data found for ifOutCollisions on swi02
---

Any ideas on tracking this down?

We have many if these switches at other sites and they display correctly
in Xymon.

Thanks

CC

list Nikolai Lifanov · Thu, 26 May 2016 08:49:12 -0400 ·
I have patches to "fix" message size limit.
This updates SNMP_Session and other external modules to
the latest "vanilla" upstream versions and changes dm_snmp
to use the API properly (and bump max_pdu_size to 64k).
quoted from Jeremy Laidman

On 05/26/2016 08:42, Jeremy Laidman wrote:
Colin

Might be the 32bit ID limit. See here:

http://lists.xymon.com/archive/2014-November/040624.html

J


On Thu, 26 May 2016, 22:21 Colin Coe <user-5b250cd7a540@xymon.invalid
<mailto:user-5b250cd7a540@xymon.invalid>> wrote:

    Hi all

    I know this is the Xymon list not Devmon but I think I'll have most
    luck on this list so apologies in advance.

    I have a few of Cisco switches that I've just started monitoring at
    a couple of remote sites. snmpwalk works from the Xymon server but
    the Devmon tests are all clear (Devmon runs on the Xymon server).

    When I look at the devmon.log I just see lots of     ---     [16-05-26 at 20:19:09] No SNMP data found for ifName on swi02
    [16-05-26 at 20:19:09] No SNMP data found for ifOutPktsSec on swi02
    [16-05-26 at 20:19:09] No SNMP data found for ifOutCollisions on swi02
    --- 
    Any ideas on tracking this down?

    We have many if these switches at other sites and they display
    correctly in Xymon.

    Thanks

    CC

list Nikolai Lifanov · Thu, 26 May 2016 08:53:33 -0400 ·
quoted from Nikolai Lifanov
On 05/26/2016 08:49, Nikolai Lifanov wrote:
I have patches to "fix" message size limit.
This updates SNMP_Session and other external modules to
the latest "vanilla" upstream versions and changes dm_snmp
to use the API properly (and bump max_pdu_size to 64k).
See attached. You can tweak max_pdu_size in dm_snmp.pm
quoted from Jeremy Laidman
On 05/26/2016 08:42, Jeremy Laidman wrote:
Colin

Might be the 32bit ID limit. See here:

http://lists.xymon.com/archive/2014-November/040624.html

J


On Thu, 26 May 2016, 22:21 Colin Coe <user-5b250cd7a540@xymon.invalid
<mailto:user-5b250cd7a540@xymon.invalid>> wrote:

    Hi all

    I know this is the Xymon list not Devmon but I think I'll have most
    luck on this list so apologies in advance.

    I have a few of Cisco switches that I've just started monitoring at
    a couple of remote sites. snmpwalk works from the Xymon server but
    the Devmon tests are all clear (Devmon runs on the Xymon server).

    When I look at the devmon.log I just see lots of 
    --- 
    [16-05-26 at 20:19:09] No SNMP data found for ifName on swi02
    [16-05-26 at 20:19:09] No SNMP data found for ifOutPktsSec on swi02
    [16-05-26 at 20:19:09] No SNMP data found for ifOutCollisions on swi02
    --- 

    Any ideas on tracking this down?

    We have many if these switches at other sites and they display
    correctly in Xymon.

    Thanks

    CC
    
list Nikolai Lifanov · Thu, 26 May 2016 08:57:07 -0400 ·
quoted from Nikolai Lifanov
On 05/26/2016 08:53, Nikolai Lifanov wrote:
On 05/26/2016 08:49, Nikolai Lifanov wrote:
I have patches to "fix" message size limit.
This updates SNMP_Session and other external modules to
the latest "vanilla" upstream versions and changes dm_snmp
to use the API properly (and bump max_pdu_size to 64k).
See attached. You can tweak max_pdu_size in dm_snmp.pm
Sorry, this should read "max_pdu_len".
quoted from Jeremy Laidman
On 05/26/2016 08:42, Jeremy Laidman wrote:
Colin

Might be the 32bit ID limit. See here:

http://lists.xymon.com/archive/2014-November/040624.html

J


On Thu, 26 May 2016, 22:21 Colin Coe <user-5b250cd7a540@xymon.invalid
<mailto:user-5b250cd7a540@xymon.invalid>> wrote:

    Hi all

    I know this is the Xymon list not Devmon but I think I'll have most
    luck on this list so apologies in advance.

    I have a few of Cisco switches that I've just started monitoring at
    a couple of remote sites. snmpwalk works from the Xymon server but
    the Devmon tests are all clear (Devmon runs on the Xymon server).

    When I look at the devmon.log I just see lots of 
    --- 
    [16-05-26 at 20:19:09] No SNMP data found for ifName on swi02
    [16-05-26 at 20:19:09] No SNMP data found for ifOutPktsSec on swi02
    [16-05-26 at 20:19:09] No SNMP data found for ifOutCollisions on swi02
    --- 

    Any ideas on tracking this down?

    We have many if these switches at other sites and they display
    correctly in Xymon.

    Thanks

    CC
    
list Nikolai Lifanov · Thu, 26 May 2016 09:01:53 -0400 ·
quoted from Nikolai Lifanov
On 05/26/2016 08:57, Nikolai Lifanov wrote:
On 05/26/2016 08:53, Nikolai Lifanov wrote:
On 05/26/2016 08:49, Nikolai Lifanov wrote:
I have patches to "fix" message size limit.
This updates SNMP_Session and other external modules to
the latest "vanilla" upstream versions and changes dm_snmp
to use the API properly (and bump max_pdu_size to 64k).
See attached. You can tweak max_pdu_size in dm_snmp.pm
Sorry, this should read "max_pdu_len".
List ate my attachments. Here:
https://lifanov.com/files/patches/devmon-use-snmp-session-api.diff
https://lifanov.com/files/patches/devmon-external-module-update.diff
quoted from Jeremy Laidman
On 05/26/2016 08:42, Jeremy Laidman wrote:
Colin

Might be the 32bit ID limit. See here:

http://lists.xymon.com/archive/2014-November/040624.html

J


On Thu, 26 May 2016, 22:21 Colin Coe <user-5b250cd7a540@xymon.invalid
<mailto:user-5b250cd7a540@xymon.invalid>> wrote:

    Hi all

    I know this is the Xymon list not Devmon but I think I'll have most
    luck on this list so apologies in advance.

    I have a few of Cisco switches that I've just started monitoring at
    a couple of remote sites. snmpwalk works from the Xymon server but
    the Devmon tests are all clear (Devmon runs on the Xymon server).

    When I look at the devmon.log I just see lots of 
    --- 
    [16-05-26 at 20:19:09] No SNMP data found for ifName on swi02
    [16-05-26 at 20:19:09] No SNMP data found for ifOutPktsSec on swi02
    [16-05-26 at 20:19:09] No SNMP data found for ifOutCollisions on swi02
    --- 

    Any ideas on tracking this down?

    We have many if these switches at other sites and they display
    correctly in Xymon.

    Thanks

    CC
    
list Paul Root · Thu, 26 May 2016 13:38:40 +0000 ·
Look in your hosts.db file, and see if the switches are identified as the proper template and that the community string is correct. The newer NX type switches do not work with the old template.

Use snmpwalk/snmpget to pull some of the OIDs of the template.
quoted from Colin Coe

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Colin Coe
Sent: Thursday, May 26, 2016 7:21 AM
To: xymon at xymon.com
Subject: [Xymon] Devmon tests clear but snmpwalk works

Hi all

I know this is the Xymon list not Devmon but I think I'll have most luck on this list so apologies in advance.

I have a few of Cisco switches that I've just started monitoring at a couple of remote sites. snmpwalk works from the Xymon server but the Devmon tests are all clear (Devmon runs on the Xymon server).

When I look at the devmon.log I just see lots of
---
[16-05-26 at 20:19:09] No SNMP data found for ifName on swi02
[16-05-26 at 20:19:09] No SNMP data found for ifOutPktsSec on swi02
[16-05-26 at 20:19:09] No SNMP data found for ifOutCollisions on swi02
---

Any ideas on tracking this down?

We have many if these switches at other sites and they display correctly in Xymon.

Thanks

CC

This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
list Colin Coe · Fri, 27 May 2016 06:46:29 +0800 ·
Hi all and many thanks for all the replies.

Jeremy: I did the tcpdump as per the thread and found that the R= value was
significantly less than the maximum 32 bit integer on some of the switches
but negative on others.  Censored, truncated output follows.  How do we
deal with the negative numbers?
---
06:35:54.584298 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 72)
    172.22.106.11.60150 > 192.168.78.254.snmp:  { SNMPv2c C=public {
GetRequest(28) R=-2032073048  .1.3.6.1.2.1.1.1.0 } }
06:35:55.153728 IP (tos 0x0, ttl 254, id 47570, offset 0, flags [none],
proto UDP (17), length 335)
    192.168.78.254.snmp > 172.22.106.11.60150:  { SNMPv2c C=public {
GetResponse(287) R=-2032073048  .1.3.6.1.2.1.1.1.0="Cisco IOS Software,
CGS2520 Software (CGS2520-IPSERVICESK9-M), Version censored, RELEASE
SOFTWARE (fc1)^M^JTechnical Support:
http://www.cisco.com/techsupport^M^JCopyright
(c) 1986-2011 by Cisco Systems, Inc.^M^JCompiled sometime by prod_rel_team"
} }
06:35:55.157010 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 78)
    172.22.106.11.60150 > 192.168.78.254.snmp:  { SNMPv2c C=public {
GetRequest(34) R=-2032073047  .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 } }
06:35:55.731062 IP (tos 0x0, ttl 254, id 47571, offset 0, flags [none],
proto UDP (17), length 79)
    192.168.78.254.snmp > 172.22.106.11.60150:  { SNMPv2c C=public {
GetResponse(35) R=-2032073047  .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1=5 } }
06:35:55.732084 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 74)
    172.22.106.11.60150 > 192.168.78.254.snmp:  { SNMPv2c C=public {
GetRequest(30) R=-2032073046  .1.3.6.1.4.1.9.2.1.2.0 } }
06:35:56.316063 IP (tos 0x0, ttl 254, id 47572, offset 0, flags [none],
proto UDP (17), length 82)
    192.168.78.254.snmp > 172.22.106.11.60150:  { SNMPv2c C=public {
GetResponse(38) R=-2032073046  .1.3.6.1.4.1.9.2.1.2.0="power-on" } }
---

Nikolai: I'm a bit hesitant to apply patches like this so I'll exhaust all
other avenues before looking at this.

Paul: Double checked Devmon's hosts.db and I can confirm the details are
correct.

Thanks


On Thu, May 26, 2016 at 9:38 PM, Root, Paul T <user-76fdb6883669@xymon.invalid>
quoted from Paul Root
wrote:
Look in your hosts.db file, and see if the switches are identified as the
proper template and that the community string is correct. The newer NX type
switches do not work with the old template.


Use snmpwalk/snmpget to pull some of the OIDs of the template.


*From:* Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *Colin Coe
*Sent:* Thursday, May 26, 2016 7:21 AM
*To:* xymon at xymon.com
*Subject:* [Xymon] Devmon tests clear but snmpwalk works


Hi all


I know this is the Xymon list not Devmon but I think I'll have most luck
on this list so apologies in advance.


I have a few of Cisco switches that I've just started monitoring at a
couple of remote sites. snmpwalk works from the Xymon server but the Devmon
tests are all clear (Devmon runs on the Xymon server).


When I look at the devmon.log I just see lots of

---

[16-05-26 at 20:19:09] No SNMP data found for ifName on swi02

[16-05-26 at 20:19:09] No SNMP data found for ifOutPktsSec on swi02

[16-05-26 at 20:19:09] No SNMP data found for ifOutCollisions on swi02

---


Any ideas on tracking this down?


We have many if these switches at other sites and they display correctly
in Xymon.


Thanks


CC
This communication is the property of CenturyLink and may contain
confidential or privileged information. Unauthorized use of this
communication is strictly prohibited and may be unlawful. If you have
received this communication in error, please immediately notify the sender
by reply e-mail and destroy all copies of the communication and any
attachments.
list Jeremy Laidman · Thu, 26 May 2016 23:49:25 +0000 ·
quoted from Colin Coe
On Fri, 27 May 2016, 08:46 Colin Coe <user-5b250cd7a540@xymon.invalid> wrote:
Hi all and many thanks for all the replies.

Jeremy: I did the tcpdump as per the thread and found that the R= value
was significantly less than the maximum 32 bit integer on some of the
switches but negative on others.  Censored, truncated output follows.  How
do we deal with the negative numbers?
Can you confirm if positive IDs are accepted by devmon? If not, this might
be a red herring.

OK, so I'm no SNMP expert, but this is what I think is happening.  When a
large 64bit integer (greater than 2^32-1) that is cast to a 32bit integer,
the highest bit is interpreted as a sign bit of 1, meaning negative. The
request ID can be a 64bit integer, but not greater than 2^32-1.  But some
switches convert into an incompatible type that causes problems with 32bit
libraries.  Some builds of SNMP libraries have an API option to avoid
sending requests with negative IDs.

In my case, I think I had a 32bit MRTG package installed on a 64bit OS, and
I had to uninstall it and reinstall the 64bit version of package. I'm
guessing that the 32bit version sent out negative IDs in queries but
couldn't cope with the 64bit values in the responses. It's also possible
that I remember things wrongly (it was a loooong time ago), so don't trust
my conjecture about the cause but of this.

I recall some people recompiled their Perl module to enable the "avoid
using negative IDs" option in the API.

More here:

https://lists.oetiker.ch/pipermail/mrtg-developers/2002-September/000103.html

This link mentions Dell switches but my problem was with Cisco switches.

J
quoted from Colin Coe

---
06:35:54.584298 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 72)
    172.22.106.11.60150 > 192.168.78.254.snmp:  { SNMPv2c C=public {
GetRequest(28) R=-2032073048  .1.3.6.1.2.1.1.1.0 } }
06:35:55.153728 IP (tos 0x0, ttl 254, id 47570, offset 0, flags [none],
proto UDP (17), length 335)
    192.168.78.254.snmp > 172.22.106.11.60150:  { SNMPv2c C=public {
GetResponse(287) R=-2032073048  .1.3.6.1.2.1.1.1.0="Cisco IOS Software,
CGS2520 Software (CGS2520-IPSERVICESK9-M), Version censored, RELEASE
SOFTWARE (fc1)^M^JTechnical Support: http://www.cisco.com/techsupport^M^JCopyright
(c) 1986-2011 by Cisco Systems, Inc.^M^JCompiled sometime by prod_rel_team"
} }
06:35:55.157010 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 78)
    172.22.106.11.60150 > 192.168.78.254.snmp:  { SNMPv2c C=public {
GetRequest(34) R=-2032073047  .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 } }
06:35:55.731062 IP (tos 0x0, ttl 254, id 47571, offset 0, flags [none],
proto UDP (17), length 79)
    192.168.78.254.snmp > 172.22.106.11.60150:  { SNMPv2c C=public {
GetResponse(35) R=-2032073047  .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1=5 } }
06:35:55.732084 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 74)
    172.22.106.11.60150 > 192.168.78.254.snmp:  { SNMPv2c C=public {
GetRequest(30) R=-2032073046  .1.3.6.1.4.1.9.2.1.2.0 } }
06:35:56.316063 IP (tos 0x0, ttl 254, id 47572, offset 0, flags [none],
proto UDP (17), length 82)
    192.168.78.254.snmp > 172.22.106.11.60150:  { SNMPv2c C=public {
GetResponse(38) R=-2032073046  .1.3.6.1.4.1.9.2.1.2.0="power-on" } }
---

Nikolai: I'm a bit hesitant to apply patches like this so I'll exhaust all
other avenues before looking at this.

Paul: Double checked Devmon's hosts.db and I can confirm the details are
correct.

Thanks


On Thu, May 26, 2016 at 9:38 PM, Root, Paul T <user-76fdb6883669@xymon.invalid>
wrote:
Look in your hosts.db file, and see if the switches are identified as the
proper template and that the community string is correct. The newer NX type
switches do not work with the old template.


Use snmpwalk/snmpget to pull some of the OIDs of the template.


*From:* Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *Colin Coe
*Sent:* Thursday, May 26, 2016 7:21 AM
*To:* xymon at xymon.com
*Subject:* [Xymon] Devmon tests clear but snmpwalk works


Hi all


I know this is the Xymon list not Devmon but I think I'll have most luck
on this list so apologies in advance.


I have a few of Cisco switches that I've just started monitoring at a
couple of remote sites. snmpwalk works from the Xymon server but the Devmon
tests are all clear (Devmon runs on the Xymon server).


When I look at the devmon.log I just see lots of

---

[16-05-26 at 20:19:09] No SNMP data found for ifName on swi02

[16-05-26 at 20:19:09] No SNMP data found for ifOutPktsSec on swi02

[16-05-26 at 20:19:09] No SNMP data found for ifOutCollisions on swi02

---


Any ideas on tracking this down?


We have many if these switches at other sites and they display correctly
in Xymon.


Thanks


CC
This communication is the property of CenturyLink and may contain
confidential or privileged information. Unauthorized use of this
communication is strictly prohibited and may be unlawful. If you have
received this communication in error, please immediately notify the sender
by reply e-mail and destroy all copies of the communication and any
attachments.
list Jeremy Laidman · Mon, 30 May 2016 02:28:57 +0000 ·
On Fri, May 27, 2016 at 9:49 AM Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
quoted from Jeremy Laidman
wrote:
Can you confirm if positive IDs are accepted by devmon? If not, this might
be a red herring.
Try applying this adjustment to devmon and see if it helps. This forces the
SNMP_Session module to avoid negative request IDs:

*** devmon.orig     2011-05-06 15:32:03.000000000 +1000
--- devmon        2016-05-30 12:26:14.000000000 +1000
***************
*** 30,35 ****
--- 30,37 ----
   # Do start-up housekeeping
    initialize($VERSION);

+   $SNMP_Session::default_avoid_negative_request_ids = 1;
• # Enter main loop
    do_log("Entering poll loop", 0);
    while (1) {
list Colin Coe · Mon, 30 May 2016 10:56:04 +0800 ·
Hi Jeremy

Yep, did this after reading through your other thread again.

Still no change in behavior.  Tis very strange as we have many of this
model switch responding correctly and the cisco router (and another type of
Cisco switch) installed on those sites are working as expected.

Still looking...

Thanks

On Mon, May 30, 2016 at 10:28 AM, Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
quoted from Jeremy Laidman
wrote:
On Fri, May 27, 2016 at 9:49 AM Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
wrote:
Can you confirm if positive IDs are accepted by devmon? If not, this
might be a red herring.
Try applying this adjustment to devmon and see if it helps. This forces
the SNMP_Session module to avoid negative request IDs:

*** devmon.orig     2011-05-06 15:32:03.000000000 +1000
--- devmon        2016-05-30 12:26:14.000000000 +1000
***************
*** 30,35 ****
--- 30,37 ----
   # Do start-up housekeeping
    initialize($VERSION);

+   $SNMP_Session::default_avoid_negative_request_ids = 1;
• # Enter main loop
    do_log("Entering poll loop", 0);
    while (1) {

list Phil Crooker · Mon, 30 May 2016 03:22:01 +0000 ·
How do the IOS versions compare between the switches that work and those that don't?
quoted from Colin Coe

?

From: Xymon <xymon-bounces at xymon.com> on behalf of Colin Coe <user-5b250cd7a540@xymon.invalid>
Sent: Monday, 30 May 2016 12:26 PM
To: Jeremy Laidman
Cc: xymon at xymon.com
Subject: Re: [Xymon] Devmon tests clear but snmpwalk works

Hi Jeremy

Yep, did this after reading through your other thread again.

Still no change in behavior.  Tis very strange as we have many of this model switch responding correctly and the cisco router (and another type of Cisco switch) installed on those sites are working as expected.

Still looking...

Thanks

On Mon, May 30, 2016 at 10:28 AM, Jeremy Laidman <user-71895fb2e44c@xymon.invalid<mailto:user-71895fb2e44c@xymon.invalid>> wrote:
On Fri, May 27, 2016 at 9:49 AM Jeremy Laidman <user-71895fb2e44c@xymon.invalid<mailto:user-71895fb2e44c@xymon.invalid>> wrote:
Can you confirm if positive IDs are accepted by devmon? If not, this might be a red herring.

Try applying this adjustment to devmon and see if it helps. This forces the SNMP_Session module to avoid negative request IDs:

*** devmon.orig     2011-05-06 15:32:03.000000000 +1000
--- devmon        2016-05-30 12:26:14.000000000 +1000
***************
*** 30,35 ****
--- 30,37 ----
   # Do start-up housekeeping
    initialize($VERSION);

+   $SNMP_Session::default_avoid_negative_request_ids = 1;
• # Enter main loop
    do_log("Entering poll loop", 0);
    while (1) {
list Jeremy Laidman · Mon, 30 May 2016 05:21:20 +0000 ·
quoted from Colin Coe
On Mon, May 30, 2016 at 12:56 PM Colin Coe <user-5b250cd7a540@xymon.invalid> wrote:
Hi Jeremy

Yep, did this after reading through your other thread again.

Still no change in behavior.  Tis very strange as we have many of this
model switch responding correctly and the cisco router (and another type of
Cisco switch) installed on those sites are working as expected.
Ah, was worth a shot.

Another possibility I seem to recall people having problems with is the use
of bulk responses.  In SNMPv2c, you can send a query for a tree walk and
get one single response (depending on size) with lots of OIDs in it, and
some devices might have problems with this.  If you switch the devmon
config to v1, you disable bulk responses, and it might be a bit more
reliable for some devices, although it'll generate a bit more outbound
traffic in some cases.

J