Xymon Mailing List Archive search

fedora upgrade from 13 to 14 crashed hobbitd_rrd

12 messages in this thread

list Jason Chambers · Thu, 11 Nov 2010 19:45:18 +0000 · 
- Program crashed

Fatal signal caught!


Anyone know how I can resolve this issue? The tail end of rrd-status.log looks like this:

2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD was created on another architecture

The funny thing is, the architecture never changed. I just upgraded Fedora Core from version 13 to 14.

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid
list Jason Chambers · Thu, 11 Nov 2010 20:23:08 +0000 · 
Just by curiosity, I tried creating a new rrd directory and things didn't improve at all:

rrd_dontneed: Argument 'rrd_file' is NULL.
2010-11-11 12:40:44 Peer at 0.0.0.0:0 failed: Broken pipe
2010-11-11 12:40:45 Peer not up, flushing message queue
2010-11-11 12:40:46 Peer not up, flushing message queue
2010-11-11 12:40:47 Peer not up, flushing message queue
2010-11-11 12:40:47 Peer not up, flushing message queue
2010-11-11 12:40:49 Peer not up, flushing message queue
▸ signature

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid
▸ quoted from Jason Chambers
From: Jason Chambers [mailto:user-3fa671c0a30d@xymon.invalid]
Sent: November-11-10 2:45 PM
To: 'xymon at xymon.com'
Subject: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd

- Program crashed

Fatal signal caught!


Anyone know how I can resolve this issue? The tail end of rrd-status.log looks like this:

2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD was created on another architecture

The funny thing is, the architecture never changed. I just upgraded Fedora Core from version 13 to 14.

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX


Visit our user-be8ce7065ec5@xymon.invalid<http://www.geosoft.com>;
list Bruce White · Thu, 11 Nov 2010 14:38:58 -0600 · 
I get the same issue on RedHat AS 5.2 (running 4.3.0-0.beta2) once in a
while.  It happened two days ago.  I find, if I don't restart all of the
Xymon processes, it happens more frequently and some of my graphs get
very screwed up.

 
     ......Bruce

 
 
 Bruce White
 Senior Enterprise Systems Engineer | Phone: XXX-XXX-XXXX | Fax: XXX-XXX-XXXX | user-58f975e8bf9d@xymon.invalid | http://www.fellowes.com/
 
 
 
Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. Fellowes, Inc.
▸ quoted from Jason Chambers
 

From: Jason Chambers [mailto:user-3fa671c0a30d@xymon.invalid] 
Sent: Thursday, November 11, 2010 1:45 PM
To: 'xymon at xymon.com'
Subject: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd

 
- Program crashed

 
Fatal signal caught!

 
Anyone know how I can resolve this issue? The tail end of rrd-status.log
looks like this:

 
2010-11-11 14:43:34 RRD error updating
/home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD
was created on another architecture

 
The funny thing is, the architecture never changed. I just upgraded
Fedora Core from version 13 to 14.

 
Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid
list Henrik Størner · Thu, 11 Nov 2010 20:45:24 +0000 (UTC) · 
In <user-dde3c96e7ed6@xymon.invalid> Jason Chambers <user-3fa671c0a30d@xymon.invalid> writes:

- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log lo=
oks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/di=
sk,H.rrd from 192.168.0.0: This RRD was created on another architecture
▸ quoted from Bruce White
The funny thing is, the architecture never changed. I just upgraded Fedora =
Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD library?

To begin with, I would definitely re-compile Xymon with the versions
of librrd that are now in place on your new system. Just to rule out
any problems with the runtime library having changed from the compile-
time setup.


Regards,
Henrik
list Jason Chambers · Thu, 11 Nov 2010 21:18:52 +0000 · 
Hi Bruce,

I have actually tried restarting the service all the time. The RRD daemon just  crashes immediately. Quite confusing.
▸ signature

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid
▸ quoted from Bruce White
From: White, Bruce [mailto:user-58f975e8bf9d@xymon.invalid]
Sent: November-11-10 3:39 PM
To: xymon at xymon.com
Subject: RE: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd

I get the same issue on RedHat AS 5.2 (running 4.3.0-0.beta2) once in a while.  It happened two days ago.  I find, if I don’t restart all of the Xymon processes, it happens more frequently and some of my graphs get very screwed up.

     ……Bruce


 Bruce White


 Senior Enterprise Systems Engineer | Phone: XXX-XXX-XXXX | Fax: XXX-XXX-XXXX | user-58f975e8bf9d@xymon.invalid<mailto:user-58f975e8bf9d@xymon.invalid> | www.fellowes.com<http://www.fellowes.com/>;


[cid:image001.jpg at 01CB81BC.20EC1A50]
▸ quoted from Bruce White


Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. Fellowes, Inc.


From: Jason Chambers [mailto:user-3fa671c0a30d@xymon.invalid]
Sent: Thursday, November 11, 2010 1:45 PM
To: 'xymon at xymon.com'
Subject: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd

- Program crashed

Fatal signal caught!


Anyone know how I can resolve this issue? The tail end of rrd-status.log looks like this:

2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD was created on another architecture

The funny thing is, the architecture never changed. I just upgraded Fedora Core from version 13 to 14.

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid<http://www.geosoft.com>;
list Jason Chambers · Thu, 11 Nov 2010 21:22:08 +0000 · 
Hi Henrik,

This is actually an in place upgrade using their PreUpgrade tool described here: http://fedoraproject.org/wiki/PreUpgrade . The bit version did not change. I also used the GUI interface (of yum I would assume) to uninstall and reinstall the RRD tool without any luck.

I have also done a make clean and started a new configuration of the xymon compilation without any luck. This is using the 4.3 beta2 code.

Thankfully I am getting alerts and the xymon tool itself is working. Just no RRD right now.
▸ signature

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid

-----Original Message-----
▸ quoted from Henrik Størner
From: Henrik Størner [mailto:user-ce4a2c883f75@xymon.invalid] 
Sent: November-11-10 3:45 PM
To: xymon at xymon.com
Subject: Re: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd

In <user-dde3c96e7ed6@xymon.invalid> Jason Chambers <user-3fa671c0a30d@xymon.invalid> writes:

- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of 
rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating 
/home/hobbit/data/rrd/xxx.yyy.com/di=
sk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded 
Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD library?

To begin with, I would definitely re-compile Xymon with the versions of librrd that are now in place on your new system. Just to rule out any problems with the runtime library having changed from the compile- time setup.


Regards,
Henrik
list Henrik Størner · Thu, 11 Nov 2010 22:06:31 +0000 (UTC) ·
▸ quoted from Henrik Størner
On Thu, 11 Nov 2010 20:45:24 +0000, Henrik "Størner" wrote:

In <user-dde3c96e7ed6@xymon.invalid> Jason
Chambers <user-3fa671c0a30d@xymon.invalid> writes:

- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log
lo= oks like this:
2010-11-11 14:43:34 RRD error updating
/home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: This
RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded
Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD
library?
I think the problem has been found, but it might be difficult to fix.
Look at the Bugzilla discussion in the Fedora mailing list:
https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29

Apparently the glibc folks implemented a new version of the glibc
routine that does memory-copies fin Fedora 14. And unlike the old
version, this one corrupts data when copying between memory areas
that overlap.

Stricly speaking, doing that is not valid. But it is apparently
quite common - the Bugzilla entry lists quite a few apps that
break because of this change - and it has worked until the change
in glibc. So the glibc maintainers is taking the "not a bug - won't fix" 
stance.

In which case the problem has to be fixed in all of the applications
and libraries that do the invalid memcpy, i.e. the rrdtool library
in this case.


Regards,
Henrik
list Xymon User in Richmond · Thu, 11 Nov 2010 18:48:01 -0500 ·
▸ quoted from Henrik Størner
On Thu, November 11, 2010 17:06, Henrik Størner wrote:
On Thu, 11 Nov 2010 20:45:24 +0000, Henrik "Størner" wrote:

In <user-dde3c96e7ed6@xymon.invalid>
Jason Chambers <user-3fa671c0a30d@xymon.invalid> writes:

- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of
rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating
/home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: This
 RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded
Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD
library?
I think the problem has been found, but it might be difficult to fix.
Look at the Bugzilla discussion in the Fedora mailing list:
https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29

Apparently the glibc folks implemented a new version of the glibc routine
that does memory-copies fin Fedora 14. And unlike the old version, this
one corrupts data when copying between memory areas that overlap.

Interesting bug thread.  So the new glibc broke Flash (and rrd), and who
knows what else.  And even Linus Torvalds thinks it's wrong and
unnecessary.  And the glibc team seem to be adamant about not changing it.

BSD, anyone?
list Jason Chambers · Fri, 12 Nov 2010 13:53:16 +0000 · 
*face palm*


OK, Let's see if I can figure out how to downgrade glibc. 
Note: not a linux user :)


Thanks for this information guys.
▸ signature

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid

-----Original Message-----
▸ quoted from Xymon User in Richmond
From: Xymon User in Richmond [mailto:user-24d6f8323faa@xymon.invalid] Sent: November-11-10 6:48 PM
To: xymon at xymon.com
Subject: Re: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd

On Thu, November 11, 2010 17:06, Henrik Størner wrote:
On Thu, 11 Nov 2010 20:45:24 +0000, Henrik "Størner" wrote:

In <user-dde3c96e7ed6@xymon.invalid>
Jason Chambers <user-3fa671c0a30d@xymon.invalid> writes:

- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: This  RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD library?
I think the problem has been found, but it might be difficult to fix.
Look at the Bugzilla discussion in the Fedora mailing list:
https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29

Apparently the glibc folks implemented a new version of the glibc routine that does memory-copies fin Fedora 14. And unlike the old version, this one corrupts data when copying between memory areas that overlap.

Interesting bug thread.  So the new glibc broke Flash (and rrd), and who knows what else.  And even Linus Torvalds thinks it's wrong and unnecessary.  And the glibc team seem to be adamant about not changing it.

BSD, anyone?
list Sebastian Auriol · Fri, 12 Nov 2010 14:07:04 -0000 · 
Hi Jason,

I'm certainly no expert on this, but in the Bugzilla discussion in the
Fedora mailing list:
https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29
That Henrik quoted below, there is a work-around posted originally by Linus
Torvalds (and reposted a bit more friendly here):
https://bugzilla.redhat.com/show_bug.cgi?id=638477#c55
This may also work for rrd. Try replacing /usr/bin/firefox in that post with
the command to execute hobbitd_rrd, or perhaps more precisely, try
prepending:
LD_PRELOAD=$HOME/Downloads/linusmemcpy.so to that command. Off the top of my
head, I'm not sure where you change that command though in the hobbit config
or code. I presume that it can be done though?

Kind regards,

SebA
▸ quoted from Jason Chambers
-----Original Message-----
From: Jason Chambers [mailto:user-3fa671c0a30d@xymon.invalid] Sent: 12 November 2010 13:53
To: 'xymon at xymon.com'
Subject: RE: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd

*face palm*


OK, Let's see if I can figure out how to downgrade glibc. 
Note: not a linux user :)


Thanks for this information guys.

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid

-----Original Message-----
From: Xymon User in Richmond [mailto:user-24d6f8323faa@xymon.invalid] Sent: November-11-10 6:48 PM
To: xymon at xymon.com
Subject: Re: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd

On Thu, November 11, 2010 17:06, Henrik Størner wrote:
On Thu, 11 Nov 2010 20:45:24 +0000, Henrik "Størner" wrote:

In <user-dde3c96e7ed6@xymon.invalid>
Jason Chambers <user-3fa671c0a30d@xymon.invalid> writes:

- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of >>> rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating >>> /home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: >>> This  RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded >>> Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD >> library?
I think the problem has been found, but it might be difficult to fix.
Look at the Bugzilla discussion in the Fedora mailing list:
https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29

Apparently the glibc folks implemented a new version of the glibc > routine that does memory-copies fin Fedora 14. And unlike the old > version, this one corrupts data when copying between memory areas that overlap.

Interesting bug thread.  So the new glibc broke Flash (and rrd), and who knows what else.  And even Linus Torvalds thinks it's wrong and unnecessary.  And the glibc team seem to be adamant about not changing it.

BSD, anyone?



Date: 11/07/10 19:34:00

19:34:00
list Xymon User in Richmond · Fri, 12 Nov 2010 10:08:21 -0500 ·
▸ quoted from Jason Chambers
On Fri, November 12, 2010 08:53, Jason Chambers wrote:
*face palm*


OK, Let's see if I can figure out how to downgrade glibc.

Note: not a linux user :)


Off the top of my head:


Copy /etc/yum.repos.d/fedora-updates.repo to
/etc/yum.repos.d/fedora-updates12.repo

Edit the new file, replacing all "$releasever" with "13" and set
"enabled=1" to "enabled=0".  Change every "updates" inside square brackets
to "updates12".  Then use
yum --enablerepo=updates12 downgrade install glibc-2.12.1-4
glibc-common-2.12.1-4 glibc-devel-2.12.1-4 glibc-headers-2.12.1-4
to do the actual downgrade.

I don't have the means to test this for you right now, so ymmv.

Next time:  clone your Xymon server on a PC with similar processor
architecture and bring it up on a different IP.  Point a couple of clients
at it (they can feed multiple Xymon servers each).  Upgrade it and test
things out.  And don't use Fedora on production machines unless you're
well prepared to troubleshoot things that are broken by cutting-edge
stuff.  I use Fedora at home, but Centos/RHEL/OEL at work.
list Jason Chambers · Wed, 17 Nov 2010 13:41:56 +0000 · 
Good news.

RRD Group released a fix for this problem. I'm able to see graphs again, but obviously lost a few weeks of data there when it didn't work. No major problem for me here. Just thought I send an update just in case someone else is using Fedora.
▸ signature

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid
▸ quoted from Jason Chambers
From: Jason Chambers [mailto:user-3fa671c0a30d@xymon.invalid]
Sent: November-11-10 2:45 PM
To: 'xymon at xymon.com'
Subject: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd

- Program crashed

Fatal signal caught!


Anyone know how I can resolve this issue? The tail end of rrd-status.log looks like this:

2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD was created on another architecture

The funny thing is, the architecture never changed. I just upgraded Fedora Core from version 13 to 14.

Jason Chambers
IT Help Desk Associate

GEOSOFT INC.
freedom to explore
T +X XXX.XXX.XXXX #344
F +X XXX.XXX.XXXX

Visit our user-be8ce7065ec5@xymon.invalid<http://www.geosoft.com>;