BigBrother migration: trending on the mem test 🔗 link

Hi All,

I am almost complete with my Proof-of-concept with out BigBrother to Hobbit migration. Yes, has been taking along time. There were a lot of BB customizations that needed porting.

Anyways. All my *NIX devices report into BB with their memory report as mem, not memory. So I have BigBrother & hobbit running in parallel, with a bbproxy sending bbclient data to both. One of my last issues is that I am missing memory trends for all my *nix servers. So is there anyway to tell the hobbit server to do memory trending on memory & mem reports ? I have tried a few things, but with no success.

If there is no way to configure hobbit to perform trend collection on memory & mem reports, I can just do a --extra-test option and script something. I have a few of those already. I was just curious if there was any inbuilt method.

Thanks,
 ~Steve

On Fri, Jul 13, 2007 at 08:34:32AM -0400, user-ce96540ed38f@xymon.invalid wrote:

I think putting a "mem=memory" in the TEST2RRD setting and restarting
the hobbitd_rrd processes (just kill them, they'll automatically
restart) should do it.


Regards,
Henrik

On Friday 13 July 2007 11:59, Henrik Stoerner wrote:

That did not work. I will be checking to see if there is anything in the logs.

 ~Steve

Correction, that did not work for my Linux devices reporting in with a mem report, but it does work with my Solaris devices. I am assuming the output from the bb-memory version I am using, does not mesh with what Hobbit is set to trend on. So I will probably just need to send mem report to a modified memory-larrd script via the extra-scripts option. This will only be a short-time fix, until I can push hobbit clients out to replace the bb clients. The first step is to get the server upgraded, then the clients. Below I have have include the Linux/Sun memory output for reference:

Linux:
green Sun Jul 15 09:56:28 EDT 2007 - Memory OK
&green real 7926M / 7948M 99%
&green swap 0M / 2047M 0%

Solaris:
green Sun Jul 15 09:56:59 EDT 2007 - Memory OK
   Memory       Used    Total   Percentage
&green Physical 50378M  65536M  76%
&clear Actual   -1M     65536M  0%
&green Swap             42658M  85191M  50%

Thanks for the guidance and help on this,
 ~Steve

I'm encountering an odd problem doing a rebuild of the Hobbit server. I'm using the same Solaris 8 system that I originally built my server on, however I no longer seem to be getting the pcre and gcc libs linked to my binaries, and they're entirely different sizes as well. Example:

Original hobbitsvc.cgi binary:
Size: 1374336
$ldd hobbitsvc.cgi
         libpcre.so.0 =>  /usr/local/lib/libpcre.so.0
         libresolv.so.2 =>        /usr/lib/libresolv.so.2
         libsocket.so.1 =>        /usr/lib/libsocket.so.1
         libnsl.so.1 =>   /usr/lib/libnsl.so.1
         libc.so.1 =>     /usr/lib/libc.so.1
         libgcc_s.so.1 =>         /usr/local/lib/libgcc_s.so.1
         libdl.so.1 =>    /usr/lib/libdl.so.1
         libmp.so.2 =>    /usr/lib/libmp.so.2
         /usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1

New hobbitsvc.cgi binary:
Size: 532124
$ldd hobbitsvc.cgi
         libresolv.so.2 =>        /usr/lib/libresolv.so.2
         libsocket.so.1 =>        /usr/lib/libsocket.so.1
         libnsl.so.1 =>   /usr/lib/libnsl.so.1
         libc.so.1 =>     /usr/lib/libc.so.1
         libdl.so.1 =>    /usr/lib/libdl.so.1
         libmp.so.2 =>    /usr/lib/libmp.so.2
         /usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1

I've tried numerous fixes. Set all the paths in LD_LIBRARY_PATH, added -R/usr/loca/lib to the Makefile for PCRE. Using the Sunfreeware SMCgcc 3.4.6 package with SMCmake installed also. I don't get any errors when running make or make install. I know it's actually using gcc:

gcc -g -O2 -Wall -Wno-unused -D_REENTRANT -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DSunOS -I. -I/build/hobbit-4.2.0/include -I/usr/local/include -o hobbitsvc.cgi hobbitsvc.o hobbitsvc-info.o hobbitsvc-trends.o ../lib/libbbgen.a -L/usr/local/lib -R/usr/local/lib -lpcre -lresolv -lsocket -lnsl

Am I missing something stupidly obvious here?

Have you tried updating your library path using crle?

=G=

-----Original Message-----
From: Matthew Epp [mailto:user-c07bdcff406c@xymon.invalid] Sent: Monday, July 16, 2007 2:22 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Missing libpcre and libgcc in compiled binaries

I'm encountering an odd problem doing a rebuild of the Hobbit server. I'm using the same Solaris 8 system that I originally built my server on, however I no longer seem to be getting the pcre and gcc libs linked to my binaries, and they're entirely different sizes as well. Example:

Original hobbitsvc.cgi binary:
Size: 1374336
$ldd hobbitsvc.cgi
         libpcre.so.0 =>  /usr/local/lib/libpcre.so.0
         libresolv.so.2 =>        /usr/lib/libresolv.so.2
         libsocket.so.1 =>        /usr/lib/libsocket.so.1
         libnsl.so.1 =>   /usr/lib/libnsl.so.1
         libc.so.1 =>     /usr/lib/libc.so.1
         libgcc_s.so.1 =>         /usr/local/lib/libgcc_s.so.1
         libdl.so.1 =>    /usr/lib/libdl.so.1
         libmp.so.2 =>    /usr/lib/libmp.so.2
         /usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1

New hobbitsvc.cgi binary:
Size: 532124
$ldd hobbitsvc.cgi
         libresolv.so.2 =>        /usr/lib/libresolv.so.2
         libsocket.so.1 =>        /usr/lib/libsocket.so.1
         libnsl.so.1 =>   /usr/lib/libnsl.so.1
         libc.so.1 =>     /usr/lib/libc.so.1
         libdl.so.1 =>    /usr/lib/libdl.so.1
         libmp.so.2 =>    /usr/lib/libmp.so.2
         /usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1

I've tried numerous fixes. Set all the paths in LD_LIBRARY_PATH, added -R/usr/loca/lib to the Makefile for PCRE. Using the Sunfreeware SMCgcc 3.4.6 package with SMCmake installed also. I don't get any errors when running make or make install. I know it's actually using gcc:

gcc -g -O2 -Wall -Wno-unused -D_REENTRANT -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DSunOS -I. -I/build/hobbit-4.2.0/include -I/usr/local/include -o hobbitsvc.cgi hobbitsvc.o hobbitsvc-info.o hobbitsvc-trends.o ../lib/libbbgen.a -L/usr/local/lib -R/usr/local/lib -lpcre -lresolv -lsocket -lnsl

Am I missing something stupidly obvious here?

Matthew Epp said the following, On 07/16/2007 01:21 PM:

I'm encountering an odd problem doing a rebuild of the Hobbit server. I'm using the same Solaris 8 system that I originally built my server on, however I no longer seem to be getting the pcre and gcc libs linked to my binaries, and they're entirely different sizes as well. Example:

When you ran configure did it have:
Checking for PCRE ...
Found PCRE include files in /usr/local/include
Found PCRE libraries in /usr/local/lib

OR...
in /usr/local/lib/ is there a libpcre.so or just a libpcre.a ?
If its just a libpcre.a file its a static lib and it won't show up as being part of a dynamic dependancy as the code will be linked it directly.

I've thought about that, but the problem is I'd have to do that also on every system that runs the server, as we build our binaries on machines separate from everything else. Also, I've never had to modify with crle before and the existing servers are working just fine. I'm trying to figure out what has changed since I last completed a successful compile.

Galen Johnson wrote:

Just for kicks I modified with crle on my build server, ran a make 
clean, make and make install. I still get the same binaries with missing 
lib links.

Galen Johnson wrote:

sladewig wrote:

Yes.

Both.
$ls -l /usr/local/lib|grep pcre
-rw-r--r--    1 root     other      460740 Jul 12  2006 libpcre.a
-rwxr-xr-x    1 root     other         792 Jul 12  2006 libpcre.la
-rwxr-xr-x    1 root     other      433948 Jul 12  2006 libpcre.so.0.0.1
lrwxrwxrwx    1 root     other          31 Jul 16 17:30 libpcre.so.1 -> /usr/local/lib/libpcre.so.0.0.1
-rw-r--r--    1 root     other      495576 Jul 12  2006 libpcrecpp.a
-rwxr-xr-x    1 root     other        1213 Jul 12  2006 libpcrecpp.la
-rwxr-xr-x    1 root     other      495492 Jul 12  2006 libpcrecpp.so.0.0.0
-rw-r--r--    1 root     other       15688 Jul 12  2006 libpcreposix.a
-rwxr-xr-x    1 root     other         853 Jul 12  2006 libpcreposix.la
-rwxr-xr-x    1 root     other       20988 Jul 12  2006 libpcreposix.so.0.0.0

So, you think it's seeing the libpcre.a and just building it statically? In that case, wouldn't the binary be slightly larger like the original build I had?

I found the issue. Very odd. For some reason, when I last built PCRE, it did not create symlinks for some of the files. These were missing:
/usr/local/lib/libpcre.so -> libpcre.so.0.0.1
lrwxrwxrwx    1 root     other          16 Jul 17 16:55 /usr/local/lib/libpcre.so.0 -> libpcre.so.0.0.1
-rwxr-xr-x    1 root     other      433948 Jul 12  2006 /usr/local/lib/libpcreposix.so -> libpcreposix.so.0.0.0
lrwxrwxrwx    1 root     other          21 Jul 17 16:56 /usr/local/lib/libpcreposix.so.0 -> libpcreposix.so.0.0.0

Once those were in place, the linker added in libpcre as well as libgcc as expected. Everything's working again. :) By the way, I was doing this so I could add a WIKI button to alert pages, which when clicked, does an auto-search in our internal wiki using that hostname and service as the keywords. Works great! One more thing I need to document and upload.

Matthew Epp said the following, On 07/17/2007 02:05 PM:

So it must have been using the static libs when it linked.
If you still have the old binary around you can run nm on it and look for the references to the functions you are interested in. In this case pcre.

nm hobbitd |grep -i pcre
[289]   |    248184|       0|FUNC |GLOB |0    |UNDEF  |pcre_compile
[228]   |    248196|       0|FUNC |GLOB |0    |UNDEF  |pcre_exec
[480]   |    255612|       4|OBJT |GLOB |0    |22     |pcre_free


In this case the hobbitd binary has UNDEF for the routines with pcre. The system linker/loader would have to resolve these by using external libs such as the .so versions of the lib or the binary is a no-go.
Any statically linked routines wouldn't have UNDEF.

--
steve