client data compression 🔗 link

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I was looking at the amount of data travelling to my hobbit server, and
figured some compression would be useful...

Given we are effectively transferring pure text data from place to
place, the CPU overhead of compressing/de-compressing the data should be
minimal, and the bandwidth advantage (for non-lan) are significant...
Taking the most recent file for each machine in my hostdata directory,
and adding it up I get 1137kB or 116720MB / year or 9.5GB/month ...

Copying each of the most recent files to a temp folder, I get:
Raw data per 5 mins: 1137	/tmp/hobbitdata
gzip data per 5 mins: 389	/tmp/hobbitdata
bz2 data per 5 mins: 385	/tmp/hobbitdata

So, from the above, I could change from using 9.5GB/mth to 9.5 *
385/1137 = 3.2 GB.... that would be a saving of around $19 per month for
me.... Anyone else feel like calculating how much this is worth to them?

Try this:
mkdir /tmp/hobbitdata
cd /var/lib/hobbit/hostdata
for host in *
do
 file=`ls -lrt $host|tail -1|awk '{print $9}'`
 cp $host/$file /tmp/hobbitdata
done
echo -en "Raw data per 5 mins: "
du -sk /tmp/hobbitdata
gzip /tmp/hobbitdata/*
echo -en "gzip data per 5 mins: "
du -sk /tmp/hobbitdata
gunzip /tmp/hobbitdata/*
bzip2 /tmp/hobbitdata/*
echo -en "bz2 data per 5 mins: "
du -sk /tmp/hobbitdata
rm -rf /tmp/hobbitdata

Might take a few minutes depending on number of hosts/etc...

So, is compression likely to be implemented into the hobbit
client/server any time soon?

Regards,
Adam

- --
Adam Goryachev
Website Managers
www.websitemanagers.com.au
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklHRMcACgkQGyoxogrTyiUf6ACfcDwQbwVuISQ70A814Z+hujvm
ezoAn1RrxkP0SMjr9ibV2iO56QI/iqh+
=fLdN
-----END PGP SIGNATURE-----

In <user-f74398147e60@xymon.invalid> Adam Goryachev <user-92fd6827f6ae@xymon.invalid> writes:

It's been in the main development trunk since March:

r5558 | henrik | 2008-03-02 13:44:57 +0100 (søn, 02 mar 2008) | 2 lines

Support on-the-fly compress/decompress of data


Regards,
Henrik

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Henrik Størner wrote:

r5558 | henrik | 2008-03-02 13:44:57 +0100 (søn, 02 mar 2008) | 2 lines

Support on-the-fly compress/decompress of data

Oops... perhaps this is a reason for me to upgrade to the dev version,
though I'd of course prefer to wait for a proper release.... oh, the
dilemma!

Thanks for your hard work, and a great system BTW! It's been a while
since we had a flurry of thank you emails :)

Regards,
Adam

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklH1A8ACgkQGyoxogrTyiXdtQCgpmwd/3FX9VFs85wEq9UsCpde
a7MAn2iy9kZmRW7aO4FWsVKIQQaqAC5D
=wlzR
-----END PGP SIGNATURE-----

I may have missed something here...but is this compression done already or
is there a need to configure it?

Also, when was 4.2.0 release?  I'm using the 4.2.0 release at the office
currently and I'd like to know if the compression has been active or not
(since Henrik has included it since March, presumably 2008).

Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX

Those who don't understand UNIX are condemned to reinvent it, poorly.
--- Henry Spencer


On Tue, Dec 16, 2008 at 11:15 AM, Adam Goryachev <

user-92fd6827f6ae@xymon.invalid> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Josh Luthman wrote:

Not sure...

August 2006 says sourceforge :)
http://sourceforge.net/project/showfiles.php?group_id=128058

I would say from Henrik's response that it is only in the cvs version
right now, ie, what will become 4.3.0 when it is released. So you
won't have it now, and even if you upgrade to 4.2.2 you won't get it.
You would need to install the development version from cvs to get this
feature, or wait for 4.3.0

Regards,
Adam

On Tue, Dec 16, 2008 at 11:15 AM, Adam Goryachev
<user-92fd6827f6ae@xymon.invalid
<mailto:user-92fd6827f6ae@xymon.invalid>> wrote:

Henrik Størner wrote:

In <user-f74398147e60@xymon.invalid

<mailto:user-f74398147e60@xymon.invalid>> Adam Goryachev
<user-92fd6827f6ae@xymon.invalid

- --

Adam Goryachev
Website Managers
Ph: +XX X XXXX XXXX                            user-eaec2ffb4cbc@xymon.invalid
Fax: +XX X XXXX XXXX                            www.websitemanagers.com.au
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJSCjFGyoxogrTyiURApT2AJ9HxvS6V+ccbuW7K3MoTAJolkKCfwCgt5xg
v51UdvDpeOLZsGSk7wnMjVc=
=crQN
-----END PGP SIGNATURE-----

In <user-b2fa9d1146f3@xymon.invalid> Adam Goryachev <user-92fd6827f6ae@xymon.invalid> writes:

It's done, but only in the development version - neither 4.2.0
nor 4.2.2 has it. It was rather non-trivial to implement and in
a key-function in Hobbit; so adding it to the 4.2.x series was
out of the question.

It requires a new hobbitd daemon on the server side, and a new
"bb" utility at the client. Compression is not turned on by
default - you'll have to invoke "bb" as "bbz", or pass it the
"--compress" option.

Aug 10, 2006 to be exact.


Regards,
Henrik

Oh, didn't think about it requiring a new binary.   Not interested it
that for my purposes if that is the case.

Thanks a bunch for clearing that up!

On 12/16/08, Henrik Størner <user-ce4a2c883f75@xymon.invalid> wrote:

-- 
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX

Those who don't understand UNIX are condemned to reinvent it, poorly.
--- Henry Spencer

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Henrik Størner wrote:

Sorry to continue this conversation, but I'd like to clarify some
implementation details.

If I have two hobbit servers, and one bbproxy server which transmits the
data to both hobbit servers, and of course multiple bb clients behind
the bbproxy.

Do I need to update *both* hobbit servers, or could I only upgrade one
hobbit server, and the bbproxy, and bbproxy would be capable of
compressing received updates, and forwarding them to the compression
capable server, and forwarding the uncompressed updates to the older server?

Also, how do old hobbit clients, or bb windows clients, or hobbit
windows clients behave when behind the bbproxy? Would I need to upgrade
each client for it's updates to be compressed, or will bbproxy handle
that for me?

Thanks,

Adam

- --
Adam Goryachev
Website Managers
www.websitemanagers.com.au
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklIekYACgkQGyoxogrTyiUI6gCfV3QJ0IhoA/ejrA6eBxkFGwy0
LfsAn2xIol3ngZh7IzrH7GtpDGblv9Nb
=XXSK
-----END PGP SIGNATURE-----

In <user-8ce088227264@xymon.invalid> Adam Goryachev <user-92fd6827f6ae@xymon.invalid> writes:

You've caught one of the "missing items" in the development version -
bbproxy hasn't been made compression-aware yet. In the end, there will
be a way of specifying whether bbproxy target servers can handle
compression or not - so yes, you could have updates sent in compressed
form to one hobbitd, and uncompressed to another.

At least, that's the plan. I haven't looked too much at the 
implementation details yet, so it might turn out differently.

If a target server supports compression, then I think bbproxy will 
perform the compression on all of the "large" messages, i.e. status-
and client-messages. Other messages are too small to be worth the 
trouble of compressing them. So that will be transparent to the 
client.


Regards,
Henrik