Alternate to msgcache/hobbitfetch? 🔗 link

I've got some servers out in the DMZ that can't talk directly to the
corporate network.  I can get out to them via socks/ssh.  I do not want
to run another listening service (msgcache) out there in the DMZ.  Is
there a way to configure the client to collect messages to a file
(without msgcache listening), and then go retrieve/process those file(s)
via ssh from the Hobbit server and process normally?
 
What I'm doing now is not installing the client in the DMZ, and instead
using a custom Hobbit server script that socks/ssh'es out to the DMZ and
runs it's own data collection commands.  The data collected is similar
to what the client would normally do, but it's all done via custom
script without any help from the Hobbit client.
 
I guess what I'm looking for is a hobbitfetch variant that uses ssh/scp
rather than connecting to a msgcache listener.  And along with that, a
way to configure the client to collect data normally, but skip starting
a listening service.

On Sat, June 2, 2007 22:50, Haertig, David F (Dave) wrote:

Why not just use msgcache bound to a localhost address and ssh port
tunneling, front-ended by autossh for persistence?

AUTOSSH_GATETIME=0
autossh -M 0 -f -f -N -L 127.0.0.2:1984:127.0.0.2:1984 \
-i <path-to-rsa-private-key>  user-df99d5171f96@xymon.invalid

Install autossh.  Do your key exchange, test ssh PKI authentication.  Set
msgcache to --listen=127.0.0.2 and --server=127.0.0.1, use
"dmzserver.example.com 127.0.0.2 # testip pulldata" in bb-hosts, and you
should be good to go, with a msgcache that cannot be fetched except via
your tunnel.  Note that the conn check for this bb-host entry will be
meaningless, and you'll need to add -L port mappings for any other
server-side tests you want.

Writing this from memory, so there might be a syntax error or misspelled
option.  But it's pretty close.  You do need a very up-to-date ssh to
specify the near-side IP on the -L port forward (which helps you to keep
up with tunnels for different hosts--otherwise you have to use different
ports; I actually put in /etc/hosts entries for the 127.0.0.x entries so I
can call them by name).  I know that the near-side ip spec is supported in
the current repo versions for FC5 and up, RHEL5, and Centos5, but not in
RHEL4 and down.  Hope this helps.

Thanks.  I'm familiar with ssh tunnels, pubkey authentication, ssh
agents, etc.  I use that quite a bit in other applications.

The issue I'm faced with is starting up a new listener out there in the
DMZ.  Any listener(even one only listening on the local adapter).  That
brings other people into this, with the associated reviews, paperwork,
and so on.  Reviews that I 100% agree with - putting on my security hat.
I have a working and approved solution now using only socks/ssh and my
custom scripts (from back in the Big Brother days when I first wrote
this).  I'm just investigating if it's possible to pull the formal
Hobbit client into my existing solution methodology.  Adding a listener
is not a total deal breaker, I could probably gain the required
approvals, but the overhead in getting that paperwork and approval done
has me researching other possibilities.

Thanks for your detailed reply.  Your solution is very much what I would
have recommended to others in similar circumstances.  I know from
experience that it can be done simply and relatively securely.  I think
ssh is my favorite application.  You can do so much with it.  Tunneling,
dynamic port forwarding, sshfs (a really cool technique to mount remote
computers as local filesystems, with no server software required other
than an ssh server).

-----Original Message-----
From: Hobbit User [mailto:user-24d6f8323faa@xymon.invalid] 
Sent: Sunday, June 03, 2007 8:34 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Alternate to msgcache/hobbitfetch?

On Sat, June 2, 2007 22:50, Haertig, David F (Dave) wrote:

Why not just use msgcache bound to a localhost address and ssh port
tunneling, front-ended by autossh for persistence?

AUTOSSH_GATETIME=0
autossh -M 0 -f -f -N -L 127.0.0.2:1984:127.0.0.2:1984 \ -i
<path-to-rsa-private-key>  user-df99d5171f96@xymon.invalid

Install autossh.  Do your key exchange, test ssh PKI authentication.
Set msgcache to --listen=127.0.0.2 and --server=127.0.0.1, use
"dmzserver.example.com 127.0.0.2 # testip pulldata" in bb-hosts, and you
should be good to go, with a msgcache that cannot be fetched except via
your tunnel.  Note that the conn check for this bb-host entry will be
meaningless, and you'll need to add -L port mappings for any other
server-side tests you want.

Writing this from memory, so there might be a syntax error or misspelled
option.  But it's pretty close.  You do need a very up-to-date ssh to
specify the near-side IP on the -L port forward (which helps you to keep
up with tunnels for different hosts--otherwise you have to use different
ports; I actually put in /etc/hosts entries for the 127.0.0.x entries so
I can call them by name).  I know that the near-side ip spec is
supported in the current repo versions for FC5 and up, RHEL5, and
Centos5, but not in
RHEL4 and down.  Hope this helps.

Haertig, David F (Dave) wrote:

I can't remember where I saw it but I thought this functionality was already part of hobbit...

=G=

Haertig, David F (Dave) a écrit :

hi,

there should be a bb-central set of user-e0894e22938e@xymon.invalid.
those scripts works the following :
the server generates scripts to collect th edatas on the clients.
the scripts are copied then executed via scp/ssh on the client.
the server parses the datas collected, and push the infos to the hobbit server.
this tool was created to work with bigbrother, but you can certainly use them with hobbit.
start with a small config, as you can have some bugs because these are quite old.

hope this'll help.

A slightly different way would be if hobbit client did "sending validation"...

Client sends it's data as usual to the server but if the connection fails it stores information locally until it can.  Could be used if your WAN link goes down.  Or in your case, intentionally use it like hobbitfetch without using msgcache (and make hobbitfetch use ssh to get the logs).


Craig Cook
--
Systems Monitoring Consulting and Support Services
http://www.cookitservices.com

On Mon, June 4, 2007 23:18, Craig Cook wrote:

This would be a wonderful feature to have, especially since hobbitfetch
appears to be seriously broken as to reliability.  Came in this morning to
find my prototype farm all purple on msgcache columns, all apparently
having ceased to get hobbitfetch-ed between 6:15 and 6:20 last evening. 
All processes still running, but nought going on.

Random thoughts on msgcache and alternatives:

1. SSH tunnels are nice, as they encrypt the data transfers*

2. SSH tunnels are a pain in some ways, imagine managing 1000+ tunnels. Even if you have autossh to help keep them up, it's fairly resource-intensive to have thousands of ssh tunnels constantly established from your server to remote hosts.

3. I would imagine it's not a simple thing to modify hobbit to do everything over an "on-demand" ssh tunnel, because Hobbit does more than just ssh to a host, running the client script, and parsing the output.  There is also all the other hobbit protocol stuff like pushing out new clients, logfile monitoring, etc.

4. If Hobbit could do things via ssh, it would be much easier to deal with firewalls, as there are usually already rules in place for ssh, and if not it usually doesn't raise any serious flags with infosec if you request ssh access.

*5. Regardless, I would like to see some sort of encryption of the hobbit protocol. Nothing extreme, just not plaintext. Even a simple XOR of the data, or, even better, the server and clients could have a hobbit-security.conf where a key was defined, and all data would be XOR'd with this key. XORing data is easy to do in C and not resource intensive. This should satisfy the folks who worry about sending the contents of their logfiles and other sensitive information over the network.

That's my ramblings for the day :)
-Charles

First - yes, there are bugs in msgcache/hobbitfetch, and I'll try to get
those sorted out.

I must disagree here.

Poorly implemented cryptography is much worse than no cryptography.
It gives people the impression that confidentiality "has been taken care
of with encryption", when in fact it hasn't. And then people tend to
forget about the *other* things they need to do to get a secure
environment.

Inventing your own crypto protocol is usually the *worst* way to
begin doing any kind of encryption.  History is full of examples. 
I do not want to become part of it.

If Hobbit is going to have an encrypted link between clients and the
Hobbit server, it will be using TLS (SSL). It's a well-tested protocol,
it has support for not only encryption but also authentication (both
server and client), and there are standard libraries available
implementing it - which Hobbit already uses for network tests.

But I'm still un-convinced that it's such a great idea. There are lots
of ways that you can attack Hobbit if you want to get at the information
it stores - if I were to attack such a system, my first attempt would be
to get access to the server and steal the hobbitd.chk file, which holds
all of the information Hobbit stores about a host.


Regards,
Henrik

On Tue, June 5, 2007 16:32, Henrik Stoerner wrote:

It's good to know that.  Considering how well most of it works, I'm sure
that a reliable solution to the msgcache/hobbitfetch issues will be
forthcoming.  There are also some troublesome things about the
initialization scripts.  I see environment files being sourced, then paths
being hard-coded that could have been constructed from the sourced values.
 There are also stop functions that don't shut down everything that the
start function starts.  I know there's a lot of work in preparing these
for the different environments.  Is the right way to help to contribute
"upgraded" scripts back to you, Henrik?  Or would you prefer to have them
posted here or to the Shire?

Absolutely the right thinking.  Trivial encryption not only gives a false
sense of security, it adds unnecessary overhead given the ease of cracking
it.

That could be useful, but I find that ssh lends itself to tunneling
hobbitfetch and the network-based tests if you want to do that, is
ubiquitous, and it's simpler to set up the PKI authentication than doing
full-blown TLS.  It's worth considering how to ride on that as an
alternative in the way Hobbit works.  Of course, I'm biased because I
already set up that relationship on all the monitored machines for general
systems management purposes.

Thanks, Henrik, for the enormous amount of time you must put into this. 
And thanks to everyone for this active support community.

regards,
j.