xymon ssh scan 🔗 link

Any ideas on how to solve the following problem.


hamilton is shown as ssh ok, status unchanged
for a week, but you can't ssh in:

% ssh -v hamilton
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to hamilton [128.210.3.42] port 22.
debug1: Connection established.
debug1: identity file /homes/jflack/.ssh/identity type -1
debug1: identity file /homes/jflack/.ssh/id_rsa type -1
debug1: identity file /homes/jflack/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.9p1
debug1: match: OpenSSH_3.9p1 pat OpenSSH_3.*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
Read from socket failed: Connection reset by peer

Apparently something goes wrong in the server just at the start
of key exchange. The xymon ssh test reports the remote protocol
and software versions, so it must converse at least that far, but
I guess it doesn't go on through the key exchange.

The ssh server going wrong that way seems to be a familiar failure
mode for our linux boxes, so it would be nice to have a test for it
in xymon. An ssh identity that's allowed to run some single very
restricted command would work. Actually ssh-keyscan does the trick
too, and doesn't require any logging in or permission on the host.
The only trick is ssh-keyscan exits with status 0 whether it
succeeded or not, so it would have to be used in a script that
actually parses its output to see if it worked.

Dustin's script that does ssh-keyscan for updating the keys list
could be a useful starting point.

Robert P. McGraw, Jr.
Manager, Computer System                    EMAIL: user-33cf07af04dd@xymon.invalid
Purdue University                            ROOM: MATH-807
Department of Mathematics                   PHONE: (XXX) XXX-XXXX
XXX N. University Street                      
West Lafayette, IN XXXXX-XXXX

Why not use simply
 ssh %host% cat /etc/SuSE-release

mfg

Andreas

-- 


Andreas Kunberger
Körschtalstraße 26, 73770 Denkendorf

What is something that can be executed to make that universal?  Is
/etc/*release on all *nix systems?  I know it is on rhel but that's
it.

On 6/11/10, Andreas Kunberger <user-6b0b54288086@xymon.invalid> wrote:

-- 

Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX

“Success is not final, failure is not fatal: it is the courage to
continue that counts.”
--- Winston Churchill

You need something that's on all *nix systems.
/etc/passwd
/etc/hosts
or even /etc/motd

But why use cat?
ssh $HOST ls -la /etc/ will do just as well.
ssh $HOST ls -la /tmp is probably even less of a risk.

If you are testing ssh, any command will do.
ssh $HOST ls -d /tmp is probably best.
You know what the result will be, on all unix systems, and if you get "/tmp"
then ssh works.

Cheers
     Vernon


On Fri, Jun 11, 2010 at 2:32 PM, Josh Luthman

<user-4c45a83f15cb@xymon.invalid>wrote:

Am 11.06.2010 08:32, schrieb Josh Luthman:

'All' ist difficult. You could of course create a link in /etc,
do a 'grep CLIENTHOSTNAME /etc/sysconfig/hobbit' or 'ls /etc/passwd'.


mfg

Andreas
-- 

Andreas Kunberger
Körschtalstraße 26, 73770 Denkendorf

I like:

ls -d /tmp

Same result, always and everywhere.

On 6/11/10, Andreas Kunberger <user-6b0b54288086@xymon.invalid> wrote:

-- 
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX

“Success is not final, failure is not fatal: it is the courage to
continue that counts.”
--- Winston Churchill

On Friday, 11 June 2010 07:15:06 Andreas Kunberger wrote:

Why not use simply
 ssh %host% cat /etc/SuSE-release

Maybe just run a command that is likely to be present on almost any host 
running ssh.

For example, almost any unix should have 'uptime' or 'true'.

Then again, cisco devices don't have that, but they do have 'who'.

On Thursday, 10 June 2010 18:35:33 McGraw, Robert P wrote:

This is quite an old version. Time to consider an upgrade?

In quite a few years in production environments with hundreds of linux 
servers, I haven't seen that myself ...

Have you managed to find a way to reproduce it? Have you filed a bug? IOW, maybe 
prevention of the problem is better than identification.

Regards,
Buchan

Simplicity is the ultimate sophistication.
   -- Leonardo da Vinci

:-)

On Fri, Jun 11, 2010 at 2:52 PM, Josh Luthman

<user-4c45a83f15cb@xymon.invalid>wrote:

Hi, I'm the guy who documented the original issue Robert forwarded.

On Friday, 11 June 2010 Buchan Milne wrote:

This is quite an old version. Time to consider an upgrade?

Red Hat Enterprise is very conservative about switching package
versions during the lifetime of a single RHEL major release, though
they do frequently issue backported patches. We tend to avoid
replacing the shipped packages unless we can't help it.

It won't be that easy to prevent until the kinks are worked out of
RHEL's NFSv4 state recovery code. I.e., it's not some bug in sshd
we are talking about.  At the times in question, the box is
partially moribund: some kernel services are deadlocked, but the
sshd is still able to run just enough to accept connections and
get as far as key exchange but no further. The details don't matter
that much; the only point of reporting the issue on this list was
to point out that xymon's current ssh test doesn't confirm the
complete handshake, and maybe it would be a stronger test if it
did.

 ...

Most of the suggestions I've seen so far on the list seem to involve
running some arbitrary command on the host. That was my first idea
too, but it needs some cautions.

- that requires giving xymon an identity with which to log in to the
  host. To avoid saving a password, that should be done with a public/
  private key pair, the public key being installed in authorized_keys
  on each machine to be tested.

- the identity should not be allowed to run arbitrary commands. an
  entry in authorized_keys can be limited to running a single fixed
  command.

- the discussions of whether certain commands or files are present
  everywhere still seem Unix-centric; you can get ssh daemons for
  other OSes too, which may or may not even have something called
  /tmp etc.

- because a fixed command should be assigned for the identity key
  anyway, OS heterogeneity isn't such a big problem; the fixed
  command could just be "echo OK" or whatever the equivalent is
  on another OS.

However, because that approach does require extra setup on each
client machine (create a uid or pick some existing one like nobody,
set up authorized_keys to accept the xymon identity and run the
fixed command), an approach based on ssh-keyscan might be a
reasonable compromise. It doesn't require extra setup on the
host or permission to log in, and it does confirm at least that
enough of the OS is working for the ssh daemon to retrieve its
keys.

Chapman Flack

On Fri, June 11, 2010 09:30, user-6b3be4007cf2@xymon.invalid wrote:

Just give the identity a login shell of /bin/true in /etc/passwd and you
won't have to be concerned about commands from a shell at all.

Yes, that works too, if you will create a new dedicated identity (or
reuse one that already has true for a shell).  command="/bin/true"
in authorized_keys will work in any event (though something like
/bin/echo OK might give a more positive confirmation).

The line in authorized_keys should also disallow all the extra
goodies like port forwarding, X tunneling, and so on.

-Chap

On Fri, Jun 11, 2010 at 11:21 AM, Xymon User in Richmond <

user-24d6f8323faa@xymon.invalid> wrote:

You can also use a command such as /bin/hostname - that would give you a way
to verify you reached the target system.

Ralph Mitchell

On Fri, June 11, 2010 12:41, Ralph Mitchell wrote:

/bin/true will return exit 0.  If you don't get that far, ssh will return
nonzero.