Detecting read-only file system in Linux 🔗 link

On Mon, Mar 09, 2015 at 12:44:03PM -0000, SebA wrote:

I have been trying to find out if there is a way of Xymon detecting that a
file-system in Linux has gone read-only as a result of a disk error (other
than reporting it just the once via monitoring /var/log/messages).  Nothing
is showing up in my Xymon server, but my xymon-client is a bit old:
xymon-client-4.3.7-26.1.el5.tnt
 
I did a bit of Googling and I came up with these two links that may be
relevant:
http://sisyphus.ru/en/srpm/Sisyphus/xymon/sources/8
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764197
 
It seems that a RPM maintainer may have made some modifications to their
version in order to catch disks in a read-only state (in the first link) and
that there is mount-ro plugin that is part of the hobbit-plugins package in
Debian / Ubuntu.  Does anyone have more information on either of
these

and whether any patches can be integrated upstream or plug-ins added
to xymonton?

CCing Axel Beckert as he seems to have committed
something to the mount-ro plugin recently:
https://www.openhub.net/p/hobbit-plugins/commits

The following was at the bottom of /var/log/messages, but it does not
suggest any very obvious alarm strings to add other than the last line
without the 'dm-0', but it would be nicer to have something more generic
still as textual messages can change between different versions of the O/S.
 
kernel: sd 0:0:0:0: Unhandled sense code
kernel: sd 0:0:0:0: SCSI error: return code = 0x08100002
kernel: Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
kernel: sda: Current: sense key: Hardware Error
kernel:    Add. Sense: Defect list error
kernel:
kernel: Buffer I/O error on device dm-0, logical block 1358756
kernel: lost page write due to I/O error on dm-0

-----Original Message-----
From: Axel Beckert [mailto:user-96d9963fe797@xymon.invalid] 
Sent: 09 March 2015 13:14
To: SebA
Cc: xymon at xymon.com
Subject: Re: Detecting read-only file system in Linux

Hi,

On Mon, Mar 09, 2015 at 12:44:03PM -0000, SebA wrote:

I have been trying to find out if there is a way of Xymon 
detecting that a
file-system in Linux has gone read-only as a result of a 
disk error (other
than reporting it just the once via monitoring 

/var/log/messages).  Nothing

is showing up in my Xymon server, but my xymon-client is a bit old:
xymon-client-4.3.7-26.1.el5.tnt
 
I did a bit of Googling and I came up with these two links 
that may be
relevant:
http://sisyphus.ru/en/srpm/Sisyphus/xymon/sources/8
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764197
 
It seems that a RPM maintainer may have made some 
modifications to their
version in order to catch disks in a read-only state (in 
the first link) and
that there is mount-ro plugin that is part of the 
hobbit-plugins package in
Debian / Ubuntu.  Does anyone have more information on either of
these

I'm one of the maintainers of Debian's hobbit-plugins package, so
yes. :-)

and whether any patches can be integrated upstream or plug-ins added
to xymonton?

I'm not sure where exactly at https://wiki.xymonton.org/ I should add
our set of plugins.

CCing Axel Beckert as he seems to have committed
something to the mount-ro plugin recently:
https://www.openhub.net/p/hobbit-plugins/commits

Hrm, OpenHub seems horribly out of date with most projects
recently... The full view on that Git repo is at
https://anonscm.debian.org/cgit/collab-maint/hobbit-plugins.git/

The source code of the mount-ro plugin is quite simple:

It's though not a direct plugin but meant for the meta-plugin "misc"
which calls all scripts in /etc/xymon/misc.d/ and summarizes their
exit codes into a single check. This is meant for checks which get
yellow/red only very seldom and where you don't want to waste a whole
column for it.

misc plugin:

Hobbit.pm used in the misc plugin and many other plugins in that
package:

		Kind regards, Axel Beckert
-- 
Axel Beckert <user-96d9963fe797@xymon.invalid>       support: +41 44 633 26 68
IT Services Group, HPT H 6                  voice: +41 44 633 41 89
Departement of Physics, ETH Zurich
CH-8093 Zurich, Switzerland		   http://nic.phys.ethz.ch/

On 09 Mar 2015, at 13:44, SebA <user-4631430d620a@xymon.invalid> wrote:

(…)

Although we have some Debian systems, I was looking for a solution for another Linux distro.
 
If I was to write something myself to do it, I would check /proc/mounts and the best command I could find was:
awk '$4~/(^|,)ro($|,)/' /proc/mounts
which outputs:
/dev/root / ext3 ro,data=ordered 0 0
with sample line:
/dev/root / ext3 ro,data=ordered 0 0

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of SebA
Sent: Monday, March 09, 2015 7:44 AM
To: xymon at xymon.com
Cc: 'Axel Beckert'
Subject: [Xymon] Detecting read-only file system in Linux

Hi,

I have been trying to find out if there is a way of Xymon detecting that a file-system in Linux has gone read-only as a result of a disk error (other than reporting it just the once via monitoring /var/log/messages).  Nothing is showing up in my Xymon server, but my xymon-client is a bit old: xymon-client-4.3.7-26.1.el5.tnt

I did a bit of Googling and I came up with these two links that may be relevant:
http://sisyphus.ru/en/srpm/Sisyphus/xymon/sources/8
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764197

It seems that a RPM maintainer may have made some modifications to their version in order to catch disks in a read-only state (in the first link) and that there is mount-ro plugin that is part of the hobbit-plugins package in Debian / Ubuntu.  Does anyone have more information on either of these and whether any patches can be integrated upstream or plug-ins added to xymonton?  CCing Axel Beckert as he seems to have committed something to the mount-ro plugin recently: https://www.openhub.net/p/hobbit-plugins/commits

Although we have some Debian systems, I was looking for a solution for another Linux distro.

If I was to write something myself to do it, I would check /proc/mounts and the best command I could find was:
awk '$4~/(^|,)ro($|,)/' /proc/mounts
which outputs:
/dev/root / ext3 ro,data=ordered 0 0
with sample line:
/dev/root / ext3 ro,data=ordered 0 0
This command also produced a nice summary output that might be good to have on a Xymon status page:
cat /proc/mounts|sort|awk '{print $1 "\011" toupper(substr($4,0,2)
The following was at the bottom of /var/log/messages, but it does not suggest any very obvious alarm strings to add other than the last line without the 'dm-0', but it would be nicer to have something more generic still as textual messages can change between different versions of the O/S.

kernel: sd 0:0:0:0: Unhandled sense code
kernel: sd 0:0:0:0: SCSI error: return code = 0x08100002
kernel: Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
kernel: sda: Current: sense key: Hardware Error
kernel:    Add. Sense: Defect list error
kernel:
kernel: Buffer I/O error on device dm-0, logical block 1358756
kernel: lost page write due to I/O error on dm-0

Kind regards,

SebA

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Thomas Eckert
Sent: 09 March 2015 14:05
To: Xymon MailingList
Subject: Re: [Xymon] Detecting read-only file system in Linux


Hi, 


On 09 Mar 2015, at 13:44, SebA <user-4631430d620a@xymon.invalid> wrote:


(.)


Although we have some Debian systems, I was looking for a solution for
another Linux distro.
 
If I was to write something myself to do it, I would check /proc/mounts and
the best command I could find was:
awk '$4~/(^|,)ro($|,)/' /proc/mounts
which outputs:
/dev/root / ext3 ro,data=ordered 0 0
with sample line:
/dev/root / ext3 ro,data=ordered 0 0


You could use the reported "Client data" / "clientlog" with a server-side
extension. I have to admit that I'm not 100% sure if error-or-remounts are
reflected properly by this - but `mount` seems to use `/proc/self/mountinfo`
as it's datasource, so it _should_ be ok.

Extract the reported `mount`-data for host bb.local:

xymon 127.0.0.1 "clientlog bb.local section=mount"

Some testing of the output:

(export host=bb.local; xymon 127.0.0.1:1984 "clientlog $host section=mount"
| gawk '/[(,]+ro[,)]+/ { print "filesystem " $1 " mounted RO!" } ')

This would not require installing and maintaining an extension on every
client. One drawback would be increased latency as the ro-check would be
"indirect".

Cheers
Thomas

On Monday, March 09, 2015 12:44:03 PM SebA wrote:

Hi,

I have been trying to find out if there is a way of Xymon detecting that a
file-system in Linux has gone read-only as a result of a disk error (other
than reporting it just the once via monitoring /var/log/messages).  Nothing
is showing up in my Xymon server, but my xymon-client is a bit old:
xymon-client-4.3.7-26.1.el5.tnt

On 09 Mar 2015, at 18:09, SebA <user-4631430d620a@xymon.invalid> wrote:

Thanks Thomas. Simulating this suggests your code will work on our server, but as you said, you are not 100% sure that the 'ro' will surface in the mount command. The output here suggests that it may not:
http://sisyphus.ru/en/srpm/Sisyphus/xymon/sources/8 <http://sisyphus.ru/en/srpm/Sisyphus/xymon/sources/8>;
#!/bin/sh

# Read data from /proc/mounts on linux and report back in the xymon client
# It gives more accurate data than the 'mount' command and can catch
# disks in a read-only state.

test -r /proc/mounts && exec cat /proc/mounts
But maybe that is only needed for some other version of linux... Our server is now fixed so I can't test it properly. But I do prefer a server-side extension, or even a patch to xymon-server, to client plugins that I have to install to every server. The issue with your code is that it only works for 1 named server ('bb.local'), so I would need to have something iterating over all hosts. Probably not hard, and probably something that someone here already does...
 In fact, I think I have found the place to add it and that is in this add-on: https://wiki.xymonton.org/doku.php/monitors:check-client <https://wiki.xymonton.org/doku.php/monitors:check-client>; - then it will work for all hosts (correct me if I am wrong David Baldwin?)
Kind regards,
SebA

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Thomas Eckert
Sent: 09 March 2015 14:05
To: Xymon MailingList
Subject: Re: [Xymon] Detecting read-only file system in Linux

Hi,

On 09 Mar 2015, at 13:44, SebA <user-4631430d620a@xymon.invalid <mailto:user-4631430d620a@xymon.invalid>> wrote:

(…)

Although we have some Debian systems, I was looking for a solution for another Linux distro.
 If I was to write something myself to do it, I would check /proc/mounts and the best command I could find was:
awk '$4~/(^|,)ro($|,)/' /proc/mounts
which outputs:
/dev/root / ext3 ro,data=ordered 0 0
with sample line:
/dev/root / ext3 ro,data=ordered 0 0

You could use the reported “Client data” / “clientlog” with a server-side extension. I have to admit that I’m not 100% sure if error-or-remounts are reflected properly by this — but `mount` seems to use `/proc/self/mountinfo` as it’s datasource, so it _should_ be ok.

Extract the reported `mount`-data for host bb.local:

xymon 127.0.0.1 “clientlog bb.local section=mount”

Some testing of the output:

(export host=bb.local; xymon 127.0.0.1:1984 "clientlog $host section=mount" | gawk '/[(,]+ro[,)]+/ { print "filesystem " $1 " mounted RO!" } ')

This would not require installing and maintaining an extension on every client. One drawback would be increased latency as the ro-check would be “indirect”.

Cheers
Thomas

On 11 March 2015 at 00:44, Thomas Eckert <user-2a86d6cd6326@xymon.invalid> wrote:

Just for future reference as you already found your solution:

From looking at the util-linux package I understand that _if_ the
/proc-filesystem is mounted the information is used from there.

On 11 Mar 2015, at 00:58, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:

On 11 March 2015 at 00:44, Thomas Eckert <user-2a86d6cd6326@xymon.invalid <mailto:user-2a86d6cd6326@xymon.invalid>> wrote:
Just for future reference as you already found your solution:

From looking at the util-linux package I understand that _if_ the /proc-filesystem is mounted the information is used from there.

I don't think this is the case.  Running "strace" on the "mount" command shows that it first looks at /etc/mtab, and if that exists, it doesn't look at /proc/mounts.  Only if /etc/mtab doesn't exist, does mount look at /proc/mounts.

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Thomas Eckert
Sent: 11 March 2015 07:42
Subject: Re: [Xymon] Detecting read-only file system in Linux

On 11 Mar 2015, at 00:58, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:

On 11 March 2015 at 00:44, Thomas Eckert <user-2a86d6cd6326@xymon.invalid>
wrote:

Just for future reference as you already found your solution:


From looking at the util-linux package I understand that _if_ the

/proc-filesystem is mounted the information is used from there.

I don't think this is the case.  Running "strace" on the "mount" command
shows that it first looks at /etc/mtab, and if that exists, it doesn't look
at /proc/mounts.  Only if /etc/mtab doesn't exist, does mount look at

/proc/mounts.

Strange, I did also check with strace and did not see `/etc/mtab` used.

This is on Debian 7.x, Debian kernel 3.2.0-4-amd64 and util-linux 2.20.1-5.3

root at bb:~# strace -e trace=open mount 2>&1 | egrep 'mtab|proc'
open("/proc/filesystems", O_RDONLY)     = 3
open("/proc/self/mountinfo", O_RDONLY)  = 3
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

So in a situation where the filesystem containing /etc is having
problems and goes read-only, /etc/mtab won't get updated because it's now
read-only.

Agreed.
So re-using the `mount`-info for the r/o-check is at least highly
dependent on the individual environment and versions.

Cheers
Thomas