Can custom client-side tests leverage the server-side value in analysis.cfg? 🔗 link

Hi,

I'm relatively new to Xymon -- working with it for three months at a new job -- and am wondering about how to have a custom client-side test leverage the value of the server-side analysis.cfg.

Do I /have/ /to/ declare a color in a `status` or `data` message to Xymon?

Is there a way to have the color be determined by a value in the server-side analysis.cfg file?

E.g. I'd like to write a custom test to monitor /var/spool/mqueue on multiple MTAs and I'd like to have different values for the green / yellow / red levels on different servers.

Ideally I'd like the custom client-side test that I put in ${XYMONHOME}/ext/ to be identical across all clients and the only difference being the value each system has in the analysis.cfg file on the server.

Am I /required/ to send a color; green, yellow, red, and then have something override / update that color server side?  Or can I omit the color and let something server side set it using values for the given client?

I'm thinking about things like DISK / LOAD / etc. where I can set warn and critical levels, or even ignore.

Or am I looking at a custom server-side xymond_channel receiver that will receive `data` (?) from the client and alter things?  --  Even that `data` is likely going to require a color.

Would someone with more experience with Xymon than me give me a pointer in a direction that I can go research?

Please and thank you.


-- 
Grant. . . .
unix || die

Date: Thu, 12 Oct 2023 19:23:17 -0500
From: Grant Taylor <user-86150bc7e4bb@xymon.invalid>
To: Xymon <xymon at xymon.com>
Subject: Can custom client-side tests leverage the server-side value
        in analysis.cfg?

Yes, a simple way is to use the "config" command, this lets you download
any .cfg (or arbitrary files, if ALLOWALLCONFIGFILES="TRUE" but don't do
that :)
files from the server's etc/ directory.

client/bin/xymon $XYMONSERVER "config analysis.cfg" > tmp/analysis.cfg

If you use "include" or "directory" directives in the .cfg files
Xymon will send you the "flattened" expanded contents.

You will need to parse that file to do anything useful though.

Alternatively, fetching the hosts.cfg and using xymongrep
  client/bin/xymon $XYMONSERVER "config hosts.cfg" > tmp/hosts.cfg
  HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep mytag:*
might be simpler for simple configuration data that you can put in a
"tag:" for each host. xymongrep is normally used server-side to loop
over multiple hosts, you will likely need to match just one client name
from its output for a client script.

Alternatively, instead of using analysis.cfg the client-local.cfg
file is logically cleaner (and automatically shipped to clients, see
client/tmp/logfetch.HOSTNAME.cfg). This specifies the client-side part of
log and file checks, although it's not documented, you can add arbitrary
lines to this file, e.g.
 [myhostname]
 x-my-specialvarA one
 x-my-specialvarB two
 log:/var/log/message:16384
 ignore MARK

Each client will pick this up and store its own "personal" stanza from
the global client-local.cfg, this is automatically refreshed each client
run cycle. Any non-documented parameters are explicitly ignored, but
best placed above any of the documented ones to make sure.

Do I /have/ /to/ declare a color in a `status` or `data` message to Xymon?

status: yes (though "clear" might be an option), data: no.
There's also a "modify" message you can use after the fact to
change a status/colour (though take care not to trigger flapping).
https://xymon.sourceforge.io/xymon/help/manpages/man1/xymon.1.html

Seems like fetching the hosts.cfg and using xymongrep should work
for that without too much trouble, at least as an exercise, and
is reasonably self-evident.
hosts.cfg:
 10.1.2.3 mailhost1 #  ssh smtps MAILQ:qwarn=200,qcrit=500

Then in a bash script (light on error checking) for example:
  client/bin/xymon $XYMONSERVER "fetch hosts.cfg" > tmp/hosts.cfg
  HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep "MAILQ:*" |
  while read hostip hostname hostcfg; do
    if [[ "$hostname" == ${MACHINEDOTS} ]]; then
      mailcfg=""
      [[ "$hostcfg" =~ MAILQ:(.*) ]] && mailcfg="${BASH_REMATCH[1]}"
      for var in qwarn qcrit; do
        [[ "$mailcfg" =~ "($var)=([^,;]+) ]] && declare
"cfg_${BASH_REMATCH[1]}"="${BASH_REMATCH[2]}"
      done
      # declare -p ${!cfg_*} ##DEBUG##
      # mailq logic using $cfg_qwarn $cfg_qcrit goes here
      # xymon status submit goes here
    fi
  done

You could dispense with the outer while loop by using
  read hostip hostname hostcfg < <(xymongrep MAILQ:* | grep "$MACHINEDOTS")

There is no xymongrep equivalent for client-local.cfg, you could adapt
the above to parse the client cached copy (tmp/logfetch.HOSTNAME.cfg) -
simple enough since the server does not provide the entire client-local
config, just the client-specific matching stanza.

I know there were some old, old mailq client scripts, there's still some
RRD
related residue for parsing their output in Xymon. I've never used it, but
likely this was one:

https://web.archive.org/web/20080718210020/http://www.deadcat.net/viewfile.php?fileid=82
The bf/nmailq-bf.sh script therein simply barfs out a bit of sendmail's
"mailq" output. The point being if you have the data in an RRD then you can
also use "DS" in analysis.cfg to alert based on a GAUGE type metric.
There's more than one way to skin a (dead)cat :)

To scale up the configuration complexity or number of discrete service
checks it would be better to dispense with xymongrep and distribute your
own easy-to-parse configuration file(s), or use the client-local.cfg
approach (my preference).

C.

Subject: Digest Footer


End of Xymon Digest, Vol 151, Issue 10
**************************************

Hi Conor,

Thank you for your reply.

The overall gist of what I've understood thus far is that it's /indirectly/ possible to have custom client-side scripts use server-side values by arranging for a copy of those values to be available (cached) on the client for local use.  --  Is that an accurate summary?

On 10/13/23 1:20?PM, Conor McCarthy wrote:

I'm glad to know that the analysis.cfg will be flattened as we're making extensive use of includes.

You will need to parse that file to do anything?useful though.

I'd rather have the values being checked against -- as opposed to what tests are run -- in the analysis.cfg file.  It seems to me like having value entries in an additional file could potentially end up in some confusion.  Which file do I check CPU load in?  What about mail queue size?  Type thing.

xymongrep is normally used server-side to loop over multiple hosts, you will likely need to match just one client name from its output for a client script.

That's the impression I had gotten.

I feel like the ability to access the config, and thus what other clients are being monitored, is a little bit of an information leak. Probably not one that matters.  And it's not like I can do anything about that.

Alternatively, instead of using analysis.cfg the client-local.cfg file is logically cleaner (and automatically shipped to clients, see client/tmp/logfetch.HOSTNAME.cfg).

ACK

Good to know.

Each client will pick this up and store its own "personal" stanza from the global client-local.cfg, this is automatically refreshed each client run cycle.

Does the server send the personalized version?  Or does the client receive the full version and only store it's own personal values?

Any non-documented parameters are explicitly ignored, but best placed above any of the documented ones to make sure.

Noted.

status: yes (though "clear" might be an option),

Interesting call on the "clear" color.

That has the added benefit of not declaring a green / yellow / read.

data: no.

Noted.

There's also a "modify" message you can use after the fact to change a status/colour (though take care not to trigger flapping).

Ya.  I had seen modify.  I figured that might be part of how the server side tweaking of client side colors might happen.

I believe there is some restriction on once a message has been modified, only the same thing can modify it again.  --  I assume that restriction is only there until a new status message comes in from the client.

Seems like fetching the hosts.cfg and using xymongrep should work for that without too much trouble, at least as an exercise, and is reasonably self-evident.

ACK

This is some very tight and nice Bash.  I like how you're using test ([[) and parameter expansion.  --  Much to come back to and look at in more detail later.

Aside:  Did you mean "fetch" or "config" when calling xymon?

I think the biggest gotcha that I have is that some of the systems in my config are using different host names and relying on CLIENT: to make things match.  But I think that's a matter of finding and grinding out the CLIENT: from the ${hostcfg}.

You could dispense with the outer while loop by using
 ? read hostip hostname hostcfg < <(xymongrep?MAILQ:* | grep "$MACHINEDOTS")

ACK

There is no xymongrep?equivalent for client-local.cfg, you could adapt

I like the idea that the server only provides the client specific stanza.

But I also like the idea that I can pull the analysis.cfg file using `${XYMON} ${XYMSRV} "config analysis.cfg"`.

Initial skim of the xymongrep man page make me think that it's intended for the hosts.cfg file and not going to work with other files.  So I'd need to parse it.

$RESEARCH++

The bf/nmailq-bf.sh script therein simply barfs out a bit of sendmail's "mailq" output. The point being if you have the data in an RRD then you can also use "DS" in analysis.cfg to alert based on a GAUGE type metric.  There's more than one way to skin a (dead)cat :)

I have seen many references to "DS", but I'm still not quite sure what that means.  I need to read the custom graphing again and work to understand it.

To scale up the configuration complexity or number of discrete service checks it would be better to dispense with xymongrep and distribute your own easy-to-parse configuration file(s), or use the client-local.cfg approach (my preference).

ACK

The ability to pull config files from the server makes this a bit easier.  It means that the client can get access to the configuration data to be able to more accurately set clear / green / yellow / red colors in reports.

Thank you Conor.  I'll re-read, research, and do some thinking.


-- 
Grant. . . .
unix || die

Grant, a few quick responses to your message:

The "DS" means "data source". It's a term used by the RRD (round-robin
database) library (and tools) that Xymon uses for storing historical data.

As a general principle, Xymon clients operate in local mode or central
mode. In (the older) local mode, the clients do all of their own analysis
and create their own data and status messages. In (the more modern) central
mode, the clients do a lot less of that, and instead simply gather data for
the server to parse and alert on.

For your requirement, what I would do is to have the client collect the
queue size (and other data) and include this in the client message and one
of the sections, eg [mailq]. The contents of this could simply be the
output of the "mailq" command. You could create a script in the "sections"
subdirectory, and the Xymon client will automatically run it and include
the output in the [mailq] section of the client message. Then on the
server, run a script (eg from tasks.cfg or something like tasks.d/mailq)
that dumps the client data message for every client, and if a [mailq]
section is found, create a status message.

And optionally turn the mail queue size into a graphable data point (either
create a trends message with the number, or use a colon-separated value
format for the Xymon parser to detect). The Xymon doco has a "how to
setup custom graphs" page with a few different ways to do this type of
thing.

J

On Sat, 14 Oct 2023 at 08:20, Grant Taylor via Xymon <xymon at xymon.com>
wrote:

---------- Forwarded message ----------
From: Grant Taylor <user-86150bc7e4bb@xymon.invalid>
To: xymon at xymon.com
Cc:
Bcc:
Date: Fri, 13 Oct 2023 16:18:08 -0500
Subject: Re: [Xymon] Can custom client-side tests leverage the server-side

---------- Forwarded message ----------
From: Grant Taylor via Xymon <xymon at xymon.com>
To: xymon at xymon.com
Cc:
Bcc:
Date: Fri, 13 Oct 2023 16:18:08 -0500
Subject: Re: [Xymon] Can custom client-side tests leverage the server-side
value in analysis.cfg?

On 10/14/23 11:28?PM, Jeremy Laidman wrote:

Grant, a few quick responses to your message:

Thank you for the reply Jeremy,

The "DS" means "data source". It's a term used by the RRD (round-robin database) library (and tools) that Xymon uses for storing historical data.

Thank you for clarifying.  I sort of deduced that as I got further into the custom graphs, which I've now got working.  But it's one of those things that when you're newer to something "DS" is unfortunately vague and lost in a sea of new information.

As a general principle, Xymon clients operate in local mode or central mode. In (the older) local mode, the clients do all of their own analysis and create their own data and status messages. In (the more modern) central mode, the clients do a lot less of that, and instead simply gather data for the server to parse and alert on.

Would it be fair to say that the older local mode uses status messages with color and newer central mode use data messages without color?  -- As a rough generalization, not a hard and fast rule.

For your requirement, what I would do is to have the client collect the queue size (and other data) and include this in the client message and one of the sections, eg [mailq]. The contents of this could simply be the output of the "mailq" command.

That's the path that I've gone with my initial swing at monitoring the Sendmail queue.

I did so by adding an ext(ension) script and adding said ext(ension) as a task to the Xymon client.  It's working quite well.

You could create a script in the "sections" subdirectory, and the Xymon client will automatically run it and include the output in the [mailq] section of the client message.

This is very intriguing.  I spent a moment looking for "section" (case insensitive) in the client directory tree (`fgrep -ir "section"` type thing) and found reference to the client/local directory.  Upon testing, I see that I end up with the output of my executable (currently a script) with a heading named `[local:test_script.sh]`.

This is EXTREMELY useful.  It will make adding / removing additional data sections trivial.

It will also be cleaner than trying to use include files via clientlaunch.cfg  --  I was worried about central management of tasks accidentally overriding local things in the clientlaunch.cfg file.

Then on the server, run a script (eg from tasks.cfg or something like tasks.d/mailq) that dumps the client data message for every client, and if a [mailq] section is found, create a status message.

I will be exploring this.

It seems like the gist of the idea is to receive the client status message, extract the section, and generate updates therefor.

And optionally turn the mail queue size into a graphable data point (either create a trends message with the number, or use a colon-separated value format for the Xymon parser to detect). The Xymon doco has a "how to setup?custom graphs" page with a few different ways to do this type of thing.

Yep.  I used that very page to get NCV based custom graphs working Friday evening.

I still need to explore the DS analysis / color declaration therefor.

Thank you again Jeremy.  I've learned a few more things that will make Xymon easier to deploy, maintain, use, and gain insightful information.  :-)


-- 
Grant. . . .
unix || die

Responding inline:

On Tue, 17 Oct 2023 at 02:47, Grant Taylor via Xymon <xymon at xymon.com>

wrote:

Agreed. Xymon/Hobbit, and its spiritual predecessor BigBrother, come from a
tradition of using a common collection of disparate tools for monitoring
networked devices and hosts. One of these foundational tools is RRD, which
is used by other network monitors (Cacti, MRTG, etc). The "culture" of
Xymon can sometimes assume knowledge of how the other tools work. And while
there's references to RRD in the Xymon doco, it couldn't hurt to add some
extra definitions or explanations within the Xymon doco, to help those who
don't have a history in this space that extends back to the last millenium
(showing my age, aren't I).

Yes and no. The central/local mode choice relates to the bulk of the core
OS resource metric collection and processing. Choosing central mode doesn't
prevent the use of client-side processing and status messages (with colour)
being sent from the client. I know you didn't ask this, but I'm just making
sure I'm not giving the wrong idea.

With local mode, the client looks for faults, and sends a status message
with the colour in the header and a text report in the body.

With central mode, client data messages are sent to the server, into the
xymond daemon. Then code on the server asks the xymond daemon for the
client message of each host, parses them for possible problems, and
constructs a status message with the colour in the header and a text report
in the body.

So the last part of the process is the same. The Xymon server receives
status messages from itself for clients in central mode, or from the
clients when they're in local mode.

To bring it back to your phrasing, local mode uses status messages with
colo(u)r and central mode also uses status messages with colo(ur) but those
messages come from the Xymon server itself. In central mode, the colours
are determined by the server (eg in analysis.cfg), whereas in local mode,
the colours are determined by the client.

Yes, the "client/sections" directory is a relatively recent feature that
was added as a "patch" to the original client-side script. My mistake for
assuming that everyone had this. The patch is applied to the packages
available from the Terabithia repository, but I don't believe it's included
when Xymon is built from original source code. The "client/local" directory
is "standard" Xymon (as long as you create the directory), but it has the
side-effect of making section names with the "local" prefix. There are
probably good reasons for this prefixing (I'm guessing to prevent
collisions with standard section names), so if it works for you, that's
great. My guess is that the next release of Xymon will include
client/sections in the source, in addition to supporting client/local.

A snippet from the README in client/sections/ from the Terabithia package:

"This directory is intended for use with automatically deployed files. Local
scripts for this host should be placed in the client/local/ directory
instead."

It seems like the gist of the idea is to receive the client status

Exactly. The client is as simple as it can be, so that it doesn't need
constant tweaking. The server does all the smarts, including parsing,
graphing, thresholding, so that the client host can spend more cycles doing
its main function.

You're well on your way, so well done! Feel free to ask more questions on
this list. The more questions the better our shared knowledge! (Although
it's easier to respond to if each question is in a separate thread.)

J

On 10/16/23 6:39?PM, Jeremy Laidman wrote:

Responding inline:

:-)

Agreed. Xymon/Hobbit, and its spiritual predecessor BigBrother, come from a tradition of using a common collection of disparate tools for monitoring networked devices and hosts. One of these foundational tools is RRD, which is used by other network monitors (Cacti, MRTG, etc).

Yep.  I was exposed to RRD via MRTG, and subsequently Cacti, probably two decades ago.  Both surprisingly and thankfully I've not needed to use RRD directly as things like MRTG, Cacti, and Xymon have abstracted them away for me.  But I've long appreciated the suite of tools.

The "culture" of Xymon can sometimes assume knowledge of how the other tools work. And while there's references to RRD in the Xymon doco, it couldn't hurt to add some extra definitions or explanations within the Xymon doco, to help those who don't have a history in this space that extends back to the last millenium (showing my age, aren't I).

Waiter / waitress, the next cup of $COFFEE is on me.

Yes and no. The central/local mode choice relates to the bulk of the core OS resource metric collection and processing. Choosing central mode doesn't prevent the use of client-side processing and status messages (with colour) being sent from the client. I know you didn't ask this, but I'm just making sure I'm not giving the wrong idea.

ACK

Sounds like use what's best for the situation, even if there might be a slight affinity one way or the other.

With local mode, the client looks for faults, and sends a status message with the colour in the header and a text report in the body.

I've been making extensive use of this.  I've been augmenting many cron jobs that historically have sent email to now also report to Xymon.  I'm making extensive use of lifetime to make lack of reports stand out by going purple.

Honestly, this work effort has gotten me the closest "single pane of glass" visibility into environments.  We're still a long way away from it.  But we're getting closer every day.

With central mode, client data messages are sent to the server, into the xymond daemon. Then code on the server asks the xymond daemon for the client message of each host, parses them for possible problems, and constructs a status message with the colour in the header and a text report in the body.

My understanding is that's one way to do it.

My understanding that another way to do it is to hook in as a channel and receive a copy of all messages (of the proper type) and act appropriately.

So the last part of the process is the same. The Xymon server receives status messages from itself for clients in central mode, or from the clients when they're in local mode.

ACK

To bring it back to your phrasing, local mode uses status messages with colo(u)r and central mode also uses status messages with colo(ur) but those messages come from the Xymon server itself. In central mode, the colours are determined by the server (eg in analysis.cfg), whereas in local mode, the colours are determined by the client.

ACK

Yes, the "client/sections" directory is a relatively recent feature that was added as a "patch" to the original client-side script. My mistake for assuming that everyone had this.

After inheriting the -- let's go with -- existing legacy install at $DAY_JOB, I've since upgraded $DAY_JOB to 4.3.30, installed 4.3.30 at home, and installed 4.3.28 (what comes in distro files).

I've got a smattering of clients from 4.3.30 down to 4.2.3 (installed yesterday on an ancient FreeBSD 4.11).  --  As in the green faces have just recently turned to green diamonds for that FreeBSD 4.11 system.

I'm trying to get my hands on a compilation of any version for AIX 5.3 (?).  For now I'm running the hobbitclient-aix.sh via telnet w/ echo scheduled via cron.  It's working and providing most of what I need and much of what I want.

I've sort of taken what previous colleagues poked once in a while and ran with it to make things better for $DAY_JOB, myself, and a friend that I help.  I've also intrigued another friend enough to install and play with it.

I think I say that I'm liking Xymon.  ;-)

The patch is applied to the packages available from the Terabithia repository, but I don't believe it's included when Xymon is built from original source code.

Please elaborate and / or point me where I can go do some more reading.

I've seen references to 4.4 being in alpha status.

The "client/local" directory is "standard" Xymon (as long as you create the directory), but it has the side-effect of making section names with the "local" prefix. There are probably good reasons for this prefixing (I'm guessing to prevent collisions with standard section names), so if it works for you, that's great.

Yep.  Naming things and knowing where / how to start counting are three of the hardest things in computers.

My guess is that the next release of Xymon will include client/sections in the source, in addition to supporting client/local.

I'll keep that in mind.

/me chuckles to himself muttering something about "/" vs "/usr" vs "/usr/local" vs "/opt".

Exactly. The client is as simple as it can be, so that it doesn't need constant tweaking. The server does all the smarts, including parsing, graphing, thresholding, so that the client host can spend more cycles doing its main function.

:-)

You're well on your way, so well done!

Thank you.

Feel free to ask more questions on this list. The more questions the better our shared knowledge!  (Although it's easier to respond to if each question is in a separate thread.)

ACK

I tend to try to keep any given thread relatively contained and start a new thread as needed.


-- 
Grant. . . .
unix || die

On Tue, 17 Oct 2023 at 12:22, Grant Taylor via Xymon <xymon at xymon.com>

wrote:

Yes. One of the things I love about Xymon is that it's extremely flexible
so that you get to choose from a range of different ways to solve a
problem, based on your skillset, environment, and preferences. It's modular
and extensible.

Yes, one way (see my previous comment ;-). But it's also the way Xymon does
it internally. Well, kind-of. Instead of "asks the xymond daemon" a xymon
module connects to a channel and receives from the xymon daemon. But the
data flow is pretty much the same, and I suppose that's what I was trying
to convey, while avoiding the technical complexities.

My understanding that another way to do it is to hook in as a channel

Yes indeed. You're really getting into the nitty-gritty of things. Most
Xymon admins never have to know that channels are a thing, because they're
mostly used for Xymon to talk to itself (eg to parse a client data message
and create status messages). So consider yourself levelled-up!

When writing server-side extensions that use client data, you can write
your own parser that periodically fetches the latest client message, or you
can tap into the channel to get it. Xymon taps into the channel.

I'm trying to get my hands on a compilation of any version for AIX 5.3

Oh my! Isn't AIX5.3 like a decade beyond end-of-support?

I've sort of taken what previous colleagues poked once in a while and

Good to see.

About the "sections" patch? I don't know much about the background to its
conception and existence, but there's probably a CHANGELOG and that might
provide some info. If you haven't yet, perhaps install an RPM from the
Terabithia website, or unpack an SRPM to review any extra README files or
the like.

I've seen references to 4.4 being in alpha status.

Yes indeed. I've not started testing this yet myself.

Yep.  Naming things and knowing where / how to start counting are three

LoL.

Cheers
Jeremy

On 10/16/23 9:10?PM, Jeremy Laidman wrote:

That's what I'm seeing.  I appreciate a good set of LEGOs.

Yes, one way (see my previous comment ;-). But it's also the way Xymon does it internally. Well, kind-of. Instead of "asks the xymond daemon" a xymon module connects to a channel and receives from the xymon daemon. But the data flow is pretty much the same, and I suppose that's what I?was trying to convey,?while avoiding the technical complexities.

Fair enough.

Yes indeed. You're really getting into the nitty-gritty of things. Most Xymon admins never have to know that channels are a thing, because they're mostly used for Xymon to talk to itself (eg to parse a client data message and create status messages). So consider yourself levelled-up!

Thank you.

I did /try/ to do my homework and do /try/ to ask intelligent questions.   Sometimes I fail.

When writing server-side extensions that use client?data, you can write your own parser that periodically fetches the latest client message, or you can tap into the channel to get it. Xymon taps into the channel.

ACK

Oh my! Isn't AIX5.3 like a decade beyond end-of-support?

At least.

Does "legacy" and "out to pasture" or even "business critical" mean anything to you?  My $DAY_JOB is care and feeding of old systems that nobody else wants to touch.  I enjoy it and it pays the bills.

Good to see.

About the "sections" patch? I don't know much about the background to its conception and existence, but there's probably a CHANGELOG and that might provide some info. If you haven't yet, perhaps install an RPM from the Terabithia website, or unpack an SRPM to review any extra README files or the like.

$READING_LIST++

Yes indeed. I've not started testing this yet myself.

I might consider standing up a 4.3.30 xymonproxy that receives messages and sends them to 4.3.30 running on another port and 4.4.x running on another system.

Though I suspect I need to have a better understanding of the client / server hierarchy talked about in my 2nd thread.

LoL.

:-)


-- 
Grant. . . .
unix || die

On Tue, 17 Oct 2023 at 13:55, Grant Taylor via Xymon <xymon at xymon.com>

wrote:

Oh dear. I'm glad you enjoy that sort of thing. I suppose if the
expectations aren't high, then you can always exceed expectations. Do you
happen to program in COBOL? ;-)

J

On 10/16/23 11:57?PM, Jeremy Laidman wrote:

Well, as for expectations, the systems are only expected to do today what they were expected to do when they were installed.

Mostly it's an inventory / materials management application that was written for a platform that the vendor has migrated off of.  Or the customers don't want to pay to migrate to the new platform.  So we are being paid to continue running the old platform as it's doing exactly what it's designed to do.

I assume you meant it as a joke, but I have done just enough COBOL to complete Master the Mainframe training a few years ago.  As a norm I don't do anything with COBOL.  ;-)


-- 
Grant. . . .
unix || die