Well, I figured it out.
I uninstalled and reinstalled xymon server to see if using all stock would help and it didn't. All I added back into hosts.cfg was ONLY the IPs and hostnames with no page definitions.
It was then either one of two things that I noticed (I didn't wait long enough between doing both of them to know which one fixed it...)
First: I commented out the default #directory /etc/xymon/hosts.d line (because I noticed everything above was working and everything below was not) and then I noticed something else while waiting for it to update:
In my history.log there was a bunch of lines: 2016-03-13 00:15:10.061160 Cannot create /var/lib/xymon/histlogs/someservername/cpu: File exists (someservername was one of the offenders)
So I went and deleted everything in /var/lib/xymon/histlogs
After that everything magically started to appear on the main page. I believe it was deleting the directories in histlogs that fixed it because they didn't all come back at once, they came back a few at a time (just as my winscp was deleting the directories serially instead of all at once) but that could be the client updates were coming in at different times as well *shrug*.
I'm sure it was something I did wrong but it was extremely frustrating to say the least. Perhaps Xymon could try to append (is that appropriate for those files?) in such a situation instead of just logging an error and giving up?
Thanks for all your help everyone that tried.
From: John Thurston <user-ce4d79d99bab@xymon.invalid>
To: "xymon at xymon.com" <xymon at xymon.com>
Sent: Friday, March 11, 2016 7:26 PM
Subject: Re: [Xymon] strange behavior
On 3/11/2016 3:00 PM, Jason Brockdorf wrote:0.0.0.0 .default.# NOCOLUMNS:files
127.0.0.1hhsiamxymon # bbd http://localhost/
123.32.139.15cent5vm # my test client @ home
This concerns me a little. The # is not a comment delimiter in this
context. I have no idea what xymonnet is going to do the those tags. If
you want comments, put them on the line above.
I don't like the non-qualified domain names, but that shouldn't prevent
this stuff from working. If you really are using short-names, I suggest
putting an explicit TESTIP in your .default. line. Again, it shouldn't
be the cause of your breakage, but it would be a good idea.
- snip -
page prod-httpservers A bunch of HTTP servers
Have you tried this without the hypen in the page name?
group-only conn|cpu|memory|disk|info|ports|procs|sslcert|clientlog HTTP
re-using the group name should not cause any trouble.
if you run
xymoncmd xymoncfg --web
or
xymoncmd xymoncfg --net
Is the hosts.cfg correctly parsed?
If you run
xymoncmd xymonnet --no-update hhsce4aveimwst1
does the output look reasonable?
Is there anything interesting in xymond.log ?
--
Do things because you should, not just because you can.
John Thurston XXX-XXX-XXXX
user-ce4d79d99bab@xymon.invalid
Enterprise Technology Services
Department of Administration
State of Alaska