Xymon Mailing List Archive search

/var/lib/xymon/rrd folders growing larger than anticipated

list Matt Vander Werf
Sat, 21 Sep 2019 08:20:21 -0400
Message-Id: <CAE6TTt6Q=user-9e830b2d251c@xymon.invalid>

I often see those kind of interface RRD files on systems that have Docker
containers running on them. Every time a new container is started, a random
interface like that is created on the host for the container network (and
thus an RRD file is created). Every time a container is restarted, a new
random interface is created for it and the old interface isn't used
anymore, but the RRD file remains for it. For me at least, these are on
systems that we don't admin but still monitor with Xymon, so I never know
when the containers will be created/restarted/etc.

Depending on how often the containers are restarted/created/etc., the
number of random veth* interface RRD files per host can be A LOT. Normally,
they aren't very large by themselves, but I can see how combined it might
take up a considerable amount of space.

I run https://wiki.xymonton.org/doku.php/addons:finit periodically to find
inactive RRD files and remove the ones that I know won't be updated again.
The script will output rm commands you can use to remove the inactive RRD
files. You just need to change the HomeDir and Threshold (in seconds)
variables to what you want to use. This script works really well to find
and clean up the inactive random veth* interfaces (and other inactive RRD
files). It can also help to find hosts that can just be removed (dropped)
completely from Xymon instead ('xymon drop X').

Alternatively, if you don't care about these network interfaces, and know
the specific interfaces you do care about, you could use the INTERFACES
option on the host entries in hosts.cfg:

       INTERFACES:REGEXP
              On systems with multiple network interfaces, the operating
system may report a number of net?
              work interface where the statistics are of no interest. By
default Xymon  tracks  and  graphs
              the  traffic  on  all  network interfaces. This option
defines a regular expression, and only
              those interfaces whose name matches the expression are
tracked.


Hope this helps!

-- 
Matt Vander Werf
HPC System Administrator
University of Notre Dame
Center for Research Computing - Union Station
XXX W. South Street
South Bend, IN XXXXX
Phone: (XXX) XXX-XXXX


On Fri, Sep 20, 2019 at 8:22 PM J.C. Cleaver <user-87556346d4af@xymon.invalid> wrote:

On Fri, September 20, 2019 5:42 am, Eliza Danyi wrote:
Morning all,

We are currently running Xymon 4.3.28-1.el7.terabithia (on CentOS 7.x)
and
I have been struggling with this since we switched to the RPMs from an
older version of xymon built from source about a year ago. We've gone
from
a few GB on the old server to now almost 40GB just for this directory.

I posed the question a little while back to the group and thought I had
found the cause in the rrddefinitions.cfg file under the line:

RRA:HWPREDICT:8064:0.1:0.0035:2016

I tried a few different things but ended up deciding to comment out this
line to see if it would slow the growth, but it does not appear to be
working. Is there any reason these files would be increasing in number or
size? Is there something I can do with rrdtool to clean up data? Is it
normal that an rrd directory for a single host is upwards of 60M on the
newer versions of Xymon?

Here is an example of the files in a folder for a host:
Hi,

RRD files by themselves do not change size (unless intentionally
manipulated), however new RRAs would have been added in I believe when it
was first run.

The ones you're seeing below, though, are a result of two things; one
easier to fix than the other. Systemd creates "/run/user/xxxx" partitions
on user login, and if the client is running when a new UID is logged in,
you'll get a new RRD file dropped in. This was changed in 4.3.29+ I
believe, but you should be able to exclude them by adding this in to your
analysis.cfg file:

CLASS=linux
        # Ignore some usually uninteresting tmpfs mounts.
        DISK    /dev IGNORE
        DISK    /lib/init/rw IGNORE
        DISK    /sys/fs/cgroup IGNORE
        DISK    %^/run/user/\d+$ IGNORE
        INODE   /dev IGNORE
        INODE   /lib/init/rw IGNORE
        INODE   /sys/fs/cgroup IGNORE
        INODE   %^/run/user/\d+$ IGNORE

After that's in there, you should be able to confirm that the /run/user/*
ones aren't being updated any more and you can purge them.

The second issue, however, is with your network interfaces:
ifstat.veth08022d9.rrd, etc... It looks like something is giving you
highly variable transient interface names?

Xymon does not currently have a mechanism to exclude interfaces by name
like this, since they just look like new things to track. Are these
changing during known-events that might be able to trigger a cleanup?

As a workaround, you should be able to purge ones that have not been
updated after a certain length of time.


HTH,
-jc


[root at xymon]# du -h --max-depth=1 /var/lib/xymon/rrd/server.example/*
216K    /var/lib/xymon/rrd/server.example/clock.rrd
424K    /var/lib/xymon/rrd/server.example/disk,boot.rrd
424K    /var/lib/xymon/rrd/server.example/disk,dev,shm.rrd
424K    /var/lib/xymon/rrd/server.example/disk,elasticsearch.rrd
424K    /var/lib/xymon/rrd/server.example/disk,home.rrd
424K    /var/lib/xymon/rrd/server.example/disk,root.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,0.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1000.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1001.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1002.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1003.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1004.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1014.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1016.rrd
424K    /var/lib/xymon/rrd/server.example/disk,sys,fs,cgroup.rrd
424K    /var/lib/xymon/rrd/server.example/disk,tmpfs.rrd
424K    /var/lib/xymon/rrd/server.example/disk,tmp.rrd
424K    /var/lib/xymon/rrd/server.example/disk,var.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.docker0.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.eno16780032.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth0506ddf.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth08022d9.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth081fb4b.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth0828eba.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth0ed858f.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth164c573.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth1740a33.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth189a45e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth197189c.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth19d7cca.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth1dfb69f.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth2090f52.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth2282b2c.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth258f7aa.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth273f639.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth28aa403.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth29630d9.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth2c50b73.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth2ccd63f.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth38ba0c9.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth38c413a.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth3a47851.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth3c5e33e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth3cb1195.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth3e20beb.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth42dcce0.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth4311e0a.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth461d830.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth4855b73.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth5074706.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth51e84f6.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth525c545.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth569cd53.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth59a3d88.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth5c4439e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth5f09582.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth5fa1dd2.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth67d3cbf.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth67dbb70.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth682379a.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth684bcf3.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth69ab431.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6ad9701.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6c80cac.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6ce4f39.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6e24fbf.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6e59721.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6fb0766.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth71f97e2.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth741d0d5.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth7c752f1.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth7f93b1f.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth8386dfb.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth8e3b4e4.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth8f68e08.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth9063983.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth91bc3c6.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth96c1fac.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth9a73f58.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth9ed52d1.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha09504c.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha184ef3.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha410e0c.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha460481.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha5a56d6.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha6cec13.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethad8eda4.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethad8faa1.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethaffd7df.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethb3c04b7.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethb426b52.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethb52aae4.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethb6937d3.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethba647c0.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethbd846e1.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethc01c9dd.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethc81bb06.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethc8ea13a.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethcc1a6d6.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethcc51afb.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd01eb33.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd105c62.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd1b1789.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd676e0b.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd947e9e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethdcb0028.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethed73659.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethf05d463.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethf4659bf.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethf84f9fc.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethf8efbee.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethfb569a5.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethfb5988e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethfeec0dd.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethffc6c75.rrd
424K    /var/lib/xymon/rrd/server.example/inode,boot.rrd
424K    /var/lib/xymon/rrd/server.example/inode,dev,shm.rrd
424K    /var/lib/xymon/rrd/server.example/inode,elasticsearch.rrd
424K    /var/lib/xymon/rrd/server.example/inode,home.rrd
424K    /var/lib/xymon/rrd/server.example/inode,root.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,0.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1000.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1001.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1002.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1003.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1004.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1014.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1016.rrd
424K    /var/lib/xymon/rrd/server.example/inode,sys,fs,cgroup.rrd
424K    /var/lib/xymon/rrd/server.example/inode,tmpfs.rrd
424K    /var/lib/xymon/rrd/server.example/inode,tmp.rrd
424K    /var/lib/xymon/rrd/server.example/inode,var.rrd
216K    /var/lib/xymon/rrd/server.example/la.rrd
216K    /var/lib/xymon/rrd/server.example/memory.actual.rrd
216K    /var/lib/xymon/rrd/server.example/memory.real.rrd
216K    /var/lib/xymon/rrd/server.example/memory.swap.rrd
3.3M    /var/lib/xymon/rrd/server.example/netstat.rrd
216K    /var/lib/xymon/rrd/server.example/procs.rrd
216K    /var/lib/xymon/rrd/server.example/tcp.conn.rrd
216K    /var/lib/xymon/rrd/server.example/users.rrd
3.5M    /var/lib/xymon/rrd/server.example/vmstat.rrd


Do these look like reasonable file sizes/number of files compared to
everyone else? How much space should we reasonably expect to use for this
given we have roughly 1k hosts being monitored (mix of Windows and
Linux).
Thanks,
*Eliza Danyi*
Operating Systems Analyst
VCU Computer Center
Phone: XXX-XXX-XXXX
E-mail: user-d7b2efa5cd13@xymon.invalid

*Don't be a phishing victim -- VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information.  For more details,
visit **https://phishing.vcu.edu/* <https://phishing.vcu.edu/>;