Need help with multiple RRD data points in one check 🔗 link

Hi Christoph,

Thanks for the details, 4.3.30 is fine for this.

Yes, I'd make one test per metric. Putting several metrics in one test is possible (the RRD side works either way, and GRAPHS_<test> can show several graphs on one test), but everything Xymon does with a
status works per test, not per metric:

- One color. A test has one color. If power is red, you can't see from the overview that memory is also yellow, and when power goes back to green the column stays yellow.
- One alert. alerts.cfg selects alerts by test (SERVICE=). You can't send power alerts to one team and memory alerts to another, or use a different delay or repeat for each.
- One ack and one disable. Acking or disabling the test silences every metric in it. If you ack a known power issue, a new memory problem on the same test goes unnoticed until the ack expires.
- One history. The history and availability reports record one color change per test. Afterwards you can't tell which metric caused an outage.

Inode is likewise its own column, separate from disk. Several instances of the same metric in one test is fine, which is what disk does with all its filesystems, and what the script below does with your GPUs.

Several metrics in one test are also fine when they always go together, for example traffic in/out on an interface, or memory used/total below. A good test is whether you would ever want a different alert or
ack for one of them. If you would, split them.

The reason only GPU 0 was stored is SPLITNCV. Every GPU's "Power Draw" line goes to the same file, nvidia-gpu,PowerDraw.rrd, and xymond_rrd keeps only the first update to a file from each message.

The devmon format, which xymond_rrd also handles, fixes that. Each test adds a block inside HTML comments, so it doesn't show on the status page:

<!-- linecount=0 -->
<!--DEVMON RRD: nvidia-mem 0 0
DS:memory_used:GAUGE:600:0:U DS:memory_total:GAUGE:600:0:U
0 4096:81920
1 2048:81920
-->

- The DS: line defines the data sources.
- Each line after it is one GPU: an id with no spaces, then the values separated by ":", in DS order.
- xymond_rrd writes one RRD file per line: nvidia-mem.0.rrd, nvidia-mem.1.rrd, and so on.
- The <!-- linecount=... --> line is needed for the status page to show the graph. Without it, 4.3.30 shows no graph for a devmon test whose name isn't in its built-in list (disk, inode, if_load, ...). Its
  value decides how the status page groups the GPUs, see the two options below.

This also corrects what I wrote before: it is not one RRD file holding all GPUs, but one file per GPU, the same way "disk" keeps one file per filesystem. The graph then puts them together.

Two options for the status page

The trends page shows each graph with at most 5 GPUs per image in both cases, on 4.3.30 and on the development version. The difference is the status page of each test:

┌──────────────────────────────┬─────────────────────────┬──────────────────────────┐
│  SPLITGRAPHS in the script   │         4.3.30          │   development version    │
├──────────────────────────────┼─────────────────────────┼──────────────────────────┤
│ 0 (linecount=0)              │ one image with all GPUs │ one image with all GPUs  │
├──────────────────────────────┼─────────────────────────┼──────────────────────────┤
│ 1 (linecount=number of GPUs) │ one image per GPU       │ at most 5 GPUs per image │
└──────────────────────────────┴─────────────────────────┴──────────────────────────┘

Option 0 looks the same on both versions. Option 1 gives the nicer layout once you upgrade, but one image per GPU on 4.3.30. Both keep the shipped "devmon::1" setting, so Devmon's own graphs are not affected.

1. Client script (one ext script, it sends one status per test):

#!/bin/sh
# One Xymon test per metric (like disk/inode), one RRD per GPU per test
# (<test>.<index>.rrd), stored by xymond_rrd's devmon handler.
# Server: TEST2RRD "<test>=devmon" for each test below.

# <test>=<nvidia-smi fields>: the fields become the DS names ("." -> "_").
TESTS="nvidia-power=power.draw
nvidia-util=utilization.gpu
nvidia-memutil=utilization.memory
nvidia-mem=memory.used,memory.total"

# Status page: 0 = one graph with every GPU; 1 = split the GPUs into graphs
# (one per GPU on 4.3.30, at most "<test>::N" per graph on newer versions).
SPLITGRAPHS=0

for t in $TESTS; do
        TEST=${t%%=*}
        FIELDS=${t#*=}
        COLOR=green
        RAW=$(nvidia-smi --format=csv,noheader,nounits --query-gpu=index,name,$FIELDS 2>&1)
        [ $? -eq 0 ] && [ -n "$RAW" ] || COLOR=red

# Set COLOR here from $RAW to apply per-test thresholds.

BODY=$(printf '%s\n' "$RAW" | awk -F', *' -v ds="$FIELDS" -v test="$TEST" -v splitgraphs="$SPLITGRAPHS" '
        BEGIN { n = split(ds, d, ","); for (i = 1; i <= n; i++) gsub(/\./, "_", d[i]) }
        NF >= n + 2 {
                txt = txt sprintf("GPU %s (%s):", $1, $2)
                v = ""
                for (i = 1; i <= n; i++) {
                        x = $(i + 2)
                        if (x !~ /^-?[0-9]+(\.[0-9]+)?$/) x = "U"       # "[N/A]" -> unknown
                        txt = txt " " d[i] "=" x
                        v = v (i > 1 ? ":" : "") x
                }
                txt = txt "\n"
                rrd = rrd $1 " " v "\n"
                gpus++
        }
        END {
                # The status page needs a linecount to show the graph: 0 draws every
                # GPU in one graph, the GPU count lets the status page split them.
                printf "%s\n<!-- linecount=%d -->\n", txt, (splitgraphs ? gpus : 0)
                printf "<!--DEVMON RRD: %s 0 0\n", test
                for (i = 1; i <= n; i++) printf "DS:%s:GAUGE:600:0:U ", d[i]
                printf "\n%s-->\n", rrd
        }')
        [ "$COLOR" = red ] && BODY="nvidia-smi failed:
$RAW"

$XYMON $XYMSRV "status $MACHINE.$TEST $COLOR $(date)
$BODY"
done

To add a metric, add one line to TESTS: the test name, then the nvidia-smi field(s) to store. DS names are limited to 19 characters. If you add a field to an existing test, delete its RRD files so they are
recreated; otherwise the updates are rejected.

If you prefer to keep your own script: your current MSG loop can stay as the human-readable part. Send one status per test, and add to each the linecount line (0, or the number of GPUs) and the <!--DEVMON
RRD: ... block with one line per GPU. In each block, use the index as the id and only that test's values.

2. xymonserver.cfg on the server:

TEST2RRD="...existing entries...,nvidia-power=devmon,nvidia-util=devmon,nvidia-memutil=devmon,nvidia-mem=devmon"
GRAPHS_nvidia-power="nvidia-power"
GRAPHS_nvidia-util="nvidia-util"
GRAPHS_nvidia-memutil="nvidia-memutil"
GRAPHS_nvidia-mem="nvidia-mem"

and add the four graphs at the end of the existing GRAPHS line, leaving "devmon::1" as it is:

GRAPHS="...existing entries...,nvidia-power::5,nvidia-util::5,nvidia-memutil::5,nvidia-mem::5"

- <test>=devmon routes each column to the devmon handler.
- GRAPHS_<test> tells the status page which graph section to use.
- <test>::5 is the maximum number of GPUs per image. It puts the graphs on the trends page, 5 per image. With option 1, the development version also uses it on the status pages; 4.3.30 does not, so there you
  get one image per GPU.
- Nothing needs to change when you upgrade.
- Remove your NCV_/SPLITNCV_ settings for the old test. You can delete the old nvidia-gpu,*.rrd files.

3. graphs.cfg: one section per test, each image showing its GPUs as lines:

[nvidia-power]
        FNPATTERN ^nvidia-power\.(.+)\.rrd
        TITLE GPU Power Draw
        YAXIS Watt
        DEF:p@RRDIDX@=@RRDFN@:power_draw:AVERAGE
        LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@
        GPRINT:p@RRDIDX@:LAST: \: %6.1lf (cur)
        GPRINT:p@RRDIDX@:MAX: \: %6.1lf (max)
        GPRINT:p@RRDIDX@:AVERAGE: \: %6.1lf (avg)\n

[nvidia-util]
        FNPATTERN ^nvidia-util\.(.+)\.rrd
        TITLE GPU Utilization
        YAXIS %
        -l 0
        -u 100
        DEF:p@RRDIDX@=@RRDFN@:utilization_gpu:AVERAGE
        LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@
        GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[nvidia-memutil]
        FNPATTERN ^nvidia-memutil\.(.+)\.rrd
        TITLE GPU Memory Bus Utilization
        YAXIS %
        -l 0
        -u 100
        DEF:p@RRDIDX@=@RRDFN@:utilization_memory:AVERAGE
        LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@
        GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[nvidia-mem]
        FNPATTERN ^nvidia-mem\.(.+)\.rrd
        TITLE GPU Memory Used
        YAXIS %
        -l 0
        -u 100
        DEF:u@RRDIDX@=@RRDFN@:memory_used:AVERAGE
        DEF:t@RRDIDX@=@RRDFN@:memory_total:AVERAGE
        CDEF:p@RRDIDX@=u@RRDIDX@,t@RRDIDX@,/,100,*
        LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@
        GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

The nvidia-mem test stores both used and total, so its graph can show memory used as a percentage.

Restart Xymon after editing xymonserver.cfg (graphs.cfg is read on each request).

Note: I tested both options on a Xymon 4.3.30 server built from the release and on the development version, with a simulated nvidia-smi for 8 GPUs (including "[N/A]" values, which become "U", i.e. unknown,
and show as a gap in the graph). Both produced one RRD file per GPU per test and the layouts in the table above. On 4.3.30 the plotted values matched what was sent. I haven't run it on a real GPU server, so
please let me know how it goes.

Bruno