DEV Community

Cover image for Detecting Missing HTCondor Nodes and Stale GPU Metrics with Prometheus and Slack
Haruki Niwa
Haruki Niwa

Posted on AI-assisted

Detecting Missing HTCondor Nodes and Stale GPU Metrics with Prometheus and Slack

Part 6 of 6 in Building and Operating an HTCondor Pool for Research Computing.

An HTCondor worker can disappear from the pool. A GPU worker can stay online while its VRAM collector stops writing. In the second case, Node Exporter may keep serving the same value on every scrape. Our lab guide routes both alerts to one Slack channel and identifies the worker and the failed check in each message.

This walkthrough starts with the pool from Part 1 and the GPU textfile writer from Part 5. On the monitoring host, the example queries HTCondor and Prometheus once per poll. A local exercise reproduces the alert and recovery decisions without contacting the pool or Slack.

Choose what to monitor

Keep two inventories on the monitoring host. Populate them from the machines you expect to serve and keep them across polls.

Inventory Identifier What one entry covers
Execute nodes Exact HTCondor Machine name, such as worker.example.org All visible slot ads for that machine. One remaining ad is enough for the presence check.
GPU workers Exact Prometheus instance label, such as 192.0.2.10:9100 The worker’s gpu_vram.prom file. The file can hold readings for several GPU UUIDs.

The lab guide adds newly observed targets every ten minutes. A worker that has never reported needs a manual entry. When retiring a target, stop its advertisement or scrape registration first so discovery cannot add it back. The example below uses two manually maintained files. Using only the latest query result as the inventory would silently drop the worker you want to detect.

Execute nodes

Run condor_status -af Machine | sort -u on the monitoring host. If a successful query omits an expected name, start counting at the first poll that observes the absence. Collector ads can outlive a lost connection. An alert can therefore arrive more than 15 minutes after the workstation lost power. A nonzero exit or timeout makes the observation unknown. HTCondor ClassAd lifetime

The lab implementation has an extra step. A one-minute writer exports htcondor_execute_node_up and retains the last good textfile if condor_status fails. The lab notifier reads that file without checking its age. A retained zero can prolong an apparent outage, and a retained one can hide one. To diagnose that version, compare the file’s mtime and contents with the command’s current output:

status_file=/var/lib/node_exporter/textfile_collector/htcondor_execute_nodes.prom
stat -c '%y %n' "$status_file"
cat "$status_file"
condor_status -af Machine
Enter fullscreen mode Exit fullscreen mode

The example queries condor_status directly. It needs neither this execute-node textfile nor the HTCondor exporter on port 9888.

GPU file updates

On the GPU worker, Part 5’s writer refreshes gpu_vram.prom. Node Exporter exposes the modification time of a successfully read file as node_textfile_mtime_seconds. In Prometheus, run this instant query. Use the job label from your scrape configuration in both selectors and in the script below. Node Exporter textfile collector

node_textfile_mtime_seconds{job="node", file=~"(.*/)?gpu_vram[.]prom"}
and on (job, instance) (up{job="node"} == 1)
Enter fullscreen mode Exit fullscreen mode

Use the sample value as the file mtime. In the HTTP API’s [timestamp, "value"] pair, using the first element would make a newly queried old file appear fresh. Use one matching file per target. Prometheus instant-query results

What the monitor receives How it treats that observation
A timestamp at least 900 seconds old The file already meets the alert threshold on this poll.
A successful vector with no row for an expected worker Start or continue the absence interval. A failed scrape, a removed target, or an unreadable file can all produce this result.
A failed API request, a warning, or an invalid result structure Mark the GPU query unknown for this poll. The execute-node check still has its own result.
Duplicate rows or an invalid timestamp for an identifiable instance Mark only that worker unknown and log the problem. Other valid rows can still be evaluated.

A zero-byte free-VRAM reading is valid. The check follows the file’s update time. A current file can omit a card, so use the UUIDs from Part 5 to investigate an individual GPU.

Decide when to alert and when to recover

The lab guide uses 15 minutes for failure and 10 minutes for recovery. Its GPU check treats any file younger than 15 minutes as healthy. A writer could update once at 14:00, stop immediately, and still satisfy that test at 14:10. The example adds a tighter freshness requirement and checks for another write.

Transition Rule in the example
Execute node absent, or GPU file timestamp missing Alert after 15 minutes of observed absence. Minute 0 is the first omission in a successful query.
GPU file present but stale Alert on the first valid observation with file age ≥ 15 minutes. For example, a 14:00 mtime qualifies at 14:16 even if 14:16 is the first poll.
Execute node present again Recover after 10 minutes of valid polls that see the expected machine name.
GPU file updating again Recover after 10 minutes during which every observed file age is 0–120 seconds and the final mtime is later than the first.

For either check, an unknown observation, a backward step in the poll clock, or a gap greater than three minutes breaks the pending time window. An open incident stays open. A single fresh write can start the recovery window. If the file ages past two minutes, that window resets. Keep the monitoring host, Prometheus host, and worker clocks synchronized. The example treats future-dated files as unknown.

The display and notification thresholds serve different purposes. Part 5 hides VRAM at 90 seconds. At 100 seconds, the graph can already show a gap while the file still qualifies for this recovery check. 15 minutes controls when a stale file warrants a Slack alert.

Set up the monitoring host

The example uses Python 3’s standard library on Linux. For both checks, use an account that can query the intended HTCondor pool and reach the Prometheus API. By default the program uses http://127.0.0.1:9090; change the endpoint in the script if Prometheus is elsewhere. Grafana is optional. Run these checks under the same account that will own the timer:

set -o pipefail
condor_status -af Machine | sort -u
curl -fsSG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=node_textfile_mtime_seconds{job="node",file=~"(.*/)?gpu_vram[.]prom"} and on(job,instance) (up{job="node"} == 1)'
mkdir -p ~/.local/bin ~/.config/systemd/user
install -d -m 700 ~/.config/htcondor-watch
touch ~/.config/htcondor-watch/{expected_nodes,gpu_instances}
Enter fullscreen mode Exit fullscreen mode

In expected_nodes, enter one HTCondor machine name per line. In gpu_instances, enter one Prometheus instance per line. Use the exact names in your inventory, keeping the port on GPU instances. An empty file disables that check. Both files must exist. Blank lines and lines beginning with # are ignored.

For delivery, enable Incoming Webhooks on your Slack app, choose Add New Webhook to Workspace, and authorize the intended channel. Save the URL as the only line of the file below. The URL contains a secret. Slack incoming webhooks

(umask 077 && touch ~/.config/htcondor-watch/slack_webhook)
chmod 600 ~/.config/htcondor-watch/slack_webhook
nano ~/.config/htcondor-watch/slack_webhook
Enter fullscreen mode Exit fullscreen mode

Monitoring script · save as ~/.local/bin/watch_pool.py
import fcntl
import json
import math
import os
import subprocess
import sys
import time
from datetime import datetime, timezone
from http.client import HTTPException
from pathlib import Path
from urllib import error, parse, request

CONFIG = Path.home() / ".config/htcondor-watch"
DATA = Path.home() / ".local/state/htcondor-watch"
FAIL, RECOVER, MAX_GAP, FRESH = 900, 600, 180, 120  # seconds
QUERY = ('node_textfile_mtime_seconds{job="node",file=~"(.*/)?gpu_vram[.]prom"} '
         'and on (job, instance) (up{job="node"} == 1)')


def inventory(filename):
    lines = (CONFIG / filename).read_text().splitlines()
    return {s.strip() for s in lines if s.strip() and not s.lstrip().startswith("#")}


def machines():
    result = subprocess.run(["condor_status", "-af", "Machine"], check=True,
                            capture_output=True, text=True, timeout=15)
    return set(result.stdout.split())


def gpu_times():
    url = "http://127.0.0.1:9090/api/v1/query?" + parse.urlencode({"query": QUERY})
    with request.urlopen(url, timeout=10) as response:
        reply = json.load(response)
    if (not isinstance(reply, dict) or reply.get("status") != "success"
            or reply.get("warnings")):
        raise ValueError("Prometheus returned an error or warning")
    data = reply["data"]
    if data["resultType"] != "vector" or not isinstance(data["result"], list):
        raise ValueError("expected a Prometheus instant vector")

    times = {}
    for row in data["result"]:
        name = row["metric"]["instance"]
        if not isinstance(name, str) or not name:
            raise ValueError("missing instance label")
        sample = row.get("value")
        if (name in times or not isinstance(sample, list) or len(sample) != 2
                or not isinstance(sample[1], str)):
            times[name] = None
            continue
        try:
            times[name] = float(sample[1])
        except (TypeError, ValueError, OverflowError):
            times[name] = None
    return times


def read_source(fetch):
    try:
        return fetch()
    except (OSError, HTTPException, ValueError, KeyError, TypeError, subprocess.SubprocessError) as exc:
        print(f"{fetch.__name__}: {exc}", file=sys.stderr)
        return None


def gpu_status(times, name, now):
    if times is None:
        return "unknown", None
    if name not in times:
        return "missing", None
    stamp = times[name]
    if stamp is None or not math.isfinite(stamp) or not 0 < stamp <= now:
        print(f"{name}: ambiguous or invalid VRAM file timestamp", file=sys.stderr)
        return "unknown", None
    age = now - stamp
    if age >= FAIL:
        return "stale", stamp
    return ("fresh" if age <= FRESH else "aging"), stamp


def transition(rec, now, status, stamp=None):
    previous = rec.get("last")
    rec["last"] = now
    if status == "unknown" or previous is None or not 0 <= now - previous <= MAX_GAP:
        rec.update(bad=None, good=None, first_stamp=None)
    if status == "unknown":
        return None

    if status in ("missing", "stale"):
        rec.update(good=None, first_stamp=None)
        if rec.get("bad") is None:
            rec["bad"] = now
        if not rec.get("alerted") and (status == "stale" or now - rec["bad"] >= FAIL):
            rec["alerted"] = True
            return "alert"
        return None

    rec["bad"] = None
    if not rec.get("alerted") or status == "aging":
        rec.update(good=None, first_stamp=None)
        return None
    if rec.get("good") is None:
        rec.update(good=now, first_stamp=stamp)
    elif now - rec["good"] >= RECOVER and (stamp is None or stamp > rec["first_stamp"]):
        rec.update(alerted=False, good=None, first_stamp=None)
        return "recover"
    return None


def save(db):
    tmp = DATA / "state.tmp"
    tmp.write_text(json.dumps(db, allow_nan=False))
    tmp.replace(DATA / "state.json")


def notify(lines):
    secret = (CONFIG / "slack_webhook").read_text().strip()
    payload = json.dumps({"text": "\n".join(lines)}).encode()
    req = request.Request(secret, data=payload, headers={"Content-Type": "application/json"})
    with request.urlopen(req, timeout=10) as reply:
        if reply.status != 200 or reply.read().strip() != b"ok":
            raise ValueError("unexpected Slack acknowledgment")


def deliver(db):
    pending = db["pending"]
    if not pending or time.time() < db.get("retry_after", 0):
        return
    batch = pending[:10]
    try:
        notify(batch)
    except error.HTTPError as exc:
        if exc.code == 429:
            delay = exc.headers.get("Retry-After", "")
            seconds = int(delay) if delay.isascii() and delay.isdigit() else 60
            db["retry_after"] = time.time() + seconds
            save(db)
        raise SystemExit(f"Slack HTTP {exc.code}; pending events retained") from None
    except (OSError, HTTPException, ValueError) as exc:
        raise SystemExit(f"Slack delivery unconfirmed ({type(exc).__name__})") from None
    del pending[:len(batch)]
    db.pop("retry_after", None)
    save(db)


def poll():
    path = DATA / "state.json"
    db = json.loads(path.read_text()) if path.exists() else {"targets": {}}
    records = db["targets"]
    pending = db.setdefault("pending", [])
    nodes, gpus = inventory("expected_nodes"), inventory("gpu_instances")
    observations = {}
    if nodes:
        visible = read_source(machines)
        now = time.time()
        for name in nodes:
            status = "unknown" if visible is None else ("present" if name in visible else "missing")
            observations["node:" + name] = (now, status, None)
    if gpus:
        times = read_source(gpu_times)
        now = time.time()
        for name in gpus:
            status, stamp = gpu_status(times, name, now)
            observations["gpu:" + name] = (now, status, stamp)

    for retired in records.keys() - observations.keys():
        del records[retired]
    for key, (observed, status, stamp) in observations.items():
        action = transition(records.setdefault(key, {}), observed, status, stamp)
        if action:
            utc = datetime.fromtimestamp(observed, timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ")
            pending.append(f"{utc}  {action.upper()}  {key}  ({status})")
    save(db)
    deliver(db)


if __name__ == "__main__":
    os.umask(0o077)
    DATA.mkdir(mode=0o700, parents=True, exist_ok=True)
    with (DATA / "monitor.lock").open("a") as lock:
        fcntl.flock(lock, fcntl.LOCK_EX)
        poll()
Enter fullscreen mode Exit fullscreen mode

What happens when Slack is unavailable?

Each poll stores the incident state and any newly detected transitions together in ~/.local/state/htcondor-watch/state.json before it tries the webhook. Each message line carries the UTC detection time. If an alert and a recovery both occur during a delivery outage, the monitor retains both in order. It sends up to ten events per poll and removes that batch only after an HTTP 200 response with the body ok. HTTP 429 defers the next send according to Retry-After; polling continues in the meantime. Slack webhook responses · Slack rate limits

The file lock serializes overlapping invocations. A timeout after Slack accepted a message, or a crash before the local acknowledgment is saved, can still cause a duplicate. Deleting the state file discards open incidents and unsent messages. Removing a target from the inventory retires it on the next poll. Any transitions already waiting for delivery retain their original timestamps.

Run the local exercise

Save this as exercise.py beside watch_pool.py, then run python3 ~/.local/bin/exercise.py. It calls the same transition code with one simulated poll per minute, using an in-memory record and no webhook.

from watch_pool import gpu_status, transition

t0 = 1_700_000_000
worker = "worker:9100"
record = {}

def check(minute, file_minute):
    now = t0 + minute * 60
    readings = {} if file_minute is None else {worker: t0 + file_minute * 60}
    status, stamp = gpu_status(readings, worker, now)
    action = transition(record, now, status, stamp)
    if action:
        print(f"{minute:2d} min  {action}  {status}")

for m in range(16):
    check(m, None)       # No file timestamp from minute 0 through minute 15.
for m in range(16, 27):
    check(m, 16)         # A single write at minute 16.
for m in range(27, 38):
    check(m, m)          # A new write at every poll from minute 27 onward.
Enter fullscreen mode Exit fullscreen mode

The local exercise produces two lines:

15 min  alert  missing
37 min  recover  fresh
Enter fullscreen mode Exit fullscreen mode

The monitor alerts at minute 15. A single GPU file update at minute 16 leaves the incident open. Updates resume every minute at minute 27, and the monitor records recovery at minute 37.

With these same synthetic observations, the evaluation function from the lab guide qualified for recovery at minute 26 after the single write at minute 16. In this example that write is already three minutes old by minute 19, so the recovery window resets. Minute 27 starts a new window. At minute 37 it is ten minutes long and the mtime has advanced.

Schedule one poll per minute

Save these units under ~/.config/systemd/user/ on the monitoring host.

htcondor-watch.service

[Unit]
Description=Check HTCondor advertisements and GPU metrics

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 %h/.local/bin/watch_pool.py
Enter fullscreen mode Exit fullscreen mode

htcondor-watch.timer

[Unit]
Description=Poll HTCondor and GPU monitoring every minute

[Timer]
OnStartupSec=30s
OnUnitActiveSec=1min
AccuracySec=1s

[Install]
WantedBy=timers.target
Enter fullscreen mode Exit fullscreen mode
systemctl --user daemon-reload
systemctl --user start htcondor-watch.service
systemctl --user enable --now htcondor-watch.timer
systemctl --user list-timers htcondor-watch.timer
journalctl --user -u htcondor-watch.service -n 30 --no-pager
Enter fullscreen mode Exit fullscreen mode

The user service inherits its manager’s environment. If a shell query succeeds but the timer logs errors, check the account’s HTCondor configuration, credentials, and PATH. To run from boot and after logout, enable lingering for the service account with loginctl enable-linger "$USER", obtaining administrator authorization if required. If the monitoring host stops, it cannot report its own outage to Slack. systemd timers · User lingering

For a delivery test, use a test-channel webhook and a separate test inventory and state (a dedicated test account is sufficient). Add htcondor-alert-test.invalid to its execute-node list. After 15 minutes of successful polls that omit it, expect one alert event. Removing that name retires the target. Use the local exercise above to check the recovery path without interrupting a real workstation.

Follow up on a notification

For an execute-node alert, repeat condor_status -af Machine | sort -u on the monitoring host. Check the exact advertised name, then pool selection, connectivity, and the worker’s condor_startd. The alert establishes that the name was missing from the monitored view.

For a message ending in ALERT gpu:192.0.2.10:9100 (missing), look up that target on Prometheus’s Targets page. Then check the GPU worker locally:

nvidia-smi --query-gpu=uuid,memory.free --format=csv
systemctl status gpu-vram-export.timer --no-pager
journalctl -u gpu-vram-export.service -n 20 --no-pager
curl -fsS http://localhost:9100/metrics |
  grep -E '^gpu_vram_free_bytes|^node_textfile_mtime_seconds.*gpu_vram[.]prom|^node_textfile_scrape_error'
Enter fullscreen mode Exit fullscreen mode

If the driver responds but the file mtime is old, inspect the writer and timer. If local readings are fresh but the series is absent in Prometheus, compare the scrape error and the job, instance, and file labels with the query and inventory. A GPU recovery message establishes that this worker’s file resumed updating. Check the per-UUID rows to see which cards contributed readings.

AI assistance was used in drafting and editing this article.

Top comments (2)

Collapse
 
holdalarms profile image
HoldAlarms •

Nice detail on separating "stale" from "query failed". That's where a lot of setups go wrong: a panel showing the last known value looks healthy until someone notices the timestamp.

One question: does the systemd timer itself have a heartbeat? If the timer or the host stops, the checks stop with it, and silence looks the same as all clear. A small ping to something off the box on every successful run, with an alert when it's late, covers that case.

Collapse
 
nha6ki profile image
Haruki Niwa •

Thanks for raising this. The example in the article doesn’t include an external heartbeat. If the timer or monitoring host stops, the monitoring script cannot report that failure.

Sending a heartbeat to an independent service after each completed run would address that gap, with the receiver alerting when a heartbeat is overdue. The script can complete with an unknown observation after a query failure, so a heartbeat would confirm that a run completed. Query failures would still need separate reporting.