Part 6 of 6 in Building and Operating an HTCondor Pool for Research Computing.
An HTCondor worker can disappear from the pool. A GPU worker can stay online while its VRAM collector stops writing. In the second case, Node Exporter may keep serving the same value on every scrape. Our lab guide routes both alerts to one Slack channel and identifies the worker and the failed check in each message.
This walkthrough starts with the pool from Part 1 and the GPU textfile writer from Part 5. On the monitoring host, the example queries HTCondor and Prometheus once per poll. A local exercise reproduces the alert and recovery decisions without contacting the pool or Slack.
Choose what to monitor
Keep two inventories on the monitoring host. Populate them from the machines you expect to serve and keep them across polls.
| Inventory | Identifier | What one entry covers |
|---|---|---|
| Execute nodes | Exact HTCondor Machine name, such as worker.example.org
|
All visible slot ads for that machine. One remaining ad is enough for the presence check. |
| GPU workers | Exact Prometheus instance label, such as 192.0.2.10:9100
|
The worker’s gpu_vram.prom file. The file can hold readings for several GPU UUIDs. |
The lab guide adds newly observed targets every ten minutes. A worker that has never reported needs a manual entry. When retiring a target, stop its advertisement or scrape registration first so discovery cannot add it back. The example below uses two manually maintained files. Using only the latest query result as the inventory would silently drop the worker you want to detect.
Execute nodes
Run condor_status -af Machine | sort -u on the monitoring host. If a successful query omits an expected name, start counting at the first poll that observes the absence. Collector ads can outlive a lost connection. An alert can therefore arrive more than 15 minutes after the workstation lost power. A nonzero exit or timeout makes the observation unknown. HTCondor ClassAd lifetime
The lab implementation has an extra step. A one-minute writer exports htcondor_execute_node_up and retains the last good textfile if condor_status fails. The lab notifier reads that file without checking its age. A retained zero can prolong an apparent outage, and a retained one can hide one. To diagnose that version, compare the file’s mtime and contents with the command’s current output:
status_file=/var/lib/node_exporter/textfile_collector/htcondor_execute_nodes.prom
stat -c '%y %n' "$status_file"
cat "$status_file"
condor_status -af Machine
The example queries condor_status directly. It needs neither this execute-node textfile nor the HTCondor exporter on port 9888.
GPU file updates
On the GPU worker, Part 5’s writer refreshes gpu_vram.prom. Node Exporter exposes the modification time of a successfully read file as node_textfile_mtime_seconds. In Prometheus, run this instant query. Use the job label from your scrape configuration in both selectors and in the script below. Node Exporter textfile collector
node_textfile_mtime_seconds{job="node", file=~"(.*/)?gpu_vram[.]prom"}
and on (job, instance) (up{job="node"} == 1)
Use the sample value as the file mtime. In the HTTP API’s [timestamp, "value"] pair, using the first element would make a newly queried old file appear fresh. Use one matching file per target. Prometheus instant-query results
| What the monitor receives | How it treats that observation |
|---|---|
| A timestamp at least 900 seconds old | The file already meets the alert threshold on this poll. |
| A successful vector with no row for an expected worker | Start or continue the absence interval. A failed scrape, a removed target, or an unreadable file can all produce this result. |
| A failed API request, a warning, or an invalid result structure | Mark the GPU query unknown for this poll. The execute-node check still has its own result. |
| Duplicate rows or an invalid timestamp for an identifiable instance | Mark only that worker unknown and log the problem. Other valid rows can still be evaluated. |
A zero-byte free-VRAM reading is valid. The check follows the file’s update time. A current file can omit a card, so use the UUIDs from Part 5 to investigate an individual GPU.
Decide when to alert and when to recover
The lab guide uses 15 minutes for failure and 10 minutes for recovery. Its GPU check treats any file younger than 15 minutes as healthy. A writer could update once at 14:00, stop immediately, and still satisfy that test at 14:10. The example adds a tighter freshness requirement and checks for another write.
| Transition | Rule in the example |
|---|---|
| Execute node absent, or GPU file timestamp missing | Alert after 15 minutes of observed absence. Minute 0 is the first omission in a successful query. |
| GPU file present but stale | Alert on the first valid observation with file age ≥ 15 minutes. For example, a 14:00 mtime qualifies at 14:16 even if 14:16 is the first poll. |
| Execute node present again | Recover after 10 minutes of valid polls that see the expected machine name. |
| GPU file updating again | Recover after 10 minutes during which every observed file age is 0–120 seconds and the final mtime is later than the first. |
For either check, an unknown observation, a backward step in the poll clock, or a gap greater than three minutes breaks the pending time window. An open incident stays open. A single fresh write can start the recovery window. If the file ages past two minutes, that window resets. Keep the monitoring host, Prometheus host, and worker clocks synchronized. The example treats future-dated files as unknown.
The display and notification thresholds serve different purposes. Part 5 hides VRAM at 90 seconds. At 100 seconds, the graph can already show a gap while the file still qualifies for this recovery check. 15 minutes controls when a stale file warrants a Slack alert.
Set up the monitoring host
The example uses Python 3’s standard library on Linux. For both checks, use an account that can query the intended HTCondor pool and reach the Prometheus API. By default the program uses http://127.0.0.1:9090; change the endpoint in the script if Prometheus is elsewhere. Grafana is optional. Run these checks under the same account that will own the timer:
set -o pipefail
condor_status -af Machine | sort -u
curl -fsSG http://127.0.0.1:9090/api/v1/query \
--data-urlencode 'query=node_textfile_mtime_seconds{job="node",file=~"(.*/)?gpu_vram[.]prom"} and on(job,instance) (up{job="node"} == 1)'
mkdir -p ~/.local/bin ~/.config/systemd/user
install -d -m 700 ~/.config/htcondor-watch
touch ~/.config/htcondor-watch/{expected_nodes,gpu_instances}
In expected_nodes, enter one HTCondor machine name per line. In gpu_instances, enter one Prometheus instance per line. Use the exact names in your inventory, keeping the port on GPU instances. An empty file disables that check. Both files must exist. Blank lines and lines beginning with # are ignored.
For delivery, enable Incoming Webhooks on your Slack app, choose Add New Webhook to Workspace, and authorize the intended channel. Save the URL as the only line of the file below. The URL contains a secret. Slack incoming webhooks
(umask 077 && touch ~/.config/htcondor-watch/slack_webhook)
chmod 600 ~/.config/htcondor-watch/slack_webhook
nano ~/.config/htcondor-watch/slack_webhook
Monitoring script · save as ~/.local/bin/watch_pool.py
import fcntl
import json
import math
import os
import subprocess
import sys
import time
from datetime import datetime, timezone
from http.client import HTTPException
from pathlib import Path
from urllib import error, parse, request
CONFIG = Path.home() / ".config/htcondor-watch"
DATA = Path.home() / ".local/state/htcondor-watch"
FAIL, RECOVER, MAX_GAP, FRESH = 900, 600, 180, 120 # seconds
QUERY = ('node_textfile_mtime_seconds{job="node",file=~"(.*/)?gpu_vram[.]prom"} '
'and on (job, instance) (up{job="node"} == 1)')
def inventory(filename):
lines = (CONFIG / filename).read_text().splitlines()
return {s.strip() for s in lines if s.strip() and not s.lstrip().startswith("#")}
def machines():
result = subprocess.run(["condor_status", "-af", "Machine"], check=True,
capture_output=True, text=True, timeout=15)
return set(result.stdout.split())
def gpu_times():
url = "http://127.0.0.1:9090/api/v1/query?" + parse.urlencode({"query": QUERY})
with request.urlopen(url, timeout=10) as response:
reply = json.load(response)
if (not isinstance(reply, dict) or reply.get("status") != "success"
or reply.get("warnings")):
raise ValueError("Prometheus returned an error or warning")
data = reply["data"]
if data["resultType"] != "vector" or not isinstance(data["result"], list):
raise ValueError("expected a Prometheus instant vector")
times = {}
for row in data["result"]:
name = row["metric"]["instance"]
if not isinstance(name, str) or not name:
raise ValueError("missing instance label")
sample = row.get("value")
if (name in times or not isinstance(sample, list) or len(sample) != 2
or not isinstance(sample[1], str)):
times[name] = None
continue
try:
times[name] = float(sample[1])
except (TypeError, ValueError, OverflowError):
times[name] = None
return times
def read_source(fetch):
try:
return fetch()
except (OSError, HTTPException, ValueError, KeyError, TypeError, subprocess.SubprocessError) as exc:
print(f"{fetch.__name__}: {exc}", file=sys.stderr)
return None
def gpu_status(times, name, now):
if times is None:
return "unknown", None
if name not in times:
return "missing", None
stamp = times[name]
if stamp is None or not math.isfinite(stamp) or not 0 < stamp <= now:
print(f"{name}: ambiguous or invalid VRAM file timestamp", file=sys.stderr)
return "unknown", None
age = now - stamp
if age >= FAIL:
return "stale", stamp
return ("fresh" if age <= FRESH else "aging"), stamp
def transition(rec, now, status, stamp=None):
previous = rec.get("last")
rec["last"] = now
if status == "unknown" or previous is None or not 0 <= now - previous <= MAX_GAP:
rec.update(bad=None, good=None, first_stamp=None)
if status == "unknown":
return None
if status in ("missing", "stale"):
rec.update(good=None, first_stamp=None)
if rec.get("bad") is None:
rec["bad"] = now
if not rec.get("alerted") and (status == "stale" or now - rec["bad"] >= FAIL):
rec["alerted"] = True
return "alert"
return None
rec["bad"] = None
if not rec.get("alerted") or status == "aging":
rec.update(good=None, first_stamp=None)
return None
if rec.get("good") is None:
rec.update(good=now, first_stamp=stamp)
elif now - rec["good"] >= RECOVER and (stamp is None or stamp > rec["first_stamp"]):
rec.update(alerted=False, good=None, first_stamp=None)
return "recover"
return None
def save(db):
tmp = DATA / "state.tmp"
tmp.write_text(json.dumps(db, allow_nan=False))
tmp.replace(DATA / "state.json")
def notify(lines):
secret = (CONFIG / "slack_webhook").read_text().strip()
payload = json.dumps({"text": "\n".join(lines)}).encode()
req = request.Request(secret, data=payload, headers={"Content-Type": "application/json"})
with request.urlopen(req, timeout=10) as reply:
if reply.status != 200 or reply.read().strip() != b"ok":
raise ValueError("unexpected Slack acknowledgment")
def deliver(db):
pending = db["pending"]
if not pending or time.time() < db.get("retry_after", 0):
return
batch = pending[:10]
try:
notify(batch)
except error.HTTPError as exc:
if exc.code == 429:
delay = exc.headers.get("Retry-After", "")
seconds = int(delay) if delay.isascii() and delay.isdigit() else 60
db["retry_after"] = time.time() + seconds
save(db)
raise SystemExit(f"Slack HTTP {exc.code}; pending events retained") from None
except (OSError, HTTPException, ValueError) as exc:
raise SystemExit(f"Slack delivery unconfirmed ({type(exc).__name__})") from None
del pending[:len(batch)]
db.pop("retry_after", None)
save(db)
def poll():
path = DATA / "state.json"
db = json.loads(path.read_text()) if path.exists() else {"targets": {}}
records = db["targets"]
pending = db.setdefault("pending", [])
nodes, gpus = inventory("expected_nodes"), inventory("gpu_instances")
observations = {}
if nodes:
visible = read_source(machines)
now = time.time()
for name in nodes:
status = "unknown" if visible is None else ("present" if name in visible else "missing")
observations["node:" + name] = (now, status, None)
if gpus:
times = read_source(gpu_times)
now = time.time()
for name in gpus:
status, stamp = gpu_status(times, name, now)
observations["gpu:" + name] = (now, status, stamp)
for retired in records.keys() - observations.keys():
del records[retired]
for key, (observed, status, stamp) in observations.items():
action = transition(records.setdefault(key, {}), observed, status, stamp)
if action:
utc = datetime.fromtimestamp(observed, timezone.utc).strftime("%Y-%m-%d %H:%M:%SZ")
pending.append(f"{utc} {action.upper()} {key} ({status})")
save(db)
deliver(db)
if __name__ == "__main__":
os.umask(0o077)
DATA.mkdir(mode=0o700, parents=True, exist_ok=True)
with (DATA / "monitor.lock").open("a") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
poll()
What happens when Slack is unavailable?
Each poll stores the incident state and any newly detected transitions together in ~/.local/state/htcondor-watch/state.json before it tries the webhook. Each message line carries the UTC detection time. If an alert and a recovery both occur during a delivery outage, the monitor retains both in order. It sends up to ten events per poll and removes that batch only after an HTTP 200 response with the body ok. HTTP 429 defers the next send according to Retry-After; polling continues in the meantime. Slack webhook responses · Slack rate limits
The file lock serializes overlapping invocations. A timeout after Slack accepted a message, or a crash before the local acknowledgment is saved, can still cause a duplicate. Deleting the state file discards open incidents and unsent messages. Removing a target from the inventory retires it on the next poll. Any transitions already waiting for delivery retain their original timestamps.
Run the local exercise
Save this as exercise.py beside watch_pool.py, then run python3 ~/.local/bin/exercise.py. It calls the same transition code with one simulated poll per minute, using an in-memory record and no webhook.
from watch_pool import gpu_status, transition
t0 = 1_700_000_000
worker = "worker:9100"
record = {}
def check(minute, file_minute):
now = t0 + minute * 60
readings = {} if file_minute is None else {worker: t0 + file_minute * 60}
status, stamp = gpu_status(readings, worker, now)
action = transition(record, now, status, stamp)
if action:
print(f"{minute:2d} min {action} {status}")
for m in range(16):
check(m, None) # No file timestamp from minute 0 through minute 15.
for m in range(16, 27):
check(m, 16) # A single write at minute 16.
for m in range(27, 38):
check(m, m) # A new write at every poll from minute 27 onward.
The local exercise produces two lines:
15 min alert missing
37 min recover fresh
With these same synthetic observations, the evaluation function from the lab guide qualified for recovery at minute 26 after the single write at minute 16. In this example that write is already three minutes old by minute 19, so the recovery window resets. Minute 27 starts a new window. At minute 37 it is ten minutes long and the mtime has advanced.
Schedule one poll per minute
Save these units under ~/.config/systemd/user/ on the monitoring host.
htcondor-watch.service
[Unit]
Description=Check HTCondor advertisements and GPU metrics
[Service]
Type=oneshot
ExecStart=/usr/bin/python3 %h/.local/bin/watch_pool.py
htcondor-watch.timer
[Unit]
Description=Poll HTCondor and GPU monitoring every minute
[Timer]
OnStartupSec=30s
OnUnitActiveSec=1min
AccuracySec=1s
[Install]
WantedBy=timers.target
systemctl --user daemon-reload
systemctl --user start htcondor-watch.service
systemctl --user enable --now htcondor-watch.timer
systemctl --user list-timers htcondor-watch.timer
journalctl --user -u htcondor-watch.service -n 30 --no-pager
The user service inherits its manager’s environment. If a shell query succeeds but the timer logs errors, check the account’s HTCondor configuration, credentials, and PATH. To run from boot and after logout, enable lingering for the service account with loginctl enable-linger "$USER", obtaining administrator authorization if required. If the monitoring host stops, it cannot report its own outage to Slack. systemd timers · User lingering
For a delivery test, use a test-channel webhook and a separate test inventory and state (a dedicated test account is sufficient). Add htcondor-alert-test.invalid to its execute-node list. After 15 minutes of successful polls that omit it, expect one alert event. Removing that name retires the target. Use the local exercise above to check the recovery path without interrupting a real workstation.
Follow up on a notification
For an execute-node alert, repeat condor_status -af Machine | sort -u on the monitoring host. Check the exact advertised name, then pool selection, connectivity, and the worker’s condor_startd. The alert establishes that the name was missing from the monitored view.
For a message ending in ALERT gpu:192.0.2.10:9100 (missing), look up that target on Prometheus’s Targets page. Then check the GPU worker locally:
nvidia-smi --query-gpu=uuid,memory.free --format=csv
systemctl status gpu-vram-export.timer --no-pager
journalctl -u gpu-vram-export.service -n 20 --no-pager
curl -fsS http://localhost:9100/metrics |
grep -E '^gpu_vram_free_bytes|^node_textfile_mtime_seconds.*gpu_vram[.]prom|^node_textfile_scrape_error'
If the driver responds but the file mtime is old, inspect the writer and timer. If local readings are fresh but the series is absent in Prometheus, compare the scrape error and the job, instance, and file labels with the query and inventory. A GPU recovery message establishes that this worker’s file resumed updating. Check the per-UUID rows to see which cards contributed readings.
AI assistance was used in drafting and editing this article.

Top comments (2)
Nice detail on separating "stale" from "query failed". That's where a lot of setups go wrong: a panel showing the last known value looks healthy until someone notices the timestamp.
One question: does the systemd timer itself have a heartbeat? If the timer or the host stops, the checks stop with it, and silence looks the same as all clear. A small ping to something off the box on every successful run, with an alert when it's late, covers that case.
Thanks for raising this. The example in the article doesn’t include an external heartbeat. If the timer or monitoring host stops, the monitoring script cannot report that failure.
Sending a heartbeat to an independent service after each completed run would address that gap, with the receiver alerting when a heartbeat is overdue. The script can complete with an
unknownobservation after a query failure, so a heartbeat would confirm that a run completed. Query failures would still need separate reporting.