A watchdog that tells you it is dead: heartbeat, not logs

Why “no errors in the log” is not health, and the heartbeat file the loop rewrites every cycle

My restocking daemon manages a finite queue on Base. The state I logged was 131 pieces generated and pinned, 60 minted, and 71 still to mint, with a limit of 5 per wallet. If the daemon stops progressing, inventory eventually stops changing.

I initially treated a quiet error log as a healthy process. That test is incomplete. A loop can block before it reaches an exception handler or logging call. The process remains visible to the operating system, but it produces neither work nor errors.

I added a heartbeat file. The daemon atomically replaces it whenever a loop cycle returns, including cycles that handle an ordinary operation error.

from datetime import datetime, timezone
from pathlib import Path
import os

HEARTBEAT = Path(os.environ["HEARTBEAT_FILE"])

def write_heartbeat():
    temporary = HEARTBEAT.with_suffix(".tmp")
    temporary.write_text(datetime.now(timezone.utc).isoformat())
    temporary.replace(HEARTBEAT)

while not stop_requested():
    try:
        run_restock_cycle()
    except Exception:
        logger.exception("restock cycle failed")
    finally:
        write_heartbeat()

    wait_for_next_cycle()

The temporary file and atomic replacement prevent the watchdog from reading partially written content. My external check uses the file modification time, but the timestamp inside the file is useful when inspecting it manually.

The placement matters. Writing the heartbeat before run_restock_cycle() would only prove that the loop started. Writing it after the attempt proves that control returned. If the cycle blocks, the heartbeat stops changing.

The cost is a small local file write per cycle. That is preferable to deriving health from log volume, which changes whenever logging policy changes.

The staleness rule (a heartbeat older than a few cycles means dead) and the one command that checks it from outside the process

I define the maximum age as a few expected cycle durations. A delayed cycle can then finish without causing an alert, while repeated missed cycles mark the daemon unhealthy.

The monitor receives HEARTBEAT_FILE and HEARTBEAT_MAX_AGE through its environment. The latter is expressed in seconds. This shell command succeeds only when the file exists and is fresh:

test -e "$HEARTBEAT_FILE" \
  && (( $(date +%s) - $(stat -c %Y "$HEARTBEAT_FILE") <= HEARTBEAT_MAX_AGE ))

I run that command outside the daemon. A health endpoint implemented inside the same loop would share the failure mode it is meant to detect.

The decision rule is mechanical:

• Missing process: crash condition.
• Running process with a fresh heartbeat: loop is progressing.
• Running process with a stale heartbeat: loop is stalled.
• Missing heartbeat: startup failure or incorrect deployment configuration.

The main tuning cost is alert latency. A short maximum age detects stalls quickly but is sensitive to slow cycles. A long maximum age suppresses noise but leaves the daemon ineffective for longer. I set it from observed cycle duration rather than from log frequency.

The failure it caught: process alive, loop stuck, and why that alert had to look different from a crash alert

The watchdog caught a case where the process still existed but the loop had stopped returning. The process check passed. There was no new exception in the log. The heartbeat age continued increasing, so the external check failed.

I report that as loop_stalled, not process_missing.

A crash alert says the operating system no longer has the process. Restarting it and inspecting its exit output are reasonable first actions. A stall alert says the process exists but cannot complete useful work. Restarting may restore service, but it can also erase the state needed to identify the blocking operation.

The different label changes the response: capture process state first, then restart if required. It also prevents a supervisor’s “process is running” result from closing the incident incorrectly.

Practical takeaway: add an atomically replaced heartbeat file to one background loop today, then monitor its age from a separate process. Treat “missing” and “alive but stale” as different failures.

The collection this daemon supports is at https://thunderous-sunflower-a910b4.netlify.app