16 August 2026 · monitoring

Count the silence: three green monitors that were lying

Three scheduled jobs reported success for weeks while doing nothing at all. In none of the three cases was the bug in the job. It was in what the monitor counted.

I run an autonomous agent that keeps a few dozen scheduled jobs alive — crawlers, watchers, a forecasting bot that has to submit a prediction inside a three-hour window or lose the question forever. Losing a window costs real money, so the jobs are monitored. They were all green.

Here are three that were green and wrong, what each one taught, and the twenty lines of Python I now put in front of anything that runs on a timer.

1. The tick that meant nothing

A GitHub Actions workflow ran every thirty minutes for weeks and passed every time. It had never once done its job. The forecasting step needed an API token that was documented as a required secret but only ever lived in a stale bundle, so the step hit a missing-credential branch that ended in exit 0:

if [ -z "$METACULUS_TOKEN" ]; then
  echo "no token configured, skipping"
  exit 0          # <- this is the bug
fi

An unconfigured bot is a broken bot, but exit 0 says otherwise, and the green tick is precisely what stopped anyone from looking. A sibling workflow was worse. Its social-post step was an unquoted YAML scalar, so a fragment ending in #agents #security" || true had everything from the # stripped as a YAML comment. Bash received an unterminated quote and died on a parse error — early enough that the step's own || true had been stripped too — which killed the three steps below it, including the one that saved state. That workflow had never saved state in its entire history. Underneath sat a second bug that could never surface: the job had no permissions: contents: write, so its push could only ever have returned 403, and the trailing || true would have swallowed that as well.

Three separate mechanisms — exit 0, || true, continue-on-error — each of which converts a dead job into a silent one. The tell was runtime, not status: a healthy forecasting run does LLM work for minutes, and the broken one crashed in about thirty seconds. Nothing was watching runtime.

2. The schedule that quietly halved itself

The cloud job was the backup. The primary was a Windows scheduled task firing every fifteen minutes. I hardened its settings one afternoon — and hardening them rebuilt the trigger. StopAtDurationEnd came back as true against an empty repetition duration, and the fifteen-minute cadence silently became roughly ninety. Five forecasting windows were skipped.

Nothing reported it. The task's State read Ready. Every setting read back exactly as configured. Every run that did fire returned 0. A broken schedule and a healthy one are identical from every angle except the runs that never happened — and nothing was counting those.

It was noticed only because the append-only log had stopped growing. That is the lesson in one line: the artefact that goes quiet when runs stop is the only honest sensor you have, whatever the cause — bad trigger, machine asleep, task disabled, script wedged. Count the silence, not the failures.

3. The alarm that measured one of two triggers

So I wrote a silence monitor. It read the job's log, computed the longest gap between runs in the last 24 hours, and alarmed when that gap exceeded the three-hour question lifetime. It was a real improvement — the metric everyone reaches for first is time since the last run, and that number reads perfectly healthy the moment the machine wakes up. Over 592 logged runs the difference was stark:

A mean of 0.77 h and a median of 0.25 h describe a perfectly healthy scheduler. The tail is what loses questions, and there was a hole on almost every single day.

Then the new alarm went red and stayed red. Every morning: blind spot 9.51 h, any question that opened and closed in that hole is unreachable forever. The measured cause was the desktop sleeping overnight, and the remedy the alarm itself printed was "cover it from the cloud or keep the machine awake."

The cloud cover had shipped weeks earlier. I went and measured it — same 24 hours, from the Actions runs API rather than from the cron expression:

The cloud job runs the same submitting script over the same tournaments. A fire there covers a question exactly the way a local fire does. The last entry in the forecast ledger sat squarely inside the "hole." Nothing had been unreachable. The monitor was measuring one lane of a two-lane road and reporting the empty lane as a closed highway.

That is a specific and repeatable failure, and it is worse than a missing alarm. A daily false red on the one thing guarding your most valuable process trains you to scroll past it.

When two independent mechanisms satisfy the same duty, a metric over one of them is not a weaker version of the truth. It is a different quantity — and the label on it ("coverage lost") is a claim about the union.

The unit bug hiding inside the fix

Merging the two timelines has a trap in it. The local log is written by PowerShell's Get-Date -Format s, which is naive local time. The Actions API returns UTC. Concatenating them raw would have manufactured a nine-hour hole that was only a timezone — and it would have looked exactly like the bug I was trying to fix. In Python, a naive datetime's .astimezone() attaches the local zone, which is the one-call fix:

local = [datetime.fromisoformat(line.split()[0]).astimezone() for line in log]
cloud = [datetime.fromisoformat(r["run_started_at"].replace("Z", "+00:00"))
         for r in runs]
merged = sorted(local + cloud)          # now they are the same quantity

The watchdog

This is the whole pattern. It takes timestamps from every mechanism that satisfies the duty, and returns the longest silence — including the open-ended one that ends at now, because a scheduler that died an hour ago has no closing timestamp and would otherwise report its last healthy gap forever.

from datetime import datetime, timedelta, timezone

def max_gap_h(stamps, hours=24):
    """Longest silence in the last `hours`, across all triggers. None if unmeasurable."""
    now = datetime.now(timezone.utc)
    stamps = sorted(stamps)                       # all tz-aware
    if len(stamps) < 2:
        return None
    cut = now - timedelta(hours=hours)
    recent = [t for t in stamps if t >= cut]
    before = [t for t in stamps if t < cut]
    if before:
        recent.insert(0, before[-1])              # anchor, or a silence that
    if len(recent) < 2:                           # starts before the cutoff
        return None                               # is invisible
    gaps = [(b - a).total_seconds() / 3600
            for a, b in zip(recent, recent[1:])]
    gaps.append((now - recent[-1]).total_seconds() / 3600)   # the open gap
    return round(max(gaps), 2)

Two details earn their lines. The anchor: without the last run from before the window, a silence that starts before the cutoff simply is not in the data — which is exactly the overnight hole the function exists to catch. And the open gap: without it, a dead scheduler reports the last healthy interval it managed.

Then alarm against the thing you actually lose, not against a round number:

WINDOW_H = 3.0        # a question is open for exactly this long

gap = max_gap_h(local_stamps + cloud_stamps)
if gap and gap > WINDOW_H:
    alert(f"blind spot {gap}h across BOTH triggers — check Actions quota "
          f"and the local task")

Three questions for any monitor

1. Does a green result require work to have happened? If the job can pass by skipping, it will, and you will learn nothing for weeks. Make the unconfigured case fail loudly. Track runtime, not just status.
2. Am I watching the mean or the tail? If the thing you lose has a deadline, the mean cannot see the loss. Alarm on the max gap against that deadline.
3. Does my denominator cover every mechanism that does this job? List them before you write the query. If two things can satisfy the duty, a metric over one is a different quantity with a misleading name.

The last one is the general form of all three. Every failure above was a monitor whose number was accurate and whose label was a claim about something it had never measured. Green is not a fact about your system. It is a fact about your query.

All numbers here are measured from the repository this describes — a long-running autonomous agent operation. The watchdog above is the one now running.