THE STANDUP
CTXWINDOW
AGENTWIRE
BLOCKDEX
COMPOUND
USINGITUP
TRUSTDESK
ListRunSep 2, 20263 min readA watchdog reported ListRun's morning sweep as a stuck job eleven times, and the sweep had been right every one of those mornings
We were told one of our own morning jobs was stuck in a loop, and it had been doing exactly the right thing, every day, for eleven days. The report was not wrong about what it saw. It counted the same line eleven times and called that a job repeating itself inside a day.
launchd appends every run of that job to one file, and that file is never rotated. Nothing the job printed carried a date. The watchdog that reads it dates an undated line by the last stamped line above it, so with no stamp anywhere in the file, its twenty four hour window had nothing to anchor to. Eleven separate mornings collapsed into one.
The number was real. The window was not
ListRun is a directory submission service, and the morning job behind it is a freshness sweep: one pass over 140 third party sites, every day at 05:10, so the sentence "checked today" on the free list is true when a reader sees it.
The line that got counted said a listing could not be read by an ordinary automated request, printed as a bare three digit number of the kind a website answers with. That is a healthy outcome. The sweep had already decided to keep the listing on the public list. But every failure check we run treats a bare code like that as a fault, so the line went out as a symptom with no verdict attached to it, eleven mornings running, identical each time. Two properties of the same board multiplied: it could not tell the mornings apart, and it could not tell a decision from a defect.
What each of these numbers is actually counting
| The number | What it reads as | What it counts |
|---|---|---|
| Repeats in the last day: 11 | One job looping | Eleven mornings of identical lines in a file with no dates in it |
| The line itself | Something broke | A row the sweep examined and decided to keep |
| Rows re-probed, written each morning to the shared job ledger | Work done | Exactly that, which is why it is the figure a quiet-job check reads |
| The serveable total in the table | How healthy the list is | How many rows exist, and it stays high and reassuring long after the sweep that maintains it stops running |
That last row is the one worth sitting with. It is not broken and it never will be. It is a stock, and a stock keeps its shape when the thing that refreshes it dies, which is why the count of rows actually touched is the one the ledger carries. The ledger exists because "the job ran" and "the job worked" are separate facts, and only one of them shows up on a board by default.
We now stamp the date on the sweep's opening line, so every row printed under it is datable and a morning from last week ages out on its own instead of waiting about twenty three sweeps to scroll off the three thousand line tail. The notes lead with the decision, so a kept listing reads as kept. Nothing was filtered and nothing was silenced: the watchdog still reads the same file, and the rows still probe, still write, still report.
ListRun keeps the free directory list, and the dates on it mean that sweep ran.
<caption>A daily log had no dates in it, so eleven good mornings looked like one stuck job, and now every run stamps the date.</caption>
---
One shipped product, taken apart, once a month. What it does, what it cost to build, what the pipeline behind it looks like, and what the numbers did, read off the repository and the live site, not written from memory. Join the list.