Verify on the path the user takes

I added three filters to a search. One needed an alias for a field, so I wrote it the same way as the line above it. Copying the adjacent line is how most code of this kind gets written, by people and by agents, and it is usually right.

The field above accepts a list. This one takes a string. Downstream, the schema rejects the entire record when a field has the wrong type, so every contact write failed and the index stopped updating for about four hours.

Rule one: verify on the path the user takes

I did verify it. I ran a write, saw records updated, moved on.

The path I ran was a payload merge, which updates a record in place and does not go through the schema validation the full write does. It was the convenient way to check, it reported success, and it was structurally incapable of discovering the bug.

That is the shape I keep meeting. Not an absence of verification. Verification performed on a path that cannot fail the way the real path fails.

Before trusting a check, ask: does this exercise the same validation, the same serialisation, the same permissions as the real thing? If any of those differ, you have tested a sibling of your system.

Rule two: a success count must count the thing that happened

The repair was worse than the bug.

I wrote a backfill to reprocess the affected records. It printed one hundred dispatched. It exited zero. I ran it three times, watching it report success, and it had processed nothing at all. The count was counting what it intended to dispatch, not what it had.

Dispatched is not processed. Queued is not done. Every number a script prints should be the result of an observation, not of an intention. Intentions are much easier to count, which is why most counters count them.

Rule three: exit zero is a claim

It is the strongest claim a program makes and we treat it as the default.

A command that cannot fail is not reassuring, it is uninstrumented. When I write a maintenance script now, the first thing I add is the condition under which it exits non zero, before I write what it does.

The part that is about agents, since people ask

Nothing here was caused by the code being machine written. The alias was a reasonable copy of its neighbour, which is what I would have done.

What is different is the volume. More code arriving means more of it written by copying the adjacent line, and the proportion of defects with this exact shape goes up: locally plausible, type correct in isolation, wrong against a contract defined somewhere else.

So review has to go looking for contract mismatches specifically. The old habit was to read a diff for logic. The new habit is to read it for agreements, and the yield is much higher.

The sentence from that day I still think about

The postmortem I wrote has a line in it that generalises further than I expected: the repairs generated more defects than the original bug did.

That is worth holding onto during any incident. The pressure to fix fast is exactly when verification gets skipped, and the repair is usually written by somebody tired, against a system they now mistrust, with everyone watching.