Catalin Bozianu

Reproduce it in the artefact you ship

A packaged build of a desktop client was crashing on startup. The error reporting library was missing from the bundle, and the code that initialises it assumed it was there.

The first fix guarded against the library being absent. That fix is correct and it is treating the symptom, which is easy to miss because the crash stops and a stopped crash is extremely persuasive.

The real question was why a declared dependency was not in the package. Getting that right took three more versions: one to contain the transitive dependencies properly, one to revert that because the override excluded files that needed to be there, and one to do it correctly.

Four version numbers in an afternoon, on a product where a version number is not a deploy.

The rule

A packaging bug must be reproduced in a packaged build before the fix is merged. Not in development. Development is precisely the environment where the bug does not exist, because that is the definition of a packaging bug.

This sounds obvious written down. In practice the development loop is seconds and the packaged loop is minutes, so everyone verifies in the fast one and discovers the truth in the slow one, after shipping.

The same rule generalises: reproduce in the artefact you ship. A container, a bundle, a minified build, a lambda package. Any step between your source and what the user runs can create defects, and it is the only place they are visible.

Why the stakes are different outside the web

When a web deploy is wrong you roll back and it is over in ninety seconds. Mostly nobody outside the team knows.

A desktop release does not work that way. Every bad build you publish is a build somebody is now running, and your only remedy is to publish another and wait for it to be picked up. The blast radius is not the next ninety seconds, it is however long it takes your users to restart.

Which is why the worst bug that month was not a crash at all. The updater reported that it had installed and then did nothing, because a background tray handle kept the process alive and the installer had nothing to replace. Users saw a success message and the old version.

An updater that reports success without verifying the swap is worse than one that fails. A failure gets reported to you. A false success gets reported to nobody and quietly splits your user base across versions.

The second rule

When a fix stops a symptom without explaining the cause, say so in the pull request in plain words.

Not as a confession. As a note that the ticket is not finished, so the next person, machine or otherwise, does not close it. Half the repeat incidents I have seen came from a symptom fix inheriting the credibility of a root cause fix, purely because nobody wrote down which one it was.

Why the gate before a release exists

Not ceremony, and not distrust of whoever wrote the code. It exists because the cost of being wrong is not symmetric. Most changes are cheap to get wrong. A few are not, and those few are rarely the ones that look risky in the diff.

An agent cannot make that call, because the call is not about the code. It is about what happens outside the code if the code is wrong.

Back to the diary