How to make static analysis usable on a decade old codebase
In November I spent a month inside a codebase whose first commit is from 2016. Twenty three thousand commits, fifteen authors over the years, a framework three major versions behind, and a business running on it right now.
Step one was obvious: point static analysis at it. It produced about six thousand findings.
Six thousand findings is the same as zero findings. Nobody reads it, nobody acts on it, and the tool gets switched off within a fortnight. Here is what actually worked.
Two configurations, not one
The single most useful decision was splitting the analysis in two.
The gate. Permissive, achievable, runs in the pipeline, and fails only on things that are genuinely new. It is green today and must stay green. Everything historical is suppressed, by category.
The survey. The same analysis with the two biggest categories un-suppressed. It is never green and is not supposed to be. Nobody runs it in the pipeline. You open it when you are deciding what to touch next.
A gate you can pass and a measurement you cannot are different tools with different jobs. Trying to make one file do both is the reason static analysis gets abandoned on old code, every time.
Make the suppression file an inventory
This is where the value is. Every suppressed category in my configuration carries a count and a verdict about whether it can break at runtime:
- around fifteen hundred undefined properties
- a thousand undefined methods
- eight hundred and sixty six conditions that are always true or always false
- six hundred lines of unreachable code
- five hundred lifecycle hooks the analyser cannot see
- nearly five hundred malformed documentation blocks
- two hundred and thirty one references to classes that no longer exist
That last line is the one that mattered. A reference to a class that does not exist is not a style opinion. It is a line that throws the moment it runs, and the only reason it has not is that nobody has taken that path in years.
Six thousand findings sorted into "will throw", "might throw" and "offends the analyser" is a plan. The same six thousand in one list is a reason to give up.
Where to point the agents and where not to
Delegate the mechanical half. Fifty three of my commits that month are batched corrections: patterned, tedious, individually verifiable. That is exactly the work to hand over, and it goes fast.
Do not delegate the triage. Deciding whether a finding is cosmetic or lethal requires knowing what the code does in production, which paths are hot, which module was rewritten in 2019 and which has not been touched since its author left. None of that is in the repository. It is in the shape of the data and in the heads of the people who were there.
The deliverable is the map
The month did not end with a clean codebase. It ended with a document that says which of six thousand complaints can actually hurt you, ranked, with counts.
That turned out to be worth far more than fixed code would have been, because the upgrade that follows is a series of decisions about risk, and now there is something to decide against.
Do not try to get the number to zero. You will not, and chasing it will burn the goodwill you need for the real work. Get the number sorted instead.