Test your gates by making them fail
For about two weeks the main branch of a project I work on was green, every build. In the same two weeks nearly every pull request failed its required end to end check.
Both facts were visible. Neither was noticed, because each looks normal alone. A green trunk looks like health. Failing pull requests look like work in progress.
The cause was one condition in the pipeline: the end to end suite was skipped on the main branch and ran only on branches. Somebody, possibly me, added that to save time when the suite was slow and the trunk was quiet.
Why that is worse than having no suite
A check you do not have is a known gap. You route around it. You test manually before a release because you know nothing else will.
A check that exists, is required, and cannot fail where it matters, is a belief. The trunk carried a green badge for a fortnight and the badge meant nothing, and everybody including me read it as evidence.
The habit
For every check you rely on, ask two questions: where can this fail, and have I seen it fail there.
If you have never watched a check go red on the branch that matters, you do not know that it can. Break it on purpose once. Push a deliberately failing test to the trunk and confirm the pipeline goes red and that somebody is told. It takes ten minutes and it is the only way to know the wiring exists.
The same question applies to alerts, to monitors, to backups, and to anything with the word verification in its name. A gate that has never failed is not proven. It is unproven in a way that looks exactly like proven.
The second mistake, which was mine
One of those failures had been looked at and dismissed as environmental. That is a word I now distrust in my own mouth.
It was a real regression. A page had been ported and lost its support chat widget, so every user landing after login had no support chat. Sitting in a failing test for days, labelled flaky.
Both things were true: that test had a history of flakiness, and this time it was right. Flakiness is not an excuse for a failure, it is a reason to look harder, because once a signal is noisy you cannot tell the two apart and you are paying that cost every time it fires.
If you have a flaky test, either fix it or delete it. Keeping it is choosing to train yourself to ignore a red light.
A practical detail that cost me an hour
Bisecting the pipeline history needed per job conclusions rather than overall run status, because a run cancelled by a newer commit looks identical to a failure in the interface.
Once I read job level results the window was obvious: green at ten to seven in the morning, red by quarter past nine, all seven failures from one commit.
Worth knowing before you need it: the summary view most of us trust conflates cancelled and failed, and that conflation hides exactly the pattern you are looking for during an investigation.
And one more thing that was quietly rotten
While in there I read the manual test plan, fifty eight cases, and found it had gone stale against behaviour we had deliberately changed. Run as written it would have filed a batch of bug reports against correct code.
Three quality mechanisms in one week, all giving confident answers, none of them connected to whether the software worked. Confidence is cheap. Connection to reality is the expensive part, and it has to be checked on a schedule like anything else.