Taking over a red pipeline
12 quarantined tests restored in 20 days, leaked test data cut from 7 records per run to zero, and a suite that stopped killing other teams' pipelines.
- quarantined tests restored in 20 days
- 12
- leaked records per run
- 7 → 0
- one folder's runtime
- 55s → ~27s
The suite ran against a shared environment, and every run created real records: organisations, API applications, tax rules, stores. Cleanup had not kept pace. Leaked organisations built up, degrading the shared graph for everyone using it. The cleanup sweep meant to fix that then ran past its time budget and terminated a different team's pipeline.
Underneath that, one nightly suite had never been fully green, another had been red for days with a mixture of genuine product defects and test-side failures, and a newly split application produced empty merge-request pipelines because the change-detection script had no tag mapping for it.
What we built
Run-scoped cleanup. Each record registers itself at creation, and the always-on teardown removes only what that run created. The full sweep moved behind an explicit opt-in flag and gained guards for the team's own records and for shared fixtures.
We stopped the leak at its source: tests were creating a throwaway organisation each, seven per run, that no teardown caught. Pointing them at two pre-seeded, idempotent fixture organisations took that to zero.
A quarantine ledger. A failing test is skipped only with a linked bug ticket and un-skipped when the fix ships.
The decision that mattered
When the sweep began killing another team's pipeline, we shipped a one-line disable the same day and the run-scoped redesign a week later.
The cost was real: the leak grew for that week. We took it anyway. Holding other teams' pipelines hostage while we designed the correct fix was the more expensive option, and it is the kind of trade a suite owner has to be willing to make out loud.
What it produced
All twelve quarantined tests restored within twenty days, in batches of three, four and five. Thirteen un-skips against eight skips over the period. Leaked organisations per run went from seven to zero, verified across two runs. One folder's runtime dropped from fifty-five seconds to roughly twenty-seven after three heavy reads sharing a single backend timeout budget were sequenced rather than run in parallel.
This is the engagement our two-week Test Suite Diagnostic is modelled on.
