Flaky Tests in CI: How to Take Over a Red Pipeline

To fix flaky tests in CI, separate genuine defects from test-side failures, stop your suite from leaking data into the shared environment, and quarantine failing tests with a linked ticket instead of deleting or ignoring them. A red pipeline is rarely one problem. It is usually several, stacked so that none of them is visible on its own.

This post walks through how to take over a pipeline in that state: what to look at first, how to clean up after your own runs, and how to bring quarantined tests back. It draws on a real engagement on a multi-tenant commerce operations platform.

Why flaky tests in CI keep the pipeline red

When one nightly suite has never been fully green and another has been red for days, the failures are a mixture. Some are genuine product defects. Some are test-side problems such as timing and leftover data. Until someone sorts them, nobody can trust either kind, and a real regression hides among the noise.

Pipeline configuration adds its own failures. A newly split application can produce empty merge-request pipelines because the change-detection script has no mapping for its tags. The symptom is a green run that tested nothing, which is arguably worse than a red one.

Read the pipeline in layers: configuration first, then environment, then the tests themselves. A test cannot be fixed if the environment under it is degraded.

Resist the urge to fix everything at once. Deal first with whatever harms other people, then with whatever hides real defects, then with the rest. Shared infrastructure problems outrank a single unstable assertion, because they make every other signal worse.

Shared test environment: clean up what your run created

A shared test environment is a commons, and a suite that creates real records in it is a tenant. Every run that leaves records behind degrades the environment for everyone who uses it, and the damage compounds over time.

Good test data cleanup is scoped to the run. Each record registers itself at the moment it is created, and an always-on teardown removes only what that run created. A broader sweep can still exist, but it should sit behind an explicit opt-in flag and carry guards for your own team's records and for shared fixtures. A cleanup job that is allowed to delete anything will eventually delete something that belongs to someone else.

Test data management: stop the leak at its source

Cleanup treats the symptom. Test data management treats the cause. Look for tests that create a throwaway record each and have no teardown that catches it. A common offender is a test that spins up its own organization just to have something to attach data to.

The durable fix is to point those tests at a small number of pre-seeded fixtures that are idempotent, meaning they can be created repeatedly without producing duplicates or side effects. The test no longer produces anything that needs to be cleaned up, so there is nothing to leak.

How to quarantine flaky tests without losing them

When you quarantine flaky tests, the rule that keeps the practice honest is a ledger. A failing test is skipped only with a linked bug ticket, and it is un-skipped when the fix ships. That turns a vague pile of ignored tests into a visible list with owners and a way out.

Track the flow in both directions. A healthy ledger shows more tests coming back than going out, and it shows how quickly the backlog shrinks. A skip with no ticket is how a suite slowly stops testing what it claims to test. Review the ledger on a fixed rhythm so the list shrinks on purpose, and agree who is allowed to add to it.

What this looked like on a real engagement

The suite ran against a shared environment, and every run created real records: organizations, API applications, tax rules and stores. Cleanup had not kept pace, so leaked organizations built up and degraded the shared graph. The cleanup sweep meant to fix that then ran past its time budget and terminated a different team's pipeline.

We built run-scoped cleanup, moved the full sweep behind an opt-in flag with guards, and stopped the leak at its source. Tests had been creating a throwaway organization each, seven per run, that no teardown caught. Pointing them at two pre-seeded, idempotent fixture organizations took that to zero. We also introduced the quarantine ledger described above.

The decision that mattered came when the sweep began killing another team's pipeline. We shipped a one-line disable the same day and the run-scoped redesign a week later. The cost was real: the leak grew for that week. We took it anyway. Holding other teams' pipelines hostage while we designed the correct fix was the more expensive option, and it is the kind of trade a suite owner has to be willing to make out loud.

All twelve quarantined tests were restored within twenty days, in batches of three, four and five. There were thirteen un-skips against eight skips over the period. Leaked organizations per run went from seven to zero, verified across two runs. One folder's runtime dropped from fifty-five seconds to roughly twenty-seven after three heavy reads sharing a single backend timeout budget were sequenced rather than run in parallel. You can read the full case study or browse all case studies.

This is the engagement our two-week Test Suite Diagnostic is modeled on. If your pipeline looks like the one above, get in touch about the Diagnostic.

A CI pipeline stability checklist

• Failures are sorted into product defects and test-side problems before anything else is fixed.
• Pipeline configuration is checked, including whether newly split applications are mapped to their tags.
• Every record a run creates registers itself, and teardown removes only that run's records.
• Any broad cleanup sweep sits behind an opt-in flag and protects shared fixtures and other teams.
• Tests that created throwaway records now use pre-seeded, idempotent fixtures.
• A skipped test always has a linked ticket, and is un-skipped when the fix ships.
• Heavy reads that share one backend timeout budget run in sequence, not in parallel.

Timing noise inside the tests themselves has its own cure, covered in how to fix flaky Playwright tests. For an outside view of your own suite, see a sample quality audit or our services.