tldr: A flaky test doesn't just annoy you. It costs real engineering hours, and most teams have never actually done the math on how many.
The ritual everyone knows
A CI job fails. Someone glances at it, shrugs, and hits rerun "just to check." It passes the second time. Nobody investigates further. The PR merges.
That fifteen-second ritual feels harmless. It isn't. Multiply it by your team size, your merge frequency, and every sprint this quarter, and you're looking at a quiet tax on engineering time that never shows up on a roadmap.
Flaky tests are common enough that most engineers have stopped treating them as bugs. Bitrise's 2025 Mobile Insights report, which analyzed over 10 million builds across 3.5 years, found the share of teams experiencing test flakiness grew from 10% in 2022 to 26% in 2025. Microsoft's own research on large-scale CI systems puts roughly a quarter of all test failures down to flakiness rather than real defects, and a widely cited developer survey from Eck et al. found 58% of developers deal with flaky tests at least monthly. This isn't a tooling inconvenience anymore. It's a persistent, growing tax that most teams have never measured.
What counts as a flaky test
A flaky test is one that passes and fails against the same code, with no changes in between. Run it ten times and it might fail twice, for no reason you can point to.
A broken test is another story. It fails consistently because something real changed. A flaky test fails unpredictably because something in its execution isn't deterministic: a race condition, a timing assumption, a shared piece of state. The unpredictability is the whole problem. You can't trust a test that sometimes lies.
The hidden cost: minutes become hours become sprints
Here's the math nobody runs. Say each engineer on your team hits 10 flaky reruns a day, and each rerun costs 5 minutes of attention: noticing the failure, deciding it's "probably flaky," clicking rerun, waiting, moving on.
That's 50 minutes per engineer, every day. Across a 10-engineer team, that's over 8 hours a day of collective attention, more than a full engineer's day, spent on a problem that produces zero new features. At a fully loaded cost of roughly $75 an hour, that's over $3,000 a week, or more than $160,000 a year, just in reruns. That number doesn't include the actual debugging time when someone finally stops to fix the root cause, or the trust erosion when engineers start ignoring real failures because "it's probably just flaky."
This isn't hypothetical. Slack's engineering team documented publicly that flaky mobile test failures ran as high as 56.76% before they built automated flaky-test detection and suppression, bringing it down to 3.85% and saving an estimated 553 hours of triage time in the process, roughly 23 days of developer time. That gap, more than 50 percentage points, is roughly the size of the problem sitting in most teams' pipelines today, unmeasured and unowned.
Why flaky tests happen
Most flakiness traces back to a handful of causes:
-
Race conditions. The test asserts before the app has finished an async action.
-
Hardcoded waits. A
sleep(2000)that works on a fast CI runner and fails on a slow one. -
Shared test environments or data. Two tests fight over the same record, and whichever runs first wins.
-
Brittle selectors. A CSS class or
data-testidthat shifts slightly between builds, and the test can't find the element in time. We've written before about why pointing tests at DOM nodes was always the wrong foundation.
Each of these is a symptom of the same root issue: the test is coupled to how the UI happens to behave right now, not to what it's supposed to do.
The fixes everyone tries that don't actually fix it
Most teams reach for the same three band-aids:
-
Auto-retry on failure. Rerun until it passes. This hides flakiness instead of fixing it, and it stretches your CI runtime with every retry.
-
Quarantine lists. Move flaky tests to a "known flaky" bucket and stop blocking merges on them. Now you have untested code shipping under the illusion of coverage.
-
More
sleep()calls. Add a longer wait and hope the timing issue goes away. It usually just changes which build the flake shows up on.
None of these touch the root cause. They all trade a visible problem for an invisible one.
What actually eliminates flakiness
Slack's fix was flaky test detection and suppression: catch which tests are flaky, then quarantine them so they stop blocking the pipeline. That's a legitimate strategy at their scale, and it saved real hours. But it manages flakiness after the fact. It doesn't stop a test from being timing-sensitive in the first place.
The other path is removing the cause instead of managing the symptom: resolving steps by intent instead of by DOM position, reading the page the way a screen reader does, through the accessibility tree, and re-resolving a step when the UI changes instead of waiting on a fixed timer or a fragile selector.
That's the model Passmark, Bug0's open-source testing engine, uses under Bug0 Managed. A step doesn't wait a hardcoded number of milliseconds for an element to exist. It resolves what "the submit button" means right now and acts on it, so a slow render or a moved button doesn't produce a false failure. It's a big part of why Bug0 Managed suites run at close to a 0% flake rate instead of the 25%-plus most teams live with, and every failure that does happen still gets a human's judgment before it's called real or noise.
Calculate your own flaky-test tax
You don't need our numbers. Run your own:
(Reruns per engineer per day) x (minutes lost per rerun) x (engineers affected) x (working days per month) ÷ 60 x (hourly rate) = your monthly flaky-test tax.
Plug in even conservative numbers, 5 reruns per engineer a day, 5 minutes each, 8 engineers, 21 working days, $75/hour, and you land north of $5,000 a month. Most teams have never run this calculation once. Five minutes with your own numbers will tell you what flakiness is costing you before you decide it's not worth fixing.
FAQs
What is a flaky test?
A test that produces different results (pass or fail) against the same code with no changes in between. The inconsistency, not the failure itself, is what makes it flaky.
What causes flaky tests?
The most common causes are race conditions, hardcoded wait times, shared test environments or data, and brittle selectors that break when the DOM shifts slightly.
How do you fix flaky tests?
Reruns, quarantine lists, and longer waits only hide the symptom. The durable fix is resolving test steps by intent, through something like the accessibility tree, instead of anchoring to fragile DOM selectors and fixed timers.
What's the difference between a flaky test and a broken test?
A broken test fails consistently because something real changed in the app. A flaky test fails inconsistently against unchanged code, which makes it much harder to trust or triage.
How much do flaky tests actually cost engineering teams?
It depends on your team size and rerun frequency, but even a modest 10-engineer team rerunning flaky tests several times a day can lose over $160,000 a year in collective engineering attention, not counting the actual root-cause debugging time.
Does Bug0 guarantee a 0% flake rate?
Bug0 Managed, a done-for-you QA service, runs Passmark-based suites that resolve steps by intent instead of brittle selectors or fixed waits, which is why they run at close to 0% flake rate. Your dedicated FDE also verifies every failure, so a real bug is never dismissed as "probably flaky."
