A Test-Flakiness Interview Exercise With an Intermittent Failure
Investigate flaky tests using preserved failures, competing hypotheses, isolation checks and targeted experiments instead of hidden retries.
TL;DR
- In this fictional checkout-test exercise, preserve evidence of an intermittent CI failure instead of trusting a rerun.
- Test competing hypotheses, replace arbitrary sleeps with real conditions, and inspect isolation and cleanup.
- If quarantine is needed, assign ownership; verify the cause and repair rather than one fortunate passing run.
Preserve the intermittent failure
Use this fictional interview prompt: a checkout test fails occasionally in continuous integration but usually passes when rerun. The team has added retries, and the interviewer asks how you would investigate the flakiness.
First preserve the original failure evidence. Record the failed assertion, test order, environment, seed where relevant and useful logs. A passing rerun does not explain the first result. It may hide a race, shared-state dependency or environmental difference that still affects the product or test suite.
Clarify what the test is intended to establish. Is it checking a user workflow, a service contract or a visual state? Without that purpose, it is easy to remove the very assertion that made the failure meaningful.
Form competing hypotheses
Create a short hypothesis table and identify a test that distinguishes each possibility. Do not change timeouts, fixtures and application code simultaneously, because an apparent improvement then says little about the cause.
| Hypothesis | Useful experiment | Evidence to retain |
|---|---|---|
| Wait depends on arbitrary timing | Wait for the intended observable state | Actual state transitions |
| Earlier test leaves data behind | Run in isolation and varied order | Fixture identifiers and cleanup |
| Concurrent workers share a resource | Isolate resource ownership | Worker and resource mapping |
| Environment differs | Compare relevant configuration | Dependency and runtime versions |
These are starting points for the fictional exercise, not a claim that all intermittent tests share the same cause. Let the first failure trace guide the order of investigation.
Replace a sleep with a real condition
Suppose the test clicks Submit, waits a fixed interval and expects a confirmation. The interval can be too short on a busy worker, but simply increasing it can make the suite slower without establishing correctness.
Identify the state that means the operation has completed. It might be a visible confirmation associated with the submitted order, or an agreed service result. Wait with a bounded timeout and preserve diagnostics if the condition does not occur. Avoid waiting for a vague page state that unrelated activity can satisfy.
Also inspect the application. If two responses race and the older one overwrites the newer state, a more patient test may merely conceal a real defect. The distinction between a test synchronization problem and application nondeterminism must be investigated rather than assumed.
Check isolation and cleanup
Now introduce a second test that uses the same account and shopping cart. When run concurrently, one test clears the other's data. A retry may pass because the competing work has ended, but the suite remains unreliable.
Give each test the resources it needs or control the shared resource deliberately. Cleanup should run through the appropriate lifecycle even after a failure, but cleanup alone cannot prevent concurrent interference while tests are active. If exclusive access is required, make it explicit and understand the cost.
Watch for external services, fixed ports, shared files and global configuration as well as database records. A unique database row does not isolate a test that still writes to the same filesystem path. Our system design frameworks guide can help make those hidden boundaries visible.
Use quarantine with ownership
If the unreliable test blocks unrelated delivery, a temporary quarantine may be reasonable under the team's policy. Preserve its signal, assign an owner and state what coverage is lost. Removing it from the blocking path should not mean forgetting the underlying behaviour.
Retries can reduce disruption, but track the original failure rate and reason rather than reporting only final green outcomes. Otherwise, the team loses the evidence needed to tell whether the problem is getting worse.
Do not replace a meaningful test with an assertion that always passes. A useful temporary measure might move part of the contract to a more reliable lower-level test while the end-to-end issue is repaired. Explain what that substitute does and does not cover.
Preserve a minimal reproduction where possible, so a future change can be checked against the specific failure mechanism rather than only the broad workflow.
Verify the cause, not merely a lucky run
After the targeted change, run the relevant conditions that previously exposed the failure: varied order, concurrency, repeated attempts or a controlled slow dependency. A finite passing run provides evidence, not mathematical proof that the failure can never recur.
Compare the diagnostics before and after. If the original cause remains unexplained, say so and define the next observation. Avoid declaring success solely because the latest pipeline happened to pass.
Have a practice partner reveal that the failure appears only around a date boundary. Reconsider time zones, clocks and date-dependent fixtures rather than forcing the concurrency theory to fit. In a mock interview session, score your answer on evidence preservation, discriminating experiments and honest coverage tradeoffs. The aim is a reliable signal with an understood purpose, not a greener dashboard obtained by hiding uncertainty.