Flaky Tests

Flaky tests pass and fail intermittently without code changes. Systematic diagnosis in Playwright — traces, shared state, network variability, and locator ambiguity — turns intermittent red builds into permanently green ones.

Common Flake Causes

Race conditions from missing awaits, order-dependent shared state, live network unpredictability, time/date dependencies, and ambiguous locators matching wrong elements.

CI-specific flakes often involve slower CPUs, missing env vars, parallel collision on shared resources, or timezone differences.

// Flaky: assumes instant API
await page.getByRole('button', { name: 'Load' }).click();
expect(await page.getByRole('row').count()).toBe(5);

// Stable: auto-retry assertion
await page.getByRole('button', { name: 'Load' }).click();
await expect(page.getByRole('row')).toHaveCount(5);

Replace one-shot reads with web-first assertions.

Fix Checklist

1. Open trace from failed CI run
2. Check missing await / wrong locator
3. Stub network with page.route
4. Reset state in beforeEach
5. Remove waitForTimeout

Run with --repeat-each=10 to reproduce locally.
Compare trace timeline fail vs pass.
Mock external services — eliminate network variance.
Use isolated test accounts or data factories.

Flake Fix Patterns

Symptom → solution mapping.

Symptom	Fix
Pass locally, fail CI	Env vars, timeouts, resources
Fails on parallel only	Shared data collision — isolate
Random wrong element	Stricter locator, filter by row
Timeout sometimes	Assert condition, stub slow API
Midnight failures	Freeze time with clock.install
Passes on retry	True flake — use trace to find race

Using Trace Viewer on Flakes

Compare network tab on fail vs retry pass.
Inspect action log timing before assertion.
Check if locator resolved to unexpected element.

Isolation Checklist

Each test creates its own data or uses unique IDs. No test reads another test's leftover cart or user. Database reset or transaction rollback in global teardown if needed.

Common Mistakes

Increasing timeout until flake rate drops slightly.
Retries without ever opening trace.
Running against shared staging with other teams' data.
Date.now() assertions without mocking clock.

Key Takeaways

Flakes are fixable — avoid normalizing them with retries only.
Traces and repeat-each reproduce and explain failures.
Isolation and network control eliminate most flakes.
Web-first assertions fix race-heavy patterns.

Pro Tip

Quarantine flaky tests with test.fixme and a linked ticket — don't let them run in main CI until fixed, or they erode team trust in the whole suite.