A Test That Never Fails Is Not Reassuring

A test can stay green because the software is reliable. It can also stay green because nothing interesting was asked of it. Those two situations look identical in a status badge.

That is why I find a newly written test less convincing than a test whose failure I have seen and understood. There should be a recognizable mistake that makes it fail. If the test passes before the behavior exists, or survives when that behavior is removed, its reassuring appearance deserves some suspicion.

A green test suite is useful only if the tests had a credible chance to turn red.

This is not an argument for manufacturing failures or making releases painful. It is a way to ask what evidence a test actually contributes. A suite should reduce uncertainty about consequences that matter, and each check should have a reasonably clear explanation of the uncertainty it reduces.

Start with the mistake you want to detect

Consider this site’s publication rule. An article must be explicitly public, must not be a demo, and must have reached its publication date. A test using one ordinary published article checks only one combination of those conditions.

A much more revealing example gives the article a future date while leaving the other fields valid. Another marks a past article as a demo. Another omits the explicit publication flag. These inputs isolate the reasons a page should remain private.

The useful question is concrete: if somebody accidentally removes the date comparison, which assertion catches it? The answer should name a test, rather than gesture toward general coverage. A temporary local mutation can help check that answer. Change the condition, observe the intended failure, then restore the implementation. The broken code should never become part of the release.

This method also exposes weak assertions. Checking that a result is an array says little about whether a future article was excluded. Checking that the future article’s identifier is absent speaks directly to the product rule.

Behavior belongs at the right boundary

Playwright’s testing guidance recommends testing user-visible behavior and isolating tests. For a publication, that suggests assertions about discoverable articles, working links and meaningful controls rather than incidental component structure.

Still, user-visible does not mean every rule needs a browser. Sorting articles by publication date is easier to challenge with small data fixtures. A keyboard-operated menu needs a browser because focus, events and rendered state are part of its behavior. Choosing the layer is part of the test design.

I would rather have a focused unit test for a publication boundary and a browser test proving that the shared rule reaches the rendered Archive than five browser tests that independently reconstruct the same algorithm. The first arrangement can identify both a broken rule and a broken connection. The second risks duplicating an assumption without examining it.

Implementation details occasionally deserve direct checks when they are themselves constraints. A static site must not ship a database file. A deployer must not upload credentials. Those are artifact and release boundaries, even though a visitor should never see them.

Coverage has a denominator problem

Coverage tells you something about code exercised by the tests. It does not tell you whether the assertions distinguish correct behavior from a plausible error. Executing a branch and checking only that no exception occurred can leave the branch’s result essentially unexamined.

Imagine a price parser that accepts a decimal string and returns a number. A happy-path test might execute nearly every line. It can still miss an empty field being converted to zero, a thousands separator being interpreted incorrectly, or a negative amount slipping through a rule that disallows it.

For this hypothetical parser, I would write a small decision table before discussing a coverage target:

The table is a product contract. Coverage can help find implementation areas that this contract has not exercised, but it cannot invent the contract.

A regression test should tell the original story

When a defect is fixed, the most useful regression test reproduces the consequential condition that triggered it. Its name should explain that condition. “Handles data correctly” will be difficult to interpret six months later. “Rejects a future article even when draft is false” records an understandable rule.

Keep the fixture small enough that a reader can see what is special about it. A copied production response containing hundreds of unrelated fields makes the cause harder to identify and may preserve information that should not be in the repository.

The regression belongs to the failed promise, not to the exact shape of the old code.

That distinction matters during refactoring. A helper may disappear while the behavior remains essential. If the regression only verifies that helper was called, the team has to rewrite the test without necessarily learning whether the original defect has returned.

An internal link can serve the same purpose for maintainers: the development workflow should make the relevant checks discoverable instead of requiring somebody to remember the history behind every command.

Failure handling deserves assertions of its own

Deployment offers a useful example from this repository. If an upload fails halfway through, the workflow should stop before deleting previously owned files or replacing the successful-deployment manifest. An assertion that the process exits with an error covers only part of that promise.

The stronger check observes which transfer operations were attempted. It proves that cleanup did not run after the interrupted upload. It also checks that diagnostic output does not contain the supplied test credential. Those assertions describe consequences, rather than simply confirming an exception.

This pattern applies to imports, background jobs and checkout integrations. A rejected record should not quietly update the live catalogue. A timed-out request should not automatically repeat an operation whose outcome is unknown. The absence of a dangerous follow-up is often the behavior worth testing.

Use clearly synthetic credentials and data in these fixtures. A test needs a distinctive value to detect accidental logging; it does not need a real password. Good failure coverage should not create a new disclosure problem.

Flakiness spends the trust the suite has earned

An intermittent failure needs investigation, even when the product appears fine. The important distinction is between an unstable environment, an unreliable assertion and an actual timing defect. Retrying everything until it passes hides that distinction.

For a browser check, waiting for the state the user needs is usually clearer than waiting an arbitrary number of milliseconds. If a search result is the outcome, assert that result becomes visible. If the application can legitimately remain in an error state, test that state separately instead of letting a broad timeout erase the difference.

Keep the evidence from a failed run: the relevant trace, response, state and environment. Avoid dumping sensitive payloads merely because a failure is difficult to reproduce. A trace should help answer a question, not become an indiscriminate archive of the user’s session.

A quarantined test needs an owner and a reason to return. Otherwise quarantine becomes another spelling of deletion, with the misleading comfort that the file still exists.

Read the release sequence as different kinds of evidence

This site’s workflow runs formatting, linting, type checks, unit tests, a production build, browser checks, editorial checks and a deployment preflight. These are complementary activities. A successful build proves neither usable keyboard navigation nor access to the intended server directory.

The preflight answers a question the local suite cannot: whether the configured credentials and remote path work in the actual hosting environment. The browser suite answers questions a successful file transfer cannot: whether the interface behaves as intended. Keeping those claims separate makes a failure easier to understand.

Before adding another check, write down the mistake it should detect and where that mistake becomes observable. Then make the mistake once in an isolated local change. Watch the check fail for the expected reason. Restore the code and watch it pass. That small exercise is often more reassuring than another row of green badges.

Sources

1. Playwright: Best practices — checked 2026-10-13