Guide / AI & Automation
AI Features Need an Evaluation Plan Before They Need a Demo
An impressive output is a starting point. Build an evaluation plan around realistic tasks, visible failures, human review and changes to prompts or models.
A generated summary can look excellent until someone asks which interview supports its strongest claim. A coding agent can produce a convincing explanation while leaving a failing route untouched. In both cases, the output is easy to demonstrate and harder to trust.
The gap matters when moving from exploration to a product feature. A demo selects a moment when the system appears useful. An evaluation plan defines the work it must perform, the conditions under which it should succeed and the failures that would make its output unsuitable.
A demo asks “can it work?” Evaluation asks “when does it stop working?”
Answering the second question early can change what gets built. It may reveal that the product needs source references, a review queue or a narrower task before it needs a more polished interface.
Define success in the user’s task
Consider a hypothetical feature that turns discovery interview notes into candidate problem statements. “Produces a readable summary” is too weak as its main success condition. The useful output should preserve what participants actually reported, distinguish observations from interpretation and expose uncertainty where evidence is thin.
A first rubric might require every factual statement to point to supporting material. It could mark invented motivations as a serious failure, even when the prose sounds plausible. It could accept several different phrasings of a problem statement if each remains faithful to the source.
Those criteria follow the task rather than the model’s output format. A syntactically valid JSON response may be necessary for the surrounding software, but it does not establish that the content is accurate or useful.
My research context concerns human-AI collaboration in problem exploration. That motivates the example; it is not a claim that this exact evaluation has been conducted or validated in that research. The proposed rubric is a practical design exercise for a feature with similar responsibilities.
Build a small collection with deliberate variation
A useful starting set contains ordinary cases as well as cases that expose important boundaries. For the interview example, include clear evidence, contradictory accounts, missing context and notes that contain an interviewer’s hypothesis beside a participant’s statement.
Record why each case exists. A short annotation such as “must not turn the interviewer’s suggestion into a reported need” makes the example maintainable. Without that explanation, a future editor may simplify the input and accidentally remove the condition it was meant to test.
Reference answers should not imply that one exact wording is the only acceptable output. Some tasks have an objectively checkable answer; others need criteria and examples of acceptable variation. A golden example is useful as a reference, but it should not become a stylistic template that the system is rewarded for copying.
Keep a portion of the cases separate from prompt development. If every change is tuned against the same visible examples, improvements may describe familiarity with that set rather than broader usefulness. New cases from legitimate, appropriately handled feedback can extend the suite over time.
Choose graders that match the claim
Anthropic’s guide to agent evaluations distinguishes code-based, model-based and human grading, and describes their different strengths and limitations. It also separates capability evaluation from regression protection. These distinctions help choose evidence rather than treat a single score as sufficient.
For the hypothetical summary feature, a deterministic check can verify that referenced source identifiers exist. It cannot necessarily tell whether a nuanced interpretation follows from those sources. A human reviewer can inspect that relationship, using a rubric that makes the decision explainable.
A model-based grader may help triage a larger set, but its judgments need calibration against human decisions. Agreement should be investigated by failure category. Two graders can agree on easy examples while consistently mishandling the cases that matter most.
| Question | Useful evidence |
|---|---|
| Is the response structurally usable? | Schema and parser checks |
| Does a reference point to real input? | Deterministic reference validation |
| Does the claim follow from that input? | Rubric-based review with source context |
| Can a person act on the result? | Task-oriented human assessment |
The table is a proposed evaluation design, not a measured comparison of grading methods.
Make failures specific enough to repair
“Bad answer” offers little direction. A failure taxonomy should separate unsupported claims, missing evidence, misclassified uncertainty, unusable structure and inappropriate action. These categories suggest different fixes.
An unsupported claim may require a stricter evidence boundary or better retrieval. A missing field may need schema validation. A confusing output may require a clearer interface rather than a different model. Combining all three into one quality score can conceal the actual product work.
Severity matters too. A minor wording issue and an invented customer quotation should not cancel each other out in an average. Define failures that block release even if other dimensions improve.
A score becomes useful when it changes a decision. Before running the evaluation, decide what will happen if a critical category regresses. Otherwise the team can rationalize almost any result after seeing it, especially when a more impressive demo is waiting to be shown.
Measure the whole attempt
The user experiences the complete workflow, including retries, corrections and waiting. An evaluation that measures only the final response can hide a costly path to that response.
Record latency, model or tool usage and whether human intervention was required. Keep the units clear: per attempt, per completed task and per accepted result answer different questions. A cheap attempt that frequently needs manual repair may have a different operating cost from the price of its first response.
Repeat selected cases when output variability matters. One successful run does not describe reliability, and one unsuccessful run may not identify the cause. Preserve the prompt, model identifier, tool configuration and relevant environment so that differences can be investigated.
For a coding-agent workflow, the final repository state is important evidence. Did the requested behavior change? Did existing tests remain intact? Was unrelated code modified? A persuasive transcript alone cannot answer those questions.
Treat changes as experiments with a baseline
A prompt revision can improve one class of input while weakening another. The same applies to model changes, retrieval settings and tool definitions. Run candidate configurations against a stable baseline and inspect differences rather than comparing isolated screenshots.
Keep the comparison narrow enough to interpret. If the prompt, model and source corpus all change together, attributing an improvement becomes difficult. Sometimes that combined change is necessary, but the uncertainty should remain visible in the decision record.
The context-engineering article explains why project facts and boundaries matter. Evaluation provides a way to test whether a context change actually helps the intended task. More context is not automatically better; it can introduce contradictions or distract from decisive evidence.
A release note can record which categories improved, which remained weak and why the resulting trade-off was accepted. This creates a useful history of decisions rather than a collection of unexplained scores.
Keep the boundary visible after release
An evaluation suite is necessarily incomplete. Once a feature is used, observability should help detect unfamiliar failures without retaining unnecessary sensitive input. Log the smallest diagnostic record that supports investigation, and define how new cases reach review.
A product may also need a way to refuse an answer, request missing information or route a result to a human. Those paths should be evaluated as useful outcomes when the input does not support confident automation. Forcing a fluent answer in every case can reward the behavior the product most needs to avoid.
The evaluation plan is part of the feature’s design. It determines what evidence the system must retain, which states the interface must explain and where human judgment belongs. Write that plan while the feature is still small. The resulting demo may become less magical, but it will show something more valuable: a system whose usefulness and limits can be examined.