Experience note · AI evaluation and observability
How I Evaluate AI Features Before They Reach Users
The evaluation habits I use to make prompt, model and tool changes without relying on a convincing demo and intuition alone.
What this note covers
- Turning failures into reference cases
- Comparing prompt and model changes
- Deciding when automated evaluation helps
- Knowing where human review still matters
Context
This note draws on work across private systems and Phemilo. Examples are generalised and contain no user data or internal implementation details.
The problem I encountered
An AI feature can look convincing in a demo and still be difficult to change safely. A prompt edit, a different model or a tool change can improve one path while quietly breaking another.
I needed a way to make those changes with more evidence than a few hand-picked conversations. The goal was never to make every output deterministic; it was to make important behaviour observable and comparable.
The approach I use
- Trace the request across retrieval, model calls and tool use.
- Turn meaningful failures into small reference cases.
- Run those cases when prompts, models or tools change.
- Use automated evaluators for repeatable signals, not as unquestioned ground truth.
- Review ambiguous or high-impact behaviour with a person.
Where it breaks down
Observability can expose sensitive inputs, so traces need deliberate data boundaries. A model-based evaluator can reproduce the same blind spots as the system it judges. A numerical score can also hide a product decision that has never been made explicitly.
That is why I treat evaluation as part of product engineering rather than a separate AI dashboard. The tests only become useful when they represent behaviour the product actually cares about.
What I would do differently now
I would create reference cases earlier, while product behaviour is still being defined, instead of waiting for a stable prompt. I would also keep the datasets smaller and more deliberate: a compact set of understood failures is often more useful than a large collection nobody regularly reviews.
What I took away
Evaluation is most useful as a conversation between product intent and observed behaviour. Traces explain what happened, reference cases preserve what matters, automated evaluators provide repeatable signals, and people still decide whether the result is acceptable.