Experience note · Retrieval and citations
Document Q&A That Can Say “I Don’t Know”
What I learned about keeping document answers grounded, traceable and willing to stop when the evidence is not enough.
What this note covers
- Keeping answers connected to evidence
- Knowing when the system should refuse
- Testing retrieval separately from generation
- Deciding what belongs outside the model
Context
This note draws on work across private systems and Phemilo. Examples are generalised and contain no user data or internal implementation details.
The problem I encountered
A document Q&A interface can look finished long before its answers are trustworthy. The difficult part was not producing fluent text. It was keeping the path from an answer back to the uploaded material visible enough to inspect and test.
I wanted to understand how much of that path could remain explicit: ingestion, retrieval, citation mapping, refusal behaviour and evaluation.
The approach I took
- I built the reference application without LangChain so the boundaries stayed visible in the code.
- Each retrieved unit keeps enough source information to map an answer back to its passage.
- Citation mapping happens as an application concern rather than being left entirely to generated prose.
- Insufficient evidence has an explicit refusal path.
- Retrieval and answer behaviour are exercised through a separate evaluation runner.
Where it broke down
Chunk boundaries can remove context. Similarity can retrieve plausible but incomplete passages. A citation can be technically valid while still failing to support the actual claim. Refusal thresholds can become either too cautious or too permissive.
Those failure modes changed how I thought about the product. The chat interface was the easy part; the real work was defining when a confident answer had stopped being a grounded one.
What I would do differently now
I would invest earlier in retrieval diagnostics and in evaluation cases where several individually relevant passages need to be combined. I would also separate citation correctness from answer usefulness instead of trying to represent both with one score.
What I took away
Grounded document Q&A is not mainly a prompting problem. It is a product and systems problem: evidence needs a traceable shape, refusal needs to be intentional, and evaluation needs to cover the places where retrieval and generation disagree.