Practice a Retrieval-System Interview With Unanswerable Questions

TL;DR

• In this fictional retrieval exercise, some questions cannot be answered from approved documents.
• Separate retrieval failure from unsupported generation and evaluate answerability and claim-level citations.
• Make abstention or partial answers useful, and distinguish missing evidence from proof that something is absent.

Test whether the sources can answer the question

Use this fictional retrieval-system interview: an internal assistant answers questions from approved product documents. A user asks whether a feature supports a deployment mode that the documents never discuss. The assistant produces a confident answer with a vaguely related citation. The interviewer asks how you would evaluate and improve the system.

Start by distinguishing a relevant-looking document from sufficient evidence. A retrieved passage can mention the feature without answering the specific deployment question. Fluent language and a clickable citation do not establish that the claim is supported.

The exercise focuses on source sufficiency and abstention, not on selecting a particular model provider. Your design should explain what the system does when it cannot support the requested answer.

Build answerability into the evaluation set

Include questions whose answers are present, absent, ambiguous, outdated or restricted to users with different access. Label the evidence needed, not just the preferred final wording. An evaluator should be able to inspect why a question is answerable from the allowed corpus.

Keep the cases tied to real product questions in a deployment, using safe test fixtures in rehearsal. A set containing only easy answerable questions will not reveal how the system behaves when evidence is absent.

Separate retrieval failure from unsupported generation

If an authoritative answer exists but was not retrieved, investigate indexing, query handling, permissions and ranking. If the answer does not exist in the allowed sources, retrieving more vaguely related text may not help. The appropriate outcome can be a clear limitation rather than a longer answer.

If the correct passage was retrieved but the answer contradicts it, the failure lies elsewhere in the response process. Inspect how the system uses evidence and whether instructions or formatting obscure the relevant constraints.

Do not use one overall “answer quality” score as the only diagnostic. Record whether eligible evidence was available, retrieved and faithfully represented. Those stages lead to different remedies.

Make abstention useful

A useful response explains the narrow limit: “The available deployment documentation does not specify this mode.” It can then point to the relevant documentation or authorised support path without inventing the missing capability.

Avoid an uninformative refusal when a clarification could resolve the ambiguity. If the user means one of two named products, asking which one may allow a supported answer. Conversely, do not keep asking questions when the corpus simply lacks the necessary information.

The interface should distinguish an unanswered question from a negative product claim. “Not documented here” does not mean “not supported.” That distinction prevents the system from turning missing evidence into a false limitation.

Verify citations at the claim level

Check whether each cited passage supports the statement attached to it. A citation to a long document can appear credible while failing to substantiate the specific assertion. Include partially supported answers in the test set, where one sentence is correct and another adds an unsupported detail.

Consider source version and effective date where relevant. A newer draft is not automatically more authoritative than an approved document, and an older source may have been superseded. The corpus needs a defined authority policy rather than a simple recency preference.

Our AI system design guide can help you connect retrieval, evidence use and user-facing behaviour. Keep authorization outside the model's discretion: a persuasive request should not expand which sources the user may access.

For a multi-part question, the system may have evidence for only one part. Test whether it can answer the supported portion and clearly identify the remainder. A blanket answer or blanket refusal can both obscure useful distinctions. The response should not fill the unsupported gap merely to sound complete.

Evaluate the tradeoff honestly

An overly cautious system may abstain on questions it could answer, while an aggressive system may fabricate support. Measure both behaviours on an appropriate held-out set and inspect the consequences for the intended task. Do not claim that abstention alone makes the system reliable.

Have a practice partner add a new authoritative document after the initial evaluation. Explain how it becomes available, how stale indexing is detected and how previously unanswerable cases are retested. Also test a malicious instruction embedded in a retrieved document; source text should be treated as evidence, not permission to change system behaviour.

Use the mock interview strategy to rehearse the distinction between absence of evidence and evidence of absence. A strong answer defines when to answer, clarify or abstain, and verifies those decisions against the sources the system is actually allowed to use.