A Search-Relevance Interview Exercise With Ambiguous Queries

TL;DR

• In this fictional search exercise, the query “leave” has several plausible workplace meanings.
• Create explicit intent judgments and distinguish missing retrieval results from poor ordering.
• Evaluate failures by intent, handle ambiguity explicitly, and retest the design when the query or available content changes.

Define relevance before changing the ranking

Use this fictional interview prompt: a workplace search product returns disappointing results for the query “leave.” Some users want the time-off policy, others want instructions for leaving a project, and some are searching for a colleague's departure announcement. The interviewer asks how you would improve search.

The first decision is what success means for these different users. Increasing a single document's score may improve one intent while making another worse. Explain how you would discover and evaluate the intents before proposing a ranking change. This exercise is a reasoning aid, not a report of a measured product improvement.

Ask about the search context: the current workspace, selected content type, language, permissions and whether the user entered a full query or a prefix. Do not assume you can collect every available personal signal just because it might improve ranking.

Build an intent set with explicit judgments

Create a small fictional evaluation set containing ambiguous and unambiguous queries. For each, describe the task and the documents that would help. Avoid choosing cases only because the existing system already handles them well.

The last row introduces an important boundary: relevance does not override access control. A document the user is not allowed to see should not become an eligible result, nor should its private contents leak through snippets or suggestions.

Have more than one knowledgeable reviewer examine ambiguous judgments. Disagreement is useful evidence that the query may need clarification or diversified results. Do not hide it by forcing every query into a single unquestioned answer.

Separate retrieval from ordering

Determine whether the useful document is missing from the candidate set or merely ranked poorly. These failures lead to different investigations. A missing document may involve indexing, field coverage, token handling or permissions. Reordering cannot recover a document that never enters the eligible candidate set.

If the document is present but low, inspect which features dominate: title matches, repeated terms, freshness, document type or another signal. Ask whether the existing result is genuinely obsolete or simply older; recency alone does not establish correctness.

Propose one change with a clear hypothesis. For example, giving approved policy titles more weight in the HR context might help policy queries. It might also suppress relevant discussion for exploratory queries. Name both the expected benefit and the regression you would watch.

Handle ambiguity without pretending it disappears

A clarification option, a visible filter or a mixture of result types may be more appropriate than aggressive personalisation. In the fictional example, a short “Time off” filter could help users express their intent without requiring a full profile of their behaviour.

Explain the cost of each option. Clarification adds a step. Diversified results may place the best match slightly lower for a user with a clear intent. Personalisation can be wrong when someone is searching on behalf of another team. The interview tests whether you can articulate these tradeoffs, not whether you can name the most sophisticated algorithm.

Keep the chosen approach connected to the product's task. A reference search for a known document differs from exploration across a large knowledge base. Our system design preparation guide can help structure the requirements conversation before selecting components.

Evaluate beyond one average

Compare the baseline and proposed change on the same judged cases. Review failures by intent, content type and query ambiguity rather than relying only on an overall score. If a change improves common queries while breaking an important infrequent task, the average may conceal the decision you need to make.

User clicks can add evidence, but a click is not automatically a successful task. A user may open several bad results or repeatedly retry. Define the observation you need and its privacy constraints before collecting more data.

Use separate examples for tuning and evaluation where practical. Repeatedly adjusting the system until one tiny set looks perfect can make the exercise look successful without demonstrating broader usefulness.

Rehearse a changed condition

Ask a practice partner to reveal that the approved policy was missing from the index. Revise the plan rather than defending the ranking adjustment. Then change the scenario again: the policy is indexed, but users still choose the wrong result because titles are indistinguishable.

Review whether your response separated access, retrieval, ranking and presentation. Write down one proposed change, one supporting observation and one possible regression. Use the mock interview strategy to repeat the exercise with a new ambiguous query. The learning goal is a defensible evaluation process, not a memorised claim that one ranking technique always wins.