An Interview Exercise About Selecting What Not to Measure
Practice choosing product metrics by connecting each signal to a decision and explaining which attractive but unhelpful numbers you would exclude.
TL;DR
- In this fictional reporting-launch exercise, retain metrics because they support decisions, not because they are easy to collect.
- Define each signal's meaning, action, and guardrails, and explain which measurements you would exclude.
- Check that reported success reflects a useful user outcome rather than merely an event name.
The practice prompt
In this fictional metrics selection interview exercise, a team is launching a new reporting workflow. Stakeholders propose tracking page views, button clicks, completed reports, support questions and several dashboard totals.
The interviewer asks which metrics you would use. Start with the decision the team needs to make, not with the available event list.
Suppose the immediate question is whether intended users can obtain a usable report and whether the workflow creates unresolved failures. State that scope before selecting signals.
Related reading: Data Analyst Interview Preparation: SQL, Metrics and a Worked Business Case.
Connect each signal to an action
For each proposed metric, ask what finding would lead the team to do differently. A number can be easy to collect and visually impressive while contributing little to the current decision.
| Candidate signal | Decision relevance |
|---|---|
| Page views | May show exposure, but not task completion |
| Button clicks | May locate an interaction step, but not a usable result |
| Completed reports | Relevant if completion is defined and verified meaningfully |
| Unresolved failures | Helps identify recovery and support needs |
| Qualitative questions | Can explain confusion that counts alone do not reveal |
These are interpretations for the fictional workflow, not universal rules about the value of each metric.
Define what the retained measure means
A completion count needs a clear event definition and population. Explain what counts as an attempt, which users are included and how repeated or interrupted actions are treated.
Do not assume an event named “success” proves that the user received a useful result. Check the behavior represented by the event and whether additional evidence is needed.
If the scenario lacks baseline data, say so. You can propose how to establish it without inventing a target or claiming a statistically meaningful effect.
Explain what you would exclude
Remove signals that do not support the current decision or whose interpretation is too weak to justify the collection and review effort. State the reason so the team can reconsider if its question changes.
For example, a cumulative total may look positive simply because it rarely decreases. If it does not distinguish current task performance or a relevant trend, it may not belong in the launch decision view.
Our system design frameworks guide can help connect measurement choices to system requirements and tradeoffs.
Keep guardrails and context
A single positive signal should not hide a consequential failure elsewhere. Explain what additional observation would prevent the team from celebrating more completed reports while ignoring a growing unresolved-error queue.
Include qualitative investigation when the numbers cannot explain the behavior. The purpose is not to replace analysis with anecdotes, but to identify questions the aggregate signals leave open.
A practice partner can ask for one headline metric. Offer a clear primary measure while naming the conditions and supporting evidence needed to interpret it responsibly.
Debrief the selection
Check whether every retained signal supports a decision, has a meaningful definition and can be interpreted within the available evidence. Explain which proposed numbers you removed and why.
Then have the interviewer change the decision from launch usability to capacity planning. Reassess the measurement set rather than claiming the first dashboard fits every question.
A strong answer shows disciplined selection. You measure enough to guide the decision and detect important problems without treating every available count as useful evidence.