8 A/B Testing Interview Questions with a Worked Conversion Example

TL;DR

• Define the decision, randomization unit and primary metric before choosing a statistical test.
• Check assignment and logging quality before interpreting an apparent treatment effect.
• Report absolute change, relative change and uncertainty together; statistical evidence and business value are different questions.
• Use a stopping plan and guardrail metrics, then explain what would make you ship, investigate or reject the change.

1. How would you turn a product idea into an experiment?

Suppose a fictional document product wants to replace a confusing onboarding screen. The idea is that clearer instructions will help more new users finish their first report. Start by defining what “finish” means and when the outcome will be observed.

An experiment plan could define the eligible population as new accounts, assign each account consistently to one version and measure whether it completes a first report within seven days. That is more precise than “increase engagement.” It identifies a unit, an event and an observation window. A similar onboarding prompt can appear in product manager execution rounds, and the PM interview questions guide lists what to cover, including guardrails and power analysis.

Then ask whether people within one account collaborate. If teammates can see or influence the same workflow, assigning individual sessions independently may mix experiences. Choose a unit that matches the product and acknowledge any remaining interference.

The interviewer should be able to tell what decision the experiment supports. A successful first report is relevant to onboarding; an unrelated increase in page views might only mean the new instructions require more navigation.

2. Which metrics would you choose?

Use one clearly defined primary success measure and a small set of checks that would prevent an apparently successful but harmful launch. For the example, completion rate is the primary measure, while processing failures, support contacts and time to a usable result may be useful guardrails.

Define denominators. “Reports completed divided by reports started” is different from “eligible accounts completing a report divided by eligible accounts assigned.” A treatment that discourages some users from starting can make the first ratio look better without helping more eligible accounts succeed.

Microsoft's research on safe feature rollout distinguishes success, guardrail and data-quality measures. Use that distinction to explain why one favorable metric is not a complete launch decision.

Choose thresholds and interpretation rules before looking at the results. A guardrail added only after an inconvenient outcome is easier to manipulate than a documented decision rule.

3. What is a sample ratio mismatch?

Before comparing outcomes, check whether the observed assignment counts are compatible with the intended allocation. An experiment configured for an even split should not quietly be analyzed as valid when logging or eligibility produces a substantial unexplained imbalance.

Microsoft's sample ratio mismatch research treats this as a data-quality warning that must be investigated before trusting the experiment's conclusion.

For an original diagnostic scenario, imagine one variant logs assignment only after a slow page finishes loading. People who leave early may disappear from its denominator. A higher measured completion rate could reflect missing observations rather than a better experience.

Check the assignment event, eligibility rules, deduplication, redirects, client versions and exclusions. An imbalance can identify a defect, but the counts alone do not tell you which defect caused it. Do not “fix” the experiment by deleting users until the allocation looks balanced.

4. How would you explain an observed conversion lift?

Consider a fictional fixed-horizon experiment with ten thousand independent eligible users per group. Eight hundred users complete in control and eight hundred eighty complete in treatment.

The observed rates are 8% and 8.8%. The absolute increase is 0.8 percentage points. The relative increase is 10%, because the change is divided by the 8% control rate. Those are different descriptions of the same observed difference.

Always give the base rate. Saying only “a ten percent improvement” can make a small absolute change sound much larger than it is. Also state the outcome window, population and whether all assigned users had enough time to mature into that window.

This is invented teaching data, not a result from PhantomCodeAI or any other product. The arithmetic describes an observation; uncertainty and experiment validity still need to be assessed.

5. Can you calculate and interpret an interval?

The following original Python exercise calculates an unpooled normal-approximation interval for the difference between two proportions. It assumes independent Bernoulli outcomes and sufficiently large success and failure counts:

from math import sqrt

def compare_rates(control_success, control_n,
                  treatment_success, treatment_n):
    counts = (control_success, control_n,
              treatment_success, treatment_n)
    if any(type(x) is not int for x in counts):
        raise ValueError("Counts must be integers")
    if control_n <= 0 or treatment_n <= 0:
        raise ValueError("Group sizes must be positive")
    if not 0 <= control_success <= control_n:
        raise ValueError("Invalid control count")
    if not 0 <= treatment_success <= treatment_n:
        raise ValueError("Invalid treatment count")
    p0 = control_success / control_n
    p1 = treatment_success / treatment_n
    difference = p1 - p0
    se = sqrt(p0 * (1 - p0) / control_n +
              p1 * (1 - p1) / treatment_n)
    return {
        "control_rate": p0,
        "treatment_rate": p1,
        "difference": difference,
        "lower_95": difference - 1.96 * se,
        "upper_95": difference + 1.96 * se
    }

For the fictional example, the interval is approximately 0.03 to 1.57 percentage points. It narrowly excludes zero under this approximation, but it includes effects smaller than one percentage point. If the business requires at least a one-point improvement, this result does not establish that requirement.

NIST's reference on differences between proportions describes interval estimation and alternative methods. The code above is a simple teaching calculation, not a universal estimator or NIST's default implementation. Sparse outcomes, clustered assignments and sequential analyses require methods appropriate to those conditions.

A 95% frequentist interval refers to the long-run coverage of the procedure under its assumptions. It is not a claim that this particular treatment has a 95% chance of being better.

6. How long should the test run?

Choose a sample-size and observation plan before starting. The required size depends on the base rate, effect you want to detect, uncertainty tolerance, power and design. “Run it for two weeks” is not a complete statistical justification. For a worked example, the data scientist interview question bank calculates the sample size needed for a test on a binary conversion metric.

Consider product cycles as well. If the metric needs seven days to mature, users assigned near the end need their full observation window before the final analysis. A test that stops collecting assignments today may not be ready for its outcome analysis today.

For a fixed-horizon design, do not repeatedly check the same conventional threshold and stop at the first favorable result. The research paper Peeking at A/B Tests addresses why ordinary inference can become unreliable under that behavior. If the business needs continuous monitoring for efficacy, choose a valid sequential approach in advance. Safety monitoring can still trigger protective action without pretending that an emergency stop is an ordinary success result.

Ask what practical change the team would make if the test is inconclusive. More data may help, but an unclear metric or broken assignment system needs a design correction rather than a longer runtime.

7. What if the primary metric improves but a guardrail gets worse?

Work through the actual tradeoff. Suppose completion improves while processing failures rise from 18 to 90 per ten thousand assigned users. Those fictional rates are 0.18% and 0.9%. Before celebrating, inspect whether the new screen sends invalid files into processing or changes the type of work attempted.

Do not merge all measures into a convenient success label. Report the primary result and the guardrail separately, including their definitions and uncertainty. A team may investigate and revise the feature instead of shipping it broadly.

Use segments to diagnose plausible mechanisms, but distinguish exploratory analysis from the original confirmatory question. Searching many segments until one looks favorable does not repair an unfavorable overall result.

Write down what was learned and what the next experiment would change. A rejected launch can still be a useful experiment when it reveals why the proposed mechanism failed.

8. How would you present the recommendation?

Give the decision-maker a compact account of the experiment:

Rehearse an answer that includes an inconvenient result, for example by saying it aloud in an AI mock interview practice session. Explaining why you would not ship a superficially positive test demonstrates stronger judgment than treating every uplift as a victory.

The aim of an A/B testing interview is to show a chain from a useful question to a defensible decision. Calculation matters, but the result is only as credible as the assignment, measurement and interpretation around it.