Together AI Interview Questions: Inference and Infrastructure Practice

TL;DR

• Match Together AI interview preparation to the actual role: inference serving, control planes and customer-facing engineering are different jobs.
• Practise latency-versus-throughput tradeoffs, overload behavior, cache correctness and recovery from partial infrastructure failures.
• The exercises below are original practice problems informed by public role descriptions, not a claim about questions Together AI actually asks.
• Confirm the interview format and permitted tools with the recruiter, and prepare evidence from systems you have personally built or operated.

Start with the current Together AI role

Together AI's careers page describes work across AI software, hardware, algorithms and models. The official openings distinguish infrastructure, inference, product and customer-facing roles. A general list of machine-learning definitions will not prepare you equally well for all of them.

Two engineering postings checked on September 14, 2026 provide useful anchors. The Senior Backend Engineer, Inference Platform role discusses routing, autoscaling, traffic fairness, prefix caching and performance analysis. The Inference / Compute Infrastructure role focuses on a declarative control plane, lifecycle management, reconciliation and dependable recovery.

Use those responsibilities as preparation clues, then return to your own vacancy. This article does not establish a fixed round count, question bank or hiring timeline. A role description explains the work; your recruiter explains the assessment.

Original question: improve throughput without hiding latency

An inference service increases batch size and reports higher total output tokens per second. At the same time, interactive customers complain that responses feel slower. Explain how both observations can be true.

Separate queue wait, time to first token, subsequent token delivery and total request time. Group observations by model, prompt length, output length and traffic class. A single average can hide a bad tail or make a workload change look like an optimisation. Check cancellation and errors too: serving more tokens is not a success if users abandon the responses.

Propose a controlled comparison using the same representative workload and a stated latency objective. Change one policy, record resource use and compare both throughput and the distribution of customer-facing delays. If an offline workload tolerates waiting while a live conversation does not, consider separate admission or scheduling policies rather than claiming one batch setting is universally best. For a related exercise, the machine learning engineer interview questions include a model-serving design prompt whose answer sets shorter batching windows for latency-sensitive endpoints than for high-throughput ones.

For a follow-up, a large customer submits a burst that fills the queue. Explain whether you reject, delay or isolate some requests, and how the customer learns what happened. Unbounded retries can amplify overload. Your answer should include a capacity boundary and a recovery signal, not only a bigger autoscaling target.

A small routing exercise you can test locally

Suppose a practice scheduler receives a snapshot of workers. Each worker advertises one model, whether it is healthy, available slots, an estimated queue delay and a measured token rate. Select the eligible worker with the smallest estimated completion time.

The following pure Python function implements that toy rule. It makes no network calls and does not represent Together AI's production routing algorithm.

from math import isfinite


def choose_worker(model, tokens, workers):
    if isinstance(tokens, bool) or not isinstance(tokens, int) or tokens <= 0:
        raise ValueError("tokens must be a positive integer")

    candidates = []
    seen = set()
    for worker in workers:
        worker_id = worker["id"]
        if not isinstance(worker_id, str) or not worker_id or worker_id in seen:
            raise ValueError("worker ids must be unique nonempty strings")
        seen.add(worker_id)
        if worker["model"] != model or not worker["healthy"]:
            continue
        slots = worker["free_slots"]
        if isinstance(slots, bool) or not isinstance(slots, int) or slots < 0:
            raise ValueError("free_slots must be a nonnegative integer")
        if slots == 0:
            continue
        rate = worker["tokens_per_second"]
        delay = worker["queue_seconds"]
        if not isfinite(rate) or rate <= 0 or not isfinite(delay) or delay < 0:
            raise ValueError("eligible worker estimates must be finite and valid")
        candidates.append((delay + tokens / rate, worker_id))
    return min(candidates)[1] if candidates else None

For 100 tokens, worker A with a one-second queue and 20 tokens per second has a six-second estimate. Worker B with a two-second queue and 50 tokens per second has a four-second estimate. The function selects B, even though its queue delay is longer. Ties resolve by worker ID. An empty or fully unavailable pool returns None, leaving admission behavior to the caller.

Test a wrong-model worker, a zero-slot worker, invalid estimates and duplicate IDs. The function takes O(n) time and O(n) space for n workers. Keeping only the best candidate avoids the candidate list, but duplicate-ID validation still uses the seen set. More importantly, the snapshot can become stale. Selection is not an atomic reservation, and two callers could choose the same remaining slot. Explain how a real admission mechanism would reserve capacity or recover from a failed reservation.

The estimate deliberately omits prefill cost, changing output length, model-specific behavior and cache effects. That limitation is part of the discussion. A simple model helps you state assumptions; it should not be promoted as a latency guarantee.

Original question: make a prefix cache safe

Two requests begin with the same text. Can an inference service always reuse the same cached computation? Start by identifying the complete context that makes a cached entry valid. Depending on the system, model revision, tokenisation, adapter configuration and other execution settings can matter in addition to visible text.

Then introduce a tenancy boundary. A cache design must not reveal another customer's private prompt or expose information through an unauthorised lookup. Describe what may be shared, what must be isolated and what evidence you would need before changing that policy. Do not assume that a performance benefit overrides the application's data boundaries.

For an evaluation, compare hit rate, memory consumption and customer-visible latency with a workload that includes repeated and unique prefixes. A rising hit rate can still be a poor result if valuable entries are evicted or the cache adds contention. Identify which traces are needed and how you would avoid collecting unnecessary sensitive content.

Original question: recover a partially provisioned cluster

A user requests a cluster. Hosts are allocated, some software installs succeed and the controller restarts before recording completion. Design the recovery behavior.

Keep desired state separate from observed state. Give each logical request a stable identity and each external action an explicit result. On restart, reconcile what exists before creating more resources. A timeout means the outcome may be unknown; it does not prove the previous action failed. This deep dive into distributed systems interview questions explains how idempotency keys let a retry return the stored result instead of duplicating work that already succeeded.

An excellent answer discusses rollback limits. Some operations can be reversed; others need a compensating action or human review. Explain how the system avoids deleting resources shared with another workload and how the requester sees progress without interpreting every retry as a final failure.

Prepare an incident story with evidence

Choose one genuine incident where an initial hypothesis was wrong. State the user impact, the measurements available, what you changed and how you checked recovery. Distinguish mitigation from the permanent fix. If another team owned a dependency, explain the coordination without claiming their implementation as yours.

Practise a follow-up where your preferred optimisation improves one metric and harms another. Describe the acceptance criterion you would agree on before rollout, the canary population and the condition that triggers rollback. You do not need a dramatic outage story; a well-investigated performance regression can show the same reasoning.

Questions to ask before the interview

Confirm which part of the platform the role owns, the coding language or environment, whether a system-design discussion is included and which tools are permitted. Ask how the team evaluates reliability alongside performance and what a successful first project might look like.

Rehearse the routing exercise and one recovery scenario aloud, then answer without suggestions. If mock-interview practice is useful, use it to challenge assumptions rather than memorise a script. The goal is to explain a design, defend its boundaries and adapt when the interviewer changes a requirement.