A Queue Backlog Interview Exercise: Diagnose Before Adding Workers

TL;DR

• Practice this fictional prompt: a background queue is growing, and someone proposes doubling the worker count.
• Compare arrival and successful completion rates with message age and processing outcomes.
• Do not use total attempt throughput as the numerator for business progress.

Start with the backlog's shape

Practice this fictional prompt: a background queue is growing, and someone proposes doubling the worker count. Before accepting the change, explain what you would measure and how different causes would change the response. The goal is to diagnose the queue, not to recite a scaling slogan.

Clarify the workload, delivery model and ordering constraints. Are jobs independent? Can they run concurrently for the same customer? Does the worker call a rate-limited dependency? Is the queue partitioned or grouped? More workers can help one bottleneck while amplifying another.

Compare arrival, completion and age

Queue depth alone tells you how many messages are waiting under the metric's definition. It does not explain whether the system is catching up, whether a small subset is stuck or whether retries are creating extra traffic. Compare arrival and successful completion rates with message age and processing outcomes.

For a primary example, Amazon SQS documents several queue metrics, including approximate counts and age-related signals. Read the definitions for the service you use; some measurements have behavior that makes them unsuitable as exact counts of unique business jobs.

In the interview, state the difference between attempts and completed operations. A worker can acknowledge many failed or retried attempts without making useful progress on the original backlog.

Test three competing explanations

Use these fictional observations as separate branches:

These are hypotheses, not automatic diagnoses. High worker CPU, a database connection limit or a producer burst could create other patterns. Explain which observation would make you revise the hypothesis.

Our system design frameworks guide can help organize the answer around constraints before selecting an intervention.

Explain when more workers helps

If jobs are independent, downstream capacity is available and workers are the limiting resource, adding workers may increase useful throughput. Verify the change with completion rate and age, not merely a lower CPU percentage per instance.

If the downstream service is already throttling, more workers can increase retries and worsen contention. You may need controlled concurrency, backoff or a smaller request rate instead. If ordering requires one active job per key, extra workers may not accelerate a single hot key; changing that constraint requires a correctness discussion.

Do not casually split an ordered group just to improve a chart. Ask what business property the order protects. The interview should show that performance changes remain subordinate to the workload's correctness requirements.

Isolate poison work without losing accountability

A message that deterministically fails can consume repeated attempts while never completing. Define a bounded retry policy and an investigation path. A dead-letter destination can preserve failed work for review, but moving messages there is not the same as resolving their business effect.

RabbitMQ's dead-letter exchange documentation describes one broker's mechanisms and conditions. Your answer should use the actual platform semantics rather than assume all queues implement the same behavior. Explain who reviews failed items, what context is retained and how replay avoids repeating an already completed external effect.

For the fictional exercise, a malformed job might be isolated after the agreed attempts while valid jobs continue. A shared dependency outage is different: quarantining every job may hide the real incident. Distinguish a message-specific failure from a systemic failure before selecting the response.

Ask whether the age metric hides repeatedly retried work or reflects only a subset of messages under the queue’s semantics. A dashboard can look healthier after difficult jobs move elsewhere even though users still await completion. Track the business operation through retries and quarantine so recovery reporting includes unresolved work, not just the visible queue that happens to be draining.

Define a recovery target and verify it

Suppose producers create fewer jobs per minute than healthy workers can complete. You can estimate a drain time from the excess useful completion capacity, but label it as a planning estimate and revisit it as workload conditions change. Do not use total attempt throughput as the numerator for business progress.

Observe whether the oldest relevant jobs are getting younger, whether successful completions exceed arrivals and whether retries or dead-letter counts remain explainable. Include downstream health so draining the queue does not overload another part of the system.

Use the mock interview strategy guide to rehearse a second version where only one customer is affected. Your final answer should name the suspected bottleneck, the evidence that distinguishes it and a reversible intervention. “Add workers” becomes one possible decision after diagnosis, rather than the default response to every growing queue.