Queue depth hides the oldest promise

The queue contains twelve messages.

That sounds fine.

Eleven arrived in the last minute. One has been retrying since yesterday.

The dashboard still says twelve.

Queue depth measures how much work is waiting. It does not measure how long the oldest customer promise has been waiting, whether one ordering key is blocked, or whether the backlog is moving. A small count can hide a serious delivery failure. A large count can be harmless when consumers are draining it faster than producers are adding work.

Treating depth as the health signal confuses inventory with delay.

1. Count the backlog, then ask how it is distributed #

A useful queue view begins with depth because capacity still matters. The mistake is stopping there.

The same depth can describe very different systems:

queue A
depth: 1,200
oldest age: 42 seconds
arrival rate: 400 per minute
completion rate: 460 per minute

queue B
depth: 12
oldest age: 19 hours
arrival rate: 3 per minute
completion rate: 3 per minute

Queue A is busy and recovering. Queue B has one promise that normal throughput is not repairing.

The next useful cut is age distribution. Show the age of the oldest message, plus enough buckets or percentiles to reveal whether delay affects the queue broadly or belongs to a thin tail. If most messages complete within a minute while a few cross an hour, an average will smooth away exactly the work that needs attention.

Age should follow the customer-facing contract where possible. Time since enqueue may be correct. In other systems, the promise began when an upstream request was accepted, before the message reached this queue. Pick the clock that represents what the customer is waiting for and name it clearly.

2. Separate a slow queue from a stuck key #

Queues often preserve order within a customer, account, or aggregate. One poison message can then block later work for that key while unrelated keys continue normally.

Global depth barely moves. Global throughput stays green. The affected customer sees every new event wait behind the same failed predecessor.

The dashboard should make that shape visible. At minimum, expose the oldest age by ordering key or tenant boundary, the number of messages waiting behind the head item, and the current retry or gap state. The purpose is not to publish sensitive identifiers on a broad screen. The purpose is to let an authorised operator move from the aggregate signal to the exact blocked sequence without searching raw logs.

A stuck-key view also changes alerting. Ten old messages spread across ten independent keys may indicate a shared dependency problem. Ten old messages behind one key may need quarantine, dead-letter review, or a supported lookup before the sequence can continue. Equal counts, different recovery.

3. Measure movement, not just arrival #

A depth chart can fall because messages completed. It can also fall because they expired, moved to a dead-letter queue, or were deleted during recovery. Those are different outcomes.

Pair depth with a flow account:

opening depth
+ accepted
- completed
- expired
- dead-lettered
- cancelled
= closing depth

This simple reconciliation makes movement explainable. If depth falls while completion remains flat, the operator can see that the system did not recover in the way the green line might imply. If depth grows while completion rate is healthy, the arrival rate explains the pressure.

Retries need their own treatment. Counting every attempt as a new accepted message inflates volume and can make a single failing promise look like broad demand. Counting only logical messages hides the work consumers are spending. Keep logical backlog and delivery-attempt load separate so capacity and customer delay remain legible.

4. Alert on the promise you are prepared to defend #

A fixed depth threshold is easy to configure. It is often hard to justify.

Why should 500 waiting messages be unsafe? On a queue that handles 20,000 messages each minute, 500 may be ordinary. On a queue that normally holds three, 500 may mean a complete stop. The threshold gains meaning only when it connects to drain time, age, or a known resource boundary.

An age alert is closer to the delivery promise. If the service says accepted messages normally receive a first delivery attempt within five minutes, alert before the oldest eligible message crosses that boundary. Exclude messages deliberately scheduled for later delivery, and keep paused or quarantined work visible under a different state. Otherwise the alert mixes broken promises with intentional waiting.

Drain time can add context:

estimated drain time = current eligible depth / recent sustainable completion rate

The estimate is imperfect, especially when message cost varies. Label it as an estimate. It is still more useful than pretending a raw count has the same meaning at every traffic level.

The alert should carry enough evidence for the first decision: oldest eligible age, affected scope, current arrival and completion rates, retry concentration, dead-letter movement, and the last known change to consumers or dependencies. It should not require the responder to rebuild queue health from six dashboards before deciding whether a customer promise is already late.

5. Keep recovery from improving the metric dishonestly #

Under pressure, it is tempting to reduce depth first and explain later.

Moving old messages to a dead-letter queue makes the primary count look better. Requeuing them with a new enqueue time makes age look better. Splitting the queue can make each individual depth look smaller. None of those actions repairs the original delivery promise by itself.

Preserve original acceptance time and logical message identity across recovery. If a message moves to quarantine, the customer-facing age should continue from the original promise unless the contract explicitly ended. If an operator cancels the work, record who or what authorised cancellation and which later messages were released. A cleaner metric should be the result of a resolved state, not a relabelled wait.

At Stack Dispatch, we treat depth as one operational fact rather than a verdict. The harder questions are how old the oldest eligible promise is, which scope is blocked, whether the backlog is moving, and where messages went when the count changed. Queue depth is useful. It simply cannot tell you who has been waiting longest.