All posts
dead lettersretriesqueuesoperations

A dead letter needs an owner before it needs a replay button

Replay is an operational decision about unresolved side effects, not a convenience action for clearing a queue

A message reaches the dead-letter queue after its delivery attempts are exhausted.

The dashboard puts a large replay button beside it.

That button looks like recovery. Without ownership and evidence, it may be a second attempt to create the same unknown side effect.

A dead letter is not merely a message that needs another chance. It is a case that normal automation could not resolve. Before replay, someone or something must own the decision about what happened, whether the original command is still wanted, and what proof would make another attempt safe.

Exhausted attempts do not explain the outcome

Retry count tells us how many times the delivery system tried. It does not tell us whether the receiver rejected the command, processed it and lost the acknowledgement, partially applied it, or never saw it.

Those cases need different actions. A stable validation error may require correcting or discarding the message. A transport timeout may require looking up the original idempotency key. A partial side effect may need compensation. A receiver outage may justify replay once health returns.

If the dead-letter view exposes only an error string and attempt count, it encourages operators to infer certainty that the system does not have. "Retries exhausted" is a scheduling fact, not a business outcome.

The case should retain the message identity, destination, attempt receipts, idempotency scope, last observed boundary, and the earliest point at which the result became uncertain. That evidence lets an owner decide what must be checked before any new delivery begins.

Ownership should follow the unresolved decision

The right owner is not always the team that operates the queue. A dispatch operator can confirm delivery mechanics, but may not be authorised to decide whether a customer notification is still appropriate. A billing team may need to decide whether a charge should be retried. A product team may need to decide whether an old command still matches current intent.

Ownership should therefore be assigned by the unresolved decision, not by whichever service emitted the dead letter.

A useful ownership record names the current owner, the decision they must make, and the time at which the case needs escalation. "Payments team to confirm whether transaction lookup shows an existing charge before replay" is actionable. "Engineering investigating" is a location without a decision.

Automated routing can suggest an owner from message type and failure boundary, but it should preserve uncertainty when the mapping is weak. Sending a case to a visible unassigned state is safer than confidently routing it to a team that cannot make the required decision.

Replay eligibility needs explicit proof

A replay button should be disabled until the case satisfies a replay policy. The policy does not have to be complicated, but it must match the possible side effect.

For an idempotent command with a receiver-backed lookup, eligibility may require a confirmed absent result and a still-valid idempotency key. For a notification, it may require proof that the event remains relevant and that no equivalent message was already delivered. For a destructive operation, autonomous replay may never be appropriate.

The decision record should state which proof was checked and when. "No matching provider receipt at 10:42 UTC" is different from "probably did not send." Evidence can expire too. A lookup made before a delayed result arrived may no longer support replay five minutes later.

Replay policy should also consider message age and current intent. A command that was valid yesterday may now be superseded, cancelled, or unsafe under changed configuration. Technical deliverability is not sufficient authority to execute it.

Replaying should create a new attempt, not rewrite history

When replay is approved, the system should append a new attempt linked to the dead-letter case. It should not reset the retry counter, move the original message back to an apparently clean queue, or erase the failures that led to review.

The new attempt needs its own timestamp, actor, reason, policy evidence, and result. If it succeeds, the history should show that normal delivery exhausted, a named review established replay eligibility, and a later attempt completed. If it fails, the original uncertainty and the new failure should remain distinguishable.

This lineage matters when the same message returns to the dead-letter queue. Operators need to know whether they are seeing the first unresolved cycle or the third manual replay. Without that context, repeated intervention can look like one stubborn incident instead of a policy that is failing to resolve the underlying condition.

Discard is also a controlled outcome

Some dead letters should never be replayed. The request may be obsolete, malformed beyond repair, duplicated by a newer command, or no longer authorised. Discarding it can be the safest result.

Discard should still preserve the reason, actor, and remaining side-effect uncertainty. "Superseded by message 9f2c" is useful. "Deleted from queue" is not. If an external effect remains unknown, removing the message from active work does not close the case. Reconciliation may still need an owner.

Retention should follow the evidence needs of the system, not the desire for an empty dashboard. A quiet queue can hide unresolved cases just as easily as a busy one can.

The queue metric should expose unresolved decisions

Dead-letter count is a weak operational measure by itself. Ten newly failed messages with clear owners may be healthier than one payment command that has been unowned for two days.

Useful measures include time to ownership, age of the oldest unresolved decision, cases awaiting external reconciliation, replays without complete eligibility proof, and repeated dead-letter cycles for the same logical command. These point toward risk and workflow quality rather than queue cleanliness.

At Stack Dispatch, we treat a dead letter as a durable operational case. The system has already admitted that automatic delivery cannot decide what should happen next. Give that decision an owner, require evidence for replay, and preserve the full attempt history. The replay button belongs at the end of that reasoning, not at the beginning.

0 comments

Join the conversation

Get the next dispatch

New writing on software architecture, AI systems, and shipping production software, sent by email. Unsubscribe anytime.