Idempotency comes before retries

Retries are easy to add and hard to trust. A worker times out, an HTTP request returns 502, a webhook receiver is slow, or a queue consumer exits halfway through a job. The common response is to try again.

That is reasonable. It is also incomplete.

A dispatch system that retries before it understands idempotency is not more reliable. It is only more persistent. If the first attempt actually created the invoice, sent the email, charged the card, published the payload, or changed the customer record, a second attempt can turn a temporary failure into a permanent data problem.

The operational question is not "should we retry?" It is "what happens if the same work is accepted twice?"

A retry is a guess #

Most retry logic starts from uncertainty. The caller knows that it did not receive a clean success response. It does not always know whether the server did nothing, did some of the work, completed the work and failed to respond, or responded but lost the response on the network.

Those states need different handling, but a retry treats them as one state: try the request again and hope the receiver can make sense of it.

For read-heavy APIs, this is often harmless. For dispatch workflows, it is not. Dispatch systems sit near side effects. They send messages, call external APIs, enqueue jobs, update state, and produce evidence that downstream systems trust. Repeating a side effect without a stable identity is how a product earns support tickets that are hard to explain.

Idempotency gives the work an identity #

Idempotency means the same logical operation can be submitted more than once without changing the result after the first successful application.

That does not mean every request is ignored after the first attempt. It means the system can recognize that two attempts represent the same intended work. The receiver can return the original result, decline a conflicting duplicate, or show that the operation is still in progress.

The key is a durable identity for the operation. In API terms, that is usually an idempotency key. In queue terms, it may be a job key, event ID, delivery ID, or domain-specific natural key. The exact name matters less than the rule: the key must represent one intended action, not one transport attempt.

The key has one job: decide whether a repeated request starts new work or points back to work already accepted.

The record matters more than the retry loop #

A serious idempotency design needs a record, not just a header.

At minimum, the system should store the operation key, request fingerprint, status, result reference, creation time, and expiry policy. The request fingerprint matters because it prevents a dangerous reuse of the same key for different payloads. If the key is the same but the payload changes, the system should return a conflict instead of pretending the duplicate is safe.

The status should distinguish accepted, processing, succeeded, failed in a retryable way, and failed permanently. Without those states, clients cannot tell whether to wait, retry, inspect the operation, or ask a person to intervene.

The result reference is what turns idempotency from suppression into a usable developer experience. If the first attempt created delivery_123, then a duplicate request should be able to return or point to delivery_123. The caller should not have to guess whether the delivery exists.

Retries still need rules #

Idempotency makes retries safer. It does not make every retry good.

Retry only failures that may recover without human action: timeouts, connection resets, 429 responses with a clear delay, and 5xx responses where the provider has not marked the operation as permanently failed. Do not automatically retry validation errors, authorization failures, malformed payloads, missing scopes, or business rule rejections.

Use backoff with jitter so a partial outage does not become a synchronized traffic spike. Put a ceiling on attempts. Record each attempt separately from the operation so an engineer can see both the logical dispatch and the transport history around it.

That separation is important. The operation is "send this dispatch once." The attempts are "we tried at 10:01, 10:02, and 10:05." Mixing those together makes the system harder to debug and easier to duplicate.

A practical dispatch checklist #

Before adding a retry policy to a dispatch workflow, answer these questions:

• What is the stable identity of the operation?
• Where is the idempotency record stored?
• What payload fields are included in the request fingerprint?
• What happens when the same key is reused with different content?
• Can the caller recover the original result after a duplicate attempt?
• Which failures are retryable, and which must stop immediately?
• How long are operation records retained?
• Can operators inspect attempts without confusing them with new work?

If those answers are unclear, the retry loop is premature.

What the dispatch layer proves #

Stack Dispatch is built around the idea that delivery is an operational contract, not a best-effort gesture. That contract needs more than queue depth and successful HTTP responses. It needs stable identities, replay-safe handlers, clear attempt history, and enough audit detail for a team to understand why a dispatch happened once, more than once, or not at all.

The product connection is deliberately narrow: reliable dispatch starts with the shape of the workflow. Faster retries are valuable only after repeated delivery is safe, observable, and explainable.

Retries help a system recover from uncertainty. Idempotency decides whether that recovery is safe.