An idempotency key can expire before its risk does

A client creates a payment with an idempotency key. The server commits the charge and stores the response for twenty-four hours. The client loses the response, goes offline, and retries two days later with the same key.

The API has forgotten the key. The financial system has not forgotten the charge.

Short deduplication windows are operationally attractive. They bound storage, keep indexes smaller, and fit the common case of immediate network retries. They can also create a false safety boundary. Duplicate risk lasts as long as a caller can plausibly repeat an operation whose first outcome still matters.

Begin with the business effect #

An idempotency record is not merely a cache entry. It is evidence that a particular caller, operation, and key already produced or attempted a business effect.

The required lifetime therefore depends on the effect. A request to refresh a transient preview may need protection for minutes. Creating an invoice, scheduling a shipment, issuing a refund, or provisioning a paid resource may remain consequential for months.

Start by asking when a repeated request can no longer create an unacceptable duplicate. The answer may involve a refund window, order lifecycle, settlement period, client offline behavior, or the lifetime of an external provider reference. Pick retention from that boundary, then solve the storage problem it creates.

Do not begin with how long Redis keeps keys by default.

Bind the key to a request shape #

Remembering a key is not enough. The server also needs to know what that key authorised.

Store a stable fingerprint of the request fields that define the operation. When the same caller presents the same key with a different amount, recipient, currency, destination, or action, reject the mismatch. Returning the first response to a materially different request can be as dangerous as executing twice.

The fingerprint should be built from a canonical representation. Field order, harmless whitespace, or omitted defaults should not create accidental differences. At the same time, do not include volatile transport fields that make every retry unique.

The scope matters too. A key may be unique per account, API credential, endpoint, or operation family. Global uniqueness sounds simple but can leak collisions across tenants and complicate ownership. A scope that is too narrow can let the same logical action execute through a neighbouring route.

Write the scope down as part of the API contract.

Preserve in-progress uncertainty #

Two matching requests can arrive at once. If both check for a completed idempotency record before either creates one, both may proceed.

Claim the key atomically before the side effect begins. The record can distinguish in progress, completed, and failed in a way that reflects what the server actually knows. A second request finding an active claim can wait, return a conflict with a lookup path, or retrieve the eventual result. It should not begin another execution merely because the first has not stored a response yet.

Crashes make the state harder. The process may die after an external provider accepts the request but before the local record becomes completed. Expiring the in-progress claim and starting again is safe only when the next attempt can reconcile the original provider reference or when the downstream operation is itself idempotent under the same stable identity.

A useful record keeps the local key, request fingerprint, operation identity, provider reference when available, current state, timestamps, and the last evidence received. That is enough for a recovery worker to investigate without inventing a new request.

Separate response retention from duplicate protection #

The full HTTP response may be expensive or inappropriate to retain for the entire risk window. That does not require forgetting the operation.

Keep a compact tombstone after the rich response expires. The tombstone can preserve the key scope, request fingerprint, terminal business identity, outcome class, and a route for authorised lookup. A late retry can then receive a stable reference or a clear already processed result without reproducing sensitive response data.

This split gives the system two retention policies. Response replay lasts as long as clients reasonably need the original representation. Duplicate protection lasts as long as the business effect can still be repeated dangerously.

Be careful with failed attempts. A validation failure before any side effect may be safe to retry after correction, which means the same key with a changed request should probably remain invalid while a new key represents the corrected attempt. A timeout after dispatch is not a clean failure. Preserve uncertainty and reconcile before deciding whether the key can be used again.

Make expiry observable #

Expiry is a policy decision, not a housekeeping detail. Record the protection horizon in API documentation and make it visible in operational data.

Measure how often retries arrive near or beyond the window. Track storage growth by operation class, mismatched-key rejections, in-progress claims that outlive their expected duration, and late retries whose original outcomes can no longer be retrieved automatically.

Those signals tell you whether the chosen lifetime matches real caller behavior. If mobile clients regularly return after several offline days, a twenty-four-hour window is not protecting the system you actually have. If a low-risk preview operation never retries after ten minutes, retaining a large response for a month is waste rather than safety.

Changing the policy needs care. Extending retention is usually straightforward. Shortening it can expose operations whose duplicate risk still exists. Apply a shorter lifetime only to new records unless you can prove older business effects have crossed their safe boundary.

Let deletion follow closure #

An idempotency record can be deleted when the business effect is no longer repeatable in a harmful way, the caller contract no longer permits the retry, and any downstream deduplication or reconciliation windows have also closed.

Those conditions may not share one date. The producer may forget a request while a payment provider still recognises its merchant reference. A consumer may retain event IDs longer than the originating API retains keys. The safest policy names these relationships instead of treating one database TTL as the entire system's memory.

Stack Dispatch treats retry safety as durable workflow evidence. Idempotency keys are one part of that record. Keep the key long enough for the risk, bind it to the request it authorised, preserve uncertainty when execution crossed a remote boundary, and retain a compact tombstone when the full response no longer deserves the space.