All posts
idempotencyreconciliationrecoverydistributed systems

Reconciliation needs its own idempotency boundary

A recovery action can be safe to retry while still being unsafe to repeat after new evidence arrives

The original request timed out.

Support compensated the customer.

A late provider receipt arrived, so the case reopened for reconciliation.

The operator chose a corrective action and pressed confirm.

Then the screen stalled.

This looks like a familiar retry problem, but the old request key is not enough. The original delivery, the compensation, and the reconciliation are three different effects. Reusing one idempotency boundary across them can suppress work that genuinely needs to happen or repeat work that has already changed the customer's position.

Reconciliation needs its own idempotency boundary.

Three effects need three identities

The original delivery has an identity. It might represent a payment, a message, a reservation, or another remote side effect.

The compensation has a second identity. It exists because the system could not establish an acceptable outcome from the first effect before the decision deadline.

The reconciliation action has a third identity. It responds to evidence that arrived after compensation and decides what, if anything, must happen now.

Those actions are related, but they are not interchangeable retries of one operation.

If the reconciliation command inherits the original delivery key, the receiver may treat it as a duplicate and ignore it. If it inherits the compensation key, it may look like another attempt to issue the same refund or replacement. If it generates a new random key every time the operator reloads the case, a stalled confirmation can create two corrective effects.

A durable case should therefore keep separate identifiers for the original attempt, the compensation decision, and the reconciliation command. The links between them belong in the case timeline, not inside an overloaded key.

Define the command before generating the key

An idempotency key is only as useful as the command it identifies.

Reconcile case 742 is too broad. The meaning can change as evidence arrives. At one moment it may mean mark the late receipt as non-customer-impacting. Later it may mean recover a duplicated charge. The same label can hide two materially different actions.

The command should be explicit enough to remain stable across transport retries:

case: 742
evidence-version: 6
action: recover-duplicate-charge
amount-source: settled-provider-receipt
customer-position-version: 3

The key can be derived from the durable command identity or stored with it. The important property is that a retry after a network timeout refers to the same evidence version, the same corrective action, and the same known customer position.

If any of those inputs changes, the system should not silently reuse the key. It should require a new reconciliation decision.

A duplicate response still needs a receipt

Idempotency prevents repeated effects. It does not explain what happened.

When a reconciliation retry reaches the server, the response should distinguish between a newly executed command and a previously completed command. Both can return success, but the operator needs the original receipt for the corrective effect.

That receipt should include the reconciliation command identity, the effect identity returned by the downstream system, the evidence version used for the decision, and the completion time. If the command is still processing, return the durable operation handle rather than asking the client to guess whether another submit is safe.

This matters after UI failure. A button spinner that disappears tells the operator nothing. A lookup by reconciliation command can show that the action completed, remains in progress, or never crossed the local acceptance boundary. The next move follows evidence instead of impatience.

The duplicate response must not reconstruct a fresh success message from current case state. Current state may have moved since the original command completed. Return the original command receipt and let the case timeline show later changes separately.

New evidence should invalidate authority, not history

Suppose another provider event arrives between the first reconciliation submit and the retry. The earlier command may still have completed correctly under evidence version 6, while the case has now advanced to evidence version 7.

The system should preserve the earlier receipt. It should also refuse to treat that command as authority for a new decision.

There are two useful checks here. First, a repeated submission with the same reconciliation key should return the original result. Second, an attempt to create a new corrective command from an old case version should fail with a stale-decision response.

That combination protects both sides of the boundary. Network retries remain safe, and newly arrived evidence cannot be ignored by replaying an earlier decision.

A stale-decision response should name what changed without exposing sensitive provider detail. Evidence advanced. Customer position changed. Another corrective action completed. The operator can then reopen the decision with the current timeline rather than guessing whether the old action remains appropriate.

Recovery tooling should expose the boundary

The operator should not have to infer which key belongs to which effect from logs.

A recovery screen can show three linked records: original attempt, compensation, reconciliation. Each record has its own identity, state, receipt, and retry boundary. The screen can then make the safe action plain.

Retry transport for reconciliation command RC-19 is different from create another reconciliation decision. The first asks the system to recover the result of one durable command. The second asks a human to judge the current evidence and authorise a new effect.

That wording prevents a common operational collapse where every uncertain button becomes retry and every retry quietly means do it again.

At Stack Dispatch, we care about preserving the identities that make recovery explainable. Reconciliation needs its own idempotency boundary because the action responds to a new decision, not merely to the failure of the original request. Give the command a stable identity, bind it to the evidence and customer position that authorised it, return its original receipt on duplicates, and reject new decisions built from stale case state.

Then a stalled screen remains a transport problem. It does not become a second customer effect.

0 comments

Join the conversation

Get the next dispatch

New writing on software architecture, AI systems, and shipping production software, sent by email. Unsubscribe anytime.