All posts
retrieslookupsreceiptsincident response

If lookup still works, do not replay yet

A retry becomes much less trustworthy when the system still has a live remote lookup path but chooses to spend certainty later instead of using it now

Replay is one of the easiest actions in an incident queue to understand.

Something looks stuck.

Somebody wants motion.

The button promises movement.

That simplicity is exactly why teams overuse it.

If the system still has a live way to ask the remote side what happened to the original attempt, replay should usually wait until that lookup path has been used or clearly ruled out.

A retry is strongest when it follows the best available evidence, not when it jumps ahead of it.

A live lookup path is already a form of progress

Teams sometimes talk as though nothing useful has happened until the queue row is either confirmed delivered or replayed.

That misses an important middle state.

If the original attempt already produced a provider reference, receipt token, request ID, or status endpoint path that still answers, the incident is no longer blind.

It is uncertain, but it is not blind.

That matters because replay and lookup solve different problems.

Lookup asks whether the first obligation already moved far enough on the remote side that a second send could duplicate or confuse it.

Replay asks the local system to act again despite that uncertainty.

When lookup still works, the product already has a narrower question available.

It should not skip that question just because replay is operationally louder.

The cost of impatience is usually duplicate ambiguity

The most common argument for immediate replay is speed.

We can always sort it out later.

Maybe.

The trouble is that later usually arrives after the evidence got worse.

If the remote side already accepted the original request under receipt rcpt_914, a replay before lookup does not only add a second action.

It adds a second story.

Now the team has to determine whether one of those attempts was unnecessary, whether both reached the provider, and which receipt actually maps to the customer-visible outcome.

That is a much harder investigation than asking the original question while the first lookup path was still warm.

Immediate replay also weakens support.

A support lead reading the incident can no longer say, with confidence, that the team first checked what the remote side already knew.

They can only say that the team acted quickly.

Quickly is not the same as carefully.

Systems that handle money movement, webhooks, dispatch, or external status handoffs should care about that distinction.

The same problem appears in reconciliation.

Once a replay has happened, the team is no longer comparing one uncertain obligation against the provider state.

It is comparing a provider state plus an extra local action against a customer-visible result that may now reflect either path.

Use the living evidence before you create new evidence

The better operational rule is plain.

  1. If the row has no remote identity, say that clearly and judge replay from local idempotency proof.
  2. If the row has remote identity but no working lookup path, classify the uncertainty honestly, then decide whether replay risk is still acceptable.
  3. If the row has remote identity and a working lookup path, spend that certainty first.

That third case is the one teams most often flatten.

They see an unresolved queue row and respond to the unresolved part while ignoring the fact that the row is also lookup-rich.

The product should make that mistake harder.

Show the lookup token.

Show the last successful query time.

Show whether the status endpoint is still answering.

Make replay feel like the second question when living evidence still exists, not the first.

A good queue protects operators from false urgency

False urgency is what turns lookup-capable uncertainty into duplicate-work incidents.

The row is red.

The backlog is growing.

Somebody wants the dashboard greener.

All of that can be real, and the correct next move can still be lookup.

A good queue should help the operator see that they are not delaying action by using the remote evidence path.

They are using the strongest action that still preserves the original truth.

That is an important design boundary.

Products often celebrate actions that change status fastest.

Better products make the lowest-regret action easiest to choose first.

At Stack Dispatch we care about this because replay mistakes rarely begin with a bad HTTP client.

They begin with a product that makes fresh action feel safer than old evidence.

If lookup still works, do not replay yet.

Use the living proof while it can still narrow the decision.

The second send will still be available in a minute.

The original certainty may not.

0 comments

Join the conversation

Get the next dispatch

New writing on software architecture, AI systems, and shipping production software, sent by email. Unsubscribe anytime.