All posts
incident responseobservabilitydebuggingoperations

When the error message lies about the failure

A production incident gets slower and riskier when the recorded failure names the wrong boundary

An operator sees a 500 and reads the message: failed to create log.

That sounds specific. It sounds like the logging step failed and the business action probably did not.

In one real incident from this monorepo, that message was wrong enough to waste time. The visible failure text named the logging path. The actual problem was a database column length limit further underneath. The system caught the database exception, wrapped it in a generic application message, and taught the next person to look in the wrong place.

That kind of error handling feels tidy when you write it. In production it creates fiction.

1. Name the boundary that actually broke

When a request crosses several boundaries, controller, service, queue, database, email provider, cache, one of them eventually decides the outcome.

If the database rejects a write because the payload does not fit the schema, the most useful first record is not that some later application step "failed to create log." The useful record is that the write hit a data-shape constraint, at the persistence boundary, with enough structure to show what kind of constraint failed.

This does not mean returning raw database errors to end users. Public responses still need restraint. It does mean the operational record must stay loyal to the real failure boundary.

Otherwise the system creates a fake story:

  • We think the log write failed.
  • We spend time inspecting the logging path.
  • We delay the schema or payload fix that would actually stop the incident.

The cost is not only slower debugging. It is worse judgment. People replay requests, widen retries, or patch the wrong subsystem because the first record already bent the truth.

2. Catch-all handlers are often translation layers for uncertainty

Many teams add a broad try/catch because they want consistent API responses. That goal is reasonable.

The mistake is letting the catch block pretend it understands the cause.

If a handler catches any thrown error and emits one fixed message, it is not simplifying the incident. It is flattening it. That flattening hides whether the request failed on validation, storage, delivery, credentials, timeout, or an internal invariant.

A better pattern is narrower:

  • Normalize the user-facing response if you need to.
  • Preserve the original failure class in logs, traces, and admin-visible records.
  • Add a stable internal code for the actual boundary that failed.
  • Include the fields that help the next operator decide whether replay is safe.

That last point matters more than teams admit. If the recorded failure says logging failed, the next person may assume the core action never landed. If the real problem happened after part of the workflow already committed, replay can duplicate side effects.

Bad error translation is therefore not only an observability flaw. It is a recovery flaw.

3. The shortest honest message beats the neatest generic one

Production systems do not need romantic prose about failure. They need compact truth.

In the incident above, a better internal record would have looked more like this:

  • request accepted by application
  • persistence write rejected
  • constraint type: value too long for column
  • affected field class: log payload
  • side-effect status: unknown until downstream check completes

That is not pretty. It is useful.

It tells the operator which layer to inspect first. It also signals that "unknown" is still part of the state, which matters when the team is deciding whether to retry, compensate, or leave the request alone.

This is the same discipline reliable dispatch systems need everywhere else. A timeout should keep its doubt. A dead-letter queue should keep the reason. An incident record should keep the true boundary of failure.

4. The record should survive the handoff

The engineer who wrote the catch block may remember what it really means. The on-call engineer next month will not. Support definitely will not. A recovery worker will only know what the system wrote down.

That is why operational records need enough structure to survive handoff:

  • the boundary that failed
  • the original error class
  • whether any earlier side effect may already have committed
  • the request identity needed for lookup
  • whether retry is expected to be idempotent

Once those facts are missing, the team starts rebuilding the incident from surrounding clues. That is slower, noisier, and more dangerous than it needs to be.

5. Product design should assume future strangers will read the incident

The person reading the record later may be tired, new to the system, or working from a support escalation instead of direct code context.

That reader needs a trail that resists wishful interpretation.

If the system knows the database rejected a value for length, it should not summarize that as a logging failure. If the system does not know whether the side effect landed, it should not write a final-sounding message that implies certainty. If the system expects replay to be safe only with a specific request ID, it should keep that ID beside the incident.

That is not overdesign. It is respect for the next handoff.

At Stack Dispatch we care about this because delivery systems and publishing systems both depend on trustworthy records. The next action is only as good as the evidence beside it. When the error message lies about the failure, the evidence starts corrupted before the investigation even begins.

0 comments

Join the conversation

Enjoyed this? Subscribe for more.

Get new essays on software architecture, AI systems, and engineering craft delivered to your inbox. No spam-ever.