Retries are not failures. They are normal.

In distributed systems, the following are routine:

• Network partitions
• HTTP timeouts
• Client retries
• Kafka re-delivery
• Webhook duplication
• Worker crashes mid-processing
• Partial commits

If your system assumes “this runs once,” it is already broken.

The correct assumption is:

Every operation may execute more than once.

Idempotency is not a feature. It is a survival mechanism.

What idempotency really means

An operation is idempotent if:

Executing it multiple times produces the same final state as executing it once.

This is about state, not response.

Example:

• Creating a payment twice → ❌ not idempotent
• Ensuring payment with ID X exists → ✅ idempotent

The difference is architectural.

The hidden retry pipeline

Most engineers think only about client retries.

But retries happen at multiple layers:

Retries can occur:

• Between client and API
• Between API and DB
• In message broker delivery
• Inside background workers

If any layer is not idempotent, your system becomes inconsistent.

Exactly-once is a myth

You may hear:

• “Kafka supports exactly-once.”
• “Our queue guarantees it.”
• “Our workflow engine retries safely.”

Even if transport is exactly-once, your side effects are not:

• Writing to DB
• Calling Stripe
• Sending email
• Updating Redis
• Decrementing inventory

True exactly-once across distributed boundaries is impractical.

The real pattern is:

At-least-once delivery + idempotent handlers.

That is how resilient systems are built.

Pattern 1: HTTP APIs with Idempotency Keys

The problem

Client sends POST /payments.

The request times out.

Client retries.

Two payments are created.

The correct model

The client generates an Idempotency-Key.

POST /payments
Idempotency-Key: 7b1e9f2a

The server:

1. Checks if key exists
2. If yes → returns stored response
3. If no → processes and stores result atomically

Storage model

CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  response JSONB NOT NULL,
  created_at TIMESTAMP DEFAULT now()
);

Processing flow:

The key must be:

• Unique per logical operation
• Persisted
• Scoped correctly (user/account if needed)

Critical discipline

• Do not store only “processed = true”
• Store the full response
• Return the same status code and body

Idempotency is behavioral consistency, not just state protection.

Pattern 2: Idempotent Message Consumers

Kafka and similar systems default to at-least-once delivery.

Your handler must tolerate replays.

Dangerous handler

func handleOrderCreated(event OrderCreated) {
    decreaseInventory(event.ProductID, event.Quantity)
}

Replay → inventory decremented twice.

Correct pattern: event ledger

Each event must have a unique ID.

CREATE TABLE processed_events (
  event_id TEXT PRIMARY KEY,
  processed_at TIMESTAMP DEFAULT now()
);

Handler flow:

The insert must occur before side effects.

If insert fails → do nothing.

This makes the handler idempotent.

Pattern 3: The Outbox Pattern (DB → Events)

The classic failure:

1. Write to DB
2. Publish to Kafka
3. Kafka publish fails
4. State exists, event doesn’t

Or worse:

1. Publish event
2. DB commit fails
3. Event exists, state doesn’t

Correct architecture

Within one DB transaction:

• Write domain change
• Write outbox event

A background publisher reads outbox and emits events.

If publisher crashes → safe retry.

Outbox rows must be idempotently marked as sent.

This is the most stable pattern for event-driven systems.

Pattern 4: Idempotency in External Integrations

Stripe, webhooks, third-party APIs they retry.

Assume duplicates.

Webhook handler pattern:

CREATE TABLE external_events (
  provider_event_id TEXT PRIMARY KEY,
  received_at TIMESTAMP DEFAULT now()
);

If event ID already exists → ignore.

Never rely on:

“They probably won’t resend.”

They will.

Pattern 5: Idempotent Workflows (Temporal, Sagas)

Workflow engines retry activities.

This means:

• An activity may run twice
• A compensation step may re-execute

Rules:

• Activities must be idempotent
• Avoid non-deterministic side effects
• External calls must be guarded

Example mistake:

id := uuid.New()
saveUser(id)

If retried → two users.

Better:

id := deterministicID(input)
saveUserIfNotExists(id)

Workflow systems amplify idempotency mistakes.

They don’t hide them.

Designing for Idempotency

When reviewing a system, ask:

• Where can retries happen?
• Where are side effects triggered?
• What boundaries cross network or process limits?
• What writes are not naturally idempotent?

Then apply guardrails:

Layer	Guard
HTTP	Idempotency keys
Message consumer	Event ledger table
DB → Event	Outbox pattern
Webhooks	Provider event storage
Workflows	Deterministic + guarded side effects

Tradeoffs

Idempotency introduces:

• Extra tables
• Slight write overhead
• Storage growth
• More careful design

But without it, you get:

• Double charges
• Inventory corruption
• Ghost orders
• Data drift
• Expensive incident response

The tradeoff is trivial.

A Practical Production Checklist

Before shipping:

• All POST endpoints support idempotency keys
• All message consumers guard duplicate events
• All external webhook handlers store provider IDs
• DB → event publishing uses outbox pattern
• Background jobs are safe under retries

If not, your system is not production-hardened.

Final Thought

Retries are not edge cases. They are steady-state behavior. Robust systems do not try to avoid retries. They assume duplication and design for it.

That is the difference between:

• Code that works in staging
• Systems that survive production