Idempotency in Messaging

[!NOTE] This module explores the core principles of Idempotency in Messaging, deriving solutions from first principles and hardware constraints to build world-class, production-ready expertise.

1. The Problem: Duplicates are Inevitable

In a distributed system, the network is unreliable.

  • Scenario: Client sends a “Charge $100” request.
  • Server: Processes it successfully.
  • Network: The “200 OK” response gets lost on the way back.
  • Client: Thinks the request failed (Timeout). Retries.
  • Result: The user is charged $200.

Idempotency guarantees that performing an operation multiple times has the same effect as performing it once.

  • f(f(x)) = f(x)

2. Why “Exactly-Once” is a Lie

You cannot achieve “Exactly-Once Delivery” in a mathematical sense (FLP Impossibility Result). What systems like Kafka mean by “Exactly-Once” is actually: At-Least-Once Delivery + Deduplication.

3. The Solution: Idempotency Keys

The client must generate a unique ID (UUID) for every intent.

  1. Client: Generates ref_id = "uuid-123".
  2. Client: Sends POST /charge { amount: 100, ref_id: "uuid-123" }.
  3. Server:
    • Checks DB: Have I seen uuid-123?
    • No: Charge card. Save uuid-123 to DB. Return Success.
    • Yes: Return the previous success response immediately. Do NOT charge again.

4. Interactive Demo: The Duplicate Slayer

Simulate a “Charge” request. Toggle Idempotency on/off to see the difference when network retries occur.

  • Scenario: You click “Pay”. The network is flaky (40% packet loss on response). The client auto-retries.
IDEMPOTENCY: ON
BAD NETWORK (Retries)
CLIENT APP
Balance: $1000
Request ID: uuid-101
SERVER (Payment Gateway)
PROCESSED_KEYS Table
Key Status Result
Empty
> Server Ready... Waiting for requests.

5. Implementation Patterns

A. The “Unique Constraint” (Database)

The simplest way. Rely on the database’s ACID properties.

INSERT INTO payments (idempotency_key, amount, user_id)
VALUES ('uuid-101', 100, 50);
-- If run twice, DB throws "Duplicate Key Violation"

The app catches this error and returns “Success” (since the work is already done).

B. Redis + Lua (The Race Condition)

A common mistake is the “Check-Then-Act” pattern, which causes a Race Condition:

# BAD CODE - Race Condition!
if not redis.exists(key):
  # <-- Another thread could insert here!
  redis.set(key, "processing")
  process_payment()

Solution: Use Redis SETNX (Set if Not Exists) or a Lua Script to make the Check+Set operation Atomic.

-- ATOMIC LUA SCRIPT
if redis.call("EXISTS", KEYS[1]) == 1 then
  return 0 -- Already exists
else
  redis.call("SET", KEYS[1], "processing")
  return 1 -- Success, lock acquired
end

6. Soft Delete vs Hard Delete

Idempotency often requires “Soft Deletes” to handle re-execution of “Delete” logic safely, although DELETE is naturally idempotent.

  • Hard Delete: DELETE FROM users WHERE id=1.
  • First call: Returns “OK”.
  • Second call: Returns “0 rows affected” (or 404). This might confuse the client.
  • Soft Delete: UPDATE users SET deleted_at=NOW() WHERE id=1 AND deleted_at IS NULL.
  • First call: Sets deleted_at. Returns “OK” (1 row affected).
  • Second call: Does nothing (0 rows affected). Returns “OK” (We treat 0 rows as ‘Already Deleted’).
  • Benefit: The record is preserved for audit trails, and retry logic becomes consistent.

7. Idempotency in REST APIs

Not all HTTP methods are equal.

Method Idempotent? Description
GET Yes Reading data doesn’t change state. Safe to retry.
PUT Yes “Replace this resource”. Calling PUT /users/1 {name: "Bob"} 10 times results in the same state (Name is Bob).
DELETE Yes “Delete this resource”. Calling it 10 times results in the same state (Resource is gone).
POST NO “Create a resource”. Calling POST /payments 10 times creates 10 payments.

How to make POST idempotent?

Use a custom header like Idempotency-Key (Stripe standard).

  1. Client generates UUID uuid-123.
  2. Client sends POST /payments with header Idempotency-Key: uuid-123.
  3. Server checks Middleware:
    • If Key exists in Redis → Return cached Response.
    • If Key missing → Process Request → Cache Response → Return.

8. Summary

  • Network failures (Timeouts) create ambiguity. You never know if the server did the work.
  • At-Least-Once delivery means you WILL get duplicates.
  • Use Idempotency Keys (UUIDs) to distinguish retries from new requests.
  • Use Atomic Operations (DB Constraints) to prevent Race Conditions.
  • Prefer Soft Deletes (deleted_at) for consistent history and safe retries.