Kafka, Postgres, and Exactly-Once Business Effects

At scale, identity verification is not a CRUD problem.

It is a distributed state machine under adversarial conditions.

• External providers retry aggressively.
• Webhooks arrive out of order.
• Consumers crash mid-transaction.
• Operators manually retry jobs.
• Entire days of traffic may be replayed.
• Regulators expect perfect audit trails.

And the core invariant is unforgiving:

A verification must have exactly one terminal outcome and that outcome must never change accidentally.

This is a case study of designing a verification system that processes millions of attempts while guaranteeing exactly-once business effects using Kafka + Postgres.

No distributed transactions. No cross-service locks. No magical “exactly-once delivery.”

Just disciplined architecture.

1. The Problem: Verification Is a Workflow, Not a Record

A verification attempt goes through multiple phases:

1. User initiates verification.
2. System sends data to external KYC provider.
3. Provider performs checks asynchronously.
4. Provider calls back via webhook.
5. Internal risk engine may enrich or override.
6. Human review may intervene.
7. Final decision is locked and auditable.

At scale, failure scenarios become common:

• Provider sends the same webhook 5 times.
• Network partition causes webhook retries.
• Consumer crashes after applying state but before committing offset.
• Operator replays 24 hours of events.
• Backfill job reprocesses historical verifications.

If you don’t design carefully, you get:

• Double approvals.
• Rejected → approved flips.
• Impossible states.
• Audit gaps.

Verification systems cannot tolerate that.

2. Business Invariants (Start Here, Always)

Before Kafka. Before Postgres. Before Go code.

We define invariants:

1. A verification has exactly one terminal state.
2. APPROVED and REJECTED are mutually exclusive.
3. LOCKED state is irreversible.
4. Provider callbacks are idempotent.
5. Replaying events must not change final outcome.
6. Every state transition is auditable.

If your architecture cannot enforce these invariants, it is wrong.

3. The Aggregate State Machine

Verification is modeled as a deterministic state machine.

Important properties:

• Only specific transitions allowed.
• Terminal states are irreversible.
• All transitions require validation inside DB transaction.
• Illegal transitions are rejected, not ignored.

4. Architecture Overview

The system is composed of:

• Verification API Service
• Provider Integration Service
• Kafka
• Postgres per service
• Outbox publisher
• Idempotent consumer
• Side-effect workers

Design rules:

• Postgres is source of truth.
• All state transitions occur inside transactions.
• Events are emitted via transactional outbox.
• All Kafka consumers are idempotent.
• External side effects are task-based.

5. Creating a Verification (Transactional Outbox)

When a verification is initiated:

Inside a single Postgres transaction:

1. Insert verification row.
2. Insert VerificationInitiated event into outbox.

Schema

CREATE TABLE verifications (
    id UUID PRIMARY KEY,
    user_id UUID NOT NULL,
    state TEXT NOT NULL,
    version INT NOT NULL DEFAULT 0,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE outbox_events (
    id UUID PRIMARY KEY,
    aggregate_id UUID NOT NULL,
    event_type TEXT NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at TIMESTAMPTZ
);

CREATE INDEX idx_outbox_unpublished
ON outbox_events (created_at)
WHERE published_at IS NULL;

Go Example

func CreateVerification(ctx context.Context, db *sql.DB, userID uuid.UUID) error {
    return withTx(ctx, db, func(tx *sql.Tx) error {
        id := uuid.New()

        _, err := tx.Exec(`
            INSERT INTO verifications (id, user_id, state, version)
            VALUES ($1, $2, 'INITIATED', 0)
        `, id, userID)
        if err != nil {
            return err
        }

        eventID := uuid.New()
        payload := map[string]interface{}{
            "verification_id": id,
        }

        _, err = tx.Exec(`
            INSERT INTO outbox_events (id, aggregate_id, event_type, payload)
            VALUES ($1, $2, 'VerificationInitiated', $3)
        `, eventID, id, toJSON(payload))
        return err
    })
}

No Kafka publish happens here. Outbox worker handles that.

6. Outbox Publisher (Reliable Delivery)

Publisher loop:

func PublishOutbox(ctx context.Context, db *sql.DB, producer *kafka.Producer) {
    for {
        rows, _ := db.Query(`
            SELECT id, aggregate_id, event_type, payload
            FROM outbox_events
            WHERE published_at IS NULL
            ORDER BY created_at
            LIMIT 100
            FOR UPDATE SKIP LOCKED
        `)

        for rows.Next() {
            var id, aggregateID uuid.UUID
            var eventType string
            var payload []byte

            rows.Scan(&id, &aggregateID, &eventType, &payload)

            err := producer.Produce(&kafka.Message{
                TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
                Key:            []byte(aggregateID.String()),
                Value:          payload,
            }, nil)

            if err == nil {
                db.Exec(`UPDATE outbox_events SET published_at = now() WHERE id = $1`, id)
            }
        }

        time.Sleep(100 * time.Millisecond)
    }
}

Key points:

• Uses FOR UPDATE SKIP LOCKED.
• Partitioned by aggregate_id to preserve ordering.
• Safe under crash.
• Horizontal scaling possible.

7. Handling Provider Webhooks (Where Most Systems Break)

Providers retry aggressively. Webhook must be idempotent.

We introduce processed_events.

CREATE TABLE processed_events (
    event_id UUID PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Safe Consumer Flow

Go Example (Webhook Processing)

func ProcessProviderResult(ctx context.Context, db *sql.DB, eventID uuid.UUID, verificationID uuid.UUID, result string) error {
    return withTx(ctx, db, func(tx *sql.Tx) error {

        // Dedup check
        var exists bool
        err := tx.QueryRow(`
            SELECT EXISTS (SELECT 1 FROM processed_events WHERE event_id = $1)
        `, eventID).Scan(&exists)
        if err != nil {
            return err
        }
        if exists {
            return nil
        }

        var state string
        err = tx.QueryRow(`
            SELECT state FROM verifications WHERE id = $1 FOR UPDATE
        `, verificationID).Scan(&state)
        if err != nil {
            return err
        }

        if state != "PENDING_PROVIDER" {
            return fmt.Errorf("illegal transition")
        }

        newState := "REJECTED"
        if result == "approved" {
            newState = "APPROVED"
        }

        _, err = tx.Exec(`
            UPDATE verifications
            SET state = $1, version = version + 1, updated_at = now()
            WHERE id = $2
        `, newState, verificationID)
        if err != nil {
            return err
        }

        _, err = tx.Exec(`
            INSERT INTO processed_events (event_id)
            VALUES ($1)
        `, eventID)
        return err
    })
}

Even if webhook arrives 5 times:

• First call updates state.
• Subsequent calls no-op.

Exactly-once business effect.

8. Manual Review as an Event

Human approval generates event:

VerificationManuallyApproved

Same consumer logic applies.

No bypass logic.

No special case.

Uniform state machine.

9. Safe Replays

We simulate replay of 48 hours of Kafka.

What happens?

• Dedup table prevents duplicate processing.
• State machine prevents illegal transitions.
• Side effects stored separately.

System converges to same state. Replay safety achieved.

10. Side Effects as Tasks

External actions (notifications, billing flags) are stored as tasks.

CREATE TABLE side_effect_tasks (
    id UUID PRIMARY KEY,
    verification_id UUID NOT NULL,
    type TEXT NOT NULL,
    status TEXT NOT NULL DEFAULT 'PENDING',
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Worker executes tasks with idempotency keys.

Replay does not duplicate.

11. Observability

We monitor:

• State transition latency
• Outbox lag
• Duplicate webhook rate
• Illegal transition attempts
• Consumer lag
• Side-effect retry rate

Without this, invariants degrade silently.

12. Failure Drills

We routinely run:

1. Kill consumer before offset commit.
2. Replay 24 hours of traffic.
3. Simulate provider retry storm.
4. Simulate partial DB outage.

System must:

• Not corrupt state.
• Not double-approve.
• Not violate invariants.

13. What We Did NOT Use

• Distributed transactions
• Two-phase commit
• Global locks
• Strong consistency across services
• Cross-service database writes

We used:

• Local DB transactions
• Transactional outbox
• Idempotent consumers
• Deterministic state machines
• Replay discipline

Final Takeaway

Verification systems are hostile environments. Retries are constant. Failures are normal. Replays are inevitable. Audit requirements are strict.

Exactly-once delivery is marketing. Exactly-once business effects are architecture. And they are achievable.

Through:

• Explicit invariants
• Aggregate boundaries
• Transaction discipline
• Idempotency everywhere
• Replay-first design

That is how you build a verification system at scale, without double approvals, ghost states, or operational chaos.