Outbox Pattern: Save Data and Publish Events Reliably

A user places an order. Your service saves it to the database, then publishes an order.created event so that billing, email and stock services can react. Two lines of code, and on most days it works.

Then one day the process is killed between the two lines. A deploy rolls out, the container runs out of memory, the broker is unreachable for a second. The order is in the database. The event never left. Billing never charges, the customer never gets an email, and nothing in your logs looks wrong, because nothing threw an error.

This is the dual-write problem, and it is one of the most common causes of "the data is inconsistent and nobody knows why". This post explains it, then fixes it with the outbox pattern: how it works, how to build the relay, why consumers must handle duplicates, and what to monitor. It is the same idea I used for the order paid event in the payment redirect post, explained on its own.

The dual-write problem

Your service needs to change two separate systems, the database and the message broker, and tell both the truth. But they cannot commit together, so there is always a gap between the two writes.

Swapping the order does not help:

• Save first, publish second. A crash in the gap leaves a saved order with no event. Other services never find out.
• Publish first, save second. A crash in the gap announces an order that does not exist. Other services act on something false.
• Wrap it in try/catch. That only handles errors that the process survives. A killed process runs no catch block.

You could use a distributed transaction (two-phase commit) across the database and the broker, but most brokers do not support it, and where they do, it is slow and fragile. There is a simpler way that needs nothing but your database.

The idea: write the event into your own database

A database transaction can make two writes atomic, as long as both are in the same database. So do not publish the event during the request. Instead, save the event as a row in an outbox table, in the same transaction as the business change.

1. The order and its event are saved together in one transaction. Either both exist or neither does.
2. A separate process, the relay, reads unpublished outbox rows and sends them to the broker.
3. Only after the broker confirms does the relay mark the row as published.

Now "the order exists but the event is lost" is impossible. If the order is there, the outbox row is there, and the relay will keep trying until it is delivered. Publishing turns from a hope into a retry loop.

The outbox table

A minimal table in PostgreSQL:

CREATE TABLE outbox (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_type text        NOT NULL,           -- e.g. 'order'
  aggregate_id   text        NOT NULL,           -- e.g. the order id
  type           text        NOT NULL,           -- e.g. 'order.created'
  payload        jsonb       NOT NULL,
  created_at     timestamptz NOT NULL DEFAULT now(),
  published_at   timestamptz
);

-- The relay only ever looks at unpublished rows, so index only those.
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;

The partial index keeps the relay's query fast even when the table holds millions of old, published rows.

Writing the order and the event together

In the request path, you only touch the database:

async createOrder(dto: CreateOrderDto): Promise<Order> {
  return this.db.$transaction(async (tx) => {
    const order = await tx.order.create({ data: { userId: dto.userId, total: dto.total } });

    await tx.outbox.create({
      data: {
        aggregateType: 'order',
        aggregateId: order.id,
        type: 'order.created',
        payload: { orderId: order.id, userId: order.userId, total: order.total },
      },
    });

    return order;
  });
}

There is no broker call here, so there is nothing that can fail halfway. If the transaction rolls back, both rows disappear. If it commits, both are safe on disk. This is the same transactional thinking as in the sale-as-one-transaction post.

The relay: two ways to build it

The relay's job is to move rows from the outbox to the broker. There are two common designs.

Polling relay

A scheduled job reads a batch of unpublished rows, publishes them, and marks them published. The key detail is FOR UPDATE SKIP LOCKED, which lets several relay instances run at once without picking up the same rows:

@Interval(1000)
async relay(): Promise<void> {
  await this.db.$transaction(async (tx) => {
    const rows = await tx.$queryRaw<OutboxRow[]>`
      SELECT id, type, payload
      FROM outbox
      WHERE published_at IS NULL
      ORDER BY created_at
      LIMIT 100
      FOR UPDATE SKIP LOCKED`;

    for (const row of rows) {
      // The row id doubles as a dedupe key for the broker or queue.
      await this.queue.add(row.type, row.payload, { jobId: row.id });
    }

    if (rows.length > 0) {
      await tx.outbox.updateMany({
        where: { id: { in: rows.map((r) => r.id) } },
        data: { publishedAt: new Date() },
      });
    }
  });
}

This uses a queue such as BullMQ as the broker, but the shape is the same for Kafka, RabbitMQ or SQS. Polling is simple, works with any SQL database, and is enough for most systems. The costs are a small delay (up to one polling interval) and a little constant query load.

Change data capture (CDC)

Instead of polling, a tool such as Debezium reads the database's write-ahead log and publishes each new outbox row as soon as it commits. Delay drops to near real time, and the database is not queried repeatedly. The cost is another system to run, secure and monitor.

My advice: start with polling. Move to CDC only when the delay or the query load is a real, measured problem.

Duplicates are normal, so consumers must be idempotent

Look at the relay again. It publishes first, then marks the row. If it crashes in between, the row stays unpublished, and the next run publishes it again. That means the broker can receive the same event twice. This is at-least-once delivery, and it is the price of never losing an event.

So the consumer must be safe to run twice. The simplest tool is a table of processed event ids with a unique key:

async handleOrderCreated(event: OrderCreatedEvent): Promise<void> {
  await this.db.$transaction(async (tx) => {
    const inserted = await tx.$executeRaw`
      INSERT INTO processed_events (event_id) VALUES (${event.id})
      ON CONFLICT (event_id) DO NOTHING`;

    if (inserted === 0) return; // seen before: skip

    await this.billing.createInvoice(tx, event.payload.orderId);
  });
}

The insert and the real work happen in one transaction, so they succeed or fail together. If the same event arrives again, the insert conflicts and the handler does nothing. For work that has an external side effect (an email, a call to another API), pass the event id along as an idempotency key so that system can dedupe too. The idempotency post covers this in detail.

Walking through every crash

A good way to trust a design is to crash it at every step:

No crash point loses an event. The worst outcome is a duplicate, and duplicates are handled by the consumer. That is the real trade: you swap "events sometimes disappear" for "events sometimes repeat", and repeating is a much easier problem to solve.

Things that bite in production

• Ordering. Rows are published in created_at order, but with several relays or consumers, order is not guaranteed across events. If order matters for one entity (all events of one order), publish them in order through a single partition or queue keyed by aggregate_id, or include a version number and let the consumer ignore stale events.
• Table growth. The outbox grows forever if you let it. Delete or archive published rows on a schedule, for example those older than a few days, in small batches so the cleanup does not lock the table.
• A poison row. One event that always fails (a bad payload) can block a batch. Count attempts, and after a limit move the row to a dead-letter table and alert, instead of retrying forever.
• Long transactions. The polling example holds a transaction while it publishes. Keep batches small, and set a timeout on the publish call.
• Schema changes. Events are a contract. Version the type (order.created.v2) or the payload when you change them, and keep old versions working until consumers move.
• Sensitive data. Do not put secrets or full personal data in payloads. Send ids and let consumers fetch what they need.

What to monitor

The outbox is easy to forget, so give it two numbers on a dashboard (see the Prometheus and Grafana post):

1. Unpublished row count. If it keeps growing, the relay is stuck or too slow.
2. Age of the oldest unpublished row. This is your delivery delay. Alert when it passes a limit you choose, such as a few minutes.

Also log the event id at every step (written, published, consumed) so one id lets you trace an event end to end. The structured logging post shows how.

When you do not need an outbox

The pattern adds a table, a relay and a consumer-side dedupe. Skip it when:

• Losing an occasional event does not matter (analytics pings, best-effort cache warmers).
• You do not publish events at all and just call another service synchronously, accepting that it can fail.
• The thing you publish is the only write. If the broker is your single source of truth, there is no second system to disagree with.

For anything where "saved but never announced" costs money or trust (payments, orders, stock, emails), the outbox is cheap insurance.

A short checklist

1. Business change and outbox row in one transaction. No broker call in the request path.
2. A relay that publishes, then marks as published. Start with polling and FOR UPDATE SKIP LOCKED.
3. A stable event id, sent with every event.
4. Idempotent consumers, with a unique processed-events key or an idempotency key.
5. Cleanup of old rows and a dead-letter path for poison events.
6. Metrics for backlog size and oldest-unpublished age.

The outbox does not make distributed systems simple. It does something smaller and more useful: it turns a failure you cannot see into one you can retry, count and alert on. That is usually all a reliable system needs.