Lk
Lab notes

Case study

The Events That Arrived in the Wrong Order

An e-commerce team migrated from synchronous order processing to Kafka. Two weeks later, orders started showing impossible states. The root cause was a missing partition key.

The bug report read: "Customer charged but order shows as cancelled." That's the kind of sentence that makes you put down your coffee.

My client was a mid-size e-commerce company, roughly 30,000 orders a day. They'd spent the previous quarter migrating their order pipeline from a synchronous REST-based flow to an event-driven architecture using Kafka. The migration had gone live two weeks before I arrived. The first week went fine. The second week, customer support started seeing tickets that didn't make sense.

An order would show as "cancelled" in the dashboard, but the payment had gone through. Or an item would show "shipped" while the inventory system still had it reserved. Small numbers — maybe 15 tickets out of thousands of orders — but each one required a support engineer to manually reconcile the state across three services.

The architecture, on paper

The old system was simple. A single POST /orders endpoint talked to the payment service, inventory service, and notification service in sequence. If payment failed, the order stopped. If inventory was low, the order stopped. Synchronous, slow, tightly coupled — but predictable.

The new system published events to Kafka. OrderCreated, PaymentProcessed, InventoryReserved, OrderConfirmed, OrderShipped. Each downstream service consumed the events it cared about and did its thing independently.

On paper, this looked clean. In practice, it was hiding a race condition.

Chasing the ghost

I started with the logs. Kafka consumer logs, order service logs, payment service logs. I picked one of the broken orders — order #88431 — and traced every event.

The timeline should have been:

  1. OrderCreated (10:14:02.331)
  2. PaymentProcessed (10:14:02.887)
  3. InventoryReserved (10:14:03.104)
  4. OrderConfirmed (10:14:03.210)

What the order service actually received:

  1. OrderCreated (10:14:02.331)
  2. InventoryReserved (10:14:03.104)
  3. OrderConfirmed (10:14:03.210)
  4. PaymentProcessed (10:14:03.892)

The PaymentProcessed event arrived after the OrderConfirmed event. By the time the order service processed it, the order had already moved to the "confirmed" state. The payment event hit a state machine transition that wasn't valid from "confirmed," so the service logged a warning and dropped it. No error. No retry. Just a silent discard.

Meanwhile, a customer cancellation came in at 10:14:05, which transitioned the order to "cancelled." But the payment had already been captured. So the system showed "cancelled" while the customer's card had been charged.

The partition key that wasn't

Kafka guarantees ordering within a partition. If two messages land in the same partition, consumers see them in the order they were produced. But across partitions, there's no such guarantee.

The team's Kafka producer configuration looked like this:

await producer.send({
  topic: 'order-events',
  messages: [{
    value: JSON.stringify(event),
  }],
});

No key field. When you don't specify a partition key, Kafka's default partitioner distributes messages across partitions using a round-robin or sticky strategy. Events for the same order could — and did — end up on different partitions.

The order-events topic had six partitions and three consumer instances. Under normal load, events for a single order usually happened to land on the same partition because they were produced in rapid succession during a sticky batch window. But under moderate load, when multiple orders were being processed simultaneously, the sticky partitioner would rotate partitions between batches. Events for order #88431 ended up on partitions 2, 4, and 2 again.

Two consumer instances each pulled from different partitions, processing events for the same order concurrently. Whichever consumer happened to be faster won the race.

The fix

The fix itself was small. Add the order ID as the partition key:

await producer.send({
  topic: 'order-events',
  messages: [{
    key: event.orderId,
    value: JSON.stringify(event),
  }],
});

With a key set, Kafka hashes it to a consistent partition. All events for order #88431 go to the same partition. Same partition means same consumer, which means guaranteed ordering.

But the fix raised a second problem. The team had six partitions and was planning to add more as traffic grew. Repartitioning a topic changes the hash-to-partition mapping, which would temporarily break the ordering guarantee during the transition. We ended up writing a consumer-side ordering buffer — a short window where the consumer collects events for an order, sorts them by a sequence number we added to the event payload, and then processes them in order.

interface OrderEvent {
  orderId: string;
  sequenceNumber: number;
  type: string;
  payload: Record<string, unknown>;
  timestamp: string;
}

Adding the sequence number was insurance. Partition keys handle ordering 99% of the time, but sequence numbers let the consumer detect and correct out-of-order delivery when things get weird — rebalances, failovers, future repartitioning.

The state machine problem

There was a deeper issue, though. The order service's state machine silently dropped events that didn't match a valid transition. That design choice — ignore what you don't understand — turned ordering bugs into data corruption. A state machine that rejected invalid transitions with an error and a dead letter queue would have caught this on day one.

We changed the state machine to push unexpected transitions to a dead letter topic with full context: the event, the current state, and the expected states. An alert fired when anything landed there. Within the first hour after deploying, we caught two more ordering edge cases we hadn't found in logs.

Tip

If your event-driven state machine silently drops events that don't fit the current state, you're turning ordering bugs into silent data loss. Reject them loudly.

What I took away from this

The team wasn't wrong to move to event-driven architecture. The old synchronous pipeline was genuinely a bottleneck, and the new system was already handling 3x the throughput with lower p95 latency. The problem wasn't the pattern — it was a missing constraint.

Event ordering isn't a feature you add later. It's a design decision you make on day one, right alongside "which events exist" and "who consumes what." The Kafka documentation talks about partition keys early and often, but it's easy to skim past when you're focused on getting the basic producer-consumer flow working.

I've seen this same class of bug in three other event-driven migrations since. The specifics differ — sometimes it's SQS FIFO queues with missing group IDs, sometimes it's a Pub/Sub subscription with concurrent consumers — but the shape is always the same. Events that must be processed in order are being processed concurrently, and the system works fine until it doesn't.

What's your strategy for handling event ordering? Do you rely on partition keys alone, or do you layer on sequence numbers at the application level?