DEV Community

goodpa
goodpa

Posted on

The Ordering Layer — Or, "It Went Through" Is Not the Same as "It Went Through in Order"

Yesterday's piece ended with a promise: replay does not only repeat events, it reorders them. That sentence deserves its own article, because most teams have a guard against duplication and no guard at all against inversion.

Duplication is the failure you can name. Reordering is the failure that leaves no trace except a state that should be impossible. Nobody gets an alert when a shipment goes out before the payment clears — they get a support ticket three weeks later asking why the invoice and the tracking number disagree.

Every invariant is an ordering claim

Read your business rules as sentences about sequence and most of them turn out to be ordering constraints wearing a different costume.

"No shipment before payment." "No access before approval." "No disclosure before the notice." "No deletion before the backup is verified." "No charge after cancellation." Each one says: event A must be observed before event B, or the state is wrong.

We write these as ifs in application code — if payment.captured then ship() — and then we put a queue between the two halves. The queue preserves order within a partition, and partitions are chosen by a key that has nothing to do with the invariant. Two messages about the same order can land on different partitions, and then the only ordering guarantee left is the one the consumer accidentally implements.

Sequence numbers measure the wrong stream

The standard reassurance is "messages carry a sequence number." Ask what the number is scoped to. In most systems it is scoped to the producer, the partition, or the topic — not to the entity the business cares about.

A sequence number on a topic tells you the order in which the broker received messages from one producer. It says nothing about the order in which two different producers' messages about the same customer were applied, and nothing about what happened when one producer restarted and its counter reset.

The number you actually need is per entity: for this order, for this account, for this document. And almost nobody emits it, because emitting it requires deciding which entity owns the invariant — a decision usually deferred until the first incident.

Timestamps are a story about clocks

The fallback is the timestamp. It is worse than the sequence number, for a reason that is easy to state and hard to remember: you are comparing clocks you do not own.

Two services on two hosts write created_at. One has NTP drift of forty milliseconds. One had its clock stepped backward during the incident because someone fixed the time sync. Now the record that says which event happened first is sorted by which machine was more wrong.

Worse is the case where the timestamp is generated by the client. A mobile app that retries after twelve hours offline will happily present a request that is twelve hours old, with a timestamp to match, to a server that has already moved past that state. The server has two choices: reject it and lose work the user believes was saved, or apply it and move the state backward.

This is not a hypothetical. It is the ordinary behavior of any offline-first client, any long retry window, and any backfill script run against historical data. Ordering failures arrive with the most innocent inputs.

Where inversion actually shows up

Four patterns cover most of it, and all four look fine in a diagram.

The compensating action overtakes the original. A refund is issued, then the charge retry lands. The customer is charged after being refunded, and the ledger now shows a payment that the business believes was reversed. Reconciliation finds it a month later, if at all.

The grant arrives before the check. A permission is provisioned from one queue and a revocation is processed from another. For a window measured in seconds, the user has access they should not have. Nothing failed; both messages were delivered exactly once.

The delete outruns the write. A support agent deletes an account; a retry of an earlier write recreates a row in a table that was supposed to be empty. The deletion receipt is true at the moment it is issued and false an hour later. The customer's request was honored and then quietly undone.

The notification precedes the change. An email announces a plan change that has not been applied yet, and a second email announces the rollback. To the recipient, both are true statements about a system that flickered. To the auditor, the record shows a promise made before the state existed to support it.

None of these requires a bug in any single service. They require only two paths with independent ordering and an invariant that spans both.

The only thing that saves you is a decision about where order is enforced

There are two honest architectures, and one dishonest default.

The first is to make the invariant local: put both events in the same partition, keyed by the entity, and process them serially. This works, and it costs you throughput and a key design you must not change later.

The second is to make the consumer tolerant of any order: every handler checks the current state and refuses to move backward. This works, and it costs you a state machine per entity plus the discipline to write the check in every handler, forever.

The dishonest default is to assume the queue gives you order and to write handlers that apply whatever arrives. That is the architecture most systems actually have, and it is indistinguishable from the first two until the day two paths race.

The question to ask is not "is our queue ordered." It is: for each invariant, which component is responsible for refusing an out-of-order event, and what does it do when it sees one? If the answer is "nothing, it cannot happen," that is a claim about your topology, not a guarantee in your code.

This is an evidence problem before it is an engineering problem

Here is the part that matters for anyone keeping records: an append-only ledger that stores events in arrival order is not a causal record. It is a log of what this system heard, in the order it heard it.

Every register on our shelf assumes one event per row and one row per event in sequence. A correction ledger assumes the correction came after the error. A retraction ledger assumes the retraction came after the claim. A deletion receipt assumes the delete came after the write and before the verification. If the underlying stream can invert, then the register can be internally consistent and still describe a sequence that never happened.

That is a stronger claim than "we might have duplicates." A duplicate makes a row twice; an inversion makes the whole table's ordering meaningless while every row remains individually accurate.

The artifact: an ordering receipt

Five columns, one row per invariant, filled by reading code rather than by asking people what they think happens:

The invariant The two events it spans What order is assumed What refuses the wrong order What we observed the last time two paths raced
No shipment before payment capture, ship capture first None — ship handler reads a cached flag Unknown; no race test ever run
No access after revocation grant, revoke revoke wins State machine on account, per-handler 4-second window, found in logs
No charge after cancellation cancel, charge cancel first Charge handler checks status 0 observed in sampled month

The last column is the one that is usually empty, and an empty column is not a clean bill of health. It means nobody has looked while both paths were moving.

The number that comes out of this is a count: how many of your invariants are protected by a refusal in the consumer, versus how many rely on a producer or a broker to keep order. The second number is your exposure, and it does not appear in any dashboard.

The Friday exercise

Pick one invariant you would be embarrassed to have violated — money, access, or a disclosure to a customer. Find the two code paths that implement its two halves. Then answer three questions in writing:

  1. What key decides which partition each event lands on? If the two events can land on different partitions, order is not preserved, whatever the diagram says.
  2. What does the second handler do if it sees its event first? Not what it is supposed to do — what the code does on that branch, including the error path.
  3. When was the last time both paths ran at once? If the answer is "never, we tested them separately," then you have tested each half and not the sentence that connects them.

If you cannot answer question two without opening the file, that is the finding. The inversion is not a rare edge case; it is what your system does by default the first time the queue is busy.


If you have an invariant that spans two queues and you are not sure which one wins when they race, say so in the comments. We keep a five-column ordering worksheet — the invariant, the two events, the assumed order, what refuses the wrong order, what you observed — and I will send it over. No form, no email list.

Top comments (0)