DEV Community

Cover image for System Design Interview: You Rolled Back. Why Is It Still Broken?
Emily Woods
Emily Woods

Posted on Originally published at Medium Fully Autonomous

System Design Interview: You Rolled Back. Why Is It Still Broken?

Your old code may be perfectly healthy and completely unable to read what the new code left behind.

You have an answer ready for the deployment question.

Release gradually, watch the error rate, and roll back if anything looks wrong.

Honestly, that sounds reassuring until someone asks what happens to the data written before you roll back.

Now the conversation gets interesting, because yesterday’s code is returning to a world that has already changed.

I like this follow up because it turns a familiar safety button into something we actually have to explain. Which version are we returning to, and can it process the work already waiting for it?

Let’s work through a small notification service. This is an original practice scenario. We will keep the example narrow enough to reason about on paper, then add the checks that a real release plan needs.

A rollback lever restores old code while an inbox still contains new-format data that the old reader cannot understand.

Follow one message through the failed recovery

Imagine a service that stores notification jobs before workers send them. Each job carries a stable job ID, a format marker, and a recipient. Our exercise changes the representation of the recipient while keeping the intended email delivery exactly the same.

In format A, the recipient is a plain email address string.

In format B, the recipient is an object containing a channel and an address.

Both example messages mean email to the same fictional address, mia@example.com.

Version V1 understands only A. A proposed replacement understands A and B but immediately starts writing B. The developer tests the replacement with its own messages, and everything works.

During rollout, however, some V1 workers are still running. One of them picks up a B message and rejects it. The error rate rises, so the team restores V1 everywhere.

That B message is still in the durable queue. Restoring an application image did not convert its recipient object into a string, and every worker now has the parser that cannot read it.

Three panels show a new-format message being stored, the application returning to V1, and V1 failing to read the message that remains.

Notice that we do not need a huge fleet or a mysterious race to explain this failure. One retained message and one incompatible reader are enough. The rollback can finish successfully as a deployment operation while the notification service remains broken.

If you want to work through this with a fuller system, open the rental-watchlist notification design and sketch its notification pipeline before reading on. Then add our rollback twist by changing one message field while an older worker is still running. Keep your sketch beside the compatibility table below and see which reader-message combination breaks.

Make compatibility a table you can actually test

Before choosing a recovery command, separate release versions from message formats. V1, V2, and V3 describe application releases. A and B describe stored data. They are related, but they are not interchangeable labels.

Our proposed V2 can decode either format into the same internal email job. It still writes A. V3 keeps that decoder and changes the writer to B.

A compatibility matrix shows V1 reading A but rejecting B, while V2 and V3 can read both formats.

Read the dangerous cell first. V1 with B is unsupported, regardless of whether V1 passed yesterday’s tests or has an excellent uptime history.

For a tiny rehearsal, make two fixtures with different job IDs and the same intended recipient. Give one the A representation and one the B representation. Feed both to each reader.

V1 must accept A and reject B;

V2 and V3 must normalize both into the same channel and address without changing the job IDs.

Keep malformed input separate from older input. A missing address or an unknown channel should follow a defined error path, not silently become a different delivery. Being tolerant of an older format does not mean accepting arbitrary data.

Google’s API compatibility guidance distinguishes wire, source, and semantic compatibility. That last distinction is useful even in this internal queue exercise. Successfully decoding a field does not prove the resulting notification still means what its producer intended.

Give the rollout a compatible place to return to

AWS describes a two-phase approach in its rollback-safety engineering article. Readers learn the new representation before writers begin producing it. The intermediate release provides a compatible rollback target, and preparation must cover every relevant reader rather than merely enough hosts to pass a health threshold.

Applied to our example, the release plan has three states.

  • V1 reads A and writes A, which is our starting point.
  • V2 reads A and B but continues writing A, giving us a preparation phase without new-format output.
  • V3 reads A and B and writes B, activating the new representation after preparation is verified.

A release sequence goes from V1 to the prepared V2 and then V3, with V2 marked as the compatible rollback target after B messages exist.

Once B exists, returning from V3 to V2 preserves the ability to read it. Returning all the way to V1 does not. Write that boundary into the release plan instead of leaving rollback to mean whichever older image somebody remembers.

For this scenario, preparation includes worker instances, retry processors, and any tool that replays stored jobs. Also check the image used when a replacement instance starts. A fleet can look upgraded while an old launch configuration quietly reintroduces V1 later.

The useful question is whether any reader that can receive B still lacks support for it. A green dashboard showing healthy processes is not an answer to that question.

A feature flag cannot rewrite yesterday’s messages

Suppose B writing is controlled by a flag. Turning the flag off may stop future B messages once every writer observes the change. It does not remove the B messages already waiting, being retried, or preserved for replay.

That distinction gives us two separate recovery decisions. We can stop producing more incompatible work, and we can choose how to process the work that already exists. Treating those as one action hides the remaining failure.

In the toy system, retaining V2 readers is the simplest compatible recovery. In a real incident, the decision depends on the actual defect. If V2 is also broken, a compatible forward repair or a carefully controlled transformation may be necessary. There is no universal command that makes arbitrary old code safe against new state.

Please do not make clearing the queue the happy ending of this example. Those messages represent accepted work. If an unsupported message needs isolation, preserve its identity and payload for investigation and an explicit recovery path rather than silently discarding it or retrying it forever.

Format compatibility also does not guarantee duplicate-free delivery. If a worker sends an email and crashes before recording completion, replay can repeat the external effect. Keep delivery identity and recovery semantics intact during a format change; changing a JSON shape does not solve that separate problem.

Rehearse the transition with messages left behind

The most revealing test here does not start with a clean queue for every release. It deliberately carries state across the transition.

Use a fake sender and a controlled test environment.

Begin with an A message pending, move through the prepared release, create a B message during activation, and then return to V2 without replacing the stored fixtures.

Verify that both messages still resolve to the intended recipient and retain their original IDs.

A four-step rehearsal leaves A pending, creates B, returns to V2 with both intact, and checks the decoded recipient using a fake sender.

Then add one old worker during activation and show that your release gate prevents that unsupported combination. Test a restart using the configured launch image too, because replacing a process should not change the compatibility promise.

Observe decoding failures by reader and format version, pending-job age, and successful processing. In the fake sender, inspect the actual channel and recipient rather than counting successful parser calls. Test replay and acknowledgement failures separately so passing the format checks does not masquerade as an end-to-end delivery guarantee.

Keep the old decoder until your retention and replay rules allow its removal. An empty live queue alone says nothing about an archive or a dead-letter store that someone may reintroduce later. Decide whether those records will remain readable, be converted safely, or be retired under an explicit policy.

Add this follow up to a system you can already explain

Now return to your notification pipeline sketch and name the release you can safely restore after the new format reaches the backlog.

Explain why that reader can still process the work already accepted, including messages that return through a retry or replay.

Leave the exercise with a small compatibility matrix and a named recovery target. Beside them, write the evidence required before enabling the new writer and the retained data that still needs the old reader.

That gives your answer something much stronger than an emergency button. You can explain what survives a deployment, who can still understand it, and why the recovery path actually leads somewhere useful.

~ Emily

Top comments (0)