Part 5 of a 5-part series on automating a multi-app Android release pipeline. Part 4 → ended on the first real production release going out cleanly. This part covers what showed up right after — a bug that had nothing to do with signing, versioning, tracks, or storage, and everything to do with a comment that stopped being true nine minutes after it was written.
TL;DR
- Right after the first real production release, both apps started showing the same login screen — staff's.
- Root cause: this org has exactly one self-hosted runner. Both apps build sequentially, on the same disk, sharing the same Metro bundler cache.
- A comment already in the codebase claimed this couldn't happen, because staff and student "each get their own runner." That was true — for eight days, on a different runner setup, before a later, unrelated commit quietly removed the isolation the comment depended on. Nobody went back to check whether the comment still held.
-
The real fix wasn't isolating the runners again — it was making the cache actually aware that "which app" is something that should invalidate it. Metro's own
cacheVersionconfig field does exactly that. - The fix stops it happening again. It didn't undo what had already shipped — both apps needed a fresh build and a fresh release after the fix landed.
Live, and Showing the Wrong Screen
The message came in a few hours after the first real production release went out: "both apps are showing me the login screen for staff app." Not a color slightly off, not an icon mismatch — the student app, downloaded fresh from the Play Store, opened straight to staff's actual login screen.
That's a different category of bug than anything else in this series. Signing, versioning, tracks, storage — all of that could delay a release or misreport one. This was the first time the pipeline had actually shipped the wrong thing to a real user.
Ruling Out the Obvious, Then Finding the Comment That Explained Everything — Except It Was Wrong
The first instinct was styling — a leftover theme value, a stale asset. That theory didn't survive five minutes: this wasn't a color, it was an entire screen. Something was bundling one app's actual code into the other's build.
Then a comment already sitting in the workflow file for this exact pipeline turned up, and it looked like the whole answer:
# Staff and student each get their own runner -- previously both built
# sequentially on ONE shared machine, which meant Metro's on-disk bundler
# cache (which staff's build populates first) could be reused for
# student's build right after, even though EXPO_PUBLIC_ROUTE had changed.
# ...
# Separate runners share no disk and no cache, so this can't happen --
# the same reason the Play Store release pipeline never could.
Word for word, this described the exact symptom being debugged — and confidently declared it impossible. The only way to actually trust that was to check it directly instead of reading it and moving on:
$ gh api orgs/<org>/actions/runners --jq '.runners[] | {name, status}'
{"name": "<our-only-runner>", "status": "online"}
One runner. Not two. The comment's entire premise — that staff and student physically couldn't share a disk — was false, right now, in production, while confidently telling anyone who read it the opposite.
Why It Was Specifically Staff's Screen, on Both Apps
With the real constraint confirmed (one shared machine, not two), the mechanism became straightforward. metro.config.js — the config for Metro, the tool that actually bundles this app's JavaScript — picks which app's styling to bundle based on an environment variable:
const ROUTE = process.env.EXPO_PUBLIC_ROUTE ?? "staff";
Metro also keeps its own on-disk cache of already-bundled work, to avoid redoing it. That cache key was never told that ROUTE was something worth caring about. The build matrix runs staff first, then student, back to back, on the one machine that exists. Staff's build populates Metro's cache. Student's build starts moments later, on the same disk — and as far as Metro's cache key is concerned, nothing relevant has changed. It hands back staff's already-bundled output.
That's the specific detail that makes this bug make sense rather than look random: it wasn't a coin-flip swap between the two apps. Staff always builds first, so staff's bundle is always what gets silently reused — which is exactly why both apps ended up showing staff's screen, not a symmetric mix-up.
The Comment Was True. Then It Wasn't, Nine Minutes Later.
This is the part worth sitting with longer than the fix itself. Checking the actual commit history:
Eight days earlier, this exact bug happened once before, in a smaller form — colors leaking between apps, not full screens. The fix at the time was real and correct: split the build into a matrix, one job per app, each on GitHub's own hosted runners. On GitHub-hosted infrastructure, each matrix combination genuinely does get its own separate, disposable machine. The comment — "separate runners share no disk and no cache, so this can't happen" — was true the day it was written.
Eight days after that, a different, unrelated change replaced the hosted runners with this org's own self-hosted one, to get off a shared billing quota entirely (that migration is its own story — see Part 3). A brief, dynamic version tried picking a hosted runner as a fallback when the self-hosted one was busy; nine minutes after that was added, it was removed again as unnecessary complexity, replaced with a flat, permanent assignment to the one self-hosted runner. That is the exact moment the comment's premise stopped being true. Both apps now pointed at the same runner label — meaning the same physical machine — and nothing about that change touched, or even mentioned, the comment sitting eight days earlier claiming isolation as settled fact.
Nobody was wrong at either point. The first fix was correct when it shipped. The infrastructure commit was a reasonable simplification on its own terms. The bug lived entirely in the gap between them — a true statement that outlived the conditions that made it true, with nothing in the process that would ever prompt someone to go back and re-check it.
The Actual Fix
Re-isolating the runners was the tempting fix — it's what worked before. It was also treating the symptom: the underlying cache still wouldn't know what it was caching. The real fix was making the cache itself aware of the one thing that actually determines its output:
const ROUTE = process.env.EXPO_PUBLIC_ROUTE ?? "staff";
// Metro's cache has no built-in notion that ROUTE changes what it should be
// bundling. cacheVersion is Metro's own mechanism for exactly this -- forces
// a genuinely separate cache whenever the value changes, so a route switch
// can never again silently reuse another route's bundle, regardless of how
// many runners this ever ends up on.
config.cacheVersion = ROUTE;
One line. It doesn't depend on runner topology at all — one machine or ten, this can't happen again, because the cache key itself now reflects the thing that actually changes. Verified it was even valid JavaScript before pushing it anywhere (node --check metro.config.js) — a small habit, and a direct one born from an earlier mistake in this same week's work (Part 4 has that story).
The stale comment got fixed too — not just deleted, replaced with what's actually true now, including the part where the old claim was wrong, so the next person reading it gets the real history instead of a second false sense of safety.
What the Fix Didn't Do
Merging the fix stopped this from happening to the next build. It did nothing for the two builds that had already shipped. Both apps' live production releases had been built under the broken cache behavior — the fix doesn't reach backward into an already-published AAB. The remaining step was the least surprising one in this whole series: rebuild both apps for real, under the fixed config, and push a fresh release to each.
Lessons From Part 5
| What broke | What we learned |
|---|---|
| A comment declaring a bug class "impossible" | Verify the claim directly (gh api orgs/<org>/actions/runners), every time — especially the ones confident enough that nobody questions them |
| Isolation as the safety mechanism | Infrastructure-level isolation (separate runners) is fragile the moment infrastructure changes; a cache key that reflects its real inputs doesn't depend on topology at all |
| A true comment, un-revisited after the infrastructure under it changed | Nothing enforces that a comment stays true after the code around it changes — the same problem as every stale-comment fix elsewhere in this series, just with a real user-facing consequence this time |
| The fix landing didn't fix what already shipped | A pipeline fix and a production incident are two different problems; closing one doesn't automatically close the other |
Where the Series Actually Ends
Five parts, and the throughline turns out to be the same idea wearing different clothes each time: a signing key nobody mentioned, a version number a local file silently owned, a track name nobody was watching, a workflow expression rule invisible to a normal syntax check, a storage quota nobody had sized, a notification relay that could fail independently of the thing it reported on, and finally a comment that was honest the day someone wrote it and silently stopped being honest over a week later. None of these were carelessness. Every one of them was a true thing that quietly stopped being true, discovered only once it actually mattered.
That's the real shape of what "finished" means for a pipeline like this: not the absence of bugs, but a system that tells you the truth the moment something breaks, instead of the moment someone finally goes looking.
The mobile apps discussed in this series are closed-source, but the backend and web frontend they talk to are public — school-be-laravel (Laravel, multi-tenant) and school-fe-nextjs (Next.js) power the same Cyfamod School Management System, alongside school-site-engine for the public-facing school sites. More on what we build at cyfamod.com.
Hit a version of any of these? I'd genuinely like to hear which theory you tried first — drop it in the comments.
I write about the debugging journeys nobody puts in the docs — more at cycy.is-a.dev.
Top comments (0)