Apple’s first foldable iPhone, the iPhone Duo, has barely hit the market, and users are already posting photos of screens with the same horizontal crack. The internet named it “Crackgate” within days, echoing “Antennagate” from 2010. Apple has acknowledged that a small number of units have inner display problems and is routing affected users to AppleCare for replacements.
I don’t build phones. But reading about it, I kept thinking that this is a story about shipping software too. The failure modes are the same, and so are the lessons.
Your test suite can’t see what only happens one time in a thousand
A former Apple screen engineer quoted in the coverage made a point every developer should tattoo somewhere visible. When a defect is rare enough, you can run a thousand units through testing and never see it. You need at least one failure to show up before you can start chasing it.
That’s the flaky-test problem and the race-condition problem, scaled up. A bug that appears in 0.1% of runs will pass your CI pipeline nearly every time. It will also fail for someone in production within hours, because production is where the volume is.
Some practical takeaways:
Treat “it passed” as evidence, not proof. A green build means you didn’t observe a failure, not that none exists.
Invest in observability before you need it. If a rare failure appears, you want the logs, traces, and device or environment data already waiting.
Take rare reports seriously. One user’s weird screenshot can be the first visible tip of a systemic problem.
Pattern recognition is a superpower
One analyst noted that the consistency of the reports, the same crack in roughly the same place, suggests something systematic, perhaps tolerances in the hinge or how the display is supported during folding. Random damage looks random. A repeated pattern points to a cause.
This is exactly how good debugging works. When dozens of bug reports look alike, resist the urge to treat them as dozens of separate problems. Look for the common factor: the same OS version, the same region, the same batch of dependencies, the same deploy window. The article also raises the possibility of a defective batch of components from a supplier, which is the hardware version of a bad dependency release.
Rollbacks are easy in software, which is exactly why we skip the discipline
Hardware teams live with a brutal constraint: you can’t push a patch to a cracked screen. That’s why Apple’s options here are replacements, service programs, and possibly something bigger. A recall would be unusual for the company, but it’s on the spectrum.
Software teams have the opposite problem. We can fix almost anything after the fact, so we get sloppy about before the fact. Feature flags, canary deploys, and staged rollouts exist because the ability to fix things quickly is not a substitute for limiting blast radius. The article mentions that earlier this year Apple shipped a software fix for a cellular connectivity problem affecting some customers on one carrier. Fast fixes are great, but the people who hit the bug first still had the bad experience.
If you can roll out to 1% before 100%, do it. The 1% will find your thousand-to-one bug for you.
The communication lesson is older than the bug
The 2010 episode is remembered less for the antenna than for the response. Steve Jobs initially brushed off complaints by implying users were holding the phone the wrong way, and Apple eventually offered free protective bumpers.
Compare that with the current statement, which acknowledges the issue plainly and tells people what to do. Whatever you think of the company, the second approach is the one engineering teams should copy in an incident:
Acknowledge quickly. Silence reads as denial.
Say what you know and don’t know. Apple admitted a problem without claiming to know its scale.
Give users a clear next step. “Contact support for a replacement” beats “we’re looking into it.”
Never blame the user for using the product as designed. If folding a foldable breaks it, that’s a design problem.
I’d add one for status pages and postmortems: don’t let a competitor tell your story first. Samsung ran an ad within days of the news. Your equivalent might be a rival’s blog post about your outage.
High expectations make small failures loud
The Duo costs upwards of two thousand dollars, and the company had publicly emphasized waiting until the engineering was right instead of shipping something half-baked. That’s a strong promise, and a strong promise raises the stakes on every defect.
Software has its own version of this. The more you market reliability, polish, or “it just works,” the less forgiveness you get for a visible failure. A scrappy side project can have bugs and be charming. A payments platform that has built its brand on trust cannot.
That doesn’t mean promise less. It means match your testing, monitoring, and incident readiness to the promises you make.
The takeaway
Even a company with enormous engineering resources and tightly controlled supply chains can ship a rare, hard-to-catch defect. If it can happen to them, it can happen to your team. The question isn’t whether a one-in-a-thousand bug will reach users. It’s whether you’ll find out fast, understand the pattern, limit the damage, and tell people the truth.
Build for that moment before it arrives.
Have you ever shipped a bug that your tests had no realistic way to catch? How did you find it, and what changed afterward? Share it in the comments.
Sources:
Michael Acton, “Apple hit by ‘Crackgate’ outcry over iPhone Duo screen problems,” Financial Times, San Francisco. https://www.ft.com/content/324875fb-8631-4339-8516-6057de9ac085 (add the publication date from the article page; FT content may require a subscription)
Top comments (10)
The retry/idempotency angle is interesting here. Imagine a payment service where the provider successfully authorizes a charge, but the response times out before reaching your backend. The retry passes through CI just fine, yet in production it can create a second authorization if the original request isn't safely deduplicated. I'd want to test that exact failure window, including concurrent retries, and verify that the idempotency record and payment state can't diverge. That's where a rare failure can turn into a real data integrity issue.
The point about “it passed” being evidence rather than proof is the one that resonates most. A green build only tells you that you didn’t observe a failure, and a one-in-a-thousand bug will sail through CI and then show up in production within hours, simply because production is where the volume is.
I’d add that the staged rollout advice and the observability advice work as a pair. A 1% canary only helps if you can actually see the problem when it appears, so the logs, traces, and environment data need to be in place before the first rare report comes in. Otherwise you get the signal but can’t trace it back to a cause.
The communication section was a nice touch too. Acknowledging the issue quickly, saying what you do and don’t know, and giving users a clear next step is exactly what a good incident update looks like.
To answer your closing question: the bugs I’ve found hardest to catch were the ones that depended on timing or on a particular combination of environment and data, and what changed afterward was usually better logging around those code paths rather than more tests.
This is a great example of how a seemingly small hardware failure can reveal some of the most important lessons in software engineering. What stands out to me is the idea that passing tests should never be confused with proving that a system is reliable. Testing provides evidence about the behavior we have observed, but it cannot guarantee that every possible failure mode has been eliminated.
The one-in-a-thousand failure scenario is particularly interesting. A defect that occurs in only 0.1% of executions might remain invisible during a relatively small test run. However, when a system operates at scale, those rare failures become inevitable. This is why reliability engineering needs to focus not only on preventing defects but also on detecting, understanding, and containing them when they inevitably reach production.
I also strongly agree with the emphasis on pattern recognition. When multiple users report similar symptoms, the natural temptation is to investigate each incident independently. But repeated failures often indicate a shared underlying cause. It could be a particular dependency version, a deployment configuration, a hardware batch, a timing condition, or an interaction between components that was never adequately tested.
This is where observability becomes essential. Logs, metrics, distributed traces, and contextual information about the environment can transform a vague production complaint into a reproducible engineering problem. Without that visibility, teams are left guessing. With it, even an intermittent failure can become an opportunity to improve the system.
The comparison between hardware and software rollbacks is another valuable point. Software teams have the advantage of being able to deploy fixes relatively quickly, but that flexibility can create a false sense of security. A fast rollback is useful, yet it does not eliminate the impact experienced by the users who encountered the defect. Canary deployments, feature flags, progressive delivery, and automated rollback mechanisms are important precisely because they limit the number of people exposed to an issue before it becomes widespread.
I would also add that the 1% rollout strategy is only effective when it is combined with meaningful monitoring and clear rollback criteria. A rare failure may not appear immediately, even in a canary deployment. Teams need to monitor the right signals, allow sufficient observation time, and understand which failures require stopping the rollout.
The communication lesson is equally important. When users report a problem, acknowledging the issue and providing a clear next step builds trust. Engineering teams do not need to have every answer immediately, but they should communicate what is known, what remains uncertain, and what they are doing to investigate. Blaming users or dismissing unusual reports only delays the discovery of potentially systemic problems.
One additional lesson I find valuable is that reliability expectations should influence engineering decisions from the beginning. A product that promises exceptional reliability, security, or seamless operation needs testing, observability, and incident-response practices that reflect those expectations. Reliability is not simply a feature that can be added after launch; it is an engineering discipline.
Ultimately, the goal of testing is not to prove that software cannot fail. It is to reduce uncertainty, expose weaknesses, and prepare the system to respond when failures occur.
A rare bug is not necessarily evidence of poor engineering. Ignoring the signals it provides, failing to investigate recurring patterns, or having no mechanism to contain its impact is a much bigger problem.
The strongest engineering teams are not necessarily the ones that never experience production failures. They are the ones that detect failures quickly, learn from them systematically, minimize their impact, and use those lessons to make the next release more resilient.
This is a reminder that reliability is not measured only by how often a system works, but also by how intelligently it responds when it doesn't.
The point about 'Green builds are evidence, not proof' hits at a fundamental flaw in modern CI/CD pipelines. We tend to treat green test suites as a binary certification of correctness rather than a statistical sample of tested states.
A 0.1% edge case—like a subtle race condition under specific concurrency loads or a non-deterministic state mutation—is mathematically invisible in a standard PR test run. But once you roll out to a million active sessions, that same 0.1% translates to 1,000 broken users within the first hour. This is precisely why staged rollouts and aggressive canary metrics are non-negotiable: the production environment isn't just where code runs, it’s the only test runner with enough volume to catch long-tail distribution failures.
This is a fantastic parallel. The comparison between hardware manufacturing tolerances and low-probability distributed software bugs hits right on target.
The point about blast radius limitation versus rapid rollback is where so many software teams stumble. Because CI/CD makes it effortless to ship a fix in 10 minutes, we get complacent about staged deployments. But to the user who hit that edge case on production, your 10-minute mean time to recovery (MTTR) still felt like a broken product. Canary releases and feature flags aren't just for preventing total outages; they’re the software equivalent of preventing a mass recall.
Another dynamic worth calling out is observability gap during silent failures. When a hinge tolerance issue or a 0.01% race condition occurs, the system rarely throws a clean 500 stack trace. It usually presents like that horizontal crack: silent data corruption, subtle state drift, or ghost metrics that look like normal variance until thousands of users collide with it at scale.
Treating green builds as "absence of evidence, not evidence of absence" is a mindset shift every engineering culture needs. Great write-up—the lessons from physical manufacturing are usually written in expensive hardware returns, which makes them sharp reminders for how we design software pipelines.
Thank you Shirley for sharing this insightful essay her on DEV.to!
Very insightful read thank you for sharing!
Very informative essay my friend appreciate you sharing with us!
Very informative piece thank you for sharing my friend!
Very informative piece thank you for sharing!