DEV Community

Cover image for The health check that took down Outlook: inside the Microsoft 365 outage of November 12
Kashif Manzer
Kashif Manzer

Posted on Edited on

The health check that took down Outlook: inside the Microsoft 365 outage of November 12

On Thursday lunchtime, Outlook stopped opening. Not for one company, and not in one country. People clicked their inbox and got nothing. They refreshed. They restarted the app. They tried their phone. Nothing worked, and the usual fixes did nothing at all, because the problem was not on their side of the screen.

By 1:09 p.m. Eastern time, Downdetector had logged 128,874 problem reports for Outlook. Independent traffic measurements put Exchange Online, the service behind Outlook mailboxes, at about 89 percent below its normal level for more than nine hours. Teams, SharePoint, Copilot and the Microsoft 365 admin centre broke along with it, because they all lean on the same mail service.

This is the story of what happened next, told from Microsoft's own incident log. It is worth reading closely, because the villain of this outage was not a hacker, a traffic spike, or a bad deploy. It was the safety system that was supposed to protect the service.

The day, hour by hour

All times are Eastern, Thursday November 12, 2026, unless stated otherwise.

11:17 a.m. Microsoft opens the incident internally. Almost nobody outside the company notices anything yet.

12:25 p.m. Outlook problem reports start climbing on Downdetector.

12:35 p.m. Kentik, a company that watches internet traffic, sees Exchange Online traffic fall off a cliff.

12:54 p.m. Microsoft's first public post. It talks about users "in the Americas", as if this is a regional problem.

1:07 p.m. The wording changes to "users in any region". It is not regional.

1:09 p.m. Downdetector peaks at 128,874 Outlook reports.

2:18 p.m. Microsoft corrects its own first theory. The elevated load, it writes, "isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure." Read that sentence twice. The traffic did not grow. The fleet shrank.

2:43 p.m. The missing servers have a cause: they are failing automated health checks, and the load balancer is pulling them out of service because of it.

2:57 p.m. The suspect narrows. The failed health checks "may originate from a specific component on the affected servers that's responsible for processing them." Microsoft starts deploying a change to bypass that component, and takes steps to stop more servers being removed from service.

3:56 p.m. An alternative health check method works on a subset of servers, and those servers come back. Progress, at last.

5:00 p.m. The setback. The fixes so far "have not provided the anticipated relief." Servers that were returned to service "can become unresponsive again once they begin receiving traffic."

8:06 p.m. A sharper finding: "a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests." In the same update, Microsoft starts reverting "a recent network configuration change that correlates with the start of impact."

Overnight. A patch to an upstream authentication and routing component starts showing positive results. Microsoft extends the window for delayed email from 24 hours to 48 hours, so messages sitting in queues are not bounced while recovery continues.

Friday morning. Recovery moves region by region, Asia first, then Europe, then North America. At 8:00 a.m. Eastern on Friday, the incident is still open. Some users are still seeing intermittent problems and delayed mail. Microsoft says there is no evidence of malicious activity, and promises a full post-incident report with root cause once the service is fully restored.

That report has not been published yet. Everything above comes from Microsoft's incident log as it stood on Friday morning, and the details may change when the report lands. The lesson does not depend on the details.

What a health check is, and how it can turn on you

A load balancer is a traffic cop. It stands in front of a fleet of servers and hands each incoming request to a server that is ready for it. To know which servers are ready, it asks them, every few seconds, a simple question: are you okay? That question is a health check, and a server answers it by running a small check and replying yes or no.

If a server answers no, or does not answer at all, the load balancer stops sending it traffic. That is the whole point. A sick server should not get more work.

Now imagine the part of the server that answers that question is itself broken. The server is fine. It could handle mail. But it cannot say so, because the component that processes the health check is the thing that failed. The load balancer hears silence, concludes the server is dead, and pulls it out of rotation.

That is what Microsoft's log describes. And here is where it turns into a cascade, which is the word engineers use when one failure causes the next failure, which causes the next.

Every server that gets pulled hands its share of the work to the servers still standing. Those servers now carry more load than they were built for. Their processors run hotter. They slow down. Some of them start failing their own health checks, and get pulled too. Each removal makes the next one more likely. The fleet does not fail all at once. It fails one server at a time, faster and faster, like a row of dominoes that leans a little harder with each fall.

Microsoft saw this happening in real time. At 2:18 p.m. it wrote that load was shifting onto the remaining infrastructure. That is the cascade, described in Microsoft's own words.

The second trap: bringing servers back

Fixing a cascade is not as simple as fixing the broken part and turning the servers back on. Microsoft learned that at 5:00 p.m., when servers it had returned to service went unresponsive again as soon as traffic reached them.

Think about what happens the moment a server comes back. Every client that has been retrying for hours, every phone that has been polling for new mail, every app that has been hammering refresh, all rush in at once. The server is healthy, but it is healthy in the same way a shop is healthy at the moment the doors open after a crowd has waited outside all day. The first minute is the hardest minute it will ever have.

There is a second reason that first minute is so hard, and it is the detail Microsoft found at 8:06 p.m., that a process for establishing secure connections was getting overwhelmed. When your app connects to a mail server, the first thing that happens is a handshake, a short exchange that sets up an encrypted connection. That handshake is the most processor hungry part of accepting a connection, far more expensive than sending the mail itself. During normal operation, handshakes are spread out across the day. After an outage, millions of them arrive in the same minute, because every client is reconnecting at the same time. The server spends all of its processor time saying hello, and has none left to do any work.

So the recovery has to be slow on purpose. Microsoft did exactly that: traffic limits in specific locations, a narrower set of connections allowed through, then gradually widening. Let a few servers prove they are stable under a little traffic before you give them a lot. It feels slow. It is the only thing that works.

The guard you can build this week: separate alive from ready

Most small services have one health endpoint, and it answers one question: is the process running? That is a liveness check, and it is the wrong question to ask a load balancer. A process can be alive and still unable to take traffic. What the load balancer needs is a readiness check: is it safe to send this server real work right now?

Here is the difference in Go. Two endpoints, two different questions.

// /healthz answers one question: is this process alive?
func healthz(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ok"))
}

type gate struct {
    sem chan struct{} // one slot per request we are willing to run
}

func newGate(limit int) *gate {
    return &gate{sem: make(chan struct{}, limit)}
}

// /readyz answers a different question: is it safe to send
// this server more work right now?
func (g *gate) readyz(w http.ResponseWriter, r *http.Request) {
    // If the server is 90 percent full, say not ready.
    // The load balancer will wait. It will not pull the
    // server out of rotation, and the server gets a chance
    // to drain instead of being declared dead.
    if len(g.sem) >= cap(g.sem)*9/10 {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ok"))
}
Enter fullscreen mode Exit fullscreen mode

Pair that with a shedder, a piece of middleware that refuses new work fast when the server is full, instead of accepting it and dying slowly:

func (g *gate) shed(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        select {
        case g.sem <- struct{}{}:
            defer func() { <-g.sem }()
            next.ServeHTTP(w, r)
        default:
            // Full. Refuse in microseconds, with a 503 the
            // client can retry, instead of queueing work
            // that will time out anyway.
            http.Error(w, "busy, try again", http.StatusServiceUnavailable)
        }
    })
}
Enter fullscreen mode Exit fullscreen mode

The threshold matters, and it is worth testing at the boundary. Run a load test that fills the server to 89 percent of its limit, then 90, then 100. At 89 it should still report ready. At 90 it should report not ready. At 100 it should refuse fast. If those three numbers do not behave, the guard is decoration.

The guard on the other side: retry like a polite guest, not a crowd

The cascade also had a client side. Every Outlook app, every phone, every integration kept retrying, and when the servers came back, all of those retries landed at once. You cannot fix Microsoft's clients. You can fix yours.

A retry that waits the same amount of time every time is a crowd that knocks on the door in unison. A retry with backoff waits longer after each failure, and jitter, a small random amount added to the wait, spreads the retries out so they do not all land in the same second.

import random
import time
import requests

def get_with_backoff(url, tries=5):
    delay = 0.5  # start by waiting half a second
    for attempt in range(tries):
        try:
            resp = requests.get(url, timeout=5)
            if resp.status_code < 500:
                return resp  # a 4xx is our mistake, not the server's
        except requests.RequestException:
            pass  # network error: treat it like a 5xx and retry
        # Wait longer each time, and add jitter so every client
        # does not retry at the same instant.
        time.sleep(delay + random.uniform(0, delay))
        delay *= 2
    raise RuntimeError("service still down after retries")
Enter fullscreen mode Exit fullscreen mode

And one step further, a circuit breaker: a small counter that stops calling a service entirely after a few failures in a row, and only tries again after a cooling off period. It is the client side version of the same idea: when something is struggling, the kindest thing you can do is stop hitting it.

Three questions to take back to your own systems

First, what does your health check actually measure? If the check runs through the same component that does the real work, a failure in that component looks like a dead server, and your load balancer will do exactly what it did to Microsoft: remove healthy capacity in the middle of a crisis. Check the check.

Second, do you separate alive from ready? A server that is alive but full should shed load, not be declared dead. A server that is dead should be declared dead. One endpoint cannot answer both questions.

Third, have you tested the return to service, not just the failure? Kill a server in staging, bring it back, and watch what happens in the first minute under real traffic. If it falls over again the moment traffic reaches it, you have the same second trap Microsoft hit at 5:00 p.m., and you can fix it before your users find it for you.

The uncomfortable part of this outage is how ordinary each piece was. A component that answers health checks. A load balancer doing its job. Clients retrying, as clients do. None of it was exotic. That is the point. The systems that take down a whole service are rarely exotic. They are ordinary parts, arranged so that the safety system itself can become the failure.

So here is the question to take back to your own systems: which of your health checks is measuring the thing that checks, instead of the thing itself, and how would you know?

Sources: Microsoft incident log for EX1550543 and MO1550821, as published in the Microsoft 365 admin centre and mirrored publicly by the University of Pennsylvania; Downdetector and Kentik figures as reported in the Enrow write-up of the outage. Microsoft has not yet published a root cause; it has promised a full post-incident report once the service is fully restored.

Top comments (0)