Technology

Microsoft service blackout caused by latent bug and server issues

A months-old bug and reduced server capacity caused a 24-hour Microsoft outage that disrupted Outlook and Office 365 services. The failure impacted hospitals, schools, and police departments before Microsoft restored access by Saturday.

Microsoft service blackout caused by latent bug and server issues

Factors in the service disruption

The 24-hour blackout of Microsoft services last week crippled Outlook, Exchange, and other Office 365 software. The disruption began Thursday when US users reported interrupted access to Outlook accounts. Millions of people worldwide lost email access by Friday. I consider the impact of this outage unacceptable, as jails could not release prisoners, hospitals turned to paper records, schools lost contact with parents, and police departments could not receive crime reports. Microsoft restored service by Saturday and did not attribute the blackout to a cybersecurity breach. On Monday, Microsoft issued an initial report identifying that a months-old bug and malfunctioning internal connections, alongside server capacity issues, led to the disruption. These factors worked together to create a major outage that was harder to mitigate.

Technical failures and recovery delays

Charles Lamanna, the executive vice-president of Copilot, Agents, and Platform, stated in a post on X that a "latent bug or defect from July" caused servers to "become overwhelmed and unresponsive". Microsoft typically uses two networking gateways to allow data to pass between separate systems, but one gateway for Microsoft Teams failed. This failure overwhelmed the primary gateway. An unrelated issue from the previous day meant that 14% of the company’s server capacity was also not working. The combined increase in load and the reduction in server capacity caused the network gateway to encounter the latent bug from July, which caused requests to build up until servers ultimately stopped responding during the peak window of traffic in the United States. I find the recovery process was poorly handled because servers were immediately overwhelmed by waiting traffic and became unresponsive again as they returned to service. Because the main code defect was a latent bug from several months previous, normal rollback or feature flags were unable to quickly revert the change. Matt Hull, vice-president of cyber intelligence and response at NCC Group, said reports that hospitals and schools were affected demonstrate the wide impact that a system outage can have.