World

Microsoft Says Months-Old Bug Caused Global Outlook Outage

Elena MarquezPublished 9m ago2 min readBased on 2 sources
Microsoft Says Months-Old Bug Caused Global Outlook Outage
Photo by Luka Franzi on Pexels

Microsoft said on Monday that a months-old software bug caused a 24-hour outage that disrupted Outlook, Exchange and other Office 365 services around the world, and that the company had identified the root cause. The Guardian Office 365 is Microsoft's subscription suite of online office software, which includes email.

The disruption began on Thursday, 12 November, when users in the United States reported that they could no longer reach their Outlook email accounts. By Friday, millions of people worldwide said they were unable to access email and other Office 365 services. Microsoft said it had resolved the issue and restored service by Saturday, and it apologised.

Charles Lamanna, executive vice-president of Copilot, Agents, and Platform at Microsoft, said on Monday that a latent bug dating to July caused Microsoft's servers to become overwhelmed and unresponsive. A latent bug is a flaw that stays hidden until conditions trigger it. Microsoft normally has two networking gateways, the systems that route traffic between services, but one was not working for Microsoft Teams. That left the primary gateway overloaded.

A separate problem compounded the strain. According to Lamanna, more than 14% of Microsoft's server capacity was not working because of an unrelated issue the previous day. Increased traffic combined with the lost capacity meant the gateway hit the July bug during the peak US traffic window. Requests built up until the servers stopped responding. The Guardian

Recovery took longer than the initial failure, Lamanna said, because servers coming back online were immediately overwhelmed by the traffic waiting for them. Automated recovery was not designed for a state in which so many servers were unresponsive at once. Rollback, which returns software to an earlier version, along with flighting, which releases changes to small groups in stages, and feature flags, which switch functions on or off, could not quickly revert the months-old defect.

Reported effects of the outage extended beyond corporate email. Hospitals turned to paper records, schools lost contact with parents, jails were unable to release prisoners, and police departments could not receive crime reports. The Guardian

Microsoft did not attribute the blackout to a cybersecurity breach. The company said it plans to publish a final report with more information about the outage.