Microsoft blames a network maintenance system bug for the massive Microsoft 365 outage

Microsoft has attributed Thursday's massive Microsoft 365 outage to a bug in its own system for automatically processing network maintenance requests. Because of it, IP routes were removed from more devices than intended, disrupting Azure and Microsoft 365 services.

The outage began on July 23 at 10:44 a.m. Eastern Time and primarily affected customers reaching Microsoft 365 through infrastructure tied to the Azure West US region. By 11:11 a.m., Downdetector had logged 2,403 reports — against a typical baseline of about 29. SharePoint was hit hardest (78% of reports), followed by Excel (11%) and the Microsoft 365 admin center (6%).

What was down

Under incident ID MO1437424, the company confirmed disruptions across several services at once: OneDrive access was intermittent, SharePoint Online returned “Something went wrong” errors, Teams saw degraded chat and images that wouldn't load, Loop pages wouldn't open, Power Automate flows wouldn't run, and Copilot Chat responded with delays or failures. The list of affected products also included Fabric and Power BI, Power Apps, Copilot Studio, Windows 365 and Microsoft Defender.

How one bug pulled extra routes

In a preliminary review of the incident for Azure, Microsoft said the failure arose during scheduled maintenance on devices in the West US region, when individual network paths were being isolated. By procedure, the system translates such requests into machine-readable instructions and verifies that at least one of two redundant paths remains healthy before work begins.

However, a bug in the request-conversion module flagged additional network devices as part of the maintenance. As a result, IP routes were removed from more devices between the West US data center and the company's global network. This disrupted traffic entering and leaving the region — while traffic that stayed entirely within the region was unaffected.

The fallout was broad: connection failures, increased latency and access problems across dozens of cloud services, among them Azure App Service, Application Gateway, Cosmos DB, Kubernetes Service, ExpressRoute, Microsoft Graph, Sentinel and VPN Gateway.

Recovery

Microsoft initially tried to mitigate the impact by rerouting traffic along alternate paths, but many services kept failing. Engineers first saw widespread route “flapping” on the global network, then traced the removals back to the West US data center and linked them to the recent maintenance. The rollback of the change began at 1:45 p.m. and completed at 2:26 p.m. Eastern Time, with all affected services fully restored by 3:41 p.m.

The company is now conducting an internal review focused on safety checks and the automated processes for carrying out maintenance requests. Microsoft promises to publish a final report once the investigation is complete — typically within two weeks.

Source: Bleeping Computer