A Firebase Server-Side Change Crashed iOS Apps Worldwide

On the morning of September 29, mobile developers around the world watched their iOS apps crash en masse. The culprit turned out to be a server-side change pushed by the Firebase team: it made the SDK crash across every session at once, leaving apps no way to protect themselves. Gergely Orosz, the writer behind a well-known publication on engineering culture, chronicled the whole thing.

What happened

The server-side update began crashing several versions of the Firebase SDK. Developers saw 100% of their app sessions failing — and could do nothing, because the fault lay not in their code. DoorDash was among the casualties. Orosz estimates the global outage lasted between two and six hours: apps that had cached the malformed server payload suffered the longest.

Developers beat Google to it

The most awkward part, in Orosz's telling, was how Firebase itself behaved. Google's engineers apparently failed to notice the outage; their own monitoring stayed silent. Outside developers piled into the GitHub issue tracker by the hundred, begging for the change to be rolled back. Only after roughly a hundred such messages did a Firebase engineer finally take notice — and it took another hour to revert the offending change.

Meanwhile, status.firebase.google.com showed no incident even five hours in, despite two Firebase staffers having already confirmed the outage. Orosz drily notes that when it came to incident management, ordinary developers proved more competent than the Firebase team.

Unanswered questions

The central puzzle for Orosz: how did a server-side change reach production with zero verification? The SDK's one job is to work; if it crashes, he argues, Firebase has no business shipping it. Someone in the thread jokingly wondered whether Gemini was behind the change.

Orosz promises a deeper write-up and says he has already sent questions to Google. For now it remains unclear whether a postmortem will appear at all — and for an outage of this scale, he says, its absence would be unacceptable.

Source: Hacker News