- Verizon blamed its outage over the weekend on “a software-related issue”
- Recon Analytics’ Daryl Schoolar suggested the outage probably was related to a mobile core network software update
- Many recent MNO outages were likely caused by software updates
Verizon was the latest mobile network operator (MNO) to suffer an outage across multiple cities in the United States this past weekend, begging the question: what went wrong?
Outage report site Downdetector said that the Verizon 5G network went down on Saturday morning in New York City, Houston, Chicago, Seattle, the Bay Area, and other major urban centers and came back online around 7.30 pm PDT on Sunday evening. Users complained that their phones and other devices couldn’t connect to the Verizon 5G network.
“On Saturday, some customers in isolated pockets around the country experienced a disruption in service caused by a software-related issue. As soon as we discovered the problem, our teams worked non-stop to get everything back up and running, and service was restored later in the day,” a Verizon spokeswoman told Fierce in an email.
She added that the operator continues to examine what went wrong and will use its learnings to strengthen its network experience for users.
“It would not surprise me if this was related to a software update involving Verizon’s mobile core network,” noted Recon Analytics analyst Daryl Schoolar. “That is what I have seen as the source of most outages.”
AT&T similarly blamed a software glitch for the problems which impacted more than 70,000 users in its major February 22, 2024 nationwide outage. Meanwhile, T-Mobile blamed “a third-party transport vendor” dealing with issues affecting subscribers in the Washington, D.C., area for its latest outage on October 2, 2024.
Verizon’s last outage took place on September 30, 2024 and lasted 9 and a half hours.
Software forward
Most of the recent MNO outages appear to stem from standard software updates that are deployed overnight. The resulting network problems only really become obvious in the early morning after the update. As the industry moves towards a cloud-based 5G-Advanced and 6G future, software issues are likely to remain a major cause of outages. After all, operators can hardly revert to traditional hardware-centric architectures.
So, if software is increasingly the culprit, what's being done to mitigate these kinds of outages? Wouldn't the digital twins we've all heard so much about enable sandbox testing that could help prevent these kinds of issues?
Well, it seems operators are using digital twins but perhaps not in ways that prevent update-related outages.
For instance, AT&T utilizes digital twins of its network for "analytical operations" such as ticket and alarm correlation, cost reduction, circuit consolidation and integrating machine learning models. It also uses digital twins of its cell towers to help with RF planning and cut down on physical site visits.
Likewise, Verizon is using digital twins in its network, though it appears to be using them largely to analyze network sites and deliver more efficient energy use, rather than to guard against software update problems.