Global IT Outage Affected 911 Systems, Broadcasters, Nielsen
The global outage of Microsoft systems caused by a software update from cybersecurity company Crowdstrike grounded airplanes globally and affected some broadcasters and 911 systems but spared others, reports from multiple companies and state agencies said.
“The FCC is aware of reports of a systems outage that is causing disruptions in service, including 9-1-1,” an agency spokesperson emailed. ”We’re closely working with other federal agencies to provide assistance and determine the extent of these service disruptions.” The Cybersecurity and Infrastructure Security Agency said it's "working closely with Crowdstrike and federal, state, local, tribal and territorial (SLTT) partners, as well as critical infrastructure and international partners to assess impacts and support remediation efforts."
Crowdstrike said a defect in an update for Windows computers caused the outage. “This was not a cyberattack,” the company said Friday. A fix for the defect was deployed Friday morning, it added. “We understand the gravity of the situation and are deeply sorry for the inconvenience and disruption. We are working with all impacted customers to ensure that systems are back up.” Microsoft didn't comment.
Authorities in roughly half the states said they received reports from public safety answering points about issues with computer aided dispatch (CAD) systems, Harriet Rennie-Brown of the National Association of State 911 Administrators said. Some states experienced call handling equipment and CAD problems but responded quickly with backup protocols, she said. “Based on the information I'm receiving, 911 calls are getting through.” Prince George’s County, Maryland, said several county systems were affected: “We are actively working with our vendors to get systems restored and operational."
The CrowdStrike problem affected New Hampshire 911 statewide around 1 a.m. Friday, said Mark Doyle, the state’s 911 director. Telecommunicators could see 911 calls coming in but were unable to answer them, he said. Since they could see the phone numbers, telecommunicators quickly returned calls to 49 people, Doyle said. At 3:30 a.m., the state implemented a backup process; by 4:30 a.m. all operations were normal, and that remains the case, he said.
New Hampshire didn’t know immediately that a CrowdStrike glitch caused the problems, Doyle said. But when 911 officials started digging, “the only thing that really rang true was the commonality associated with what others around the world were starting to see in some type of cascading effect,” he said. “It is concerning because you think you have all your bases covered” regarding the 911 system’s resiliency, robustness and redundancy, "and then there’s that one thing you have no control over at all that could actually put a halt on your system, and worse yet, prevent calls from coming into your 911 center.”
The Iowa IT division's security center identified the CrowdStrike error at 12:30 a.m. CST and applied a temporary fix at 1:04 a.m., said the governor’s office: CrowdStrike issued an official fix at 1:30 a.m. “The quick action by the security team limited the initial impact of the outage to approximately 1,400 servers and 3,300 workstations, or less than 20 percent of all devices across executive branch departments.”
Not all states’ emergency-number systems appeared to be affected. For example, the Nebraska Public Service Commission received no reports that the CrowdStrike IT issue affected 911 in the state, a PSC spokesperson said. In a news release Friday morning, Dallas said that it was assessing the impact of the outage on city systems but that 911 and emergency calls were functioning.
Gray Television Chief Legal and Development Officer Kevin Latek told us Friday its stations and systems were hit, “like seemingly everyone else” but “work-arounds deployed overnight and this morning” resolved most issues. A Nexstar spokesperson told us the outage caused “no real issues here.” Cumulus stations “have had no operational impact,” a Cumulus spokesperson said. IHeartmedia declined to comment, while Sinclair and E.W. Scripps didn’t respond. CNBC host David Faber said Friday the network’s computers would require a manual update to address the issue in a segment on the outage. Mediacom, in an email, said the Microsoft crash had limited impact early Friday, but the cable internet service provider had most of its systems up and running before the workday started. Comcast and Charter didn’t comment.
Nielsen sent a notice to clients Friday morning warning that its systems were affected. “We have been working through the night to assess and address the impact to our servers and are prioritizing the restoration of client production servers. We have recovered a majority of the impacted servers and will provide an update on the scope of the impact and timeline for resolution as soon as possible.”
Verizon said in a news release that the outage isn't affecting its networks but could limit service and operations in the company's physical stores, leading to longer wait times.
Communications attorneys told us they hadn’t experienced issues uploading filings to FCC databases, and the electronic comment filing system appeared to be functioning normally Friday. The FCC didn’t respond to questions about whether the outage disrupted work at the agency.