911 Officials Say CenturyLink Didn't Immediately Tell Them About Outage
Three coordinators of state and local 911 systems said they didn't get warning or immediate direct information from CenturyLink as a network outage last month disrupted such systems nationwide. Officials in Washington state, Colorado and Wyoming's state capital told us they relied on their own information, news reports and Twitter in the early stages as they decided how to respond to problems including static, loss of automatic location data and, in Washington state, hours-long 911 outages.
Sign up for a free preview to unlock the rest of this article
Communications Daily is required reading for senior executives at top telecom corporations, law firms, lobbying organizations, associations and government agencies (including the FCC). Join them today!
The FCC and states are probing CenturyLink problems, which the company blamed on a faulty network card. The carrier said it will cooperate (see 1901020022). The FCC didn't comment Monday.
"We are diligent to keep our customers and other stakeholders informed when dealing with these sort of events," a CenturyLink spokesperson said. "But we also use them as an opportunity to review and improve our processes, including exploring ways to keep stakeholders and customers better informed.” The "network event" affected voice, IP and transport for some customers Dec. 27-29, and the carrier has acted to "prevent the issue from reoccurring," she said. "Where CenturyLink is an emergency 911 service provider and voice services were functioning, 911 calls largely completed during the incident. However, address location information (ALI) did not accompany calls to a small number of public safety answering points."
At one point, 47 of about 60 Washington PSAPs weren’t receiving 911 calls, and the outages lasted from the evening of Dec. 27 into the wee hours the next morning, said the state's enhanced 911 coordinator, Adam Wasserman. Washington suffered a big outage in 2014, but it was much less of a mystery, he said. “Something had happened in the 911 network” in 2014, but “here, the 911 network was working but calls weren’t getting into it.”
Wasserman became aware of a problem with long-distance and 800 numbers at CenturyLink early Dec. 27, a minor issue affecting PSAPs only if they had to transfer a call to a long-distance number. Wasserman’s phone “started blowing up” around 8 p.m. while he was celebrating his birthday, he said, with many PSAPs reporting getting no calls. The state’s new ESInet provider Comtech then reported it wasn’t getting calls from the older CenturyLink ESInet, he said. The state in 2016 decided to change providers and expects to migrate most PSAPs by the end of February, he said. PSAPs on both networks saw impact, and the state didn’t decide to change companies due to any technical problems, he clarified.
Early on, the coordinator set up a phone bridge for the state’s 60 PSAPs to find out what was going on, with Comtech and CenturyLink representatives on the line, Wasserman said. The state alerted the public with the emergency alert system, wireless emergency alerts and through the media, asking people use 10-digit public safety numbers, Wasserman said. Those longer numbers aren't as easy and location information isn’t automatically sent to dispatchers, he said.
The state 911 program is providing information for the Washington Utilities and Transportation Commission probe and will do the same for the FCC investigation, which paused during the shutdown, Wasserman said. The 911 program is compiling lessons learned. While statewide wireless alerts worked, some people were angry about receiving messages as late as 11 p.m., and some got multiple messages if their county also did a more localized alert, he said.
Colorado's 911 office learned first about disruptions from Eagle County, which Dec. 27 reported dropped and failed calls on administrative lines, though not 911 calls, said state 911 Program Manager Daryl Branson. Next morning, Weld County reported “a lot of static” on 911 lines, he said. Branson learned from news reports CenturyLink was having a problem and got updates “the same time as everyone else” from the company’s statements, he said. The telco is required to notify the state only for a 911 outage, but “they usually contact me anyway and let me know when they’re having problems.” The state official conceded “it was an all-hands-on-deck kind of situation for CenturyLink."
The geographic reach of the 911 problems was unusual, affecting PSAPs across Colorado, Branson said. No 911 centers went down, but PSAPs reported static, and one lost automatic location data for about 15 minutes, Branson said. With static, 911 dispatchers lack context from background noises and voice tone.
“The first indication we had a problem” was the 911 system “losing a lot of packets” and communications issues within the network, said Chuck Trimble, assistant director of the Laramie County Combined Communications Center in Cheyenne. “I was initially thinking it was maybe an internal network issue” but after reading media reports, he learned it was CenturyLink’s problem, he said. Emergency calls came, but the county lost automatic number and location information the last day of the outage, Trimble said. That data “is a big part of the process for verifying emergency location,” he said. Operators used backup procedures, verbally asking callers for their location while data wasn’t coming through, he said.
December’s 911 problems “seemed larger” than previous incidents witnessed by Trimble in his nearly 20 years of experience. The PSAP received no notification from CenturyLink about the outage until a day in, he said. “I really had expected to get an email or something through our notification system that they were experiencing problems.” Other providers tend to notify the county even if they only predict a possible outage, he said, “so that we can start watching and preparing.”