August 24, 1995 Nightmare at Oakland Center

Email this article |Print this article

by	Mike Busch

About the Author ... Mike Busch is editor-in-chief of AVweb, a member of the technical staff at Cessna Pilots Association, and in a prior lifetime was a contributing editor for The Aviation Consumer and IFR Magazine. A 6,000-hour commercial pilot and CFI with airplane, instrument and multiengine ratings, Mike has been flying for 36 years and an aircraft owner for 33. For the past 14 of those years, he's owned and flown a Cessna T310R turbocharged twin, which he maintains himself. In his never-ending quest to become a true renaissance man of aviation, Mike's on the verge of earning his A&P mechanic certificate. Mike and his wife Jan reside on the central coast of California in a semi-rural area where he can't get DSL or cable TV.

FAA tiger teams are still trying to sort out exactly what happened in the wake of an unprecedented, 45-minute power outage that shut down all radar and radio communications at Oakland Center on Wednesday morning, August 9th.

All ARTCCs are equipped with three independent sources of power for critical equipment. Each includes an emergency diesel generator, plus a bank of batteries capable of operating the center for 30 minutes. Consequently, a complete power failure was considered by FAA planners to be virtually impossible. But it happened anyway.

How It Happened

Although the outage is still under investigation, the probable cause is now pretty clear. One of the three power sources was down for testing and maintenance at the time of the episode. The second power source failed unexpectedly. When technicians attempted to bring the third power source on-line, a faulty circuit board in Critical Power Panel failed, preventing power from being restored.

 The power-failure was widely misreported in the popular press as being related to the FAA's chronic problems with computers and related ATC equipment. A spokesman for NATCA, the controller's union, wasted no time in blaming the problem on obsolete equipment and reduced staffing at the Center.

Although Oakland Center is only the 16th busiest ARTCC in the country, it's responsible for the largest block of airspace of any ATC facility — 18 million square miles extending from San Luis Obispo to the Oregon border, and including most Pacific oceanic routes.

The failure occurred, at 7:13 a.m. local time during the morning "departure push" period. Controllers estimate that 60 to 80 aircraft were under ZOA control when the power died. All radar screens went dark and all radio communications were cut off. Lights and telephones were unaffected. It took 45 minutes to restore radio and the backup DARC (direct access radar channel) radar system and more than an hour before the NAS computerized radar was restored.

What Do We Do Now, Boss?

Controllers were stunned as their screens went dark, their headsets silent, and the perpetual whirr of equipment cooling fans ceased. Once controller described it as "the loudest silence I'd ever heard."

Another controller told AVweb that there was "panic on everyone's face" as the controllers realized that they had been rendered deaf, dumb, and blind.

"We don't train for a situation like this," the controller said.

It took a minute or two for controllers to realize that the failure was facility-wide. Most turned to their supervisors for guidance, and the supes turned to their supes. But there's no book procedure for handling a situation like this. So in the end, most controllers wound up improvising.

Controllers at adjoining Los Angeles, Salt Lake City, and Seattle ARTCCs and at various TRACONs were asked to take over surveillance and control of all airspace within their radar coverages, and to divert aircraft under their control bound for destinations in Northern California. Control towers at San Francisco, Oakland, San Jose, Sacramento, and other airports in the area were instructed to hold all IFR departures on the ground.

The most difficult problem was getting notification to airborne flight crews. In one case, said controller Mike Seko, "we had the Napa Airport tower telling planes that Oakland Center had lost everything, and to switch to emergency frequency." But most aircraft who were airborne on Oakland Center frequencies were left in a state of lost-comm unless they figured out what was happening and switched to another ARTCC or TRACON facility.

Meantime, In the Cockpit...

Flight crews apparently did a fair amount of improvising of their own. Some pilots squawked VFR and proceeded on that basis. Many others continued to follow their last clearance, and some climbed into or descended out of Class A airspace without a clearance. A lot of ASRS forms were undoubtedly filed that day. All indications are that the FAA is not pursuing any actions against the pilots involved in this mess.

In retrospect, one of the biggest problems was that nobody believed that a prolonged outage like this was possible. Controllers and supervisors alike operated on the assumption that their radios and radar would come back "any moment now." The same was true of controllers at adjacent facilities like Bay TRACON that were paralyzed by the Center's blackout.

It's impossible to say how many losses of separation and TCAS resolution advisories occurred during the hour-long episode. Some near-midair reports will undoubtedly be filed, but the vast majority of loss-of-separation situations will probably go unreported.

After power was restored and the primary radar system returned to normal operation, extensive air traffic delays, diversions, and flight cancellations persisted for many hours at Bay Area airports, particularly airline departures from San Francisco International.

Aftermath

Every controller involved had a war story to tell: Controller Jonathan Smith told AVweb, "My shift started moments after power was restored. I will never forget the atmosphere upon my arrival. Talking to my fellow controllers was much like talking to accident victims in various states of shock. All were hyper-active, some were exhibiting bravado or gallows humor, others repeating incidents and experiences. All were immensely proud of their having survived a controller's worst nightmare."

It will be months before we know the full aftermath of this incident. Clearly, changes will be made in the way critical power feeds are wired and maintenance is done. Contingency plans will be rewritten and controllers will receive training in how to execute them. The FAA may lobby Congress for funding to provide redundant links to radio and radar sites so that when one ARTCC has a catastrophic failure, adjacent facilities can take control of the airspace.

Meantime, you might want to brush up on your non-radar and no-radio procedures...