Nightmare at Oakland Center

FAA tiger teams are still trying to sort out exactly what happenedin the wake of an unprecedented, 45-minute power outage that shutdown all radar and radio communications at Oakland Center on Wednesdaymorning, August 9th.

All ARTCCs are equipped with three independent sources of powerfor critical equipment. Each includes an emergency diesel generator,plus a bank of batteries capable of operating the center for 30minutes. Consequently, a complete power failure was consideredby FAA planners to be virtually impossible. But it happened anyway.

Although the outage is still under investigation, the probablecause is now pretty clear. One of the three power sources wasdown for testing and maintenance at the time of the episode. Thesecond power source failed unexpectedly. When technicians attemptedto bring the third power source on-line, a faulty circuit boardin Critical Power Panel failed, preventing power from being restored.

 The power-failure was widely misreported in the popular pressas being related to the FAA's chronic problems with computersand related ATC equipment. A spokesman for NATCA, the controller'sunion, wasted no time in blaming the problem on obsolete equipmentand reduced staffing at the Center.

Although Oakland Center is only the 16th busiest ARTCC in thecountry, it's responsible for the largest block of airspace ofany ATC facility — 18 million square miles extending from SanLuis Obispo to the Oregon border, and including most Pacific oceanicroutes.

The failure occurred, at 7:13 a.m. local time during the morning"departure push" period. Controllers estimate that 60to 80 aircraft were under ZOA control when the power died. Allradar screens went dark and all radio communications were cutoff. Lights and telephones were unaffected. It took 45 minutesto restore radio and the backup DARC (direct access radar channel)radar system and more than an hour before the NAS computerizedradar was restored.

Controllers were stunned as their screens went dark, their headsetssilent, and the perpetual whirr of equipment cooling fans ceased.Once controller described it as "the loudest silence I'dever heard."

Another controller told AVweb that there was "panicon everyone's face" as the controllers realized that theyhad been rendered deaf, dumb, and blind.

"We don't train for a situation like this," the controllersaid.

It took a minute or two for controllers to realize that the failurewas facility-wide. Most turned to their supervisors for guidance,and the supes turned to their supes. But there's no book procedurefor handling a situation like this. So in the end, most controllerswound up improvising.

Controllers at adjoining Los Angeles, Salt Lake City, and SeattleARTCCs and at various TRACONs were asked to take over surveillanceand control of all airspace within their radar coverages, andto divert aircraft under their control bound for destinationsin Northern California. Control towers at San Francisco, Oakland,San Jose, Sacramento, and other airports in the area were instructedto hold all IFR departures on the ground.

The most difficult problem was getting notification to airborneflight crews. In one case, said controller Mike Seko, "wehad the Napa Airport tower telling planes that Oakland Centerhad lost everything, and to switch to emergency frequency."But most aircraft who were airborne on Oakland Center frequencieswere left in a state of lost-comm unless they figured out whatwas happening and switched to another ARTCC or TRACON facility.

Flight crews apparently did a fair amount of improvising of theirown. Some pilots squawked VFR and proceeded on that basis. Manyothers continued to follow their last clearance, and some climbedinto or descended out of Class A airspace without a clearance.A lot of ASRS forms were undoubtedly filed that day. All indicationsare that the FAA is not pursuing any actions against the pilotsinvolved in this mess.

In retrospect, one of the biggest problems was that nobody believedthat a prolonged outage like this was possible. Controllers andsupervisors alike operated on the assumption that their radiosand radar would come back "any moment now." The samewas true of controllers at adjacent facilities like Bay TRACONthat were paralyzed by the Center's blackout.

It's impossible to say how many losses of separation and TCASresolution advisories occurred during the hour-long episode. Somenear-midair reports will undoubtedly be filed, but the vast majorityof loss-of-separation situations will probably go unreported.

After power was restored and the primary radar system returnedto normal operation, extensive air traffic delays, diversions,and flight cancellations persisted for many hours at Bay Areaairports, particularly airline departures from San Francisco International.

Every controller involved had a war story to tell: ControllerJonathan Smith told AVweb, "My shift started momentsafter power was restored. I will never forget the atmosphere uponmy arrival. Talking to my fellow controllers was much like talkingto accident victims in various states of shock. All were hyper-active,some were exhibiting bravado or gallows humor, others repeatingincidents and experiences. All were immensely proud of their havingsurvived a controller's worst nightmare."

It will be months before we know the full aftermath of this incident.Clearly, changes will be made in the way critical power feedsare wired and maintenance is done. Contingency plans will be rewrittenand controllers will receive training in how to execute them.The FAA may lobby Congress for funding to provide redundant linksto radio and radar sites so that when one ARTCC has a catastrophicfailure, adjacent facilities can take control of the airspace.

Meantime, you might want to brush up on your non-radar and no-radioprocedures...