Regional Accident Analysis: Know Your Local Risk Factors
Savvy pilots are always interested in learning how to avoid potential risks. However, most accident data is summarized on a national basis and may understate the risk of some factors in your local area. Max Trescott found profound differences in accident causes in the San Francisco Bay Area and explores how you can discover unique risks in your area.
Flying is a lot of fun and isn't particularly dangerous for pilots who actively manage the risks. Since three-quarters or more of accidents are pilot-induced, we're largely in control of our fate if we pay heed to the missteps of others and avoid risky behaviors. Hence, it's totally appropriate that many government and private safety programs focus on analyzing accidents and educating pilots on the causes.
Most accident analysis is done on a national basis, and this makes sense for a lot of accident causes. For example, engine failures (which are fortunately relatively few in number) are probably distributed relatively equally around the country. Likewise, other types of mechanical problems, and perhaps some pilot errors such as fuel management, probably happen everywhere with about the same frequency.
However, it's reasonable to assume that some types of accidents are more prevalent in certain regions of the country, depending upon variations in topography and weather. In the simplest case, density-altitude related accidents undoubtedly occur more often in the West because of higher terrain.
As pilots, we probably would benefit from regional accident analysis if there are unique risks in our particular region. Where local risk factors exist, educating pilots about them should help push down the general aviation accident rate even further.
Nuts and Flakes: Not Just for Breakfast
For example, the San Francisco Bay Area is rather unique, and not just for the "nuts and flakes" that we joke about living here. (OK, so it's not a joke.) The geography is varied in that we have a series of mountain ranges going up to 4400 feet that are adjacent to many valleys near sea level. The weather is also unique because we're next to the ocean, and on most evenings a marine layer of clouds forms above many of the valleys. This results in many micro-climates, where the weather can vary substantially over just a few a few miles. It's not hard to imagine that the accident risks in this region might vary from the rest of the country and that other regions of the country, with their own unique terrain and weather, could have their own risk factors.
The preliminary conclusion of this research is that indeed Bay Area fatal accident factors do differ substantially in several important ways from the U.S. averages, and that there are clusters of accidents in local areas that may be more dangerous than the rest of the region. In particular, a much higher percentage of our fatal accidents in the San Francisco Bay Area occur at night and involve VFR into IMC. Communicating these results and the locations of the accident clusters is now a high priority so that pilots flying in and out of the Bay Area are aware of the higher probability of being involved in an accident with these factors.
What does this imply for those of you who don't fly in and out of the Bay Area? Since accident results for our area are so highly skewed from national statistics, then it follows that accident results in other areas of the country are also skewed in other ways. As a savvy pilot, you'll want to know what unique risks that you face in your area, and I'll describe here how you can do some analysis for accidents in the areas in which you fly.
It All Started With A Joke
My interest in researching Bay Area accidents was triggered by a conversation with Jack Hocker, the local FAA Safety Program Manager. I had asked him about the local accident rate, and he commented with his typical sense of humor that it couldn't get much lower since local pilots, "... were crashing in other districts." This piqued my interest because, in statistics, data is generally assumed to have a normal distribution; yet pilots crashing "elsewhere" seemed like an abnormal distribution. Either Jack, who is excellent at his job, had figured out how to convince pilots to crash in other districts; or more likely, there were unique accident factors operating across the FAA FSDO boundaries. If these could be identified and communicated to pilots, then perhaps accidents could be further reduced.
One of the great things about the Internet is that now everyone has access to accident data, and can analyze it in any number of ways. Thus, anyone can define their own region of interest, go to the NTSB Web Site and analyze accidents for a particular area. If you find major differences in the accident statistics in your area, you'll want to get the word out to local pilots any way you can. In our local case, when results were so startling, I shared them with the local FAA FSDO, started giving FAA Wings seminars to local pilots, wrote articles such as this one, and began an online newsletter focused on aviation safety in our area.
For those who don't care how the data was derived, you're welcome to skip the next few paragraphs and jump straight to the Results section below. I've included the following information so that those who may want to do their own regional analysis will have a blueprint to follow. Be prepared, however, to spend many hours on it. At the time, I was just transitioning from a high-tech job to a full-time teaching career as a Master CFI, and so I was able to devote a lot of time to the project. Also, please contact me if you need help or find any interesting results for your local region.
Defining the Data
|Low Clouds and fog move east through the same passes that pilots use. (Click for larger version.)
From "Weather of the S.F. Bay Region" by Harold Gilliam, used with permission.
The first issue was defining the set of accidents to be included in the study. Since local pilots were crashing "elsewhere," I couldn't just sort on accidents that occurred in the local region. Instead, I needed to find accidents for which the origination or destination was a Bay Area airport. This required separate keyword searches of the NTSB database searching for references to every Bay Area airport. Also, I analyzed only fatal accidents, since they were fewer in number (and hence easier to analyze) and of greater consequence than accidents that resulted in injuries and not fatalities.
Defining which airports to include in the "Bay Area" was also important. This was done by choosing all airports (except for the few private airports) that were within an area that was more or less defined by one of the mountain ranges that surrounds the region. This included airports as far south as Watsonville, as far east as Hollister and Livermore, and as far north as Napa and Petaluma. The premise was that having to cross one of these mountain ranges to enter or leave the region might be relevant to the accident data.
To get sufficient data required going back multiple years. Whereas the Air Safety Foundation's Nall Report showed over 300 fatal accidents in 2003, it took over 10 years of data to come up with around 90 fatal accidents in the local area. Initially I went back to 1990, but later noticed that my search had turned up disproportionately fewer accidents for the years 1990 through 1992. By looking through them, I concluded that the NTSB database isn't fully indexed for keyword searching in these years, as the search results only returned accidents that actually occurred within the region and not those where the accident occurred elsewhere.
Scrub A Dub Dub
Next, the data was reviewed or scrubbed to remove non-airplane accidents, because the Nall Report focuses on airplane accidents, and I needed to compare local results with national data. You'd be surprised at the large number of helicopter, balloon and even blimp accidents that occur over 10 years. Next, I removed public-use accidents, since most GA pilots are not flying on behalf of the government to fight fires or crime. Also removed were the many accidents that referred to "Oakland Center" but didn't involve a Bay Area airport, and accidents from several trans-Pacific ferry flights destined for the Bay Area, but crashed in the Pacific short of their destination.
Next, the Part 121 air carrier accident (one well-publicized crash) was removed, but the four Part 135 accidents were included. Although the Nall Report doesn't include Part 135 accidents, these were initially included since there were so few of them and the causes were similar to the Part 91 accidents. Ultimately, this resulted in analyzing a total of 88 accidents for the years 1993 through 2004
The next task was reading all of the accident reports. This included confirming that the departure or destination airport was in the Bay Area as previously defined, noting the phase of flight that the accident occurred (e.g., takeoff, cruise, etc.), and noting whether the accident occurred during the day or at night. For those accidents that didn't note whether they occurred in the day or at night, I used an internet sunrise/sunset calculator and entered the accident site coordinates, date and time. For accidents that occurred about an hour or more after local sunset time, I categorized them as night accidents.
Where's the Beef?
The real meat of the analysis was identifying the major factors involved with each accident. In some cases, it was remarkably easy. For example, when an engine quits or someone is pulling a low-level buzz job, it's fairly simple. Also, in some cases it was easy to infer from the facts of the report that, for example, high-density altitude was involved even if it wasn't explicitly mentioned in the report.
In other cases it was more difficult. For example, in one night accident, a witness described poor weather in the area; but it's impossible to determine if the weather was directly involved in the subsequent flight into terrain. In cases like this where poor weather was noted, I listed weather as one of perhaps multiple factor involved in that accident.
The Nall Report, as I understand, assigns a single cause to each accident; thus in the case cited above, weather wouldn't have been listed as the cause. Thus my analysis may overstate the involvement of some factors (such as weather) relative to Nall Report statistics. However, I think it's important for our local pilots to know how often a factor, such as weather, is involved in an accident, even if it isn't the sole cause.
I should note, however, that my findings on night accidents should be directly comparable to Nall Report data, and would have no overstatement. Day versus night is considered a "condition" as opposed to a cause of an accident; thus that condition is listed for most accidents, and is separate from the cause of the accident.
Results: Darkness And Clouds Are Not Your Friends
The biggest differences found were that a much higher percentage of fatal accidents for Bay Area flights occur at night and/or involved VFR to IMC than the national statistics. In the 2003 Nall report, 21.6%of fatal accidents occurred at night vs. the Bay Area, where fully 43% of fatal accidents occurred at night.
VFR into IMC, which was cited as the cause of only 5.4% of the fatal accidents in the Nall report, was a factor in 33% of the fatal accidents in the Bay Area, about six times the national statistics. So, fully two-thirds of all fatal Bay Area accidents involve night, weather or both.
Interestingly, the deeper you dig into the night numbers, the worse it gets. Looking at the period from 1993-2004, there were 88 fatal accidents of aircraft that either departed a Bay Area airport or were headed to a Bay Area airport. As I mentioned, 38 (43%) occurred at night. But if you looked at just those 33 that actually crashed in the local Bay Area -- as opposed to those that crashed outside the immediate area -- 18 (55%) occurred at night.
When you take out the homebuilt fatal accidents, the statistics for those of us flying certified aircraft in the Bay Area get even worse. Five of 33 fatal accidents that occurred in the immediate area were homebuilt aircraft, but none of these were at night. Thus of the 28 fatal accidents for certified aircraft, 64% occurred at night. This is more than triple the national statistic, and points to the much greater danger of flying at night in the Bay Area versus the rest of the country.
VFR into IMC accidents also have some interesting twists. Surprisingly, about half of the VFR-IMC accidents occur during the daytime, when it should be much easier to avoid entering a cloud. And entering a cloud inadvertently happens far more than you'd guess. I've asked the audience at several recent presentations how many of them have accidentally entered a cloud; depending upon the audience, anywhere from 10 to 50 percent raised their hand to indicate that they've had accidental encounters with a cloud, and around 60 percent of these encounters were at night.
But the encounters with a cloud at night are much more deadly. When asked, only a small percentage of the audiences flew more than 5 percent of their total flying hours at night. However, half of the fatal VFR into IMC accidents occurred at night. That suggests that you're about 10 times more likely to have a fatal VFR into IMC accident at night than in the daytime. This probably shouldn't be surprising, since clouds are so much harder to spot at night and pilot performance is probably poorer due to fatigue.
To put this in perspective: Typically around three-quarters of all accidents are deemed to be the result of pilot error. If that's true for the Bay Area, than the two-thirds of accidents that involve night or weather account for almost 90% of all pilot induced accidents in the Bay Area. Avoiding both substantially reduces one's risk of being involved in a fatal accident.
Significant Results ... But Statistically Significant?
But just because there are differences in Bay Area crash statistics, we don't automatically know whether they are statistically significant. Since we are dealing with a smaller number of crashes in the Bay Area and because there is always some random variation, we can't tell just by looking at the numbers whether they are statistically significant.
Here is where some fancy statistics techniques, specifically the Chi Square test, come in handy. It allows us to compare subsets of numbers from the same population (and accidents in California are part of the population of accidents in the U.S.), and determine whether any differences are significantly different. Most social scientists consider a 95% probability to be significant (thank goodness engineers use higher percentages when designing airplanes!), and when the Chi Square test is run on the Bay Area statistics for night and VFR into IMC accidents, the results are statistically significant at the 95% level. Interestingly, none of the other results -- such as a much lower percentage of maneuvering accidents in our area -- were quite statistically significant, although the maneuvering accidents were close.
Where Are The Hot Spots?
|Virtually every Night and Weather accident occurred in mountainous terrain. (Click for larger version.)|
If weather and night are recognized as critical factors, the next important question becomes where are they the most critical? Are there particular geographical locations where these accidents occur more frequently and where we should exercise greater caution? Fortunately, most of the accident reports give accurate crash-site locations, making it relatively easy to map the data.
To create the map of accident locations, only night and weather-related accidents were included, since mechanical and other causes can be assumed to be distributed relatively randomly. Also, the night and weather-related accidents for 1990-1992 were included in the maps, even though the data for these years only identified crashes that occurred in the Bay Area region, and not for Bay-Area flights that crashed outside the region.
The first striking thing about the data was the number of accidents that were not only outside the Bay Area, but actually outside the state. Eight of the 88 accidents occurred in other states. Oddly, many of these accidents were for fixed gear aircraft on very long flights (and yes, one of them was a fuel-exhaustion accident). Two were in Seattle, two were in Nevada, and one each occurred in Wyoming, Colorado, Washington and New Mexico.
Looking at just night and weather accidents from 1990-2000 (since probable causes aren't yet out for all accidents in subsequent years), 17 of the accidents were in California but outside the Bay Area, while 21 of the accidents occurred in the immediate Bay Area.
Two things are particularly significant about the crash locations. First, almost every night or weather accident occurred in mountainous terrain. In looking at the California state map, one can see that virtually none of these accidents occurred in the very large but mostly flat Central Valley that stretches for 400 miles up the middle of the state.
Second, when you look at a map of the San Francisco Bay region, a disproportionate number of the accidents (7 total) occurred in the Livermore Valley, in the eastern part of the region. For local flyers, this probably comes as little surprise. Because of high surrounding mountains, many flights to the Bay Area pass through the Livermore Valley because of its lower terrain, and because a few passes breach the hills west of Livermore.
Unfortunately, more than a few people have met their demise looking for these passes; or, if they were not searching for a pass, failing to appreciate the height of the hills. How could this happen? Very easily at night or in deteriorating weather.
The hills represent the last barrier to returning to the Bay Area, and may be the first place that pilots encounter low ceilings. Generally clouds form at night over the Bay itself. Because the prevailing winds are from the northwest, they tend to blow this stratus layer into the same passes that pilots use to cross the hills. Thus, just as pilots are nearly home (remember get-home-itis?) they encounter declining visibilities and higher terrain. To compound problems, the generally excellent radar coverage in the Bay Area is very poor in the Livermore Valley, and planes below 2500 feet are below NorCal Approach Control's radar coverage; you need to be over 4000 feet to get above their Minimum Vectoring Altitude, where you can begin receiving vectors. Thus, even if you're instrument-rated, you cannot get a pop-up clearance below these altitudes.
If you find yourself in this situation, what should you do? If you're not instrument-rated, you want to make an immediate U-turn, land at Livermore airport and get a hotel room or rental car. If you're instrument-rated, you also need to land at Livermore. Once on the ground, you can get a clearance to fly a departure procedure that will keep you safe until you climb into radar coverage. But if, instead, you continue to plow on through the murk hoping that visibilities will get better, you may be on your way to becoming the next statistic.
One of the best ways to cope with these dual dangers of night and clouds is to get an instrument rating. According to the book The Killing Zone, instrument-rated pilots have many fewer accidents compared to private pilots with the same number of hours of flying experience. In reviewing the Bay Area accident data, 43% of the VFR into IMC fatal accidents and 59% of the night fatal accidents involved an instrument-rated pilot, so while the rating undoubtedly reduces the accident rate, it by no means eliminates these particularly risky categories of accidents. Instead we need to rely upon taking a thoughtful approach to the risks involved with each particular flight.
Wherever you fly, you'll want to understand any unique factors that affect crashes in your area. One of the best ways to get this data is to analyze past data on crashes in your region. And if you live in the San Francisco Bay Area, you want to be particularly cautious when it comes to flying at night or near a cloud.
More articles about safety are available in AVweb's Safety section.