Home > News & Events > MITRE Publications > The Edge >

A steady flow of information about safety-related incidents during day-to-day operations is constantly reported to the air safety officers of various airlines. These reports run the gamut from the critical, such as a report of a near-miss collision, to the seemingly trivial, as with a passenger smoking in a lavatory. Keeping on top of this steady stream of information and identifying important patterns is a challenging task.

MITRE is sponsoring research into developing and applying data mining tools for identifying safety-related trends and patterns. Use of such tools would provide a safety officer with information needed to formulate appropriate corrective actions, ultimately contributing to reduced aviation accident rates. Our work attempts to answer the two questions most often asked by aviation safety professionals:

(1) What is the current safety status of our planes and operations?

(2) Are there any trends I should be looking out for?

Simple descriptive statistics are helpful in answering the first question, but we have found that new approaches in both text analysis and anomaly detection are useful in answering the second question. Those new approaches have also resulted in a more comprehensive answer to question one.

Simple Descriptive Statistics
The first question noted above can be partially answered using simple statistics. The total counts of incidents in a given time period, the most common types of incidents and their locations, plane types, and other factors all give the safety officer a coarse view of the current status. Graphs of these frequencies over time allow the officer to see the “big” trends. In addition, newly available COTS tools for online analytical processing (OLAP) make this kind of analysis even easier. Recent OLAP tools encourage exploratory analysis by allowing the safety officer to quickly and easily view different arbitrary combinations of attributes dynamically and with little programming effort.

When safety officers and data mining analysts collaborate, we have found, the benefits of this type of initial analysis are speed and ease of interpretation for the safety officer, and a quick lesson in the domain and suggestions of areas for deeper analysis for the data mining analyst. What this type of initial analysis doesn’t tell us, however, is whether or not anomalies exist, or how to exploit the information hiding in the text descriptions that are part of safety reports--both elements of the second question noted earlier. Therefore, additional data mining is required.

Text Classifications and Human Factors Issues
We used text classification as a means of filling in missing Human Factors (HF) information to get a more complete answer to question one. To do this, we used records of accidents and incidents from the National Transportation Safety Board (NTSB). Although these records have a field for explicitly identifying HF as playing a role, this field is often left empty. Because the narrative description of the incident often suggests HF as a factor in the incident or accident, we felt we should try to exploit this resource.

Using a set of 444 long narratives extracted from NTSB records detailing 383 inadvertent slips and 61 mistakes from 1991 to 1997, we experimented with a naïve-Bayes classifier to see if it could be trained to discriminate between HF mistakes and slips. After some data preparation, which included forcing a single canonical form for each word, we obtained a classifier with an average predictive accuracy of 92 percent. This result showed us that textual descriptions provide information that goes beyond what is available from such traditional methods as simple descriptive statistics.

Consequences of runway incursions.

Anomaly Detection
While simple descriptive statistics and text classifications do provide certain aviation safety patterns, they do not provide for detection of anomalies. To do this, MITRE developed a system called SMITHERS, which is based on an attribute-focusing technique, to detect anomalies. SMITHERS makes the detection of subtle differences in distribution easier.

SMITHERS compares the overall distribution of the values of a given “focus” attribute against its distribution in various subsets of the data. If a certain subset has a statistically different distribution of that focus attribute, then the condition that defines the subset is marked as “interesting.” Note that the overall distribution is our baseline rule and the distributions for the subsets are the potential exceptions.

For testing purposes, we applied SMITHERS to Aviation Safety Reporting System database reports on incidents categorized as “runway incursions” occurring between 1988 and 1997. We focused SMITHERS on an attribute of the database that denotes the consequences of a single runway incursion, with four possible outcomes:

1. Damage: Damage to the aircraft or injury or emotional trauma to a passenger.
   
2. Reprimand: Incident triggered Federal Aviation Administration (FAA) penalties, the threat of FAA penalties, an FAA follow-up investigation, or a flight crew/Air Traffic Controller review.
   
3. Other: Something happened, but it was not damage or a reprimand.
   
4. None: Nothing happened.

SMITHERS produced the results in the figure above.

The first row shows the overall frequency of the consequences of runway incursions in the database. The second row shows, on the basis of the relative number of such aircraft in the fleet, the expected frequency of consequences for aircraft with advanced displays such as a glass cockpit, which uses either LED displays (in place of analog dials) or a head-up display. The third row shows the actual frequency for aircraft with advanced displays.

Comparing the actual frequencies to the expected frequencies, we found that for aircraft with advanced displays the number of cases with the outcome “damage,” “reprimand,” or “other” was less than expected. The number of cases where the outcome was “none” was greater than expected. This finding suggests that the presence of an advanced display in a cockpit may be correlated with reducing damage in runway incursion incidents. Only further study will confirm that such displays themselves definitely help reduce damage in incursions.

Improved Pattern Matching
In addition to the development of SMITHERS and the other successful data mining work described above, MITRE is currently studying how to use information from both text and structured fields together to perform improved pattern matching. This work appears quite promising. A recent application of the hybrid text method showed how both text and structured fields could be exploited simultaneously. For a given “probe” report, we identified another report similar to it: the text descriptions of both reports described bad weather and, in both reports, the structured field called “lighting” had the same value. Such work will ultimately help contribute to reduced aviation accident rates by helping air safety officers swiftly and accurately determine if reports signify an anomaly or are part of a trend that can be corrected.