First-Time Measures of Pilot Performance Have Potential to Transform Aviation Safety

If an aircraft stalls in mid-air, how long does it take a pilot to execute the multi-step recovery process, and was that process performed correctly? What if an onboard system alerts the pilot that another aircraft is too close? How quickly can the pilot take the steps necessary to avoid a collision?

Capturing specifics like these has the potential to transform aircraft equipment design and certification, flight procedure development, and pilot training in ways that enhance overall flight safety. As the longtime operator of the Federal Aviation Administration’s (FAA) federal R&D center, MITRE understands just how important those goals are.

“The aviation industry has never been able to obtain broad measures of what actually happens when safety issues arise during commercial flights,” explains Valerie Gawron, Ph.D., an expert in human-centered engineering. “We’ve been able to measure pilot performance in a simulator, but that’s quite different from the real world, when the pressure is on to respond quickly and correctly at a moment’s notice.”

That means avionics and flight procedure design, as well as pilot training, are based on estimates or rules of thumb rather than real-world performance. “That’s less than optimal, so the FAA and industry have been asking for in-flight measures for a long time,” Gawron says.

Real-World Data Paints a More Precise Picture

Now, through an independent research project, MITRE has demonstrated it’s possible to provide just that, along with the models that make the data easy to apply.

The research team—led by data scientist Houda Kerkoub—began by sifting through airline and FAA data from 30 million flights over a 5-year span to identify the safety issues they elected to study: aircraft “loss-of-control” events that result in a stall and “loss-of-separation” events, where two aircraft come too close together, creating a risk of collision.

“That narrowed the research to 16,000 flight records,” says Darren Neal, a pilot and multidiscipline systems engineer who provided flight operations expertise to the team. “We then used ‘black box’-like data from those flights’ quick access recorders—which capture everything from alerts to button pushes and other flight control inputs—to measure exactly what happened during the identified safety events.”

“We measured how long it took to complete the entire recovery procedure,” Kerkoub adds. “And, since these procedures can involve multiple steps—such as pitching the aircraft’s nose downward, increasing power, and rolling the wings level—we also measured how long it took to complete each component of the procedure.”

Additional analysis enabled the team to determine whether the flight crews followed the procedures as designed or if they deviated from them in some way, such as performing the steps out of the prescribed order. They also examined how much time was available to avert an accident and how the flight crew’s performance compared.

By looking at all those factors together, the team calculated the probability of error in the selected safety scenarios.

“Our approach looks at the overall likelihood that pilots would fail to execute a procedure either because they ran out of time, made a mistake, or a combination of the two,” Kerkoub explains.

Research Lays the Foundation for Multiple Applications

Using their combined expertise in black-box-like data, flight operations, human-centered design, and statistics, the team incorporated their findings into predictive models of flight crew performance that are designed to support a variety of needs.

“There are multiple applications for the research we’re doing,” Kerkoub says. “It can enable aircraft designers to evaluate equipment before it gets installed in the cockpit. It can help airlines fine-tune their pilot training. It can be used in aircraft certification processes. And it can help the FAA evaluate new flight procedures before they’re implemented.”

While the potential of the existing findings is great, Kerkoub acknowledges there’s more work to be done. “We’ve broken new ground, but there are many more scenarios to be studied,” she notes.

Another aspect of the team’s plan is to leverage the software and tools the U.S. Nuclear Regulatory Commission already uses to conduct human reliability analysis. “If we can adapt those capabilities to aviation and incorporate our models into them, it would accelerate our research significantly,” Kerkoub says.

Ultimately, the team hopes to put the models into the hands of analysts at the FAA, airlines, and avionics manufacturers. “We need these organizations to vet our models and help us refine them to ensure they meet all stakeholders’ needs,” Gawron says. “They’ve dreamed of having such a capability. This is an opportunity to partner with us to make it a reality. We can’t wait to move this transformational research forward.”

To partner with MITRE and help move this work forward, please contact Houda Kerboub at hkerkoubkourdali@mitre.org.