Can Training in Critical Analytic Thinking Improve Job Performance?August 2018
Topics: Education and Training, Management (General), Social and Behavioral Sciences
For the government, making sure analysts do their jobs well is especially important. Agencies want to confirm they don't miss crucial clues within data or make assumptions that hinder critical analytic thinking. Several agencies have developed training programs in critical analytic thinking. But do these training programs work?
Under government sponsorship, MITRE researchers have conducted a first-of-its-kind study on a test that shows promise in evaluating the effectiveness of critical analytic thinking training. The findings indicate that critical analytic thinking skills are a predictor of job performance for positions involving analytical skills. In addition, the findings suggest that although critical analytic thinking skills are closely aligned with general intelligence, they're not one and the same.
A team of MITRE experts in behavioral sciences originally created the test to help a government sponsor assess its training courses for developing critical analytic thinking skills. MITRE has written a report on its findings recently published in Personnel Assessment and Decisions, a peer-reviewed human resources journal.
Defining Critical Analytic Thinking
What are critical analytic thinking skills? The ability to analyze information and reach a conclusion. Critical analytic thinking involves skills such as causal reasoning (the ability to figure out whether one action caused another) and identifying assumptions (determining whether something is a fact or merely an existing belief). As a company that works to make the world safer, we understand how vital such skills are for mission success.
Sara Beth Elson, a behavioral scientist and the lead writer for MITRE’s report, says that "critical analytic thinking is a tendency to engage in effortful processing. For example, this could include questioning your assumptions about the connections between A and B. If you’re making assumptions about the connection, you’d identify those assumptions as such and gather information to test them."
"Our study suggests that the tendency to engage in effortful thinking stands above and beyond sheer intelligence itself in predicting job performance," she says. "We hope that future researchers will examine the question further to see whether our findings hold up with different populations, different measures of cognitive ability, and different measures of job performance."
Amber Sprenger, who specializes in human cognition and decision-making at MITRE, says the team's work differs from other studies on critical analytic thinking. Some of the other studies made a connection between having strong critical analytic thinking skills and doing well on the job. But they hadn’t looked at the predictive value of critical analytic thinking over and above general intelligence.
"Other studies looked at whether critical thinking predicts job performance, but they didn’t add in a comparison of other existing measures of cognitive ability," Sprenger says. "Honestly, ahead of time, I would have guessed that critical thinking is pretty similar to cognitive ability. It's a component of cognitive ability, and cognitive ability is the strongest predictor out there of job performance.
"So, I wasn’t sure that it would have any incremental benefit or not. Our findings, in my opinion, were really surprising."
Sprenger cautions that MITRE's critical analytic skills test is only a predictor for how well one might do in jobs that require critical analytic thinking.
"For other jobs, for other job performance measures, we might not find the same result," she says. "It might be very specific to the type of analytical work that these government analysts are doing."
How the Team Weighed the Results
MITRE's Critical Analytic Thinking Skills (CATS) test was administered to 140 government analysts across 16 different government agencies. The test included questions involving the four major components of critical analytic thinking: causal reasoning, identifying assumptions, evaluating hypotheses, and logical reasoning. The MITRE team looked for correlations between test results and performance on an exercise designed to resemble the tasks these analysts do at their jobs.
The exercise that resembled job tasks was called the Analytic Work Sample Task. In comparing the results of the CATS test with the results of the exercise, MITRE found a strong correlation between doing well on the CATS test and doing well on the work sample task, Elson says.
"Along with the correlation between critical analytic thinking skills and performance on the work sample task, the results suggest that critical analytic thinking is a separate construct from the construct of general intelligence," she says. "Not only that, but critical analytic thinking might play a separate role in predicting performance on an analytic work sample task."
"The test we developed can be used to gauge the effectiveness of training in CATS," Elson adds. "We gave them a tool for measuring the construct of critical analytic thinking."
A Benefit for Government Agencies and Businesses
"The findings of this study hold implications for both academic researchers investigating the predictors of job performance and for businesses," MITRE’s report stated. "For academic studies, the findings suggest that it is worth measuring critical thinking in appropriate contexts. For businesses, the findings substantiate the interest shown in critical thinking skills when taking an evidence-based decision-making approach toward business management."
In summarizing the implications of the research, Elson notes, "Our study is the first to suggest that critical analytic thinking plays a separate role from intelligence in predicting performance. We hope our study inspires others to probe the question further by looking at different populations, using different measures of performance, and different indicators of intelligence.
"Ultimately, we hope the findings benefit government agencies and businesses wanting to improve job performance by teaching critical analytic thinking."
—by Tom Nutile
Members of the MITRE CATS study team included Sara Beth Elson, Amber Sprenger, Robert Hartman, Adam Beatty, Matt Trippe, Kerry Buckley, John Bornmann, Elaine Bochniewicz, Mark Lehner, Liliya Korenovska, Jessica Lee, Les Servi, Alison Dingwall, Paul E. Lehner, Maurita Soltis, Mark Brown, and Brandon Beltz.