Conclusion of First MITRE Challenge Brings New Way to Fast-Track Ideas
MCLEAN, Va., December 14, 2011—The MITRE Corporation has announced the successful conclusion of the first in a series of open competitions called The MITRE Challenge™. The Challenge was developed to encourage innovation in technologies of interest to the federal government. The inaugural competition, Multicultural Person Name Matching, launched in January 2011 and closed in September. The top performing teams—whose names were self-selected and whose identities could remain anonymous—were revealed at a MITRE-government technical exchange meeting (TEM) held October 5 at the company's McLean, Va., campus. Teams known as "Mean Mr Teach," "Riffraff," and "Beethoven" finished in the lead.
Open to academic institutions, commercial companies, government laboratories, and individuals, Challenge #1 focused on multicultural name matching—a technology that is a key component of identity matching, which involves measuring the similarity of database records referring to people. Uses include verifying eligibility for Social Security or medical benefits, identifying and reunifying families in disaster relief operations, vetting persons against a travel watchlist, and merging or eliminating duplicate records in databases.
Nine out of 10 of the highest scoring teams presented at the TEM. Participants included graduate students from Washington and Georgia, a corporate team from the Netherlands, and a team of Lebanese engineers that sent their results to be presented by a member of the Challenge Squad—the MITRE staff who worked with project lead Keith J. Miller to structure and administer the competition. The event also included representatives from the Centers for Medicare & Medicaid Services and the Department of Homeland Security, both of which rely on name and identity matching technology to carry out their missions.
"The whole idea behind this was the concept of 'bringing the world to bear,'" said Miller, a principal artificial intelligence engineer. "We know a lot of the big computational linguistics and identity matching vendors, but we didn't want to stay limited to what those vendors were doing. If Joe in his garage in Wyoming had an idea, we wanted him to have as much of a chance as a team from a major corporation."
Challenge participants were required to match a query file and an index file, each containing a list of names, against one another to produce a list of scored matches for each query name. Registered teams received a dataset and task guidelines, submitted responses, and received immediate feedback on their performance in the form of scores on several metrics. The names of the best performing teams were posted on a continuously updated leaderboard. The goal was to attract submissions that would deliver the greatest performance improvement over the baseline algorithm, ultimately driving improvements in the overall accuracy of the name matching process.
During the nine months of the competition, 140 people registered from across the globe. In total, 40 teams submitted results to the Challenge. The Challenge Squad—comprised of Miller, James Finley, Chris Johnson, Sarah McLeod, Liz Richerson, and Percy Schmidt—received 3,276 submissions (there were no limits on the number of submissions per team). All sectors were represented, including unaffiliated individuals, corporations, educational institutions, not-for-profit organizations, government, and the military.
"One of our main goals was to foster a spirit of collegial competition," added Miller. "We saw that happen throughout the Challenge; teams improved their scores over time, and the leaderboard was a powerful motivator. We were impressed with the methodologies used to get at this complex problem."
The work of the top teams demonstrated improved capability to perform identity matching for use cases of interest to the government.
"The Challenge is an exciting and innovative way to accelerate a broader set of ideas and solutions to the government," said Rich Byrne, senior vice president and general manager of MITRE's Command and Control Center, whose idea it was to create the open competition. "Our hope is that MITRE can start providing this real-time experimentation venue so that the time from development to application is dramatically reduced. You never know where the next innovation's going to come from."
About The MITRE Corporation
The MITRE Corporation is a not-for-profit organization that operates research and development centers sponsored by the federal government. Learn more about MITRE.