Several U.S. federal government agencies are modernizing their legacy, mission-critical systems. MITRE’s independent research & development program is investigating the use of large language models (LLMs) to accelerate IT modernization. Our findings provide evidence that LLMs show promise to accelerate legacy code modernization and currently require appropriate human intervention and oversight.
Government agencies need to transition their legacy IT into less expensive, more agile systems that provide more-efficient service delivery. They also must retain decades of federal laws and regulations in legacy software to ensure continuity of services. During the past decades, the federal government has been working to modernize systems—some of which are more than 60 years old. These agencies are exploring how new AI technology, such as LLMs, can accelerate and reduce the cost of modernization. To understand the best practices and unforeseen risks with AI-assisted modernization, MITRE’s IT Modernization (ITMOD) team is exploring LLM logic extraction from legacy systems to aid in modernization.
The ITMOD team generated and quantitatively evaluated intermediate representations (IRs), showing that LLMs can reliably generate IRs at scale from legacy code. However, the evaluations show that current LLM performance measures do not match human subject-matter experts’ perception of quality and are therefore not suitable for quality evaluation. LLM performance for the unique combination of legacy IT languages and complex government systems remains unproven. The ITMOD team recommends that LLMs be used in a highly supervised manner for legacy modernization efforts on mission-critical systems.