Telephone Caption Quality Measures and Metrics Working Group Final Report

While captioned telephone services have been in use in the U.S. since 2004, well-defined industry standard metrics for evaluating telephone caption quality do not exist. Currently, Internet Protocol (IP) Captioned Telephone Services (CTS) are not measured against standards related to caption accuracy, caption delay, or overall communication effectiveness.

To help address this need, in February 2023, MITRE established the Telephone Caption Quality Measures and Metrics Working Group (the Working Group), comprised of community advocates, IP CTS providers, academia, and subject matter experts (SMEs) from related industries, to identify caption quality measures and metrics related to IP CTS.

The Working Group met every two weeks to identify information that can inform the Federal Communications Commission's Disability Rights Office and Office of the Managing Director about important considerations for defining quality of service for captioned telephones. 

According to the charter, the mission and vision of the working group was to:

• Identify and define measures that can be used to quantify and compare caption quality as it relates to effective communication
• Propose methods for assessing IP CTS using these measures
• Identify potential criteria for establishing meaningful thresholds for acceptable caption quality

The Working Group identified recommendations for immediate and longer-term actions that will lead to a more complete understanding of how to measure caption quality and, ultimately, define thresholds for acceptable caption quality. 

The Working Group identified six recommendations:

• Work with an American National Standards Institute (ANSI)-certified standards developer to initiate a process to formalize caption quality standards
• Continue to refine measures and metrics
• Adopt a more transparent testing framework
• Use the adopted framework to measure characteristics of caption accuracy, caption delay, non-speech information, and punctuation and formatting
• Share research and testing information
• Perform additional research to improve measures, identify appropriate metrics, and establish thresholds for acceptable caption quality