Follow Us:

Test and Evaluation

Definition: Test & Evaluation (T&E) is the process by which a system or components are compared against requirements and specifications through testing. The results are evaluated to assess progress of design, performance, supportability, etc. Developmental test and evaluation (DT&E) is an engineering tool used to reduce risk throughout the acquisition cycle. Operational test and evaluation (OT&E) is the actual or simulated employment, by typical users, of a system under realistic operational conditions [1].

Keywords: analysis, DT&E, evaluation, OT&E, performance, testing, verification

Context

Testing is a mechanism to assure quality of a product, system, or capability (e.g., right product, built right). To be effective, testing cannot occur only at the end of a development. It must be addressed continuously throughout the entire life cycle.

Test and Evaluation involves evaluating a product from the component level, to stand-alone system, integrated system, and, if appropriate, system-of-system and enterprise. Figure 1 highlights these levels of evaluation and how they align with government DT, OT, and accreditation and certification testing.

Figure 1. Product Life-cycle Test Phases

The articles in this topic are arranged according to a notional V-model life cycle [2] chronology of a program acquisition: (1) creating and assessing T&E strategies, (2) assessing T&E plans and procedures, (3) verification and validation, and (4) creating and assessing certification and accreditation strategies throughout the process. As noted elsewhere in the Guide, the system life cycle is rarely, if ever, as linear as the simplified V-model might imply.

MITRE SE Roles & Expectations: MITRE systems engineers (SEs) are expected to be able to create test and evaluation strategies to field effective, interoperable systems that include making recommendations on certification and accreditation processes. They assist in developing and defining test and evaluation plans and procedures. MITRE SEs participate in developmental and operational testing, observe and communicate test results, influence re-test decisions, recommend mitigation strategies, and assist the customer/sponsor in making system acceptance decisions.

Best Practices and Lessons Learned

Employ prototypes and M&S to advantage. Prototypes and/or modeling and simulation (M&S) used early in a program can help predict system performance and identify expected results, both good and bad. Both techniques can be used in designing, evaluating, or debugging portions of a system before incurring the expense of "bending metal."

Common sense—sometimes a rarity. Use common sense in testing. For example, while it is necessary to ensure that environment testing covers the environment that your system is expected to operate in, specifying and testing a system to operate to -70°C when it will be used in an office-like environment is a sure way to either fail the test or drive the cost of the system beyond reason. This is an especially common pitfall when designing systems for mobile or airborne environments. Extreme temperature, vibration, radiated emissions, etc. are not always encountered in these environments. Ensure that the tests are realistic.

Match testing method with purpose. There are many forms of testing. Some involve instrumented measurements of system performance during "live" operations. Others, in decreasing order of complexity, are analysis, demonstration, or inspection. Select the testing method that suits the purpose. The performance of a critical operational capability (e.g., identification of airborne objects as friend, hostile, or neutral) will likely require all or most of the methods and culminate in a "live" test. Analysis is suited to testing requirements like long-term reliability of electronic components, and when assessing inspection is appropriate (e.g., number of operator consoles in a command center). Selecting the right verification methods produces the right results and saves time and cost.

Test strategy—start early and refine continuously. Plan the test strategy from the onset of the program and refine it throughout the program's life cycle. Involve the right stakeholders in the development and review of the test strategy and plans.

Don't overlook the basics. Ensure that tests have been developed to be objective and capable of assessing compliance with a requirement. Make sure that if one test is intended to validate many lower level requirements, you are sufficiently versed with the details of the system design and have the results of the component level tests available. This is particularly important in preparing for operation testing.

Ready or not? Determining suitability to enter a test is a key decision that can substantially affect the overall success of the program. Know when you are ready, and know when you are not. When unsure, postponing or deferring testing may be the most prudent long-term course of action.

References & Resources

1. Defense Acquisition University website.
2. Wikipedia contributors, "V-Model," Wikipedia, The Free Encyclopedia, accessed January 13, 2010.

Additional References & Resources

Defense Acquisition University, "Test and Evaluation," Defense Acquisition Guidebook, Chapter 9.

Haimes, 1998, Risk Modeling, Assessment, and Management, Wiley.

International Test and Evaluation Association website.

Stevens, et al., 1998, Systems Engineering: Coping with Complexity, Prentice Hall.

The MITRE Institute, September 1, 2007, "MITRE Systems Engineering (SE) Competency Model, Version 1," Section 2.6.