Verification and Validation
Definition: Verification is the process for determining whether or not a product fulfills the requirements or specifications established for it. Validation is the assessment of a planned or delivered system to meet the sponsor's operational need in the most realistic environment achievable.
Keywords: systems engineering life cycle, validation, verification
MITRE SE Roles & Expectations: MITRE systems engineers (SEs) are expected to understand where verification and validation fit into the systems engineering life cycle and how to accomplish them to develop effective and suitable systems. They are expected to help develop and define verification and validation requirements and specifications for test and evaluation plans and procedures. MITRE SEs are expected to participate in developmental and operational testing; observe, evaluate, and communicate test results; influence retest decisions; recommend mitigation strategies; and assist the sponsor in making system acceptance decisions. They are expected to evaluate test data and verify that requirements and specifications are met and validated to confirm operational capabilities.
The systems engineering life-cycle model may be described differently by MITRE's government sponsors. The Department of Defense (DoD) sponsor uses the DoD 5000.02 process to describe a "five stage" systems engineering life cycle . This DoD 5000.02 life-cycle model maps to other equivalent models described (e.g., International Organization for Standardization [ISO] 15288 Systems and Software Engineering Life Cycle Processes , and Institute of Electrical & Electronics Engineers [IEEE] 1220-2005 Standard for Application and Management of the Systems Engineering Process ). Figure 1, as depicted in Engineering for Systems Assurance, Ver. 1.0, published by the National Defense Industrial Association (NDIA), shows the interrelationships among the different life-cycle processes .
Regardless of the life-cycle model our sponsors use, they all track to three basic systems engineering stages: concept development, engineering development, and post-development. Each of these engineering stages may be separated into supporting phases. The concept development phase is critical because it describes the ultimate operational requirements that will be used to "validate" the ultimate material solution. The supporting system, subsystem, and component level specifications leading to preliminary design and critical design will be iteratively verified through various types of testing and analysis during materialization, integration, and testing. Verification is the critical feedback element that confirms the specifications were satisfied. Validation is confirmation that the user's needs will be or are satisfied in the final material solution. It cannot be overemphasized that Verification and Validation (V&V) and Test and Evaluation (T&E) are not separate stages or phases, but integrated activities within the SE process. Figure 2, from the Washington State Department of Transportation (DOT), illustrates how V&V provide feedback to the systems engineering process .
Government Interest and Use
Using the ISO 15288 Systems and Software Engineering Life-Cycle Processes  as a model, V&V are critical activities that are executed continuously throughout the process. During the initial concept development stage, verification activities confirm that the operational and performance requirements and functional specifications are viable. These requirements and specifications may be developed by the government, by MITRE, and/or by other Systems Engineering and Technical Assistance (SETA) contractors, and they must be verified. The government will use the operational requirements for ultimate validation of the material solution. The performance and functional requirements must also be validated because the developing contractor will use them to drive preliminary and critical design and to develop the material solution. During the engineering development stage, the subcomponents and components that make up the material solution must be verified, integrated, and tested. Operational testing is the venue that gathers data to validate that the ultimate material solution satisfies required operational capabilities. V&V are critical activities that confirm that the "contracted for" material solution provides the required operational capability.
Best Practices and Lessons Learned
- Continuously coordinate process execution. In many cases, the capabilities development, systems acquisition, and systems engineering processes, although interdependent, are executed independently by different organizations (or different parts of the same organization, such as the prime contractor). Disconnects between the activities executed by the different stakeholders can create serious problems. This can lead to cost and schedule overruns as well as to reduced operational capability. Active verification of the output from each step of the process and an active risk management program can go far to identify and address disconnects as they occur. The earlier a problem is identified and addressed, the less expensive the resolution will be in terms of cost, schedule, and performance. Early and continuous involvement of subject matter experts is required.
- Operational requirements verification—A team sport. Verified operational requirements are critical to the validation of the system. In many cases, the operational requirements are poorly documented or change during the course of an acquisition. Verification of operational requirements must involve all potential stakeholders, including the acquisition program manager, systems engineering team, and validation agent (operational tester).
- Smart contracting. The government develops operational capability needs, functional requirements, and systems specifications that are placed on contract to develop preliminary and critical designs and to materialize the system. The contract should include Contract Data Requirements Listings (CDRLs) that require the contractor to develop and the government to approve test plans, monitor test execution, and deliver reports that support subcomponent, component, and system verification. This may involve additional upfront cost for the program. However, failure to do so is likely to result in additional cost, longer schedule, and performance shortfalls to the required operational capability. The acquisition program manager, SE, and government contracting officer must work carefully to shape requests for proposal, evaluate responses, and form the contract to include these CDRLs.
- Harmonize use of modeling and simulation (M&S) in verification. M&S can be used as part of the V&V process to V&V subcomponents, components, and systems. The program manager should involve the contractor as well as development and operational test agencies to identify where M&S can be used to generate data for use in V&V. Establish the intended use of M&S by each of the testing stakeholders at the beginning of the systems engineering process. The M&S approach can then be harmonized across several intended users and phases of V&V.
- Integrated testing supports continuous verification and operational validation. The goal of Operational Test and Evaluation (OT&E) is to confirm that the "concept" developed on the left side of the systems engineering "V" can be validated in the "material solution" on the right side. The Operational Testing Agent (OTA) often seeks contractor and developmental test data to validate capabilities. Often requirements cannot be validated because CDRLs were not specified in the contract and/or developmental test data were not clearly specified by the OTA or delivered by the program manager/developer. In some cases, the verification activities were haphazard or not properly executed. If an undisciplined approach to verification is taken, test and evaluation (missing entry or exit criteria for each step), significant gaps, and holes may exist in the material solution that are not evident until OT&E is executed. CDRLs, integration, security, interoperability, development test events, and supporting data requirements should be clearly specified in T&E Master Plans. Time to fix and retest also would be included in the process. The ultimate goal is to execute an integrated testing approach in which one stakeholder can execute a component/system/system-of-systems testing and verification and all stakeholders can accept the data. Any such testing/verification approach must be documented in test and evaluation plans, and resources to execute must be documented.
References & Resources
- U.S. Department of Defense, January 7, 2015, "Operation of the Defense Acquisition System," DoD Instruction 5000.02.
- ISO/IEC 15288, 2008, Systems Engineering—System Life Cycle Processes.
- IEEE 1220-2005, 2005, reaffirmed in 2011, Standard for Application and Management of the Systems Engineering Process.
- National Defense Industrial Association System Assurance Committee, October 2008, Engineering for System Assurance, Ver. 1.0.
- Washington State DOT, July 2010, WSDOT Design Manual, Chapter 1050.03, Systems Engineering: Systems Engineering "V."