Follow Us:

Verification and Validation

Definition: Verification is "the process for determining whether or not the products of a given phase of development fulfill the requirements established during the previous phase [1]. Validation is the "evaluation of the capability of the delivered system to meet the customer's operational need in the most realistic environment achievable [1].

Keywords: systems engineering life cycle, validation, verification

MITRE SE Roles & Expectations: MITRE Systems Engineers (SEs) are expected to understand where verification and validation fit into the systems engineering lifecycle, and how to accomplish them to develop effective and suitable systems. They are expected to assist in developing and defining requirements, specifications for test and evaluation plans, and verification and validation procedures. MITRE SEs are expected to participate in developmental and operational testing, observe and communicate test results, influence retest decisions, recommend mitigation strategies, and assist the customer/sponsor in making system acceptance decisions. They are expected to evaluate test data and verify that specified requirements are met and validated to confirm operational capabilities.

Background

The systems engineering life cycle model may be described differently by MITRE's government customers. The Department of Defense (DoD) customer uses the DoD 5000.02 process to describe a "five stage" systems engineering life cycle [2]. This DoD 5000.02 life cycle model maps to other equivalent models described (e.g., International Organization for Standardization [ISO] 15288 Systems and Software Engineering Life Cycle Processes, and Institute of Electrical & Electronics Engineers [IEEE] 1220-2005 Standard for Application and Management of the Systems Engineering Process). Figure 1, as depicted in the "Engineering for Systems Assurance" Manual Version 1.0 published by the National Defense Industrial Association (NDIA), shows the interrelationships between the different life-cycle processes [3].

Figure 1. DoD Life Cycle Framework, and National Institute of Standards and Technology Information Security and the System Development Life Cycle

Regardless of the life-cycle model our customers uses, they all track to three basic systems engineering stages: concept development, engineering development, and post development. Each of these engineering stages may be separated into supporting phases. The concept development phase is critical because it describes the ultimate operational requirements that will be used to "validate" the ultimate material solution. The supporting system, subsystem, and component level requirements leading to preliminary design and critical design will be iteratively verified through various types of testing and analysis during materialization, integration, and testing. Verification is the critical feedback element that confirms the requirements specified in the previous phase were satisfied. Validation is final confirmation that the user's needs were satisfied in the final material solution. It cannot be overemphasized that Verification and Validation (V&V) and Test and Evaluation (T&E) are not separate stages or phases, but integrated activities within the SE process. Figure 2, from the Washington State Department of Transportation (DOT), illustrates how V&V provide feedback to the systems engineering process [4].

Figure 2. Systems Engineering "V," Washington State DOT

Government Interest and Use

Using the ISO 15288 Systems and Software Engineering Life Cycle Processes as a model, V&V are critical activities that are executed continuously throughout the process. During the initial concept development stage, verification activities confirm that the operational and performance requirements and functional specifications are viable. These requirements and specifications may be developed by the government, by MITRE, and/or by other Systems Engineering and Technical Assistance (SETA) contractors and must be verified. The operational requirements will be used by the government for ultimate validation of the material solution. The performance and functional specification must also be validated, because they will be used by the developing contractor to drive preliminary and critical design and to develop the material solution. During the engineering development stage, the subcomponents and components that comprise the material solution must be verified, integrated, and tested. Operational testing is the venue that gathers data to validate that the ultimate material solution satisfies required operational capabilities. V&V are critical activities that confirm that the "contracted for" material solution provides the required operational capability.

V&V Best Practices and Lessons Learned

• Continuously Coordinate Process Execution: In many cases, the capabilities development, systems acquisition, and systems engineering processes, although interdependent, are executed independently by different organizations (or different parts of the same organization, such as the prime contractor). Disconnects between the activities executed by the different stakeholders can create serious problems. This can lead to cost and schedule overruns, and reduced operational capability. Active verification of the output from each step of the process and an active risk management program can go far to identify and address disconnects as they occur. The earlier a problem is identified and addressed, the less expensive the resolution will be in term of cost, schedule, and performance. Early and continuous involvement of subject matter experts is required.
• Operational Requirements Verification—A Team Sport: Verified operational requirements are critical to validation of the system. In many cases, the operational requirements are poorly documented or change during the course of an acquisition. Verification of operational requirements must involve all potential stakeholders, including the acquisition program manager, systems engineering team, and validation agent (operational tester).
• Smart Contracting: The government develops operational capability needs, functional requirements, and systems specifications that are placed on contract to develop preliminary and critical designs, and to materialize the system. The contract should include Contract Data Requirements Listings (CDRLs) that require the contractor to develop and the government to approve test plans, monitor test execution, and deliver reports that support subcomponent, component, and system verification. This may involve additional upfront cost for the program. However, failure to do so is likely to result in additional cost, longer schedule, and performance shortfalls to the required operational capability. The acquisition program manager, systems engineer, and government contracting officer must work carefully to shape requests for proposal, evaluate responses, and form the contract to include these CDRLs.
• Harmonize Use of Modeling and Simulation (M&S) in Verification: M&S can be used as part of the T&E process to verify subcomponents, components, and systems. The program manager should involve the contractor, and development and operational test agencies to identify where M&S can be used to generate data for use in verification. Establish the intended use of M&S by each of the testing stakeholders at the beginning of the systems engineering process. The M&S approach can then be harmonized across several intended users and phases of V&V.
• Integrated Testing Supports Continuous Verification and Operational Validation: The goal of Operational Test and Evaluation (OT&E) is to confirm that the "concept" developed on the left side of the systems engineering "V" can be validated in the "material solution" on the right side. The Operational Testing Agent (OTA) often seeks contractor and developmental test data to validate capabilities. Often requirements cannot be validated because CDRLs were not specified in the contract and/or developmental test data were not clearly specified by the OTA or delivered by the program manager/developer. In some cases, the verification activities were haphazard or not properly executed. In cases where there has been an undisciplined approach to verification, test and evaluation (missing entry or exit criteria for each step), significant gaps, and holes may exist in the material solution that are not evident until OT&E is executed. CDRLs, integration, security, interoperability, development test events, and supporting data requirements should be clearly specified in T&E Master Plans. Time to fix and retest also would be included in the process. The ultimate goal is to execute an integrated testing approach where a component/system/ system-of-systems testing and verification can be executed by one stakeholder, and the data accepted by all stakeholders. Any such testing/verification approach must be documented in test and evaluation plans and resources to execute documented.

References & Resources

1. Kossiakoff, A. and W. Sweet, 2003, Systems Engineering: Principles and Practice, John Wiley and Sons, ISBN 0-471-23443-5
2. U.S. Department of Defense, December 8, 2008, "Operation of the Defense Acquisition System," DoD Instruction 5000.02.
3. National Defense Industrial Association System Assurance Committee, "Engineering for System Assurance," Version 1.0.
4. Washington State DOT, July 2010, WSDOT Design Manual, Chapter 1050.03, Systems Engineering: Systems Engineering "V."