Verification and Validation of Simulation Models
Definitions: Verification is the process of determining that a model implementation and its associated data accurately represent the developer's conceptual description and specifications .
Validation is the process of determining the degree to which a simulation model and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model .
Keywords: accreditation, analysis, data, historical, SME, statistical, subject matter expert, validation, verification
MITRE SE Roles & Expectations: The MITRE systems engineer (SE) is expected to have a sound knowledge of the system being modeled and the software process for developing the model in order to provide effective technical guidance in the design and execution of plans to verify and/or validate a model, or to provide specialized technical expertise in the collection and analysis of varying types of data required to do so. A MITRE SE is expected to be able to work directly with the developer of the system and the simulation model to provide technical insight into model verification and validation. In most cases, the MITRE SE will be responsible for assisting the government sponsoring organization in the formal accreditation of the model.
Modeling and simulation (M&S) can be an important element in the acquisition of systems within government organizations. M&S is used during development to explore the design trade space and inform design decisions, and in conjunction with testing and analysis to gain confidence that the design implementation is performing as expected, or to assist troubleshooting if it is not. M&S allows decision makers and stakeholders to quantify certain aspects of performance during the system development phase, and to provide supplementary data during the testing phase of system acquisition. More important, M&S may play a key role in the qualification ("sell-off”) of a system as a means to reduce the cost of a verification test program. Here, the development of a simulation model that has undergone a formal verification, validation, and accreditation (VV&A) process is not only desirable, but essential.
Consider the following definitions for the phases of the simulation model VV&A process :
- Verification: "The process of determining that a model implementation and its associated data accurately represent the developer’s conceptual description and specifications."
- Validation: "The process of determining the degree to which a [simulation]model and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model."
- Accreditation: "The official certification that a model, simulation, or federation of models and simulations and its associated data are acceptable for use for a specific purpose."
- Simulation Conceptual Model: "The developer's description of what the model or simulation will represent, the assumptions limiting those representations, and other capabilities needed to satisfy the user's requirements."
Verification answers the question "Have we built the model right?" whereas validation answers the question "Have we built the right model?” . In other words, the verification phase of VV&A focuses on comparing the elements of a simulation model of the system with the description of what the requirements and capabilities of the model were to be. Verification is an iterative process aimed at determining whether the product of each step in the development of the simulation model fulfills all the requirements levied on it by the previous step and is internally complete, consistent, and correct enough to support the next phase . The validation phase of VV&A focuses on comparing the observed behavior of elements of a system with the corresponding elements of a simulation model of the system, and on determining whether the differences are acceptable given the intended use of the model. If agreement is not obtained, the model is adjusted in order to bring it in closer agreement with the observed behavior of the actual system (or errors in observation/experimentation or reference models/analyses are identified and rectified).
Typically, government sponsoring organizations that mandate the use of a formal VV&A process do not specify how each phase should be carried out. Rather, they provide broad guidance that often includes artifacts required from the process. Independent Verification and Validation (IV&V) activities occur throughout most of the systems engineering development life-cycle phases and are actively connected to them, as depicted in Figure 1, rather than being limited to integration and testing phases.
A variety of methods are used to validate simulation models, ranging from comparison to other models to the use of data generated by the actual system (i.e., predictive validation). The most commonly used methods are described in Table 1 .
Table 1. Common Simulation Model Validation Methods
Model Validation Method
Comparison to Other Models
Various results (e.g., outputs) of the simulation model being validated are compared to results of other (valid) models. For example, (1) simple cases of a simulation model are compared to known results of analytic models and (2) the simulation model is compared to other simulation models that have been validated.
Asking individuals knowledgeable about the system whether the model and/or its behavior are reasonable. For example, is the logic in the conceptual model correct and are the model’s input-output relationships reasonable?
Historical Data Validation
If historical data exist (e.g., data collected on a system specifically for building and testing a model), part of the data are used to build the model and the remaining data are used to determine (test) whether the model behaves as the system does.
Parameter Variability – Sensitivity Analysis
This technique consists of changing the values of the input and internal parameters of a model to determine the effect on the model’s behavior of output. The same relations should occur in the model as in the real system. This technique can be used qualitatively—directions only of outputs—and quantitatively—both directions and (precise) magnitudes of outputs. Those parameters that are sensitive (i.e., cause significant changes in the model’s behavior or output) should be made sufficiently accurate prior to using the model.
The model is used to predict (forecast) the system’s behavior, and then the system’s behavior and the model’s forecast are compared to determine if they are the same. The system’s data may come from an operational system or be obtained by conducting experiments on the system, e.g., field tests.
With the exception of face validity, all the methods detailed in Table 1 are data-driven approaches to model validation, with predictive validation among the most commonly used method. The use of predictive validation generally requires a significant amount of effort to acquire and analyze data to support model validation.
Best Practices and Lessons Learned
Develop and maintain a model VV&A plan. Develop a detailed model VV&A plan before the start of the acquisition program VV&A process. Distinct from the specification for the model itself, this plan provides guidance for each phase of VV&A and clarifies the difference between the verification and validation phases. The plan should also map model specification requirements to model elements, identify those model elements that require validation, and develop model validation requirements. The plan should describe the method(s) used for validation, including any supporting analysis techniques. If a data-driven approach is used for model validation, detail should be provided on either the pedigree of existing data or the plan to collect new data to support validation. Prototype simulations and ad hoc models are often developed where a VV&A plan is not considered beforehand. Then, when the prototype becomes a production system, an attempt is made to perform V&V. It is extremely difficult to perform verification after the fact when normal system development artifacts have not been created and there is no audit trail from concept to product.
Establish quantitative model performance requirements. Often performance requirements are neglected while developing the domain model. Complex systems can have interactions that produce unexpected results in seemingly benign situations. Prototypes developed with small problem sets may not scale to large problems that production systems will deal with. More model detail does not necessarily generate a better answer and may make a simulation intractable. In discrete event simulation, the appropriate event queue implementation, random number generators, and sorting/searching algorithms can make a huge difference in performance.
Establish quantitative model validation requirements. Specify the degree to which each element (component, parameter) of the model is to be validated. For model elements based on a stochastic process, validation is often based on statistical tests of agreement between the behavior of the model and that of the actual system. If a hypothesis test is used to assess agreement, both allowable Type I and Type II risks (errors) need to be specified as part of an unambiguous and complete validation requirement. If, instead, a confidence interval is used to assess agreement, the maximum allowable interval width (i.e., precision) and a confidence level need to be specified.
In certain instances, the decision to use either a hypothesis test or a confidence interval may be a matter of preference; however, there will be instances when using a hypothesis test is necessary, due to the nature of the model element being validated. Regardless, the development of a validation requirement is often an involved task requiring the use of analytic or computational methods to determine allowable levels of validation risk or precision. Sufficient time, resources, and expertise should be allocated to this task.
Develop model validation "trade-off” curves. If predictive validation is used, the amount of data required to achieve a quantitative model validation requirement (see above) will need to be determined. Determining the amount of data required (N) to achieve, say, a maximum allowable confidence interval width at a specified confidence level for any model element may require extensive analytical or computational methods. Related to this is solving the "inverse” problem: determining the maximum (or expected) confidence interval width given a particular value of N. From this, a set of curves may be constructed to allow for the "trade-off” between the amount of validation data required (which, generally, drives the cost of the validation effort) in order to achieve a validation requirement. This problem should be addressed as early as possible during the validation phase of the VV&A process.
Not every model element requires validation. Model elements associated with model functional requirements usually do not require validation. These elements are only dealt with in the verification phase of the VV&A process. Trivial examples of this are model elements that allow the user to select various model options (i.e., "switches” or "knobs”). Non-trivial examples are elements that have been identified as not being relevant or critical to the intended use of the model. However, a model element that has not been deemed critical may, in fact, be fully functional when the simulation model is deployed. In this case, the model element may still be exercised, but the model accreditation documentation should note that this particular element has been "verified, but not validated.”
Allow for simplifications in the model. Almost always, some observed behaviors of the actual system will be difficult to model or validate given the scope and resources of the model development and validation efforts. In such cases, using simplifications (or approximations) in the model may provide an acceptable way forward. For example, if a model element requires a stochastic data-generating mechanism, a probability density function with a limited number of parameters (e.g., a Gaussian distribution) may be used in place of what appears to be, based on analysis of data from the actual system, a more complex data-generating mechanism. In doing this, a conservative approach should be used. That is, in this example, employing a simplified data-generating mechanism in the model should not result in overly optimistic behavior with respect to the actual system.
Plan for parallel (iterative) model development and VV&A. The model development and its VV&A process should be carried out in parallel, with sufficient resources and schedule to allow for several iterations. Figure 1 depicts a notional relationship between model development and the VV&A process. The focus here is the gathering and analysis of validation data for a particular model element and the resulting decision to: (1) adjust the value of one or more parameters associated with the model element to obtain closer agreement with observed system behavior, (2) redesign the model element by factoring in insights obtained from the analysis of validation data, or (3) accredit the model with respect to this element.
Although Figure 1 depicts the relationship between model development and VV&A in its entirety, it may be applied to individual, independent model elements. A model VV&A plan should identify those model elements that may be validated independently of others.
Consider the partial model validation of selected model elements. When the validation of a particular model element becomes problematic, it may be acceptable to validate the model element over a subset of the element’s defined range. This partial validation of a model element may be a viable option when either insufficient data is available to enable validation or the actual system behavior is difficult to model even if reasonable simplifications are made. However, the resulting model will be valid with respect to this element only within this limited range. This fact should be noted in accreditation documentation.
Use multiple approaches to model validation. When data-driven model validation is not possible or practical, face validity may be used. However, even when data-driven validation can be carried out, face validity (i.e., expert review) may be used as a first step in the validation phase. If a similar but already validated model is available, performing a comparison of the model being developed to this existing model may provide an initial (or preliminary) model validation. Following this initial validation, a more comprehensive approach such as predictive validation may be employed.
Subsystem verification and validation do not guarantee system credibility. Each submodel may produce valid results and the integration of models can be verified to be correct, but the simulation can still produce invalid results. Most often this occurs when the subsystem conceptual design includes factors that are not considered in the system conceptual design, or vice versa. For example, in a recently reviewed simulation system with multiple subcomponents, one of the subcomponents simulated maintenance on vehicles, including 2½ ton trucks. There was no need for the submodel to distinguish between truck types, however, vehicle type distinction was important to another module. In the system, tanker trucks used for hauling water or petroleum were being returned from the maintenance submodel and became ambulances—alphabetically the first type of 2½ ton trucks.
Know what your simulation tool is doing. If a simulation language or tool is used to build the simulation, "hidden” assumptions are built into the tool. Here are four common situations that are handled differently by different simulation tools:
- Resource Recapture—Suppose a part releases a machine (resource) and then immediately tries to recapture the machine for further processing. Should a more highly qualified part capture the machine instead? Some tools assign the machine to the next user without considering the original part a contender. Some tools defer choosing the next user until the releasing entity becomes a contender. Still others reassign the releasing part without considering other contenders.
- Condition Delayed Entities—Consider two entities waiting because no units of a resource are available. The first entity requires two units of the resource, and the second entity requires one unit of the resource. When one unit of the resource becomes available, how is it assigned? Some tools assign the resource to the second entity. Other tools assign it to the first entity, which continues to wait for another unit of the resource.
- Yielding Control Temporarily—Suppose that an active entity wants to yield control to another entity that can perform some processing, but then wants to become active and continue processing before the simulation clock advances. Some tools do not allow an entity to resume processing at all. Other tools allow an entity to compete with other entities that want to process. Still others give priority to the relinquishing process and allow it to resume.
- Conditions Involving the Clock—Suppose an entity needs to wait for a compound condition involving the clock, e.g., "wait until the input buffer is empty or it is exactly 5:00 p.m.” Generally the programmer will have to "trick” the system to combine timed events with other conditions. An event at 5:00 p.m. could check to see if the buffer was empty, and if so, assume that the buffer-empty event occurred earlier.
Test for discrete math issues. Computer simulation models use discrete representations for numbers. This can cause strange behaviors. When testing models, always include tests at the extremes. Examples of errors found during testing include overflow for integers and subtraction of floating point numbers with insufficient mantissa representation. Conversion from decimal to binary representation could cause rounding errors that are significant in the simulation.
Establish a model validation working group. A regular and structured working group involving the sponsoring government organization, the system developer, and the model developer will result in a superior simulation model. The function of this group should be the generation and review of artifacts from the model validation phase. When data-driven validation is used, the majority of the effort should focus on the information products produced by the statistical analysis of validation data. This group may also provide recommendations regarding the need to collect additional validation data, should increased quality in the validation results be necessary.
Invest in analysis capabilities and resources. Plans for the availability of subject matter experts, sufficient computing and data storage capability, and analysis software should be made early in the VV&A process. For many VV&A efforts, at least two subject matter experts will be required: one who has knowledge of the system being modeled and another who has broad knowledge in areas of data reduction, statistical analysis, and, possibly, a variety of operations research techniques. If analysis software is needed, consider using open-source packages—many provide all the data reduction, statistical analysis, and graphical capability needed to support validation efforts.
The successful validation of a simulation model requires the government sponsoring organization to address VV&A early in the life of an acquisition program. Key activities include:
- The development of a model VV&A plan with quantitative model validation requirements (where appropriate).
- Detailed planning for the collection of validation data (if data-driven validation is needed).
- The assembly of a working group that includes both domain experts and analysts.
Model development and model validation should not be carried out sequentially, but in a parallel and iterative manner.
References & Resources
- DoD Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A), ), DoD Instruction 5000.61, December 9, 2009.
- Cook, David A., and Skinner, James M., How to Perform Credible Verification, Validation, and Accreditation for Modeling and Simulation, CrossTalk: The Journal of Defense Software Engineering, May 2005.
- Lewis, Robert O., Independent Verification and Validation, Wiley & Sons, 1992.
- Law, Averill M., How to Build Valid and Credible Simulation Models, paper presented at the Winter Simulation Conference, 2008.
Additional References & Resources
Law, Averill M., Simulation Modeling and Analysis, McGraw Hill, 2007.