Assess the Design's Ability to Meet the System Requirements


Definition: The ability of a system design to meet operational, functional, and system requirements is necessary to accomplishing a system's ultimate goal of satisfying mission objective(s). One way of assessing the design's ability to meet the system requirements is through "requirements traceability." This is the process of creating and understanding the bi-directional linkage between requirements (operational need), organizational goals, and solutions (performance).

Keywords: assessment, concept of operations (CONOPS), functional requirements, mission and needs, operational requirements, performance verification, requirements, requirements traceability, requirements traceability matrix, system requirements, traceability, verification

MITRE SE Roles and Expectations: MITRE systems engineers (SEs) are expected to understand the importance of system design in meeting the government's mission and goals. They are expected to be able to review and influence the contractor's preliminary design so that it meets the overall business or mission objectives of the sponsor, customer, and user. MITRE SEs are expected to be able to recommend changes to the contractor's design activities, artifacts, and deliverables to address performance shortfalls and advise the sponsor or customer if a performance shortfall would result in a capability that supports mission requirements whether or not the design meets technical requirements. They are expected to be thought leaders in influencing decisions made in government design review teams and to appropriately involve specialty engineering.

In requirements traceability and performance verification, MITRE SEs are expected to maintain an objective view of requirements and the linkage between the system end-state performance and the source requirements and to assist the government in fielding the best combination of technical solution, value, and operational effectiveness for a given capability.

Background

Traceability and Verification Process: A meaningful assessment of a design's ability to meet system requirements centers on the word "traceability." Traceability is needed to validate that the solution delivered fulfills the operational need. For example, if a ship is built to have a top speed of 32 knots, there must be a trail of requirements tied to performance verification that justifies the need for the additional engineering, construction, and sustainment to provide a speed of 32 knots. The continuum of requirements generation and traceability is one of the most important processes in the design, development, and deployment of capability.

Traceability is also the foundation for the change process within a project or program. Without the ability to trace requirements from end to end, the impact of changes cannot be effectively evaluated. Furthermore, change should be evaluated in the context of the end-to-end impact on other requirements and overall performance (e.g., see the Guide section on Enterprise Engineering). This bi-directional flow of requirements should be managed carefully throughout a project/program and be accompanied by a well-managed requirements baseline.

Requirements Flow: The planned functionality and capabilities offered by a system need to be tracked through various stages of requirements (operational, functional, and system) development, and evolution. Requirements should support higher level organizational initiatives and goals. It may not be the project's role to trace requirements to larger agency goals. However, it can be a best practice in ensuring value to the government. In a funding-constrained environment, requirements traceability to both solutions as well as organizational goals is essential in order to make best use of development and sustainment resources.

As part of the requirements traceability process, a requirement should flow two ways: (1) toward larger and more expansive organizational goals, and (2) toward the solution designed to enable the desired capability.

Requirements Verification: Because the ability to test and verify is a key element of project/program success, requirements tied to operational needs should be generated from the outset and maintained throughout the requirements life cycle with test and verification in mind. Advice on the ability to test requirements can be extremely effective in helping a project or program. Techniques such as prototyping and experimentation can help assess requirements early and provide a valuable tool for subsequent verification and validation. Refer to the SE Guide article Competitive Prototyping for related information.

Test and verification plan development and execution should be tied directly back to the original requirements. This is how the effectiveness of the desired capability will be evaluated before a fielding decision. Continual interaction with the stakeholder community can help realize success. All test and verification efforts should relate directly to enabling the fielding of the required capability. Developing test plans that do not facilitate verification of required performance are an unnecessary drain on resources and should be avoided.

Before a system design phase is initiated, it should be ensured that the system requirements, captured in repositories or a system requirements document, can be mapped to functional requirements (e.g., in a functional requirements document [FRD]). The requirements in the FRD should be traceable to operational requirements (e.g., in an operational requirements document or a capabilities development document). If all this traceability is ensured, there is a better likelihood that the design of the system will meet the mission needs articulated in a concept of operations and/or the mission needs statement of the program.

Design Assessment Considerations

The design of a system should clearly point to system capabilities that meet each system requirement. This two-way traceability between design and system requirements will enable higher probability of a successful test outcome of each system requirement, the system as a whole, and delivery of a useful capability.

As the service-oriented architecture (SOA) approach matures, there is increased emphasis on linking system requirements to specific services. Therefore, the artifacts or components of system design should be packaged in a manner that supports provisioning of services offered within SOA (assuming that deployment of SOA is a goal of an enterprise).

For example, assume that the system requirements can be grouped in three categories: Ingest, Analysis, and Reporting. To meet system requirements within these categories, the design of the system needs to point to each component of the system in a manner that addresses how it would fulfill the system requirements for Ingest, Analysis, and Reporting, respectively. The design information should include schematics or other appropriate artifacts that show input, processing, and outputs of system components that collectively meet system requirements. Absent a clear road map that shows how input, processing, and outputs of a system component meet a given system requirement, there is risk in whether that specific system requirement will be met.

The Requirements Traceability Matrix: Where the Rubber Meets the Road

Typically the project team develops a requirements traceability matrix (RTM) that shows linkage between functional requirements, system requirements, and system capabilities of system design components. An RTM that can clearly point to system components that are designed to meet system requirements is more likely to result in a well-designed system, all other considerations being equal. Additional linkages can be included in an RTM to show mechanisms to test functionality of system components for testing the design of the system to meet system requirements. An RTM as described above (i.e., one that ranges from statement of a requirement to methodology to test the system component that satisfies the system requirement) will go a long way in successfully assessing a system design to meet system requirements.

A traceability matrix is developed by linking requirements with the design components that satisfy them. As a result, tests are associated with the requirements on which they are based and the system is tested to meet the requirement. These relationships are shown in the figure below:

Relationships are shown
Relationships are shown

A sustained interaction is needed among members of a requirements team and those of design and development teams across all phases of system design and its ultimate development and testing. This kind of dialogue will help ensure that a system is being designed properly with an objective to meet system requirements. In this way, an RTM provides a useful mechanism to facilitate the much needed interaction among project team members.

Shown below is a sample RTM that spans "Requirement Reference" to "Design Reference." The matrix can be extended to include testing mechanisms for further assurance that the system design will meet system requirements. The RTM, as shown below, links a system requirement to a design component (e.g., a name of a module).

Project Name:

 

Author:

 

Date of Review:

 

Reviewed By:

 

 

Req. ID

Requirement Reference

Requirement Description

Design Reference

System
Feature
Module Name

APP 1.1

APP SRS Ver 2.1

Better GUI

APP Ver 1.2

Module A

APP 1.2

APP SRS Ver 2.1

Send Alert messages

APP Ver 1.2

Module B

APP 1.3

APP SRS Ver 2.1

Query handling

APP Ver 1.2

Module C

APP 1.4

APP SRS Ver 2.1

Geospatial Analysis

APP Ver 1.2

Module D

This RTM, shown below, links a test case designed to test a system requirement.

Unit Test Case #

System Test Case #

Acceptance
Test Case #

Requirement Type

APP_GUI.xls

TC_APP_GUI.xls

UAT_APP_GUI.xls

New

APP_MSG.xls

TC_APP_MSG.xls

UAT_APP_MSG.xls

Change Request

APP_QRY.xls

TC_APP_QRY.xls

UAT_APP_QRY.xls

New

APP_GA.xls

TC_APP_GA.xls

UAT_APP_GA.xls

Change Request

The assessment of a system design should consider how well the design team presents the linkage of its design to system requirements (i.e., through documentation, presentations, and/or critical design reviews). A traceability matrix can be an important device in communicating a design's ability to meet system requirements.

As part of the system design approach, the design team may develop mock-ups and/or prototypes for periodic presentation to the end users of the system and at design reviews. This approach provides an opportunity for system designers to confirm that the design will meet system requirements. Therefore, in assessing a system design, the design team's approach needs to be examined to see how the team is seeking confirmation of its design in meeting system requirements.

Best Practices and Lessons Learned

Traceability and Verification

Development of Project/Program Scope: The overall goals or desired impact for a project/program must be understood and delineated from the beginning of the effort. The solutions and technologies required can and should evolve during the systems engineering process, but the desired capability end state should be well understood at the beginning. "What problem are we trying to solve?" must be answered first.

Quality of Written Requirements: Poorly written requirements make traceability difficult because the real meaning is often lost. Ambiguous terminology (e.g., "may," "will") is one way requirements can be difficult to scope, decompose, and test. Assist with written requirements by ensuring a common and clear understanding of terminology.

Excessive Reliance on Derived Requirements: As work moves away from a focus on original requirements, there is a danger of getting off-course for performance. Over-reliance on derived requirements can lead to a loss of context and a dilution of the true nature of the need. This is an area where traceability and the bidirectional flow of requirements are critical.

Unique Challenges of Performance-Based Acquisition: Performance-based projects present a unique set of issues. The nature of performance-based activity creates ambiguity in the requirements by design. This can be extremely difficult to overcome in arriving at a final, user-satisfactory solution. As a matter of practice, much greater scrutiny should be used in this environment than in traditional project/program development.

Requirements Baseline: A requirements baseline is essential to traceability. There must be a trail from the original requirements set to the final implemented and deployed capability. All of the changes and adjustments that have been approved must be incorporated in order to provide a seamless understanding of the end state of the effort. It should also include requirements that were not able to be met. In order to adequately judge performance, the requirements must be properly adjusted and documented.

Project/Program Risk Impacts: The influence of requirements on project/program risk must be evaluated carefully. If sufficient risk is generated by a requirement, then an effective mitigation strategy must be developed and implemented. Eliminating a requirement can be an outcome of this analysis, but it must be weighed carefully. This is an area where an FFRDC trusted agent status is especially critical. Chasing an attractive yet unattainable requirement is a common element in project/program delays, cost overruns, and failures. Refer to the SE Guide topic Risk Management for detailed information.

Watch for Requirements That Are Difficult to Test: If requirements are difficult or impossible to test, the requirements can't be traced to results if the results can't be measured. System-of-systems engineering efforts can greatly exacerbate this problem, creating an almost insurmountable verification challenge. The language and context of requirements must be weighed carefully and judged as to testability; this is especially true in a system-of-systems context. See the Guide article on Test and Evaluation of Systems of Systems.

Requirements Creep: Requirements creep——both up and down the spectrum——is an enduring conundrum. As requirements flow through the systems engineering process, they can be diluted to make achieving goals easier, or they can be "gold plated" (by any stakeholder) to provide more than is scoped in the effort. Increasing capability beyond the defined requirements set may seem like a good thing; however, it can lead to difficulty in justifying program elements, performance, and cost. Adding out-of-scope capability can drastically change the sustainment resources needed to maintain the system through its life cycle. Requirements creep is insidious and extremely detrimental. On the other hand, the evolution of needs and requirements must be accommodated so that flexibility in creating capabilities can match changing operations and missions and provide timely solutions.

Interaction with End Users: Interaction with end users is critical to the requirements traceability and verification cycle. The ability to get feedback from people that will actively use the project or program deliverables can provide early insight into potential performance issues.

Bidirectional Requirements Traceability: There must be a two-way trace of requirements from the requirements themselves to both larger organizational goals and to applicable capability solutions.

Verification of Test Plans: Pay careful attention to the development of the requirements verification test plans. An overly ambitious test plan can portray a system that completely meets its requirements as lackluster and perhaps even unsafe. On the other hand, a "quick and dirty" test plan can miss potentially catastrophic flaws in a system or capability that could later lead to personnel injury or mission failure.

Design Assessment

Importance of Documentation and Team Commitment: A thorough review of documentation and an evaluation of the design team's commitment to engage with the stakeholders in the design process are key to conducting a meaningful assessment of whether the system design meets the system requirements.

  • Review system development team's strategy/approach to assess team's commitment in meeting system requirements.
    • Interview design team lead and key personnel
    • Review system documentation
  • Focus assessment review on:
    • Existence of an RTM and its accuracy and currency (this does not have to be exhaustive, but a systematic audit of key system functionality will suffice)
    • Participate in critical design reviews
    • Design team's approach toward outreach to system concept designers and user community (stakeholders)
    • Design team's procedures to capture stakeholder comments
    • Design team's methodology to vet system requirements and process change request

Importance of Documented and Validated Findings: Document your assessment and validate your findings.

  • Re-validate the audit trail of how you arrived at each finding and make any corrections (if needed)
  • If possible, consult with design team representative and share key findings
  • Document your re-validated findings and make recommendations.

References & Resources

  1. The MITRE Institute, September 1, 2007, "MITRE Systems Engineering (SE) Competency Model, Version 1," section 2.4.
  2. Agouridas, V., H Winand, A. McKay, and A. de Pennington, June 2005, Early Alignment of Design Requirements with Stakeholder Needs.
  3. Chairman of the Joint Chiefs of Staff, March 1, 2009, Joint Capabilities Integration and Development System (CJCSI 3170.01G).
  4. Federal Aviation Administration, May 14, 2007, Systems Engineering Manual.
  5. NASA Systems Engineering Handbook, viewed March 12, 2010.
  6. Ramesh, Bala, May 1999, Towards Reference Models for Requirements Traceability.

Additional References & Resources

Arkley, Paul, Steve Riddle, and Tom Brookes, 2006, Tailoring Traceability Information to Business Needs.

Chairman of the Joint Chiefs of Staff, March 1, 2009, Joint Capabilities Integration and Development System (CJCSI 3170.01G).

Defense Acquisition Guidebook, viewed March 12, 2010.

Federal Aviation Administration, May 14, 2007, Systems Engineering Manual.

International Council on Systems Engineering (INCOSE), January 2010, INCOSE Systems Engineering Handbook Version 3.2, INCOSE-TP-2003-002-03.2.

NASA Systems Engineering Guide, viewed March 12, 2010.

Ramesh, Bala, Curtis Stubbs, Timothy Powers, and Michael Edwards, April 1995, "Lessons Learned from Implementing Requirements Traceability," CrossTalk.

Ramesh, Bala, May 1999, Towards Reference Models for Requirements Traceability.

Tran, Eushiuan, Spring 1999, Requirements and Specifications.

Case Studies

Government Accountability Office, March 2006, Business Systems Modernization, IRS Needs to Complete Recent Efforts to Develop Policies and Procedures to Guide Requirements Development and Management, GAO-06-310.

Minardo, Katie, September 24, 2007, Tier 1 Case Study of TBMCS/TBONE (1999 - 2006), Social Contexts of Enterprise Systems Engineering MITRE-Sponsored Research Project MITRE Technical Report 070074.

Nordmann, John C., January 2007, Overview of the Single Integrated Air Picture (SIAP) Enterprise and Lessons Learned to Date (2000 - 2007), An Enterprise Systems Engineering (ESE) Perspective.

Toolkits

FAA Acquisition System Toolset, http://fast.faa.gov, viewed March 12, 2010.

Nordmann, John C., January 2007, The SEPO Requirements Process Toolkit, viewed March 12, 2010.

Publications

Download the SEG

MITRE's Systems Engineering Guide

Download for EPUB
Download for Amazon Kindle
Download a PDF

Questions?
Contact the SEG Team