Assess Integration Testing Approaches
Definition: When the components of a system are developed in isolation, all the pieces must be brought together to ensure that the integrated system functions as intended in its operational configuration. Integration testing should exercise key interfaces between system components to ensure that they have been designed and implemented correctly. The total operational architecture, including all the segments of the system that are already fielded, should be included in an end-to-end test to verify system integration success.
Keywords: end-to-end testing, integration testing, operational architecture, SoS, system-of-systems testing, systems of systems
MITRE SE Roles and Expectations: MITRE systems engineers (SEs) are expected to understand the purpose and role of integration—or system-of-systems (SoS)—testing in the acquisition process, where it occurs in systems development, and the benefits and risks of employing it. MITRE SEs are also expected to understand and recommend when integration testing, or SoS testing, is appropriate within a program development. They should be able to take a broader look at the system acquisition within the context of its intended operational environment, beyond simply the core piece of equipment or software that is being developed, to the overarching operational architecture. MITRE SEs should develop and recommend integration testing strategies and processes that encourage and facilitate active participation of end users and other stakeholders in the end-to-end testing process. They are expected to monitor and evaluate contractor integration testing and the acquisition program's overall testing processes and to recommend changes when warranted.
From a software development perspective, system integration testing (SIT) is defined as the activities involved with verifying the proper execution of software components and the proper interfacing between components within the solution. The objective of SIT is to validate that all software module dependencies are functionally correct and that data integrity is maintained between separate modules for the entire solution. Whereas functional testing focuses on testing all business rules and transformations and ensuring that each "black box" functions as it should, SIT principally focuses on testing all automated aspects of the solution and integration touch points .
Modern systems provide great value through multifunctionality. However, for SEs, the multifunctionality brings the challenge of increased complexity. Humans deal with complexity by partitioning the challenge into smaller pieces—sometimes called components or modules, although at times these are full systems in and of themselves. The downside of partitioning the problem into manageable pieces is that the pieces have to be put together (integration) and shown to work together. This integration is best achieved through a disciplined systems engineering approach containing good architecture, interface definitions, and configuration management.
In most cases, systems being acquired through the government's acquisition process are not complete, stand-alone entities. The newly acquired system will almost always need to fit into a larger operational architecture of existing systems and/or operate with systems that are being separately acquired. To be completely effective and suitable for operational use, the newly acquired system must interface correctly with the other systems that are a part of the final operational architecture. Integration testing, or SoS testing, verifies that the building blocks of a system will effectively interact and that the system as a whole will effectively and suitably accomplish its mission. This article expands the strict software-focused definition of system integration testing to a broader look at complete systems and the integration, or SoS, testing that should be conducted to verify that the system has been "assembled" correctly.
The conundrum that a MITRE SE, or any independent party charged with assessing a system's integration test strategy, will encounter in attempting to recommend or develop integration test strategies is the lack of requirements written at an SoS or operational architecture level. By way of example, although the Department of Defense (DoD) Joint Capabilities Integration and Development System (JCIDS) was developed "to support capability analysis, strategy development, investment decisions, capability requirement portfolio management, and capabilities-based force development and operational planning"  and to address shortfalls in the DoD requirements generation system—including "not considering new programs in the context of other programs" —operational requirements documents continue to be developed without an SoS focus. A typical capabilities development document will provide requirements for a system, including key performance parameters, but it will not provide requirements at the overarching architecture level. As a result, developing a recommendation for integration testing requires some creativity and a great deal of pulling information from diverse sources. Once the test is developed, the task of advocating and justifying the test's need within the system development process will be the challenge at hand.
The following discussion provides examples of systems of systems, the recommended integration testing that should be conducted, and of both good and bad integration testing. Note that best practices and lessons learned are generally interspersed throughout the article. A few cautionary remarks are also listed at the end.
Systems of Systems: Definition and Examples
Though the individual systems constituting an SoS can be very different and operate independently, their interactions typically deliver important operational properties. In addition, the dependencies among the various systems are typically critically important to effectively accomplishing the mission. The interactions and dependencies must be recognized, analyzed, and understood. Then the SoS test strategy can be developed to ensure that the individual systems have been successfully integrated to deliver a fully effective and suitable operational capability.
The examples used in this article are drawn from a particular domain, but most MITRE SEs should see a great deal of similarity in their essentials, regardless of the sponsor they support.
The Global Positioning System (GPS) is an example of an SoS. Typical GPS users—ranging from a hiker or driver using a GPS receiver to navigate through the woods or local streets, to a USAF pilot using GPS to guide a munition to its target—don't usually consider all the components within the SoS required to guide them with GPS navigation. The constellation of GPS satellites is only a small piece, albeit an important one, within the SoS required to deliver position, navigation, and timing information to the GPS user. Other essential pieces include the ground command and control network that maintains the satellite's proper orbit, the mission processing function that processes the raw collected data into usable information for the end user, the external communication networks that disseminate the information to the end user, and the equipment with which the end user interfaces with the system and uses its information. The dependencies and interfaces among all these elements are just as critical to accomplishing the user's goal as is the proper functioning of the constellation of GPS satellites.
A second SoS example is an interoperable and information assurance (IA) protected cross-boundary information sharing environment where federal government users from different departments and agencies, commercial contractors, allies, and coalition members can share information on a global network. Multiple separate but interrelated products make up the first increment suite of information technology services, including Enterprise Collaboration, Content Discovery and Delivery, User Access (Portal), and a Service-Oriented Architecture Foundation that includes Enterprise Service Management.
Finally, an example of a more loosely coupled SoS, i.e., a surveillance SoS for which a single government program office is not responsible for acquiring and sustaining the entire SoS. The surveillance network comprises multiple sensors that contribute information in the form of observations to a central processing center (CPC) that uses the sensor-provided observations to maintain a database containing the location of all objects being monitored. One organization updates and maintains the CPC, but each type of surveillance network contributing sensor has its own heritage and acquisition/sustainment tail. A new sensor type for the surveillance network is currently being acquired. Although it will be critically important for this new sensor type to integrate seamlessly into, and provide data integrity within, the overall surveillance network, the road to SoS integration testing is fraught with difficulty primarily because there are no overarching requirements at the surveillance network level to ensure adequate integration of the new sensor.
Although they are challenging to plan and execute, SoS tests for programs where a single government program office is responsible for the entire SoS are generally accomplished better as part of the system acquisition process. If nothing else, the development contractor typically plans and executes a final system integration test prior to turning the system over for operational testing. Then the operational test community plans, executes, and reports on an operationally realistic end-to-end system test as a part of the system's congressionally mandated Title 10 Operational Test and Evaluation.
A good example of an integration/SoS test is that being done to inform some GPS upgrades. As the new capability is fielded within the GPS constellation, the development community will combine their Integrated System Test (IST) with the operational test community's Operational Test and Evaluation into an integrated test that will demonstrate the end-to-end capability of the system. This SoS/end-to-end test will include the full operational process, from user request for information, through command generation and upload to the constellation, to user receipt of the information through the user's GPS receiver. During the final phase of the IST, several operational vignettes will be conducted to collect data on the end-to-end system performance across a gamut of operational scenarios.
Best Practices and Lessons Learned
Include scenarios. Integration testing should include scenarios that demonstrate the capability to perform mission essential tasks across the SoS segments.
A single program office doesn't ensure integration testing. Don't assume integration testing will necessarily happen or be adequate just because the full SoS is under the control of a single program office. There are examples of such single program office SoS acquisitions comprising a number of different products and segments that only tested each product separately.
The potential for catastrophic failure is real. Failure to conduct adequate SoS integration testing can lead to potentially catastrophic failures. If the new sensor type in the surveillance network example provides the quality and quantity of data anticipated, there is the real possibility that it will overwhelm the CPC's processing capability, thus degrading the accuracy and timeliness of the surveillance database.
Summary and Conclusions
A strict software-development view of integration testing defines it as a logical extension of unit testing . In integration testing's simplest form, two units that have already been tested are combined into a component and the interface between them is tested. A component, in this sense, refers to an integrated aggregate of more than one unit. In a realistic scenario, many units are combined into components, which are in turn aggregated into even larger parts of the program. The idea is to test combinations of pieces and eventually expand the process to test your modules with those of other groups. Eventually all the modules making up a process are tested together. Integration testing identifies problems that occur when units are combined. By using a test plan that requires you to test each unit and ensure the viability of each before combining units, you know that any errors discovered when combining units are likely related to the interface between them. This method reduces the number of possibilities to a far simpler level of analysis.
This article has focused on making the logical extension of this definition to a full-up system and expanding the integration testing definition to one of SoS testing. MITRE SEs charged with assessing integration testing approaches should ensure an SoS view of the program and develop and advocate for full end-to-end testing of capabilities within the complete operational architecture.
References and Resources
- MIKE 2.0, System Integration Testing, accessed February 3, 2016.
- Joint Chiefs of Staff, January 23, 2015, Joint Capabilities Integration and Development SystemWikipedia, paragraph 3.a (1).
- DISA/JITC, October 2008, Net-Centric Enterprise Services Initial Operational Test and Evaluation Plan, p. i.
Additional References and Resources
Microsoft Developer Network, Integration Testing, accessed February 3, 2016.