Competitive Prototyping


Definition: Competitive prototyping is an approach in which two or more competing teams (organizations) develop prototypes during the early stages of a project (acquisition or procurement phase). The competing prototypes are compared, and ultimately the one that best addresses the issue(s), problem(s), or challenge(s) is chosen.

Keywords: competitive prototype program, competitive prototype tests, competitive prototyping strategy, competitive teams, prototyping

MITRE SE Roles & Expectations: MITRE systems engineers (SEs) are expected to understand the purpose and role of competitive prototyping (CP) in the acquisition process, where it occurs in systems development, and the benefits and risks of employing it. MITRE SEs are also expected to understand and recommend when CP is appropriate to a situation. They are expected to develop and recommend technical requirements for CP efforts as well as strategies and processes that encourage and facilitate the active participation of end users and other stakeholders in the CP process. They are expected to monitor and evaluate contractor CP technical efforts and the acquisition program's overall CP processes and to recommend changes when warranted.

Background

Prototyping is a practice in which an early sample or model of a system, capability, or process is built to answer specific questions about, give insight into, or reduce uncertainty or risk in many diverse areas. This includes exploring alternative concepts, technology maturity assessments, requirements discovery or refinement, design alternative assessments, and performance or suitability issues. It is part of the SE's toolkit of techniques for managing uncertainty and complexity and mitigating their effects.

The exact form and focus of prototyping is driven by where it is used in the acquisition management system life cycle and the nature of the problem the prototype is intended to address. Prototyping may be used immediately after a decision to pursue a materiel solution to meet an operational need (see Figure 1). In this situation, prototypes are used to examine alternative concepts as part of the analysis of alternatives leading to a preferred solution concept. Prototyping to explore and evaluate the feasibility of high-level conceptual designs may be performed early in technology development as part of government activities to assess and increase technology maturity, discover or refine requirements, or develop a preliminary design. A prototype may even be developed into a reference implementation and provided to a commercial contractor for production [1].

Competitive prototyping may serve any of the just-cited purposes, but it typically involves two or more competing teams, usually from commercial contractors, and is often used as a factor in source selection evaluations leading to a formal acquisition program start and contract award. This application of competitive prototyping is depicted in Figure 1.

Historically, much of competitive prototyping focused on building tangible prototypes such as weapons, aircraft, and automobiles. The earliest documented modern use of competitive prototyping dates back to just after the War of 1812, when the United States Army became interested in a breech-loading rifle [2]. Numerous contractors submitted actual hardware examples for the Army's evaluation. After a hardware design was accepted, the contractor was given an order for a limited production of these rifles.

The U.S. aircraft industry used CP extensively throughout the 20th century. Some of the early prototypes were developed by Chanute, the Wright brothers, Curtiss, and Sikorsky and, more recently, by General Dynamics, Boeing, and McDonnell Douglas. Similarly, the U.S. auto industry uses competitive teams to design concept cars of the future.

The employment of CP in software-intensive system developments is a relatively recent phenomenon.

Government Interest and Use

In several recent acquisition reform initiatives, the U.S. government encouraged or required competitive prototyping as a tool to assess technology maturity and reduce program risk [3, 4]. Although mentioned less frequently as a primary reason, competitive prototyping can also illuminate undiscovered or uncertain requirements before engineering and manufacturing development (EMD).

The Office of Management and Budget has identified competitive prototyping as a risk mitigation tool to be used in procurement and cites five advantages for its use [5]:

  • Proves concepts are sound.
  • Allows efficient and effective communication (among operational users, procurement agency, and commercial contractors) to identify the best fit between agency (operational user) needs and marketplace capabilities.
  • Provides for competition during the development effort.
  • Where appropriate, ensures development remains constrained.
  • Facilitates firm fixed-price contracting for production.

The strongest and most recent government support for competitive prototyping comes from the Department of Defense (DoD), which requires all programs to formulate acquisition strategies and funding that provide for two or more competing teams to produce prototypes through Milestone B [6], as shown in Figure 1. High-level DoD acquisition officials from previous administrations have also endorsed this required use of competitive prototyping in government acquisitions [7].

Figure 1. The Defense Acquisition Management System. (Click image to enlarge)

Reasons noted in the DoD decision to require competitive prototyping are to enable government and industry teams to discover and solve technical issues before EMD so that EMD can focus on producing detailed manufacturing designs, not on solving myriad technical issues. Other anticipated advantages include reduced technical risk, validated design, validated cost estimates, insight into contractor manufacturing processes, refined requirements, and reduced time to fielding. But note that not all of these advantages can be expected to automatically accrue from all instances of competitive prototyping. For example, it is unlikely that substantial insight into contractor manufacturing processes to be used later for full-scale production will result from a prototyping effort unless considerable funding is devoted to tooling up for the prototype activity.

Best Practices and Lessons Learned [7, 8, 9]

  • When size (and skill) matters: Acquisition program offices that employ CPs successfully tend to require a larger contingent of government systems engineers with greater than average technical competence. Although this may appear counterintuitive, remember that CPs offer advantages to programs that use them, but the government team must skillfully plan, monitor, and manage them.
  • Right-sizing CP requirements: CP is an investment that buys information to reduce uncertainty and risk. But CP adds up-front costs to a program right at a time when funding may be scarce and support for the program is often weak. A CP may run into opposition from the least expected stakeholders—staunch advocates of a program who believe that it must be pushed at great speed to fill capability gaps. To navigate these external forces on CP efforts, the program CP requirements must be right-sized. They must focus on areas that have substantial risk or offer a high reward-risk ratio, whatever and wherever those areas may be—high-level capabilities/levels-of-service, low-level detailed requirements at the subsystem level, or issues in between. It is also important to make sure that likely performance bottlenecks are identified in the prototype process that are measurable and measured as part of prototype testing.
  • Make sure your CP learns from antecedent activities: One focus of recent government acquisition reform initiatives is on the importance of early systems engineering. Some departments and agencies are strongly recommending or mandating prototyping in advance of technology development, during materiel solution analysis (see Figure 1). Results or lessons learned from these very early prototypes should be used to shape and inform CP activities.
  • Have your CP do double duty: The primary purpose of CP is to illuminate and eliminate technology maturity risks. But don't lose sight of the fact that a CP can give important insight into other risk areas such as contractor manufacturing processes (if the CP is resourced appropriately) and undiscovered operational user requirements. Look for and collect information on all issues and areas that a CP can illuminate, especially important downstream systems engineering activities and assessments for which CP information can form the basis of refined government assessments of system design or estimates of program cost.
  • Ensure persistent, active engagement of all stakeholders: CP is not a competition between two gladiators in an arena slugging it out until one gives in, at which time everyone else in the coliseum looks up and applauds the winner. CP efforts must be structured to encourage active participation of end users and other stakeholders throughout the CP life cycle. To facilitate that involvement, CP efforts should emphasize frequent demonstrations of progress and evidence that a prototype can scale. Ideally CPs should be developed iteratively or in an evolutionary fashion and be able to demonstrate interim operational capabilities. Active operational user stakeholder engagement is particularly critical to CPs intended to address requirements discovery and refinement.
  • Remember those without "skin in the game": Important stakeholders in the eventual outcome of a program, like certification and accreditation authorities, are frequently forgotten during CP. Identify and bring these stakeholders into CP planning early so they can advise on "non-starters" and be engaged through the entire process.
  • Commercial competitors are stakeholders too: Commercial industry views CPs as investments. To attract the best commercial competitors for your program, CP goals must be clearly defined and any basis for industry investment (e.g., internal research and development) must be convincing. In particular, the production potential of the contract must be visible and attractive to would-be competitors.
  • Don't stop competition too quickly: Make sure there is sufficient information to make informed decisions before terminating a competition. This is a form of the wisdom, "measure twice, cut once." The Joint Strike Fighter program began with a competition involving prototypes built by Boeing and Lockheed Martin. During the CP phase, it appeared that both prototypes had been extensively flown before the government chose the Lockheed Martin variant, but rising costs and schedule slips of the F-35 now suggest that competition may have been closed too quickly.
  • Beware the Potemkin Village: Ensure each competitor is presenting an actual prototype and not simply performing a demonstration. A demonstration can be scripted to perform a few things well when conducted by a knowledgeable person. On the other hand, a prototype should be able to be operated without a script and by a domain expert with very little experience with the prototype.
  • Keep your eyes on the prize: Acquisitions using CPs can overemphasize the operator functional features of the prototypes at the expense of other critical, less obvious requirements. If competitors hit the mark on "operator functionality" or "user interface look and feel" to more or less the same degree, there may be a risk of eliminating the better contractor for production. Carefully evaluate the potential risks of a prototype's becoming the actual product. Prototypes often do not have robust architectures, standard designs, a full set of requirements, or complete documentation. These weaknesses may become deployment risks, such as lack of maintainability, scalability, or reproducibility. Also, consider how big a challenge it would be for the functionality or look and feel of the other contractor's prototype to be modified and approved, as well as the contractor's ability and willingness to do so.
  • Retain competitors' core skills: CPs involve down-selects, but the intellectual capital and experience built up by "losers" need not and should not be lost to the program or the government. During the evaluation interval between each phase of a CP down-select, the government should continue to fund all competitors at a level of effort sufficient to retain the core engineering skills of the competing teams. Not doing so risks losing key members of competitors' teams and can weaken the overall government acquisition effort. One reference [8] suggests that investments in down-selected bidders can be capitalized on by structuring the ensuing acquisition so that unsuccessful bidders are offered "consolation prizes," such as independent verification and validation contracts. If this tack works at all, it is more likely to be attractive to small contractors or others attempting to break into the business area represented by the system being acquired.

References & Resources

  1. Rebovich, G., and J. K. DeRosa, November 2011, Patterns of Success in Systems Engineering: Acquisition of IT-Intensive Government Systems, pp. 53–57, Bedford, Mass., The MITRE Corporation.
  2. Patnode, C.A., Jr., LTC, February 1973, Problems of Managing Competitive Prototype Programs Study Report. Defense Systems Management School (Program Management Course Student Study Program), PMC 73-2, p. 3.
  3. Department of Defense, November 25, 2013, Interim DoD Instruction 5000.02, "Operation of the Defense Acquisition System," pp. 16­­­–18, Washington, DC.
  4. Department of Defense, December 16, 2013, Defense Acquisition Guidebook, Ch. 2.8.7.1, "Competition Strategy” and Ch. 4.2.4, "Technology Development Phase," Washington, DC.
  5. Office of Management and Budget, June 2006, Capital Programming Guide, V 2.0, Supplement to OMB Circular A-11, Part 7, Section II.3.3, "Competitive Prototyping."
  6. Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, September 19, 2007, Memorandum on Prototyping and Competition, Washington, DC, Pentagon.
  7. DuPont, D. G., February 2008, "Proactive Prototypes," Scientific American, Vol. 298, No. 2, pp. 24–26.
  8. Boehm, B., and D. Ingold, July 2008, Initial Competitive Prototyping Survey Results, University of Southern California Center for Systems and Software Engineering. Presented at July 2008 OUSD/AT&L/SSE—USC/CSSE Workshop on Integrating Systems and Software Engineering with the Incremental Commitment Model.
  9. Ireland, B., July 2008, Competitive Prototyping: Industry Roundtable, Presented at July 2008 OUSD/AT&L/SSE—USC/CSSE Workshop on Integrating Systems and Software Engineering with the Incremental Commitment Model.

Additional References & Resources

Grgurich, J. M., March 2013, Conducting a Competitive Prototype Acquisition Program: An Account of the Joint Light Tactical Vehicle (JLTV) Technology Development Phase, Naval Postgraduate School, Monterey, Calif.

Copeland, E. J., October 2013, Effects of System Prototype Demonstrations on DoD Weapon Systems Development, The George Washington University, Washington, DC.

Publications

Download the SEG

MITRE's Systems Engineering Guide

Download for EPUB
Download for Amazon Kindle
Download a PDF

Questions?
Contact the SEG Team