Performing Analyses of Alternatives
Definition: An analysis of alternatives (AoA) is an analytical comparison of the operational effectiveness, cost, and risks of proposed materiel solutions to gaps and shortfalls in operational capability. AoAs document the rationale for identifying and recommending a preferred solution or solutions to the identified shortfall(s) .
Keywords: analysis of alternatives, AoA, baseline alternative, cost analysis, criteria, evaluation, materiel solutions, risk analysis
MITRE SE Roles & Expectations: MITRE systems engineers (SEs) are expected to understand the purpose and role of AoA and where it occurs in the acquisition process. MITRE SEs are also expected to understand and recommend when an AoA is appropriate to a situation. They are expected to develop, recommend, lead and conduct technical portions of an AoA, including strategies and best practices for execution. SEs are expected to monitor and evaluate AoA technical progress and recommend changes when warranted.
An AoA is one of the key documents produced in preparation for major milestones or programs reviews. It is especially key at the start of a new program. Very often, MITRE's systems engineering support to programs that are in pre-Milestone A or B phases involves support to AoA efforts. Recommendations from the AoA determine the procurement approach for either a new program or the continuance of an existing program. MITRE systems engineers involved in program planning are frequently called on to participate in these analyses, and lead the technical efforts associated with assessing other existing programs for applicability to the mission, cost-effectiveness, and risk. MITRE systems engineers provide necessary systems engineering skills (requirements analysis, technical evaluation, architecture, etc.), freedom from bias, and access to subject matter expertise required for AoAs.
Why do we do AoAs?
AoAs are performed to allow decision makers to understand choices and options for starting a new program or continuing an existing program. The bottom line is cost-effectiveness through non-duplication of effort and lowest risk to successful program delivery. An example is a MITRE customer who determined there were gaps in their current suite of systems to meet anticipated needs. One of the systems was limited in capability and nearing end-of-life with no follow-on program of record to continue or improve it. The customer needed an analysis of existing systems and technologies to make informed decisions on the best path forward to provide an integrated solution of systems to meet their emerging needs. The Departments of Defense (DoD), Homeland Security, and Treasury require AoAs for new procurements. The USAF (Air Force Materiel Command) has one of the more robust documented processes for performing an AoA . Per DoDI 5000.02, the purpose of the AoA is to assess the potential materiel solutions to satisfy the capability need documented in an initial capabilities document . Approval to enter into the development phase of a program is contingent on completion of an AoA, identification of materiel solution options by the lead DoD component that satisfy the capability need, and satisfaction of phase-specific entrance criteria for the development milestone.
Commercial industry also uses "alternative analyses," but they are usually more focused on life cycle cost. An equally interesting and valuable feature of an AoA is an assessment of risk—including operational, technical, and programmatic risks. This kind of assessment is not widely seen in AoA processes, and is not always employed . As has been seen repeatedly with other systems engineering disciplines, a good process is a good start, but does not guarantee success unless it is actually followed.
Best Practices and Lessons Learned
The plan is important. A major step leading to a successful AoA is the creation of a well-considered study plan. The study plan establishes a roadmap for how the analysis should proceed, who is responsible for doing what, and why it is being done . It should include the following information:
- Understand the technology gaps and capability gaps—what needs is the intended system supposed to meet?
- Develop viable alternatives
- Define the critical questions
- List assumptions and constraints
- Define criteria for viable/non-viable
- Identify representative solutions (systems/programs)
- Develop operational scenarios to use for comparisons/evaluation
- Identify, request, and evaluate data from the representative systems/programs (determined to be viable)
- Develop models
- Work through scenarios
The AoA should "assess the critical technology elements associated with each proposed materiel solution, including technology maturity, integration risk, manufacturing feasibility, and, where necessary, technology maturation and demonstration needs ."
Sufficient resourcing is key. Allocate sufficient resources and time for completing each of the actions and assessments and for preparing the final analysis product. The biggest risk to success is the lack of time to adequately perform the AoA. Compressed schedules associated with preparing for a new procurement, and the associated execution of funding, can present major problems for the AoA team. If faced with this issue, resources may need to be increased and allocated full-time to the effort.
Know the baseline before starting the AoA. For many AoAs, an existing capability exists but it is either nearing its end of life or no longer satisfies current needs. In these cases, it is critical to first understand the existing capability baseline. The set of alternatives considered in the AoA must include an upgrade path from the "status quo." Unless information about the existing capability (referred to as the "baseline alternative") is already in hand, it is necessary to ensure that sufficient effort is planned during the AoA to capture that baseline fully enough for the comparison analysis to be performed. The point of comparison is likely to be in the future (at which time it is projected that the new capability would field), so there may be a need to project an upgrade path for the existing capability for fair comparison. Systems engineers, design engineers, and software engineers should get involved early to review the baseline, understand it, and project an upgrade path for the "baseline alternative."
Know your stakeholders. Understand the stakeholders and decision makers involved with the AoA and how they will use the results. Assess the political, operational, economic, and technical motivations of the stakeholders to inform the scope of the analysis and its focus. Use community and user stakeholders to assist in determining the objectives, and then the measures and metrics that will be used to identify the "best" solution. Not only does this leverage their knowledge, it provides a means to obtaining their buy-in on the result.
Beware premature convergence. A recent GAO (Government Accountability Office) report on defense acquisitions attributes premature focus on a particular solution or range of solutions as a failing of AoAs . If stakeholders are already enamored of a particular solution, completing a full AoA may be difficult. The intention is to survey a broad range of alternatives to ensure the best value and technical match to the need. A narrow scope or attention paid to a particular solution renders the AoA ineffective for decision making and leads to increased risk in the resulting program.
Know your AoA team. Establish standards for the minimum level of expertise and experience that AoA study participants in each integrated project team/working group must meet. Subject matter experts (SMEs) should truly be recognized experts in their area, rather than just adding a particular organizational affiliation.
Understand the mission. It takes focusing on both the mission and the technical capabilities to be able to perform an adequate assessment. AoAs should make use of simulation and modeling to help determine the best solution; but a simulation is only as good as its fidelity and the input parameters (i.e., subject to "garbage in, garbage out"). Make use of community SMEs and users to ensure a good understanding of the objective operations needed to meet the gaps. MITRE systems engineers need to possess both the operational knowledge and technical skills to adequately analyze technical solutions. The objective is to be credible, thorough, and comprehensive. A good AoA needs to address the full spectrum of mission, operating environment, technologies, and any other unique aspects of the program. It also needs to be articulated well enough to enable decision makers to make an informed decision.
Obtain technical descriptions of the materiel solutions. Frequently the selection of alternatives for the study is made by team members who are focused on the operational mission and capability gaps. Yet technical knowledge must also be applied by the AoA team or time and resources may be wasted investigating alternatives that are not technically feasible. Early application of technical criteria will avoid this. Ask for technical descriptions (e.g., technical description documents) as well as operational descriptions of the alternatives before starting the AoA. Doing an early DOTMLPF (Doctrine, Organizations, Training, Materiel, Leadership and Education, Personnel, and Facilities) analysis makes it possible for the AoA to focus its efforts primarily on dealing with feasible material solutions.
Anticipate problems. Analyzing a broad range of solutions involves collecting a considerable amount of information on the representative systems/programs as well as on other details like operating environment or communications infrastructure. Industry requests for information can be useful, but do not always produce the detail needed. Look for other data sources through government, contractor or MITRE contacts. This issue can also be exacerbated by a compressed schedule, leading to inaccurate or incomplete analyses due to lacking detailed information. In any case, a good assumption going into an AoA is that all necessary information will not be forthcoming and will necessitate creating some workarounds (e.g., facsimile, assumptions). Be persistent!
Leverage MITRE. A wide range of technical expertise and system/program expertise are required for an AoA. Determine what skills are required for the AoA plan and leverage what you can from the broader MITRE team. Not only should MITRE expertise be used for the technical expertise in the areas of technology applicable to the solution set, but also for analysis techniques, and modeling and simulation to aid in the evaluation process. MITRE has in-house expertise in engineering cost modeling and life cycle cost tradeoff analysis that can be leveraged as well. As an example, MITRE's participation in a particular AoA involved creating and using models to assist in the evaluation as well as providing subject matter expertise in the technical areas of radio frequency, electro-optical, infrared, high-power microwave, and electronic warfare technologies. This led to a very successful AoA. Lastly, nothing can replace a good face-to-face discussion among novice and experienced engineers on an AoA.
Beware the compressed schedule. As mentioned above, an inadequate timeframe to conduct an AoA can render its conclusions ineffective. The GAO found that many AoAs have been conducted under compressed time frames—six months or less—or concurrently with other key activities that are required for program initiation in order to meet a planned milestone decision or system fielding date. Consequently, AoAs may not have enough time to assess a broad range of alternatives and their risks, or may be completed too late in the process to inform effective trade discussions prior to beginning development .
Incorporate the risk analysis. Risks are important to assess because there may be technical, programmatic, or operational uncertainties associated with different alternatives that should be considered in determining the best approach .
The GAO reported that some of the AoAs they reviewed did not examine risks at all; they focused only on the operational effectiveness and costs of alternatives. Other AoAs had relatively limited risk assessments. For example, several AoAs did not include integration risks even though the potential solution set involved modified commercial systems that would require integration of subsystems or equipment. Based on a recent Defense Science Board report on buying commercially based defense systems, programs that do not assess the systems engineering and programmatic risks of alternatives do not understand the true costs associated with militarizing commercial platforms or integrating various commercial components . Other AoAs did not examine the schedule risks of the various alternatives, despite accelerated schedules and fielding dates for the programs. They also found that programs with AoAs that conducted a more comprehensive assessment of risks tended to have better cost and schedule outcomes than those that did not . For more information on risk identification and management, see the Risk Management topic and articles within this section of the Guide.
References & Resources
- Analysis of Alternatives Handbook: A Practical Guide to Analyses of Alternatives, July 2008, Air Force Materiel Command Office of Aerospace Studies.
- Air Force Materiel Command Office of Aerospace Studies, July 2008. Analysis of Alternatives Handbook: A Practical Guide to Analyses of Alternatives.
- Department of Defense Instruction 5000.02, December 8, 2008 (revised), Operation of the Defense Acquisition System.
- Many Analyses of Alternatives Have Not Provided a Robust Assessment of Weapon System Options, September 2009, GAO Report.