Affordability, Efficiency, and Effectiveness (AEE)
Definitions: Affordability, efficiency, and effectiveness (AEE) are three success measures that guide systems engineers in developing and shaping engineering solutions, making program recommendations, and evaluating engineering efforts.
Keywords: analysis of alternatives, budgets, cost benefit analysis, cost capability analysis, life-cycle cost, portfolio analysis, program cost, return on investment
MITRE SE Roles and Expectations: MITRE systems engineers (SEs) are expected to incorporate and assess measures of affordability, efficiency, and effectiveness in engineering solutions and supporting the acquisition activities of sponsors. They are expected to:
- Work with users to understand their mission needs, capability gaps, and functional and performance requirements.
- Develop alternative solutions or courses of action, and evaluate them for mission effectiveness as well as life-cycle affordability and efficiency.
- Understand operational and technical domains; recommend and conduct engineering-based tradeoffs of requirements, design, performance, cost, and schedule to address affordability constraints.
- Understand life-cycle cost, schedule, risk, and affordability implications of alternatives under consideration, and incorporate these dimensions in engineering products and recommendations.
- Encourage and facilitate active participation of the broad stakeholder community and acquisition decision makers in exploring alternatives; help them understand and use the trade space to achieve program affordability and evaluate the merits of alternatives from operational as well as business perspectives.
- Monitor and evaluate contractor system development efforts; identify affordability risks and recommend mitigation strategies or cost capability changes when warranted as an acquisition program progresses.
- Assist government sponsors in developing, adjusting, and implementing strategies, at program and enterprise levels, to ensure affordability and improve efficiency/effectiveness.
- Communicate AEE best practices and lessons learned.
The measures of affordability, efficiency, and effectiveness can be characterized as follows:
- Affordability: Ability to fund desired investment. Solutions are affordable if they can be deployed in sufficient quantity to meet mission needs within the (likely) available budget.
- Efficiency: A measure of the "bang for the buck" or "unit benefit per dollar." Solutions are efficient if they measurably increase the "bang" or "unit benefit" for the amount of resources required to deliver the capability.
- Effectiveness: "The bang"; the ability to achieve an organization's mission. Solutions are effective if they deliver capability of high value to accomplishing the user's missions .
MITRE's mission is to work in partnership with our government sponsors in applying science and advanced technology to engineer systems of critical national importance. MITRE's systems engineering is conducted in a system acquisition context by developing solutions to meet the needs and challenges of our sponsors in executing their missions, and in aiding sponsors in planning and managing programs to acquire such solutions. Sponsor success is achieved if the systems they deploy are effective and available when needed to achieve their mission. Systems to be procured, deployed, and sustained must be affordable, that is, within the means of the sponsor's available resources, and should be efficient, providing high value for the resources to be expended.
Sponsor acquisition environments present many challenges. A number of acquisition programs have failed to deliver needed capabilities, or delivered reduced capability with expenditure of time and funds well beyond what was planned. A constrained federal fiscal environment places considerable economic stress on government agencies. Budget reductions mandate difficult decisions about where to invest limited resources, how to make current programs more affordable, and whether to terminate poorly performing programs. Investments for new capabilities, replacements, or enhancements to existing systems as well as simple continuation of existing programs require careful analysis and evaluation of their affordability, efficiency, and effectiveness. Systems engineering seeks to apply current and emerging technologies flexibly to address sponsors' dynamic threats and mission needs. At the same time, affordability engineering helps sponsors respond to fiscal realities and be effective stewards of taxpayer dollars. (Affordability engineering is a systems engineering competency that treats affordability as a requirement that is continuously measured, managed, and assessed throughout an acquisition program’s life.)
As depicted in Figure 1, affordability challenges exist at different sponsor levels and are addressed by varying engineering, analysis, or management approaches.
Enterprises invest in and sustain systems, infrastructures, and organizations to accomplish multiple missions. Investment decisions made by agency heads, senior acquisition executives (SAEs), chief information officers (CIOs), and program executive officers (PEOs) require a holistic view of the enterprise's missions and objectives. Investment decisions are required for new or enhanced systems that provide additional capabilities as well as for sustainment of existing systems, infrastructures, manpower, and operations. A portfolio management approach evaluates the benefits of or return on investment choices relative to filling identified capability gaps or improving mission functions. Efficiency initiatives and business process reengineering (BPR) look to reduce redundancy, overlap, or inefficient processes. Enterprise-level analysis and decision frameworks can be applied to help an agency achieve greatest mission effectiveness, ensuring that highest priority/highest value needs are met with its allocated budget .
Each acquisition program is an element of the integrated capability that the enterprise delivers. As such, it is vital that each be executed to deliver its target capability within available technology, funding, and time, i.e., be affordable. Each program and capability must contribute high value to users in terms of mission effectiveness achieved in the most efficient way. Not doing so can, like a chain reaction, have serious impacts across the enterprise. At this level, engineering solutions for affordability and adopting best acquisition systems engineering and management practices within the Program Management Office (PMO) are the keys to achieving success.
Technical and operational innovation contributes to AEE at both the acquisition program and enterprise levels. Adaptive systems and composable capabilities provide an enterprise with the flexibility to respond to rapidly changing mission needs with existing resources and minimal new investment. Application of advances in IT and network design offer potential for great efficiency in the delivery of information and services to end users at the enterprise and program levels. Advances within technology domains afford an opportunity to reduce life-cycle costs in the acquisition of new or enhanced capabilities and to transform operations and sustainment approaches for greater efficiency.
Government Interest and Use
The U.S. economy has been experiencing an era of slow growth, an anemic jobs market, historic high debt, and mounting budget deficits at federal and lower levels. All government agencies are feeling considerable economic stress as they strive to accomplish their missions and deliver services with constant or shrinking budgets. Agencies across the federal government are implementing new strategies to promote AEE in their acquisition decisions and management practices.
In June 2010, the Office of the Secretary of Defense/Acquisition, Technology and Logistics (OSD/AT&L) launched the Better Buying Power (BBP) 1.0 initiative  . In November 2012, OSD/ATL memo "Better Buying Power 2.0: Continuing the Pursuit for Greater Efficiency and Productivity in Defense Spending"  was followed by OSD/ATL memo "Implementation Directive for Better Buying Power 2.0—Achieving Greater Efficiency and Productivity in Defense Spending," which expanded the scope of BBP 1.0 and set the significance and breadth of this updated mandate . This 26-page memo outlined 34 specific initiatives within seven key areas:
- Achieve Affordable Programs
- Control Costs Throughout the Product Life Cycle
- Incentivize Productivity and Innovation in Industry and Government
- Eliminate Unproductive Processes and Bureaucracy
- Promote Effective Competition
- Improve Tradecraft in Acquisition of Services
- Improve the Professionalism of the Total Acquisition Workforce
AT&L has issued further guidance with specific requirements and management practices to address affordability in program planning and execution and milestone decisions. In response, acquisition leadership in the various services has also issued implementation directives. The Secretary of Defense has set targets for cost, budget, and personnel reductions in many areas of DoD's operations as well. Additional references on BBP are available on the AT&L website . Additional references on the topic of affordability are available in Section 3 of "Affordability Engineering Capstone (Phase I) Volume 1 - Basic Research" .
The Office of Management and Budget and the Government Accountability Office (GAO) have also been targeting AEE in government spending, addressing acquisition and contracting practices, duplicative capabilities and services, and inefficient business operations . Although the practice of AEE in acquisition is not new (acquisition guidance and regulation pre-BBP have long been concerned with delivering capabilities within cost and schedule targets), what is new is the sense of urgency given the current economic conditions.
At program levels, achieving AEE requires constant effort across the acquisition life cycle to examine cost and schedule implications of choices. Decisions about the system design, what requirements to meet, and how to structure and manage the acquisition impact affordability and may introduce potential affordability risk. As illustrated in Figure 2, cost and schedule analysis is integral to the systems engineering process. It will reveal affordability risks and define the trade space that includes mission needs, cost, schedule, and performance in which to explore and critically evaluate alternatives. Engineering and cost analysis must be closely coupled in the process. Systems engineering and technical skills must be paired with cost estimating skills to examine affordability trades and recommend analytically based courses of action.
Consistency of a sound technical baseline with the program cost estimate, both maturing in comprehensiveness and detail through the acquisition systems engineering process, is essential to identifying affordability risk. Many different definitions, usage contexts, and references to technical baselines exist in systems engineering. For purposes of measuring AEE, "technical baseline" refers to a comprehensive Acquisition Systems Engineering Baseline (ASEB)—the definition, description, characteristics, and details (function, design, requirements) of the system, the program, the acquisition approach, and all their dependencies and external interfaces that account for all aspects relevant to cost and schedule.
As the program moves through its acquisition stages, divergence of the technical baseline and cost estimate must be carefully monitored. Early discovery of divergence permits early intervention and correction, avoiding potential affordability risk.
MITRE developed an Affordability Engineering Risk Evaluation (AERiE) tool  to facilitate the identification of affordability risk at multiple points across the acquisition life cycle. AERiE is an integral part of an Affordability Engineering Framework (AEF) that facilitates the validation of the technical baseline and program cost estimate, suggests and analyzes tradeoffs to address affordability disconnects, and recommends alternative courses of action.
GAO studies have identified proven acquisition practices to minimize the risk of cost growth on DoD programs. Such practices help "establish programs in which there is a match between requirements and resources—including funding—from the start and execute those programs using knowledge-based acquisition practices ." Although referring to DoD, these practices are generally applicable to acquisition programs of any government agency. They require that a strong systems engineering foundation be established early in a program and that there be greater reliance on a government systems engineering team to set and manage objectives. Practices include:
- Early and continued systems engineering analysis: Ideally beginning before a program is initiated, early systems engineering is critical to designing a system that meets requirements (or negotiates requirements) within available resources, such as technologies, time, money, and people. A robust analysis of alternatives and a preliminary design review (PDR)—which analyze the achievability of required capabilities before committing to a program—can help ensure that new programs have a sound, executable business case that represents a cost-effective solution to meeting the critical user needs. Such engineering knowledge can identify key tradeoffs in requirements and technology that are essential to managing cost. Systems engineering analysis continues to be an important activity through the program's critical design review (CDR) and system demonstration.
- Leveraging mature technologies and processes: Programs often have insufficient knowledge about the maturity of technology. Prototyping early in programs can provide confidence that a system's proposed design can meet performance requirements. Further, having predictable manufacturing processes before decisions are made to move into production can reduce unknowns. Naturally, this assumes that the manufacturing process is used to develop the prototype.
- Establishing realistic cost and schedule estimates that reflect the technical baseline: Cost and schedule estimates are often based on overly optimistic assumptions. Without the ability to generate reliable cost estimates, programs are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls. Inaccurate estimates do not provide the necessary foundation for sufficient funding commitments. Engineering knowledge and more rigorous technical baselines are required to achieve more accurate, reliable cost estimates at the outset of a program. Established cost estimating, schedule estimating, work-breakdown structures, risk management techniques, engineering analyses, and past performance analyses help achieve realism in AEE assessments.
- Clear, well-defined stable requirements: Government department and agency cultures and environments sometimes allow programs to start with too many unknowns, for example, entering the acquisition process without a full understanding of requirements (technical, training, integration, fielding environment, etc.). Minimizing requirements changes could prevent or reduce the amount of cost growth experienced by acquisition programs, but this has to be carefully managed and balanced in an evolving environment to ensure continued effectiveness against, for example, new or advanced adversary threats.
- Incremental approach to acquiring capabilities: Programs can put themselves in a better position to succeed by implementing incremental/evolutionary acquisition strategies that limit the time and reduce the complexity in each incremental development. Up-front planning is required to understand the full system and how the increments will evolve to achieve the end state.
At enterprise or portfolio levels, a variety of analyses and approaches are applied to assess affordability and promote efficiency and effectiveness in investment decisions. Each is appropriate to a decision or management context. MITRE SEs are expected to understand key aspects of the analyses that will need to be performed. They are expected to know the objectives of the analysis, the decisions to be supported, and the general approaches that can be applied. They are expected to enlist the support of and engage with analysts in conducting analyses supporting AEE portfolio level objectives.
SEs are frequently called on to perform or support a number of different investment analysis types (analysis of alternatives, business case analysis, and cost benefit analysis—to name a few). These are focused on informing sponsor funding and expenditure decisions, and they provide critical analysis for assessing affordability, efficiency, and effectiveness of alternatives in deciding to select a solution or course of action.
- Analysis of Alternatives (AoA): An AoA is a technical assessment using distinct metrics and different criteria to objectively evaluate different potential courses of action (or alternatives). Typically, the emphasis is focused on an analysis of alternative technical approaches, measuring their effectiveness in meeting a given set of functional requirements or mission needs. The AoA also includes a life-cycle cost estimate for each alternative, a risk assessment for each alternative, and a recommendation(s) regarding a preferred alternative, pending the results of a more rigorous business case analysis.
- Business Case Analysis (BCA): A BCA is used to determine if a new approach and overall acquisition should be undertaken. A BCA results in a justification, one way or the other, based on the comparison of life-cycle costs and benefits and the results of financial analysis techniques such as return on investment (ROI), net present value (NPV), and payback for each alternative. A BCA may evaluate a single or multiple alternatives against the status quo. Based on the results of the financial analysis, a BCA will help determine if a potential new acquisition is warranted and if the effort should go forward.
- Cost Benefit Analysis (CBA): A cost benefit analysis is a structured assessment of alternative courses of action for achieving some objective. A CBA looks forward and evaluates specific courses of action to determine which would yield the maximum ROI. The assessment informs a decision maker about financial, non-financial, and other non-quantifiable impacts—costs and benefits—of each course of action.
Best Practices and Lessons Learned
AEE is not achieved through the application of any single analytic approach or engineering or management practice, or even a small set of the same. AEE practices need to be integrated throughout enterprise and program engineering and acquisition management activities. Achieving AEE in acquisition programs or in enterprise operations requires a continuous conscious effort on the part of all stakeholders. The following best practices are derived from lessons learned and are fundamental to engineering for AEE and achieving successful acquisitions.
Understand the operational mission, its context, and the current systems or solutions employed. Understand what is changing and what is influencing these changes. What do these changes imply in terms of new operational needs? As an engineer, understand the current program architecture and system operations to be able to evaluate impacts of these changes. Also understand the principles of the enterprise architecture, the data and system interdependencies, and the required interoperability. Affordability considerations extend beyond the system boundaries—to understand them, talk with end users and participate in operational exercises and experiments.
Understand the operational gaps, mission deficiencies, or enhanced/new capabilities being sought by users. What are the users' imperatives (threat, time, consequences) to meet these needs? Determine required vs. desired capabilities and performance levels. At what performance level would an improved capability provide no substantive value beyond current capabilities? At what performance level would an improved capability exceed that required to accomplish the mission? Resources spent delivering performance in excess of that needed might be more effectively applied to other needs. You can gain an understanding of operational gaps through discussions with the end users/operators, the after-action assessments of operations, various operational lessons learned, etc.
Derive solutions by considering DOTMLPF (Doctrine, Organization, Training, Materiel, Leadership, Personnel, Facilities) alternatives, not just materiel solutions. Where can non-materiel solutions fill a capability gap, desired enhanced capabilities, and the resulting operational benefit? If a materiel solution is deemed necessary, determine the non-materiel changes also needed to achieve the desired capability. Are these changes accounted for in requirements documents and reflected in program plans and life-cycle cost estimates? Understanding the full DOTMLPF impact early in the program life cycle of a solution is key to avoiding affordability surprises later in the program.
Conduct market research to determine where exploiting or adapting commercial products or services in devising solutions may be possible. Understand the product marketplace, product maturity, and the business as well as the technical/operational and logistics risks of reliance on commercial or government products. Many technology and capability assessments as well as product reviews exist and can help. Reaching out to SMEs and others who have used these technologies can also help.
Assess the value proposition of an enhanced or new capability. From a portfolio point of view, evaluate the cost effectiveness of solutions compared to alternative expenditures of available resources on other needs or capabilities. Is the expenditure of resources "worth it?" Does the enhanced or new capability provide value to users higher than addressing other important needs? Engineering assessments (e.g., analysis of alternatives) provide techniques for evaluating the value proposition.
Use early systems engineering to define the trade space in which concept alternatives can be developed and evaluated. Define multiple concepts and characterize them technically with sufficient information to support rough order of magnitude cost estimation. If applicable, use concept modeling, modeling and simulation, prototyping, or experimentation to examine concept feasibility. Identify the cost and schedule drivers of the concepts as they relate to specific requirements. Conduct cost capability analyses involving system users to identify technical or performance requirements that can be traded off to achieve cost and schedule objectives, or to define what capabilities can be affordably delivered. Identify the requirements that drive cost and/or schedule and that impose greater risk to timely delivery of needed capabilities. Work with the users and other stakeholders as needed to define evolutionary approaches to meeting these requirements.
Assess and compare the life-cycle cost, effectiveness, and risks of alternatives in selecting a solution. Ensure that decision processes drive efficient and effective solution choices. Measure the affordability of each solution against a current budget profile and assess the affordability risk if the budget is changed. Understand and use established cost estimating tools to help determine cost drivers and major risks associated with the AEE of a capability. (See the SEG article Life-Cycle Cost Estimation.)
Assess user stakeholder expectations against realism of budgets, time, and technology maturity. Understand the basis of budgets and funding profiles. Ensure they are consistent with the chosen solution/technical approach, based on a cost estimate of a suitable technical baseline, and include assessment of cost and schedule risk. Be wary of downward directed schedules. Develop engineering-based timelines showing the critical paths and dependencies; ensure that risks and uncertainty have been incorporated. For developmental items, ensure that a technology readiness assessment (see the SEG article Assessing Technical Maturity) accurately characterizes the technology maturity and that the effort and time to advance maturity to achieve the desired performance or other requirements are adequately assessed. To help stakeholders assess tradeoffs, present the realism in cost as well as operational terms of which mission aspects will be and might not be totally satisfied by the recommended approach along with the feasibility/projection of capability satisfaction over time/future evolutions. Integrate into the program planning the recommended AEE strategy for implementation of capabilities.
Establish, document, and maintain a comprehensive, stable technical baseline to support timely cost analysis and design trades. The technical baseline of a chosen solution becomes the foundation for the program cost estimate and program planning and execution. Through program implementation, it serves as the basis for performance of design and strategy tradeoffs, risk management, and mitigation analyses. For these purposes, the technical baseline must provide a holistic description of the system that includes its technical and functional composition, its relationships and interdependencies with other elements of the enterprise, and its acquisition strategy and program implementation.
Communicate the technical baseline to ensure cost analysts understand it. Work with the cost analysts to develop a comprehensive work breakdown structure that captures all aspects of the technical baseline. Provide a credible engineering basis and clarify any assumptions regarding input to the technical baseline. Ensure that stakeholders—user community, acquisition community, oversight organizations, etc.—are aware of, are familiar with, and understand the tradeoffs of the technical baseline and its role in AEE.
Assess the completeness and realism of the program's cost and schedule estimate. Consider the program's alignment and completeness with respect to the technical baseline and any changes to it as well as the adequacy with which uncertainty and risk have been integrated. Update the technical baseline and the program cost estimate as system requirements and program strategies change.
Integrate management of cost and technical baselines throughout the program. Ensure that cost, engineering, and management teams work together (ideally collocated) to keep the technical baseline and program cost estimate current, and maintain a list of risks, cost drivers, and alternative courses of action/mitigations to address moderate/high risk areas.
Treat cost and schedule as part of the design-capabilities trade space, just like size, weight, power, security, throughput, and other engineering parameters. Understand user expectations/targets for total system cost, particularly unit procurement and sustainment costs for systems with large quantities to be installed or fielded. Assess the ability of the chosen design to meet these targets.
Understand and document all system interfaces, interoperability requirements, dependencies on other systems, programs, and resources, and assess their associated risk as it would impact the program. The interfaces and dependencies of capabilities from independent, yet associated, efforts can be a big contributor to cost due to schedule mismatches, reworking of misunderstood interface exchanges, increased complexity in testing, etc. Include consideration of these tasks and dependencies in the technical and cost baselines along with the operational utility/value of the interfaces, dependencies, and interoperability. Various crown jewels and map-to-mission techniques can help you accomplish this. These techniques are frequently used for cyber mission assurance assessments and are equally valuable to these AEE analyses.
Manage affordability as a key risk parameter in the contractor's system development effort. Use periodic design reviews  to ensure that each component of the system is on track from a risk perspective (technical, cost, and schedule) to meet functional, performance, and interface requirements. Monitor design change for impacts to production and sustainment costs.
Inform key design and programmatic decisions with assessment and understanding of affordability implications and associated risks. Maintain and measure progress against AEE objectives (metrics) in design, engineering, and management reviews and decision processes. Ensure "affordability" is communicated to decision makers. Conduct independent assessments when confronted with significant change in affordability risk.
Keep users well informed and involved in major engineering decisions affecting requirements satisfaction, tradeoffs, and affordability. Present the AEE risks (as highlighted earlier in "Achieving AEE") to the user community for their decisions in accepting the risks (e.g., increased costs balanced against increased effectiveness) to achieve an overall best value solution.
References and Resources
- Janiga, M., December 2011, Affordability, Efficiency, and Effectiveness, Corporate Brief, The MITRE Corporation.
- OSD/AT&L Memorandum for Defense Acquisition and Logistics Professionals, June 28, 2010, Better Buying Power: Mandate for Restoring Affordability and Productivity in Defense Spending, accessed October 10, 2017.
- OSD/AT&L Memorandum for Defense Acquisition and Logistics Professionals, September 14, 2010, Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending, accessed October 10, 2017.
- OSD/AT&L Memorandum for Defense Acquisition Workforce, November 13, 2012, Better Buying Power 2.0: Continuing the Pursuit for Greater Efficiency and Productivity in Defense Spending, accessed October 10, 2017.
- OSD/AT&L Memorandum for Secretaries of the Military Departments, Deputy Chief Management Officer, DoD Chief Information Officer, Directors of the Defense Agencies, AT&L Direct Reports, April 24, 2013, Implementation Directive for Better Buying Power 2.0—Achieving Greater Efficiency and Productivity in Defense Spending, accessed October 10, 2017.
- OSD/AT&L Website, Better Buying Power, accessed October 10, 2017.
- MITRE Working note WN110058V1 Affordability Engineering Capstone (Phase I) Volume 1 - Basic Research, J. Duquette et al., September 2011.
- The MITRE Corporation, Affordability Engineering Risk Evaluation (AERiE) prototype tool.
- Government Accountability Office, March 2011, Opportunities to Reduce Potential Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue, GAO-11-318SP.
- GAO Testimony before the Committee on Homeland Security and Governmental Affairs, Subcommittee on Federal Financial Management, Government Information, Federal Services and International Security, United States Senate, March 29, 2011, DoD Cost Overruns—Trends in Nunn-McCurdy Breaches and Tools to Manage Weapon Systems Acquisition Costs, pp. 6–8.
- DAU, Technical Reviews and Audits, Defense Acquisition Guidebook, 4.2.8
Additional References and Resources
- DAU, "Affordability," Defense Acquisition Guidebook.
- Duquette, J., April 2012, Affordability Engineering Framework, The MITRE Corporation, accessed October 10, 2017.
- GAO, December 22, 2015, GAO Schedule Assessment Guide: Best Practices for Project Schedules, GAO-16-89G, accessed October 10, 2017.