Affordability, Efficiency, and Effectiveness


Definitions: Affordability, Efficiency, and Effectiveness (AEE) are three success measures that guide systems engineers in developing and shaping engineering solutions, making program recommendations, and evaluating engineering efforts.

  • Affordability: Ability to fund desired investment. Solutions are affordable if they can be deployed in sufficient quantity to meet mission needs within the (likely) available budget.
  • Efficiency: A measure of the "bang for the buck" or "unit benefit per dollar." Solutions are efficient if they measurably increase the "bang" or "unit benefit" for the amount of resources required to deliver the capability.
  • Effectiveness: "The bang"; the ability to achieve an organization's mission. Solutions are effective if they deliver capability of high value to accomplishing the user's missions [1].

Keywords: analysis of alternatives, budgets, cost benefit analysis, life-cycle cost, portfolio analysis, program cost, return on investment

MITRE SE Roles & Expectations: MITRE systems engineers (SEs) are expected to incorporate and assess measures of affordability, efficiency, and effectiveness in engineering solutions and supporting acquisition activities of sponsors. They are expected to:

  • Work with users to understand their mission needs, capability gaps, and performance requirements.
  • Develop alternative solutions or courses of action, and evaluate them for mission effectiveness as well as life-cycle affordability and efficiency.
  • Understand operational and technical domains; recommend and conduct engineering-based tradeoffs of requirements, design, performance, cost, and schedule to address affordability constraints.
  • Understand life-cycle cost, schedule, risk, and affordability implications of alternatives under consideration, and incorporate these dimensions in engineering products and recommendations.
  • Encourage and facilitate active participation of the broad stakeholder community and acquisition decision makers in exploring alternatives; help them understand and use the trade space to achieve program affordability and evaluate merits of alternatives from operational as well as business perspectives.
  • Monitor and evaluate contractor system development efforts; identify affordability risks and recommend changes when warranted as an acquisition program progresses.
  • Assist government sponsors in developing, adjusting, and implementing strategies, at program and enterprise levels, to ensure affordability and improve efficiency/effectiveness.
  • Communicate AEE best practices and lessons learned.

Background

MITRE's mission is to work in partnership with its government sponsors in applying science and advanced technology to engineer systems of critical national importance. MITRE's systems engineering is conducted in the context of developing solutions to meet the needs and challenges of our sponsors in conducting their missions, and in aiding sponsors in planning and managing programs to acquire such solutions. Sponsor success is achieved if the systems they deploy are effective and available when needed to achieve their mission. Systems to be procured, deployed, and sustained must be affordable, that is, within the means of the sponsor's available resources, and should be efficient, providing high value for the resources to be expended.

Sponsor acquisition environments present many challenges. A number of system development programs have failed to deliver needed capabilities, or delivered reduced capability with expenditure of time and funds well beyond what was planned. Current mounting federal budget deficits place considerable economic stress on government agencies. Budget reductions mandate difficult decisions about where to invest limited resources, how to make current programs more affordable, and whether to terminate poorly performing programs. Investments for new capabilities, replacements, or enhancements to existing systems as well as simple continuation of existing programs require careful analysis and evaluation of their affordability, efficiency, and effectiveness. Systems engineering seeks to apply current and emerging technologies flexibly to address sponsors' dynamic threats and mission needs. At the same time, affordability engineering helps sponsors respond to fiscal realities and be effective stewards of taxpayer dollars.

As depicted in Figure 1, affordability challenges exist at different sponsor levels and are addressed by varying engineering, analysis, or management approaches.

AEE Construct
Figure 1. AEE Construct

Enterprises invest in and sustain systems, infrastructures, and organizations to accomplish multiple missions. Investment decisions made by agency heads, Senior Acquisition Executives (SAEs), Chief Information Officers (CIOs), and Program Executive Officers (PEOs) require a holistic view of the enterprise's missions and objectives. Investment decisions are required for new or enhanced systems that provide additional capabilities as well as for sustainment of existing systems, infrastructures, manpower, and operations. A portfolio management approach evaluates the benefits of or return on investment choices relative to filling identified capability gaps or improving mission functions. Efficiency initiatives and business process reengineering (BPR) look to reduce redundancy, overlap, or inefficient processes. Enterprise-level analysis and decision frameworks can be applied to help an agency achieve greatest mission effectiveness, ensuring highest priority/highest value needs are met with its allocated budget [1].

Each acquisition program is an element of the integrated capability delivered by the enterprise. As such, it is vital that each be executed to deliver its target capability within available technology, funding, and time, i.e., be affordable. Each program and capability must contribute high value to users in terms of mission effectiveness achieved in the most efficient means. Not doing so can, like a chain reaction, have serious impacts across the enterprise. At this level, engineering solutions for affordability and adopting best acquisition system engineering and management practices within the Program Management Office (PMO) are key to achieving success.

Technical and operational innovation contributes to AEE at both the acquisition program and enterprise levels. Adaptive systems and composable capabilities provide an enterprise with the flexibility to respond to rapidly changing mission needs with existing resources and minimal new investment. Application of advances in IT and network design offer potential for great efficiency in the delivery of information and services to end users at the enterprise and program level. Advances within technology domains afford an opportunity to reduce life-cycle costs in the acquisition of new or enhanced capabilities and to transform operations and sustainment approaches for greater efficiency.

Government Interest and Use

The U.S. economy is experiencing an era of very slow growth, high unemployment, historic high debt, and mounting budget deficits at federal and lower levels. Considerable economic stress is being felt by all government agencies as they strive to accomplish their missions and deliver services with constant or shrinking budgets. Agencies across the federal government are implementing new strategies to promote AEE in their acquisition decisions and management practices.

In June 2010, the Office of the Secretary of Defense/Acquisition, Technology and Logistics (OSD/ATL) memo Better Buying Power (BBP): Mandate for Restoring Affordability and Productivity in Defense Spending set an "important priority" for DoD: "delivering better value to the taxpayer and improving the way the Department does business" [2]. OSD/ATL memo Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending, September 14, 2010, followed, setting the significance and breadth of this mandate [3]. This 17-page memo outlined five key areas, with more than 20 specific initiatives. The five areas are:

  1. Target Affordability and Control Cost Growth (mandating affordability as a requirement and implementing "should-cost" based management)
  2. Incentivize Productivity and Innovation in Industry
  3. Promote Real Competition
  4. Improve Tradecraft in Services Acquisition
  5. Reduce Non-Productive Processes and Bureaucracy

AT&L has issued further guidance with specific requirements and management practices to address affordability in program planning and execution and milestone decisions. In response, acquisition leadership in the various services has also issued implementation directives. The Secretary of Defense has set targets for cost, budget, and personnel reductions in many areas of DoD's operations as well. Additional references on the topic of affordability are available in Section 3 of Affordability Engineering Capstone (Phase I) Volume 1 - Basic Research [4].

Office of Management and Budget and the General Accounting office have also been targeting AEE in government spending, addressing acquisition and contracting practices, duplicative capabilities and services, and inefficient business operations [4]. Although the practice of AEE in acquisition is not new (acquisition guidance and regulation pre-BBP have long been concerned with delivering capabilities within cost and schedule targets), what is new is the sense of urgency given the current economic crisis.

Achieving AEE

At program levels, achieving AEE requires constant effort across the acquisition life cycle to examine cost and schedule implications of choices. Decisions about the system design, what requirements to meet, and how to structure and manage the acquisition impact affordability and introduce potential affordability risk. As illustrated in Figure 2, cost and schedule analysis is integral to the system engineering process. It will reveal affordability risks and define the trade space of mission needs, cost, schedule and performance in which to explore and critically evaluate alternatives. Engineering and cost analysis must be closely coupled in the process. System engineering and technical skills must be paired with cost estimating skills to examine affordability trades and recommend analysis-based courses of action.

As the program moves through its acquisition stages, divergence of the technical baseline and cost estimate must be carefully monitored. Early discovery of divergence permits early intervention and correction, reducing affordability risk.

An Affordability Engineering Risk Evaluation (AERiE) tool [5] is being developed to facilitate identification of affordability risk at multiple points across the acquisition life cycle. AERiE is also envisioned as part of an Affordability Engineering Framework (AEF) that will facilitate the validation of the technical baseline and program cost estimate, suggest and analyze tradeoffs to address affordability disconnects, and recommend alternative courses of action.

Affordability in Systems Engineering
Figure 2. Affordability in Systems Engineering [6]

From a number of studies, the GAO identified proven acquisition practices to minimize the risk of cost growth on DoD programs. Such practices help "establish programs in which there is a match between requirements and resources—including funding—from the start and execute those programs using knowledge-based acquisition practices [7]." While referring to DoD, these practices are generally applicable to acquisition programs of any government agency. They require a strong system engineering foundation be established early in a program and greater reliance on a government systems engineering team to set and manage objectives. Practices include:

  • Early and continued systems engineering analysis: Ideally beginning before a program is initiated, early systems engineering is critical to designing a system that meets requirements (or negotiates requirements) within available resources, such as technologies, time, money, and people. A robust analysis of alternatives and a preliminary design review (PDR)—which analyze the achievability of required capabilities before committing to a program—can help ensure that new programs have a sound, executable business case that represents a cost-effective solution to meeting the critical user needs. Such engineering knowledge can identify key tradeoffs in requirements and technology that are essential to managing cost. Systems engineering continues to be an important tool through a program's critical design review (CDR) and system demonstration.
  • Leveraging mature technologies and processes: Programs often have insufficient knowledge about the maturity of technology. Prototyping early in programs can provide confidence that a system's proposed design can meet performance requirements. Further, having predictable manufacturing processes before decisions are made to move into production can reduce unknowns. Naturally, this assumes that the manufacturing process is used to develop the prototype.
  • Establishing realistic cost and schedule estimates matched to available resources: Cost and schedule estimates are often based on overly optimistic assumptions. Without the ability to generate reliable cost estimates, programs are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls. Inaccurate estimates do not provide the necessary foundation for sufficient funding commitments. Engineering knowledge and more rigorous technical baselines are required to achieve more accurate, reliable cost estimates at the outset of a program. Established cost estimating, schedule estimating, work-breakdown structures, risk management techniques, engineering analyses, and past performance help achieve realism in AEE assessments.
  • Clear, well-defined requirements: Government department and agency cultures and environments sometimes allow programs to start with too many unknowns, for example, entering the acquisition process without a full understanding of requirements (technical, training, integration, fielding environment, etc.). Minimizing requirements changes could decrease the amount of cost growth experienced by acquisition programs, but this has to be carefully managed and balanced in an evolving environment to ensure continued effectiveness against, for example, new or improved adversary threats.
  • Incremental approach to acquiring capabilities: Programs can put themselves in a better position to succeed by implementing incremental/evolutionary acquisition strategies that limit the time in each incremental development.

At enterprise or portfolio levels, a number of analyses and approaches are applied to assess affordability and promote efficiency and effectiveness in investment decisions. Each is appropriate to a decision or management context. MITRE systems engineers are expected to understand key aspects of the analyses that will need to be performed. They are expected to know the objectives of the analysis, the decisions to be supported, and the general approaches that can be applied. They are expected to enlist the support of and engage with analysts in conducting analyses supporting AEE objectives.

Systems engineers are frequently called on to perform or support a number of different investment analysis types (Analysis of Alternatives, Business Case Analysis, and Cost Benefit Analysis —to name a few). These are focused on informing sponsor funding and expenditure decisions and they provide critical analysis for assessing affordability, efficiency, and effectiveness of alternatives in deciding to select a solution or course of action.

  • Analysis of Alternatives (AoA): An AoA is a technical assessment using distinct metrics and different criteria to objectively evaluate different potential courses of action (or alternatives). Typically the emphasis is focused on an analysis of alternative technical approaches, measuring their effectiveness in meeting a given set of functional requirements or mission need. The AoA also includes a life-cycle cost estimate for each alternative, a risk assessment for each alternative, and a recommendation(s) regarding a preferred alternative, pending the results of a more rigorous business case analysis.
  • Business Case Analysis (BCA): A BCA is used to determine if a new approach and overall acquisition should be undertaken. A BCA results in a justification, one way or the other, based on the comparison of life-cycle costs and benefits and the results of financial analysis techniques such as Return on Investment (ROI), Net Present Value (NPV), and Payback for each alternative. A BCA may evaluate a single or multiple alternatives against the status quo. Based on the results of the financial analysis, a BCA will help to determine if a potential new acquisition is warranted and if the effort should go forward.
  • Cost Benefit Analysis (CBA): A cost benefit analysis is a structured assessment of alternative courses of action for achieving some objective. A CBA looks forward and evaluates specific courses of action to determine which would yield the maximum ROI. The assessment informs a decision maker about financial, non-financial, and other non-quantifiable impacts—costs and benefits—of each course of action.

These analyses are described in more detail in the SEG articles on Performing Analyses of Alternatives and Comparison of Investment Analyses.

Best Practices and Lessons Learned

AEE is not achieved through the application of any single analytic approach or engineering or management practice, or even a small set of the same. AEE practices need to be integrated throughout enterprise and program engineering and acquisition management activities. Achieving AEE of acquisition programs or in enterprise operations requires a continuous conscious effort on the part of all stakeholders. The following practices are fundamental to engineering for AEE and achieving successful acquisitions. They reflect some examples of best practices and are derived from lessons learned:

Understand the operational mission, its context, and the current systems or solutions employed. Understand what is changing, and what is influencing these changes. What do these changes imply in terms of new operational needs? As an engineer, understand the current program architecture and system operations to be able to evaluate impacts of these changes. Also understand the principles of the enterprise architecture, the data and system interdependencies and required interoperability. Affordability considerations extend beyond the system boundaries. This understanding can be gained through discussions with end users and participation in operational exercises and experiments.

Understand the operational gaps, mission deficiencies, or enhanced/new capabilities being sought by users. What are the users' imperatives (threat, time, consequences) to meet these needs? Determine required vs. desired capabilities and performance levels. At what performance level would an improved capability provide no substantive value beyond current capabilities? At what performance level would an improved capability exceed that required to accomplish the mission? Resources spent delivering performance in excess of that needed might be more effectively applied to other needs. The understanding of operational gaps can be gained through examining the after-action assessments of operations, various operational lessons learned, etc.

Derive solutions from consideration of DOTMLPF (Doctrine, Organization, Training, Materiel, Leadership, Personnel, Facilities) alternatives, not just material solutions. Where can non-material solutions effect the desired enhanced capabilities and operational benefit? If a material solution is deemed necessary, determine the non-material changes also needed to achieve the desired capability. Are these changes accounted for in program plans and life-cycle cost estimates? Understanding up front the full DOTMLPF impact of a solution is key to avoiding affordability surprises later in the program.

Conduct market research to determine where exploiting or adapting commercial products or services in devising solutions may be possible. Understand the product marketplace, product maturity, and the business as well as the technical/operational and logistics risks of reliance on commercial or government products. Many technology and capability assessments as well as product reviews exist and can help. Reach out to others via social media.

Assess the value proposition. From a portfolio point of view, evaluate the cost effectiveness of solutions compared to alternative expenditures of available resources on other needs or capabilities. Is the expenditure of resources "worth it?" Does the enhanced or new capability provide value to users higher than addressing other important needs? Engineering assessments highlighted below (e.g., analysis of alternatives) provide techniques for evaluating the value proposition.

Use early systems engineering to define the trade space in which alternatives can be developed and evaluated. Define multiple concepts and characterize them technically with sufficient information to support rough order of magnitude cost estimation. Use concept modeling, modeling and simulation, prototyping, or experimentation to examine concept feasibility. Identify the cost and schedule drivers of the concepts as they relate to specific requirements. Involve system users to identify technical or performance requirements that can be traded off to achieve cost and schedule objectives, or to define what capabilities can be affordably delivered. Identify the requirements that drive cost and/or schedule and that impose greater risk to timely delivery of needed capabilities. Work with the users and other stakeholders as needed to define evolutionary approaches to meeting these requirements.

Assess and compare the life-cycle cost, effectiveness, and risks of alternatives in selecting a solution. Ensure decision processes drive efficient and effective solution choices. Measure the affordability of each solution against a current budget profile and assess the affordability risk if the budget is changed. Understand and use established cost estimating tools to help determine cost drivers and major risks associated with the AEE of a capability. (See SEG article on Life-Cycle Cost Estimation.)

Assess user stakeholder expectations against realism of budgets, time, and technology maturity. Understand the basis of budgets and funding profiles. Ensure they are consistent with the chosen solution/technical approach, based on a cost estimate of a suitable technical baseline, and include assessment of cost and schedule risk. Be wary of downward directed schedules. Develop engineering-based timelines showing the critical paths and dependencies; ensure that risks and uncertainty have been incorporated. For developmental items, ensure that a technology readiness assessment (see SEG article on Assessing Technical Maturity) accurately characterizes the technology maturity, and that the effort and time to advance maturity to achieve desired performance or other requirements is adequately assessed. Present the realism in cost as well as operational terms of what mission aspects will be and might not be totally satisfied by the recommended approach along with the feasibility/projection of capability satisfaction over time/future evolutions to help stakeholders assess tradeoffs. Create a time-phased roadmap highlighting the recommended AEE strategy for implementation of capabilities.

Establish, document, and maintain a comprehensive, stable technical baseline to support timely cost analysis and design trades. The technical baseline of a chosen solution becomes the foundation for the program cost estimate and program planning and execution. Through program implementation, it serves as the basis for performance of design and strategy tradeoffs, risk management, and mitigation analyses. For these purposes, the technical baseline must provide a holistic description of the system that includes its technical and functional composition, its relationships and interdependencies with other elements of the enterprise, and its acquisition strategy and program implementation.

Communicate the technical baseline to ensure cost analysts understand it. Work with the cost analysts in developing a comprehensive work breakdown structure that captures all aspects of the technical baseline. Provide a credible engineering basis and make clear any assumptions regarding input to the technical baseline. Ensure that stakeholders—user community, acquisition community, oversight organizations, etc.—are aware of, familiar with, and understand the tradeoffs of the technical baseline and its role in AEE.

Assess the completeness and realism of the program's cost and schedule estimate. Consider the program's alignment and completeness with respect to the technical baseline and any changes to it as well as the adequacy with which uncertainty and risk have been integrated. As system requirements and program strategies change, the technical baseline as well as the program cost estimate should be updated.

Integrate management of cost and technical baselines throughout the program. Ensure that cost, engineering, and management teams work together (ideally collocated) to keep the technical baseline and Program Cost Estimate current, and maintain a list of risks, cost drivers, and alternative COAs/mitigations to address moderate/high risk areas.

Treat cost and schedule as part of the design-capabilities trade space, just like size, weight, power, security, throughput, and other engineering parameters. Understand user expectations/targets for total system cost, particularly unit procurement and sustainment costs for systems with large quantities to be installed or fielded. Assess the ability of the chosen design to meet these targets.

Understand and document all system interfaces, interoperability requirements, dependencies on other systems, programs, and resources, and assess their associated risk as it would impact the program. The interfaces and dependencies of capabilities from independent, yet associated, efforts can be a big contributor to cost due to schedule mismatches, reworking of misunderstood interface exchanges, increased complexity in testing, etc. Include consideration of these tasks and dependencies in the technical and cost baselines along with the operational utility/value of the interfaces, dependencies, and interoperability. Various crown jewel and map-to-mission techniques can be used to help accomplish this. These techniques are frequently used for cyber mission assurance assessments and are equally valuable to these AEE analyses.

Manage affordability as a key risk parameter in the contractor's system development effort. Use periodic design reviews to ensure that each component of the system is on track from a risk perspective (technical, cost, and schedule) to meet functional, performance, and interface requirements. Monitor design change for impacts to production and sustainment costs.

Inform key design and programmatic decisions with assessment and understanding of affordability implications and associated risks. Maintain and measure progress against AEE objectives (metrics) in design, engineering, and management reviews and decision processes. Ensure "affordability" is communicated to decision makers. Conduct independent assessments when confronted with significant change in affordability risk.

Keep users well informed and involved in major engineering decisions affecting requirements satisfaction, tradeoffs, and affordability. Present the AEE risks (as highlighted earlier in "Achieving AEE") to the user community for their decisions in accepting the risks (e.g., increased costs balanced against increased effectiveness) to achieve an overall best value solution.

References & Resources

  1. Affordability, Efficiency, and Effectiveness, Corporate Brief, December 2011.
  2. OSD/AT&L Memorandum for Defense Acquisition and Logistics Professionals, Better Buying Power: Mandate for Restoring Affordability and Productivity in Defense Spending, June 28, 2010.
  3. OSD/AT&L Memorandum for Defense Acquisition and Logistics Professionals, Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending, September 14, 2010.
  4. MITRE Working note WN110058V1 "Affordability Engineering Capstone (Phase I) Volume 1 - Basic Research," Joseph Duquette et al., September 2011.
  5. AERiE prototype tool Community Share site.
  6. GAO-11-318SP Opportunities to Reduce Potential Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue, March 2011.
  7. GAO Testimony before the Committee on Homeland Security and Governmental Affairs, Subcommittee on Federal Financial Management, Government Information, Federal Services and International Security, United States Senate. DOD COST OVERRUNS Trends in Nunn-McCurdy Breaches and Tools to Manage Weapon Systems Acquisition Costs, March 29, 2011, pp. 6–8.

Publications

Download the SEG

MITRE's Systems Engineering Guide

Download for EPUB
Download for Amazon Kindle
Download a PDF

Questions?
Contact the SEG Team