Acquiring and Incorporating Post-Fielding Operational Feedback into Future Developments
Definition: The Post-Implementation Review (PIR) is an evaluation tool that compares the conditions prior to the implementation of a project (as identified in the business case) with the actual results achieved by the project. With reference to information technology solution projects, the Information Systems Audit and Control Association (ISACA) defines the PIR as the first or subsequent review of an information technology (IT) solution and/or the process of its implementation, performed after its implementation .
Keywords: benefit analysis, evaluation, implementation review, investment management, performance management, post-technology use
MITRE SE Roles & Expectations: MITRE systems engineers (SEs) are expected to understand the purpose and role of a Post-Implementation Review (PIR) and the benefits and costs of employing them. They are expected to be able to recommend techniques for PIRs, assist the government in tailoring PIR procedures, lead PIRs, or perform individual PIR tasks, as appropriate, e.g., post-implementation technical performance analyses. However, since PIRs should be conducted by individuals not directly involved in the previous steps of the acquisition process, MITRE SEs are frequently precluded from participating in the reviews themselves, other than as subject matter experts because of their role in recommending appropriate technology or providing guidance during earlier phases of the life cycle.
The PIR is used to evaluate the effectiveness of system development after the system has been in production for a period of time. The objectives of the PIR are to determine if the system does what it is designed to do: Does it support the user as required in an effective and efficient manner? The review is intended to assess how successful the system is in terms of functionality, performance, and cost versus benefits, as well as to assess the effectiveness of the life-cycle development activities that produced the system. The review results can be used to strengthen the system as well as system development procedures. However, while the systems engineering community is in general agreement that the PIR is a laudable thing to do, in practice the review generally remains an ignored step- child at best, as departments and agencies content themselves with simply "running the numbers," i.e., doing statistical analyses of performance deltas.
Government Interest and Use
The President's Office of Management & Budget (OMB) outlines the purpose of the PIR in Step IV.3. Post-Implementation Review (PIR), of the Capital Programming Guide, first published in 1997, as a supplement to Circular A-11, Part 3: Planning, Budgeting, and Acquisition of Capital Assets. It is described as a diagnostic tool to evaluate the overall effectiveness of the agency's capital planning and acquisition process as a complement to operational analysis. This control mechanism is used during the operational life cycle of an asset to enable resource managers to optimize the performance of capital assets over the course of their life cycle and eventual disposition [2, 3]. OMB stipulates that PIRs should be conducted by individuals not directly involved in the acquisition of the asset, but that it may include owners and users of the asset or other personnel and consultants.
In response to the OMB mandate, agency system development lifecycle (SDLC) documentation now specifies the preparation of a PIR report as part of the final phase of the SDLC, Operations and Maintenance. However, in its IT Investment Management Framework [4, 5], the General Accounting Office (GAO) observes that agencies will continue to have difficulty performing an effective PIR unless they have more comprehensively established policies and procedures to assess the benefits and performance of their investments. This lack of documented processes is typically one of GAO's primary criticisms of agency performance in this area, e.g., GAO's 2007 report on DoD Business System Modernization (GAO-07-538), which criticized the limited nature of the Defense Acquisition System (DAS) PIR procedures . In its recently updated Acquisition Directive 102-01, Department of Homeland Security (DHS) requires a PIR of every program implemented in the agency. The most recent DoD Instruction 5000.02 extends the PIR requirement to acquisition category (ACAT) II and below .
According to OMB, PIRs should be conducted three to twelve months after an asset becomes operational. However, agencies vary in the length of that period, with a range of anywhere from three to eighteen months after implementation. GAO points out that the timing of a PIR can be problematic. A PIR conducted too soon after an investment has been implemented may fail to capture the full benefits of the new system, while a PIR conducted too late may not be able to draw adequately on the institutional memory of the development/investment process. GAO indicates that PIRs should also be conducted for initiatives that were aborted before completion, to assist in the identification of potential management and process improvements.
To ensure consistency of evaluations, OMB recommends the use of a documented methodology for the conduct of PIRs, requiring that the chosen methodology be in alignment with the organization's planning process. The required level of specificity for PIR reports varies from agency to agency. According to OMB, PIRs should address :
- Customer/User Satisfaction
- Business process support
- Investment performance
- Internal Business
- Project performance
- Infrastructure availability
- Standards and compliance
- Security issues and internal controls
- Evaluations (accuracy, timeliness, adequacy of information)
- Strategic Impact and Effectiveness
- System impact and effectiveness
- Alignment with mission goals
- Portfolio analysis and management
- Cost savings
- Workforce competency
- Advanced technology use
- Methodology expertise
- Employee satisfaction/retention
- Program quality
In assisting government sponsors to tailor the PIR to a particular project, MITRE systems engineers should cast the net as broadly as possible, ensuring that the PIR also examines potential weaknesses and risks and the long-term maintainability of the solution.
Best Practices and Lessons Learned
In 2004, the Information Systems Audit and Control Association (ISACA) published a set of best practices specific to the PIR, the IS Auditing Guideline: Post-Implementation Review.
Properly leveraged, the PIR provides an important control mechanism and tool for the continuous improvement of the acquisition process. However, PIR templates are all too frequently limited to checklists comparing baseline expectations to results on a numerical value scale or summary bullets rather than providing serious analyses of root causes and contributory factors. By contrast, the Systems Engineering: Post-Implementation Review (PIR) Document Template, published by the U.S. Army Information Technology Agency (ITA) as draft in October 2008, devotes the majority of its review to the system itself rather than to the acquisition process, and emphasizes that the review should result in a free form report, thereby highlighting the analytical potential of the review .
That agencies should underestimate the value of PIRs is not surprising, given the fact that managers frequently feel they are all too aware of the limitations of system releases as a result of compression of schedules and inevitable scope creep prior to deployment. The typical strategy is simply to "move on and fix the bugs" in the next release. "Why do we need a PIR if we already know what is wrong?," a rationale that is based on a failure to recognize the potential benefit of the review when it is conducted as a true audit of technical and process performance. Significantly, the PIR is not tied to any life-cycle milestone, effectively robbing it of any determinative value.
An additional reason for the failure to fully leverage the PIR as a process improvement tool for IT acquisitions may well be that because of the capital planning and investment control (CPIC) context in which OMB and GAO place the review, it is seen more as a capital management/project management tool than as a tool to improve system development from a technical perspective.
If, from a CPIC perspective, the PIR can be seen as the initial iteration of the operational assessments conducted over the life cycle of an operational system, from the SDLC perspective it is only loosely tied in to the systems life cycle, being relegated to the O&M phase rather than seen as a close out report for the systems implementation phase of the life cycle. From the SE perspective, one could argue that the PIR is condemned to a technology limbo—too late for developers to care, too early for maintenance people to be interested.
If the PIR presents an excellent opportunity for MITRE systems engineers to assist their sponsors across all programs to improve the way they conduct business, this opportunity is even more acute for those MITRE programs whose major focus is providing systems engineering support to agencies involved in business systems modernization (BSM) efforts. These efforts generally encompass the entire enterprise and frequently, as in the case of DHS, involve the insertion of technologies that revolutionize the way the agencies do business and provide challenges far exceeding those faced in the implementation of systems in the past. The issues resulting from the challenges posed by BSM and major technology insertions are compounded by the fact that civilian agencies have increasingly relied on contractor support not just to deliver systems, but to provide end-to-end services. This is an environment where the PIR can be particularly useful not only from a project and portfolio management perspective but also as an analysis of the operational impact of new technology, e.g., the use of biometrically enabled passports to verify traveler identity or the use of digitized documentation in a traditionally paper-based environment.
The effectiveness of PIRs as a process improvement tool in general, particularly in the case of BSM deployments that transform the operating environment, is dependent on the willingness to address issues of workforce competence, advanced technology use, and methodology expertise, e.g., how databases driven and service- oriented architectures are received in organizations previously responsible for the maintenance of long-outdated hardware and software. While these areas of investigation are identified in OMB's PIR guidance, they are generally absent from agency PIR procedures. However, it is frequently these areas where implementations fail to achieve the expected results. This is either because the workforce is not ready to operate or maintain the new systems, those responsible for maintaining the new systems lack the expertise required to support next generation technology delivered through the BSM program, or a mismatch between the advanced technology deployed and the requirements of the operational environment. If the PIR is not designed to explore the root cause of potential issues, the resulting report will remain a paper exercise. However, failure to confront difficult issues leaves agencies with limited tools in their dialog with oversight bodies when it comes to requesting additional resources or postponement of mandates.
References & Resources
- ISACA, October 15, 2004, IS Auditing Guidelines Post-Implementation Review.
- OMB, June 2006, "Post Implementation Review and Post-Occupancy Evaluation," Section III.3.3, Capital Programming Guide, Version 2.0, Supplement to Office of Management and Budget Circular A-11, Part 7: Planning, Budgeting, and Acquisition of Capital Assets.
- OMB Circular No. A-130, February 8, 1996, Management of Federal Information Resources, revised.
- GAO, 2004, Information Technology Investment Management. A Framework for Assessing and Improving Process Maturity (GAO-04-394G), pp. 83–89.
- Internal Revenue Service, June 30, 2000, IRS Enterprise Life Cycle, Investment Decision Management, Post Implementation Review (PIR) Procedure.
- GAO, May 2007, Business Systems Modernization: DoD Needs to Fully Define Policies and Procedures for Institutionally Managing Investments (GAO-07-538), p. 31.
- Department of Defense, December 8, 2008, Department of Defense Instruction Number 5000.02.
- U.S. Army Information Technology Agency, October 2008, ITA Systems Engineering. Post-Implementation Review (PIR) Document Template, Version 1.0.