Quality Assurance and Measurement
Definition: Quality assurance is "a planned and systematic means for assuring management that the defined standards, practices, procedures, and methods of the process are applied." "The purpose of [quality] measurement and analysis (MA) is to develop and sustain a measurement capability used to support management information needs ."
Keywords: continuous improvement, measurement, metrics, process improvement, quality, standards
There are multiple perspectives on both quality and its measurement that depend on the stakeholder's point of view. Knowledge of these perspectives is important when recommending quality or measurement programs for a government organization.
MITRE SE Roles & Expectations: MITRE systems engineers (SEs) are expected to be able to recommend how to establish a quality assurance program in the government systems acquisition or government operational organization. They are expected to be able to guide the establishment and direction of quality assurance programs, conduct process and product reviews, and influence the resolution of corrective actions to ensure adherence to documented processes. MITRE systems engineers are expected to be able to help develop measurement capabilities to monitor processes and products .
Perspectives on Quality
Some of the perspectives on quality are as follows :
- Judgmental. When referred to as the transcendent definition of quality, it is both absolute and universally recognizable, a mark of uncompromising standards and high achievement. You can't really measure or assess it—you just know it when you see it. Lexus and Ritz-Carlton are examples.
- Product-based. In this view, quality is a function of a specific, measurable variable and differences in quality reflect differences in quantity of a product attribute, such as threads per square inch or pixels per square inch. Bed sheets and LCD high definition televisions are examples.
- User-based. Quality is defined as fitness for intended use, or how well the product performs its intended function. If you want an off-road vehicle for camping, a Jeep might suit your needs. If you want a luxury vehicle with lots of features, a Cadillac might better suit your needs.
- Value-based. From this perspective, a quality product is as useful as a competing product and sold at a lower price, or it offers greater usefulness or satisfaction at a comparable price. If you had a choice between two "thumb drives" and one offered one gigabyte of storage for $39.95 and another offered two gigabytes of storage for $19.95, chances are you would choose the latter.
- Manufacturing-based. Quality is the desirable outcome of engineering and manufacturing practice, or conformance to specifications. For Coca-Cola, quality is "about manufacturing a product that people can depend on every time they reach for it."
- Customer-driven quality. The American National Standards Institute and the American Society for Quality (ASQ) define quality as "the totality of features and characteristics of a product or service that bears on its ability to satisfy given needs ." A popular extension of this definition is "quality is meeting or exceeding customer expectations."
Quality Assurance versus Quality Control
There is an important distinction between quality assurance (QA) and quality control (QC). The ASQ defines QA as "the planned and systematic activities implemented in a quality system so that quality requirements for a product or service will be fulfilled." ASQ defines QC as "the observation techniques and activities used to fulfill requirements for quality ." Thus, QA is a proactive, process-oriented activity whereas QC is a reactive, manufacturing-oriented activity. The focus of QA is putting good processes in place so that the quality will be "built into" the product rather than trying to "inspect quality into" the finished product.
Quality Standards and Guidance
The International Organization for Standardization (ISO) 9000 family introduces the concept of quality management, processes, certification, and continual improvement. ISO 9000 is the internationally accepted standard for quality management. It looks at manufacturing and customer-based perspectives of quality. The ISO 9000:2000 family is built on eight quality management principles: (1) Customer Focus, (2) Leadership, (3) Involvement of People, (4) Process Approach, (5) System Approach to Management, (6) Continual Improvement, (7) Factual Approach to Decision Making, and (8) Mutually Beneficial Supplier Relationships .
The ISO 9001:2000 (the basis for ISO 9001 certification or registration) states, "This International Standard specifies requirements for a quality management system where an organization a) needs to demonstrate its ability to consistently provide a product that meets customer and applicable regulatory requirements, and b) aims to enhance customer satisfaction through the effective application of the system, including processes for continual improvement of the system and the assurance of conformity to customer and applicable regulatory requirements ." ISO 9001 registration is critical to securing and maintaining business for both private and public sector contractors. The government recognizes ISO 9001:2000 as a "higher level" quality requirement and may invoke it in the contract under the conditions stated in Federal Acquisition Regulation Part 46.202-4, Higher Level Contract Quality Requirements. In these situations, the government is more interested in the contractor's quality management system (and its certification) than government inspection of a product under development.
Government acquisition organizations rarely have an independent quality assurance organization that oversees the quality of the government's work products or processes. The Capability Maturity Model Integrated for Acquisition (CMMI-ACQ) includes a process area (Product and Process Quality Assurance) that provides a set of goals and specific practices for quality assurance in an acquisition organization . Both government and contractor development organizations have a similar CMMI for Development (CMMI-DEV) process area .
The government occasionally introduces a quality or process improvement initiative that receives emphasis for a while and is then overcome by events or forgotten. Several of these past initiatives include the "Suggestion Program," "Zero Defects," "Quality Circles," and "Total Quality Management." Some of the most recent initiatives are Department of Defense (DoD) Six Sigma, DoD-wide Continuous Process Improvement/Lean Six Sigma, and Air Force Smart Operations for the 21st Century [9, 10, 11]. Most of these initiatives deal with process improvement by eliminating waste, streamlining the processes, or instituting a more efficient way to perform a required task that results in cost avoidance or cost savings.
Perspectives on Measurement
All three CMMI models—CMMI-ACQ , CMMI for Services (CMMI-SVC) , and CMMI-DEV —include a process area for Measurement and Analysis. "The purpose of Measurement and Analysis (MA) is to develop and sustain a measurement capability that is used to support management information needs ." There are eight specific practices recommended in the models: 1) Establish Measurement Objectives; 2) Specify Measures; 3) Specify Data Collection and Storage Procedures; 4) Specify Analysis Procedures; 5) Collect Measurement Data; 6) Analyze Measurement Data; 7) Store Data and Results; and 8) Communicate Results .
Four categories of measurement are common to many acquisition programs in which MITRE is involved:
- The quantitative performance requirements of user requirements and system performance requirements are measured in the operational and development test programs. Suggestions on key performance parameters can be found in Enclosure B of the Chairman of the Joint Chiefs of Staff Manual on the Operation of the Joint Capabilities Integration and Development System .
- Technical Performance Measurement (TPM) monitors the developer's progress in meeting critical performance requirements over the life of the program where there is a development risk. The concept is further explained on the Office of the Secretary of Defense (OSD) TPM web site .
- Earned Value Management (EVM) monitors a developer's cost and schedule performance in cost reimbursement development contracts. Additional information can be found in the Earned Value Management article in this Systems Engineering Guide and in EIA-748A and the OSD EVM web site [17, 18].
- Process Metrics are associated with development processes like software development. A good approach to identifying the type of measurement needed and the proven metrics that support that measurement can be found on the Practical Software and System Measurement web site .
There is a fifth category that may be involved if MITRE is assisting in developing performance-based logistics criteria for operations and maintenance efforts. OSD recommends five performance parameters: 1) Operational Availability; 2) Operational Reliability; 3) Cost Per Unit Usage; 4) Logistics Footprint; and 5) Logistics Response Time. These are cited in an OSD Memo on the subject .
For additional information, refer to the articles Acquisition Management Metrics and How to Develop a Measurement Capability in this Systems Engineering Guide.
Articles Under This Topic
The article Establishing a QA Program in the Systems Acquisition and/or Government Operational Organization provides guidance on processes to assure that the right product is being built (customer-driven quality), that the product being built will meet its specified requirements (product-based quality), and that the product is suitable for its intended use (user-based quality).
The article How to Conduct Process and Product Reviews Across Boundaries provides guidance on assisting government and contractor organizations in documenting quality processes and work product specifications, and reviewing those processes and products.
References & Resources:
- CMMI-DEV, Version 1.2.
- MITRE Systems Engineering (SE) Competency Model, Version 1, September 1, 2007, The MITRE Institute, Section 3.7, Quality Assurance and Measurement, pp. 45-46.
- Evans, J. R. and W.M. Lindsay, 2008, Managing for Quality and Performance Excellence, 7th Edition, Thomson, Southwestern.
- The American Society for Quality (ASQ) website.
- ISO 9000:2000, 2000. Quality Management Systems, Fundamentals and Vocabulary, Second Edition.
- ISO 9001:2000, 2000, Quality Management Systems, Third Edition.
- CMMI-ACQ, Version 1.2.
- CMMI-DEV, Version 1.2.
- Matchette, Daniel R., "Six Sigma for the DoD," Defense AT&L Magazine, July-August 2006, Vol. 35, No. 4, p. 19–21.
- DoD 5012.42, May 15, 2008, DoD-Wide Continuous Process Improvement (CPI)/Lean Six Sigma (LSS) Program.
- The Secretary of the Air Force and Chief of Staff of the Air Force, February 7, 2006, Air Force Smart Operations for the 21st Century CONOPS and Implementation Plan, Version 4.
- CMMI-ACQ, Version 1.2.
- CMMI-SVC, Version 1.2.
- CMMI-DEV, Version 1.2.
- CJCSM 3170.01C, Operation of the Joint Capabilities Integration and Development System.
- OSD Technical Performance Measurement web site.
- Earned Value Management, ANSI EIA-748A Standard (June 1998 ed.).
- OSD Earned Value Management web site.
- Practical Software and System Measurement web site.
- OSD AT&L Memo, 16 Aug 2005, Performance Based Logistics: Purchasing Using Performance Based Criteria.
Additional References and Resources:
Metzger, L., May 2009, Systems Engineering Quality at MITRE, The MITRE Corporation.
The MITRE Corporation, August 21, 2009, "Quality," MITRE Project Leadership Handbook.
For more information on the Systems Engineering Guide, or to suggest an article, please Contact Us.