MITRE FFRDC Independent Assessments


Definition: An independent assessment is a tool that can be used at any point in a program life cycle to provide insight into progress and risks.

Keywords: audit, baseline assessment, independent expert review, independent technical assessment, red team, SCAMPI appraisal, Tiger Team

MITRE SE Roles and Expectations: MITRE systems engineers (SEs) are expected to be able to lead or participate in independent assessment teams, particularly when program processes are being evaluated or there are concerns about program progress or contractor performance [1]. MITRE SEs are expected to apply strong domain and technical expertise and experience and perform with objectivity consistent with MITRE's role in managing federally funded research and development centers (FFRDCs).

Context

An Independent Assessment is a team activity in which the team leader and team members are not members of the organization being assessed. This allows the team to more readily fulfill its charter objectively and without conflicts of interest. Ideally, assessments are proactive and intended to provide an early look into what potential problems may be on the horizon in time to take action and avoid adverse impact to the program. For example, an assessment may be used to take an early look at the challenges associated with technologies and provide feedback to the technology development strategy. In other cases, an assessment might be intended to assess progress with a process improvement framework such as CMMI (Capability Maturity Model Integration) [2], or it may be intended to examine causes of concerns with program performance.

In MITRE's work, independent assessments are known by several names, including:

  • Independent Reviews
  • Red Teams
  • Blue or Green Teams
  • Technical Assessments (Tiger Teams)
  • Appraisals (against a model or standard)
  • Independent Expert Reviews (IER)
  • Audits and Compliance Assessments
  • Accident Investigations

For related information, see the article "Planning and Managing Independent Assessments" in this topic, and "Data-Driven Contractor Evaluations and Milestone Reviews" and "Earned Value Management" in the Contractor Evaluation topic in the "Acquisition Systems Engineering" section.

MITRE SEs are frequently asked to lead and participate in independent assessments because the characteristics of an FFRDC, as chartered under the Federal Acquisition Regulation (FAR) 35.017 [3], promote independence, objectivity, freedom from conflicts of interest, and technical expertise. These characteristics support the management goals of the assessment team.

For example, the FAR describes the special relationship between a sponsor and its FFRDC [3]. The FFRDC:

"... meets some special long-term research or development need which cannot be met as effectively by in-house or contractor resources...to accomplish tasks that are integral to the mission and operation of the sponsoring agency."

"...has access, beyond that which is common to the normal contractual relationship, to government and supplier data, including sensitive and proprietary data, and to employees and facilities."

"...conduct[s] its business...to operate in the public interest with objectivity and independence, to be free from organizational conflicts of interest, and to have full disclosure of its affairs to the sponsoring agency."

Best Practices and Lessons Learned

MITRE's assessment teams have experienced some challenges, including: [4]

  • Sponsor oversight has been delegated leading to a lack of clarity about what needs to be accomplished.
  • The review is additional work and seen as a lower priority.
  • Members of the organization being reviewed are not able to participate as planned.
  • Subjects of the review are not prepared for the review.
  • Objective evidence is difficult to locate.
  • Appraisal team working space is rarely available.

MITRE practitioners have found that these and other challenges can be mitigated through communication and following the process described in the Planning and Managing Independent Assessments article.

References and Resources

  1. The MITRE Institute, September 1, 2007, MITRE Systems Engineering (SE) Competency Model, Version 1, p. 39.
  2. CMMI Institute, December 2014, Standard CMMI Appraisal Method for Process Improvement (SCAMPI): Method Definition Document for SCAMPI A, B, and C, Ver. 1.3b.
  3. Federal Acquisition Regulation (FAR) 35.017, Federally Funded Research and Development Centers, accessed July 21, 2014.
  4. Clapp, J. A., and P. G. Funch, March 5, 2003, A Guide to Conducting Independent Technical Assessments, MITRE Center for Air Force C2 Systems.

Additional References and Resources

The Project Management Institute, 2013, A Guide to the Project Management Body of Knowledge, (PMBOK Guide), 5th Ed.

Publications

Download the SEG

MITRE's Systems Engineering Guide

Download for EPUB
Download for Amazon Kindle
Download a PDF

Questions?
Contact the SEG Team