• Download in Microsoft Word

Title: Risk Metrics for Initial Investment Analysis (JRC 2(a))
Type (Standard or Guideline): Standard
Approval Date: Approval July 2002

Purpose: The Standard Risk Metrics form the investment risk standards. They should be used to ensure quality and uniformity of practice, unless there is a compelling documented case to do otherwise.

Applicability: The Standard Risk Metrics are required to be used by all FAA entities and organizations pursuing an Initial Investment Decision (JRC 2(a)) for any system acquisition. For a facility acquisition, these standard metrics would need to be tailored and modified prior to beginning the risk analysis. Work continues on modifying this standard to address facility, security, and environmental issues. Accordingly, this standard should be viewed as an evolving one, and constantly updated to remain more relevant to the context and nature of all IAs. For the Initial Investment Decision, the Joint Resource Council considers the merits of all investment alternatives which meet the requirements and down-selects to the best candidate solution.

Description: The standard metrics are the risk facets of technical, operability, producibility, supportability, cost estimating, benefit estimating, schedule, management, funding, stakeholder, information security, human factors, and safety. Within each facet is a checklist of potential issues to be considered for each alternative. The number of issues identified as affecting the alternative under consideration is an indicator of magnitude of risk. This measure of risk forms the basis for the application of a methodology to assess the level, consequence, and mitigation of risk.

Content: The Standard Risk Facets, and Risk Checklist are defined and presented below.

Configuration Management: This Standard will be maintained by the Investment Analysis and Operations Research Directorate (ASD-400) in the FAA Acquisition System Toolset (FAST). It will be reviewed annually by ASD-400 for currency with acquisition environment. If any changes are necessary, they will be proposed by ASD-400 through the normal ASAG/ FAST change process.

Risk is defined as an event whose outcome is unknown in advance. In this context, risk implies some probability that an acquisition alternative will fail to deliver the benefits projected, either in whole or in part, and that failure may have consequences to its success. The risk can derive from uncertainties in an alternative's concept or problems encountered during design, development, implementation, or operation.

For the Initial Investment Analysis Decision (JRC-2a), the critical issue to be answered is which of the viable acquisition alternatives is worth examining in more detail for implementation. Consequently, the risk assessment for the Initial Investment Decision is necessarily done at the macroscopic, and comprehensive levels.

The process includes

• Identification of risk issues,
• Development of potential mitigation strategies,
• Assessment of investment alternatives, and
• Affirmation of mitigation strategies to be examined for cost or benefit estimation impact.

In general, within the IAT Alternatives Risk Assessment there are thirteen facets that constitute an exhaustive range of potential risk areas, including: technical, operability, producibility, supportability, benefit estimate, cost estimate, schedule, management, funding, stakeholder, information security, human factors, and safety. Risk issues may be identified by discussion with program staff, users, stakeholders, or derived from secondary written sources. Usually, in the course of identifying risk issues, mitigation actions come to mind and are written down as preliminary mitigating actions. Also at this point, the discussions or secondary information suggest the likelihood of the issue surfacing and the potential severity of impact of the issue, if unmitigated.

The next step is to convene a broad team of stakeholders, programmatic, investment analysis, union representatives, and other interested colleagues. A broad team is important because information is needed from multiple perspectives. Only in so doing is it possible to get an objective assessment. Together the team reviews the material and by double blind voting and justifying discussions arrives at consensus of each facet’s rating for each alternative, and affirms or complements issue mitigation strategies or actions. In a double blind voting procedure, each group member votes before and after group discussions without knowing how other group members are voting. At the end of these discussions, the relative risk levels of alternative s emerge.

This information is provided to the IAT. In turn, the IAT uses this information and other inputs from cost, benefit, and other analyses, to put together its recommendation for down-selection.

The risk analyst then identifies whether the mitigation strategies are likely to impact the cost estimation or the benefit estimation of the down-selected alternative, and works with appropriate staff to estimate the impact within the appropriate work breakdown structure categories.

The product of this effort may be a technical memorandum, but most often is:

• A stand-alone summary section in the Investment Analysis Report;
• Solid input for creating cost and benefit high confidence ranges around the most likely numbers, and
• Direct input into the Risk Management Matrix in the Initial Acquisition Program Baseline. The Risk Management Matrix may be found in the FAA Acquisition System Toolset (www.fast.faa.gov , APB Template, Section 6, Table 6.1).

Lastly, the issue ratings are summarized into a matrix for the JRC, and for continued analysis in the Solution Implementation Phase.

The life cycle risks are broken down into thirteen components, or facets, of risk, which are used to assess the overall risk. These risk facets have been selected to reflect the comprehensive risks associated with an acquisition alternative’s development, implementation, operation, in such a way as to achieve the projected benefits. The facets are intended to facilitate the identification of risk and its quantification. The thirteen risk facets are defined as follows:

• RiskTechnical is the risk associated with (1) developing a new or extending an existing technology to provide greater performance than previously demonstrated, or (2) achieving a level of performance. It also refers to how well the system operates to design or safety specifications.
• RiskOperability is the risk associated with how well the system to be produced will operate within the NAS and interact with other systems. It addresses NAS or other system interfaces, the degree to which they are known and complete, and the degree to which the operational concept has been demonstrated and evolved to the point of a design baseline.
• RiskProducibility is the risk associated with the capabilities to manufacture and produce the desired system.
• RiskSupportability is the risk associated with fielding and maintaining the resulting systems.
• RiskBenefit Estimate considers the difficulty in estimating the benefits and in realizing benefits. This risk facet addresses the accuracy and uncertainty of the benefit estimate, including such issues as inadequate methods to estimate the benefits, lack of data to estimate the benefits, uncertainty of assumptions, and whether the alternative is defined enough to estimate the benefits.
• RiskCost Estimate considers the difficulty in estimating the cost and in adhering to the cost. This risk facet addresses the accuracy of the cost estimate, including such issues as inadequate methods to estimate the cost, lack of data to estimate the cost, uncertainty of assumptions, and whether the alternative is defined enough to estimate the cost.
• RiskSchedule considers the likelihood that the alternative will be completed within the specified schedule.
• RiskManagement refers to complexity of the alternative to manage (e.g., number of sub-tasks and/or number of performing organizations) and considers the risks of obtaining and using applicable resources and activities that may be outside of the alternative's control but can affect the alternative's outcome.
• RiskFunding addresses the availability of funds when they are needed and a confidence in management and Congress that those funds will continue to be provided.
• RiskStakeholder is the risk associated with various stakeholders supporting the development and operation of the alternative, such as internal FAA organizational users, Congress, airline and general Aviation users, and potential equipment and aircraft manufacturers.
• RiskSecurity addresses a system's vulnerability to external threats and the risks likely to occur in employing countermeasures. This pertains to both information and physical issues.
• RiskHuman Factors focuses on the effectiveness and suitability of the human-in-the-loop aspects of the system with respect to both operations and maintenance.
• RiskSafety considers the risk associated with the performance (or lack thereof) of appropriate safety risk management activities and in implementing the identified safety requirements. This risk facet shall not be confused with the operational risk of system hazards. The operational risk of system hazards are documented in other analyses.

For the Initial Investment Analysis, measuring risk begins with the identification of the number of specific risk issues for each risk area. The identification of risks is conducted in an organized and comprehensive manner.

Risk identification involves enumerating and describing the risks pertaining to each of risk facet for successfully carrying out each of the alternatives being considered. The number of risks identified constitutes the measure, or quantification of risk. The measure of risk is not to be confused with the assessment of risk. The assessment of risk for each alternative follows a set methodology which considers the probability of occurrence of risk and the severity of impact should that risk materialize. Assessment of risk will be covered in a subsequent product of the risk standardization effort, called Risk Methodology.

For purpose of risk enumeration, Table 1: Facet Risk Checklist, is included in the following several pages. For each of the thirteen (13) risk facets already identified as being relevant to the Initial Investment Analysis Decision, a comprehensive set of risk issues are listed. The analyst should examine each alternative to determine whether each of the bulleted issues is present. The number of risk issues present and their detailed description determine the metric for risk. The risk issues that are found to be present are then the basis of further research. Further research would include a risk assessment to determine the level of risk, usually by understanding the probability of the occurrence of risk, and the severity of their impact.