8.1 Aviation Environment Changes

8.2 Methodological Changes

1.1 Acquisition Management System

The Federal Aviation Administration (FAA) Acquisition Management System (AMS) establishes policy and guidance for all aspects of the acquisition life cycle. The AMS simplifies, integrates, and unifies the elements of life cycle acquisition management. It increases the quality, reduces the time, and decreases the cost of delivering needed services to its customers.

The AMS has several phases including:

• Mission analysis,

• Investment analysis,

• Solution implementation,

• In-service management, and

• Service life extension.

The Investment Analysis (IA) Team Alternatives Risk Assessment is conducted during the Investment analysis phase.

1.2 Investment Analysis Phase

The IA phase begins when the FAA determines a potential need to expend funds to meet a mission capability shortfall or to take advantage of a technological opportunity. During the IA phase, the sponsoring and acquiring organizations are partners in ensuring the critical needs of the users are satisfied by an affordable solution. The IA phase ends with a decision on how the requested funds are allocated.

Major activities in the IA phase are:

• Identifying and analyzing candidate solutions,

• Developing an Investment Analysis Plan (IAP),

• Assessing affordability,

• Developing an Acquisition Program Baseline (APB),

• Writing an Investment Analysis Report (IAR), and

• Recommending the best investment alternative.

In the IAR, the IA Team analyzes alternatives and candidate solutions. Of the alternatives identified, those determined viable become candidates for detailed analysis. The IA Team evaluates candidate solutions by compiling and analyzing such factors as cost, benefit, schedule, and risk. The IA Team documents the results in an IAR that contains comprehensive, quantitative data for each candidate solution.

During the IA phase, the IA Team conducts the Alternatives Risk Assessment, as referenced in AMS policy sections 2.9.13, 2.4.4.1 and 2.4.4.2, which identifies risks and helps to establish realistic baselines that include funding for the risk mitigation.

1.3 Risk Analysis throughout the AMS

The following describe key elements of risk management at phases in the AMS:

Mission Analysis: Identify and characterize risks to the agency’s ability to execute its legislated responsibilities and satisfy customer demands for service. Typically, these risks arise from changes in the operational environment and shortfalls in operational capability.

Investment Analysis: Ensure that primary risks associated with candidate solutions are fully identified and evaluated. Sufficient time and money must be included for each candidate solution in the APB to mitigate risk and achieve program success.

Program Implementation: Detect and reduce risks early to avoid greater cost of risk consequences later in the life cycle. The Integrated Product Teams (IPTs) manage risks throughout the implementation phase of their products and services. A Risk Management Plan or risk planning section is developed for the Acquisition Strategy Paper (ASP), and risk-mitigations are documented in the Integrated Program Plan (IPP). Risk management requirements and activities are also included in any prime contract for products or services.

In-Service Management: Assess and mitigate the risks continually and adjust mitigation plans. The Risk Mitigation Plan contains actions to scan the program’s operational environment to understand any changes in trends that impact the following:

• The operation of the program in the NAS,

• The timing and progress of implementing the committed mitigations, and

• The risk status of the program.

If the risk status of the program worsens, the In-Service Management organization will develop additional mitigations.

2.1 General

Risk analysis anticipates uncertainties in a potential capital investment, assesses the degree of the risk, and identifies mitigations. Risk mitigation is critical to an investment because it increases the odds of a program being successful.

Paragraph 2.9.13 of the Life Cycle AMS states:

"Risk management is applied throughout the acquisition management life cycle to identify and mitigate risks associated with achieving agency goals and objectives. Each line of business shall institute risk management processes that: (1) identify and assess risk areas; (2) develop and execute risk mitigation or elimination strategies; (3) track and evaluate mitigations; and (4) continue mitigations until risk is eliminated or its consequences reduced acceptably."

2.1.1 Risk at Each Phase of the AMS

Risk considerations vary in specificity and focus at each phase of the AMS as knowledge of the investment varies. These phases include:

• Mission Analysis - ATS and AIO have begun to explore the use of a portfolio management approach.

• Investment Analysis - ARA’s ASD has adopted an investment risk analysis approach.

• Solution Implementation - ARA’s IPT’s have adopted a programmatic risk assessment approach.

2.1.2 Extending Across the Phases of the AMS

Extending across all the phases of the AMS are areas that warrant separate assessments. These areas include:

• Human factors (AAR),

• Safety (ASD and ASY), and

• Security (ASD, AIO, AIS, and ACS).

These areas call for a more detailed examination of risks in each phase of a potential investments acquisition.

This report recognizes that the AMS has many approaches to risk assessments, and seeks to harmonize and integrate them. Figure 2-1 on the next page illustrates an approach to harmonization.

.

Risk assessment during the mission analysis is a component of portfolio management. The portfolio view examines the overall range of programs, i.e., the big picture. It allows the managers to pick the optimum set of programs by taking into account the organization’s goals and how the organization can meet those goals.

Early in the AMS, at the Joint Resources Council (JRC-1) mission need decision; the JRC approves a project for further development if it is consistent with the FAA mission, and if there is a need for that project. The decision is made with a full awareness of other projects being implemented, and it implies that the project is consistent with the package of programs already underway.

Portfolio management identifies the possible benefits of a program or a package of programs and facilitates tracking the delivery of the benefits.

A questionnaire, which addresses the following questions, is used to help determine program risk.

• Are we doing the right things?

• Are we doing them the right way?

• Are we getting them done well?

• Are we getting the benefits?

2.3 Risk Assessment during Investment Analysis

For the Initial IA Decision at JRC-2a, the critical issue is determining the most viable acquisition alternatives worthy of further analysis. Consequently, the risk assessment for the Initial IA Decision is macroscopic and comprehensive. The process includes the following:

• Identifying risk issues,

• Identifying mitigation strategies,

• Assessing investment alternatives, and

• Affirming mitigation strategies that will affect cost or benefit estimates.

2.3.1 The IA Team Alternatives Risk Analysis – JRC-2a (Initial IA Decision)

In general, within the IA Team Alternatives Risk Assessment, there are thirteen facets that constitute an exhaustive range of potential issues, which include: technical, operability, producibility, supportability, benefits estimate, cost estimate, schedule, management, funding, stakeholder, security, human factors, and safety process. Discussion with program staff, users, and stakeholders can help identify risk issues. In the course of identifying issues, the IA Team often formulates and records preliminary mitigations, and estimates the likelihood of the issue surfacing and the potential severity of the issue.

The next step may be to convene a team of stakeholders, investment and program analysts, union representatives, and other interested colleagues. A team comprised of a broad skill-set helps to form an objective assessment. The team reviews the material and arrives at consensus of each facet’s rating for each alternative by double blind voting (independent votes not visible to others) and justification discussions. The team also affirms or complements issue mitigations. At the end of these discussions, the least-risk alternative emerges. Depending on the complexity of the investment, a full team analysis may not be necessary (see Section 5.0, Tailoring the Process).

The analyst then coordinates with the cost, benefit, and other analysts to link the impact of the risks and their mitigation on the cost and/or the benefit estimate of the recommended alternative. This impact contributes to developing ranges around the most likely cost or benefit element in the respective Work Breakdown Structure (WBS).

The products of this effort may be a technical memorandum, provided by the Risk Assessment Team Lead, but most often are:

• A standalone summary section in the IAR,

• Solid input for creating cost and benefit high confidence ranges around the most likely numbers,

• Direct input into the Risk Management Matrix in the Initial APB, and

• Risk Issues Database.

Lastly, the IA Team summarizes the issue ratings in a matrix for the JRC, and the risk issues database forms the foundation for continued analysis in the Solution Implementation phase.

2.3.2 The IA Team Life Cycle Risk Analysis - JRC-2b (Final IA Decision)

For the Final IA Decision at JRC-2b, the risks for the preferred alternative are identified and costed in detail to ensure successful implementation of the program. The detailed examination includes: (1) providing a technical description of the risk issues, (2) quantifying the impact on cost, schedule, benefit, and performance baselines, and (3) describing the reason for the expected change. The detailed analysis is based on either past performance history or estimates from a simulation or scenario analysis. The Final APB includes the funding needed to mitigate risks threatening the program’s success. Consequently, the risk analysis for the Final Investment Decision is more focused. At this stage, the IPT Programmatic Risk Assessment is complemented with the remaining IA risk issues.

The IPT Programmatic Risk Assessment reviews the JRC-2a risk issues and incorporates the relevant issues. In addition, since the APB includes benefit and performance elements, the IPT working with the IA Staff, complements the focus on cost, schedule, and technical elements by adding the benefits and the performance elements.

As envisioned, the IPT Programmatic Risk Assessment begins with the information gained for the Initial IA Decision at JRC-2a and examines all issues from the perspective of the Final IA Baseline Metrics. The metrics are cost, schedule, benefit, and performance. For the Final IA at JRC-2b, the focus is on the issues implications to cost, schedule, benefits, and performance. In the Final IA phase, any interdependency risks (i.e., risks linked to the implementation of one program on the success of other programs) will also be examined and mitigated.

The overall IPT Programmatic Risk Management Process entails the following steps:

• Identifying risks,

• Assessing risks,

• Selecting risk mitigation options,

• Developing a Risk Mitigation Plan, and

• Establishing the groundwork for monitoring and tracking risk.

Identifying risks focuses on staffing problems, design changes, operational issues, test changes, negative trends, etc. Assessing risks focuses on understanding the size and impact of the risk. Risk mitigation focuses on what to do with the risk (avoid, transfer, assume, or mitigate). The Risk Mitigation Plan includes necessary changes in budgets, mitigations, and communication with stakeholders to get support for mitigations.

If there are insufficient resources to mitigate the risk issues, the program will probably rebaseline and review all the risk issues.

Lastly, the groundwork for monitoring and tracking includes expected timing for the mitigation action and expected impact for reducing risk. This allows tracking of mitigation actions, assessment of their impacts, and the additional follow-up remedial actions. The IPT Programmatic Risk Assessment focuses on cost, schedule, performance, and technical aspects of the preferred alternative from the JRC-2a decision.

The Risk Mitigation Plan is carried out during the Solution Implementation and In-Service Management Phase. It is a product of JRC-2b and contains actions to scan the program’s operational environment to understand any changes in trends that impact the following:

• The operation of the program in the NAS,

• The timing and progress of implementing the committed mitigations, and

• The risk status of the program.

The fact that each phase of the AMS has a risk assessment process underscores the importance of risk assessment in the AMS. While each phase’s risk assessment fulfills functional needs, there remains a compelling need to ensure an integrated framework with a smooth, continuous flow of risk and mitigation information.

Portfolio management, as of this writing, is only beginning to be applied. Typically, a questionnaire is used to interview project knowledgeable experts about potential risks. With little effort, the survey can be expanded to cover all of the thirteen facets of the IA phase, e.g., operability, human factors, benefit estimate, supportability, and producibility.

The IA Staff may use the output of portfolio management to determine the more significant risks of a program and focus on the mitigation of these risks.

The IA Team Alternatives Risk Assessment considers all risk facets, including human factors, safety process, and security risks. Nevertheless, human factors, safety, and security may conduct separate assessments. The consideration of issues in each of those separate assessments should involve those organizations responsible for conducting them, either directly through a special meeting, and/or by their designated representatives participation in the IA Team Alternatives Risk Assessment, or indirectly through guidance materials from their offices.

The Risk Team Lead for the IA is responsible for alerting the appropriate human factors, safety, and security offices in the FAA to ensure that their assessments or inputs can be adequately integrated into the IA Team Alternatives Risk Assessment.

3.2.1 Human Factors Assessment

The Human Factors Assessment for Investment Analysis entails a process that provides essential and inter-related components to the products of the Investment Analysis Team, including: a) the human-system performance contribution to program benefits, b) an assessment of the human-system performance risks, and c) the estimated costs associated with mitigating human factors risks and with conducting the engineering program support. The human factors components related to benefits, risks, and costs are integrated with other program components in the IA products and documentation to ensure that

3.3.1 Initial Investment Decision at JRC-2a

The IA Team Alternatives Risk Assessment contains a high-level comparative assessment between all alternatives and across all risk facets. The products of the assessment are a risk assessment briefing, technical memorandum, and/or a risk management matrix. The risk management matrix is required for the Initial APB.

These products should be comparable in detail with the IPT Programmatic Risk Assessment. The list of major risks and mitigation strategies identified for the preferred alternative are examined in greater detail at the Final IA Decision to form an APB that is stable throughout the Solution Implementation phase of AMS.

3.3.2 Final Investment Decision at JRC-2b

The products of the IA Team Alternatives Risk Assessment feed into the IA Team Life Cycle Risk Assessment and gives the IPT a starting point for the IPT Programmatic Risk Assessment.

The Office of Architecture and System Engineering and the Office of Investment Analysis and Operations Research, in collaboration with the IA Team, will provide input on the assessment of cost, benefit, schedule, performance, and interdependency risks.

The product of the IA Team Life Cycle Risk Assessment is the identification of detailed cost, benefit, schedule, technical, and performance risks, and respective mitigation strategies. These mitigation strategies are discussed with subject matter experts to estimate their impacts on cost, schedule, benefit, and performance baseline that the JRC is to approve in the Final IA Decision.

Another product of the IA Team Life Cycle Risk Assessment is the Risk Mitigation or Management Plan, which guides the understanding and mitigation of risks throughout the Solution Implementation and subsequent phases of the AMS.

• RiskTechnical is the risk associated with (1) developing a new or extending an existing technology to provide greater performance than previously demonstrated, or (2) achieving a level of performance. It also refers to how well the system operates to design or safety specifications.

• RiskOperability is the risk associated with how well the system to be produced will operate within the NAS and interact with other systems. It addresses NAS or other system interfaces, the degree to which they are known and complete, and the degree to which the operational concept has been demonstrated and evolved to the point of a design baseline.

• RiskProducibility is the risk associated with the capabilities to manufacture and produce the desired system.

• RiskSupportability is the risk associated with fielding and maintaining the resulting systems.

• RiskBenefit Estimate considers the difficulty in estimating the benefits and in realizing benefits. This risk facet addresses the accuracy and uncertainty of the benefit estimate, including such issues as inadequate methods to estimate the benefits, lack of data to estimate the benefits, uncertainty of assumptions, and whether the alternative is defined enough to estimate the benefits.

• RiskCost Estimate considers the difficulty in estimating the cost and in adhering to the cost. This risk facet addresses the accuracy of the cost estimate, including such issues as inadequate methods to estimate the cost, lack of data to estimate the cost, uncertainty of assumptions, and whether the alternative is defined enough to estimate the cost.

• RiskSchedule considers the likelihood that the alternative will be completed within the specified schedule.

• RiskManagement refers to complexity of the alternative to manage (e.g., number of sub-tasks and/or number of performing organizations) and considers the risks of obtaining and using applicable resources and activities that may be outside of the alternative's control but can affect the alternative's outcome.

• RiskFunding addresses the availability of funds when they are needed and a confidence in management and Congress that those funds will continue to be provided.

• RiskStakeholder is the risk associated with various stakeholders supporting the development and operation of the alternative, such as internal FAA organizational users, Congress, airline and general Aviation users, and potential equipment and aircraft manufacturers.

• RiskSecurity addresses a system's vulnerability to external threats and the risks likely to occur in employing countermeasures. This pertains to both information and physical issues.

• RiskHuman Factors focuses on the effectiveness and suitability of the human-in-the-loop aspects of the system with respect to both operations and maintenance.

• RiskSafety considers the risk associated with the performance (or lack thereof) of appropriate safety risk management activities and in implementing the identified safety requirements. This risk facet shall not be confused with the operational risk of system hazards. The operational risk of system hazards are documented in other analyses.

4.2 Risk Facet Inclusion and Flexibility

If the risk assessment team determines a facet to be inconsequential, the facet can be eliminated, but only if no perceptible risks for the facet can be determined. A risk checklist (see Table 6-2) can be used to assist the analysis in determining whether a facet should be excluded. The rationale for excluding any facet should be documented.

The risks associated with each facet should consider the whole life cycle of the investment. Therefore, the thirteen risk facet definitions must be flexible enough to include the development of the investment through the full life cycle. Facet definitions need to be flexible because they must cover all potential risk situations or program evolutionary stages. Some programs may be at such a preliminary stage that the risk facet definitions may be difficult to apply. The risk team may need to adjust and agree to definitions to handle more mature programmatic risks and preliminary investment risks.

4.3 Clarification of Risk Facets for Benefits and Costs Estimates

4.3.1 Symmetrical Risks Associated with Uncertainty

Risk may increase or decrease an estimate for benefit performance and cost values because they may either ameliorate or worsen the behavior of a system. Therefore, risk considerations should include both sides of the uncertainties associated with the estimates. One way to capture this in the IA is to include high and low estimates or a probabilistic distribution around both costs and benefits. If the benefits are underestimated, the risk lies in the denial for requested funding of the program. Similarly, if costs are overestimated, it is possible that the program will not be funded or another alternative will be chosen with fewer capabilities. If both cases occurred simultaneously, benefits were underestimated and costs overestimated, either the alternative will be different (as well as the accompanying capabilities) and/or the project may not be funded at all.

4.3.2 Uncertainty of Cost and Benefit Assumptions, and Qualitative Benefits

Risk benefit estimate considers the difficulty in estimating the benefits and in realizing benefits. This risk facet addresses the accuracy of the benefit estimate and issues such as:

• Inadequate methods to estimate the benefits,

• Lack of data to estimate the benefits,

• Whether the link of the alternative to projected benefits is tenuous,

• Whether the alternative is defined enough to estimate the avoided cost,

• Uncertainty of the major assumptions used in both cost and benefit estimates, and

• Realization of benefits.

Qualitative benefits should be characterized in as much detail as possible, and their uncertainty projected. Often qualitative benefits cannot be measured in financial terms, or in physical numerical terms. This becomes important when there are potentially large unquantifiable qualitative benefits, such as enabling benefits from advanced technology. Therefore, the risks to the accuracy of the benefits calculations should be included in the risk assessment.

4.3.3 Cost and Benefit Facet Risk Uses

The IA Team Alternatives Risk Assessment provides more information about the uncertainty in the cost and benefit estimates. This information helps to establish the cost and benefits ranges, which in turn are used to develop the high confidence cost and benefits.

The risk mitigation costs can be included in the total cost figures and for traceability and validation should be closely linked to the WBS for costs. Similarly, risk ranges can be developed for the benefits calculations by estimating uncertainties for all benefits assumptions.

4.4 IA Team Alternatives Risk Assessment Process

4.4.1 Matrix Table Results of Probability and Severity

A frequency analysis using a matrix risk table with probabilities and severities on the X and Y coordinates should be constructed. With three levels of probabilities and three levels of severity, development of the risk matrix will yield a 3-row by 3-column table. According to the Risk Facet Weighting Score Assignment in Section 6, those risks that are high correspond to (high probability with substantial severity); (medium probability with substantial severity); and (high probability with moderate severity). The definitions of medium and low overall ratings are in Section 6. This can be done for individual risks, as well as by risk facet. By applying this technique to all of the individual risks and placing their tallied number in each box for probability and associated severity, the decision-maker can see the total number of individual

risks and their distribution through various combinations of both probability and severity. For example, in Figure 4-1, the number 6 in the cell corresponding to high probability and substantial severity denotes that there were 6 total risks with this probability and severity. The sum of all of the numbers in the table represents the total number of risks evaluated. Please see Section 6 for more detail on the definitions of probability and severity associated with each risk facet.

Figure 4-1: Example of Risk Facet Rating Matrix Severity of Impact

Alternatively, the matrix could be applied to risk facets, and by tallying the facets for each box in the matrix table, a distribution of the thirteen facets can be clearly evaluated. Furthermore, the risk facets can be given an initial corresponding to the risk facet placed in each box in the table. Additional information will allow the decision-maker to view the highest-ranked risk facets.

4.4.2 Principles to Ensure Objectivity

Objectivity is essential to a successful risk assessment. To ensure objectivity, the risk team evaluators must have an open mind and have appropriate knowledge and experience. Moreover, the IA Team must be balanced by a broad perspective of members (i.e., users, airline representatives, unions, sponsor(s), implementers, and any other stakeholder).

Group review of individual risks is important to ensure objectivity as it allows those with broader experience to add a different perspective and provide specific information regarding risk mitigation strategies. Part of the review may include briefing other interested parties or stakeholders throughout the agency on the results.

All risks should be verified by either data and/or supporting documentation via risk databases, studies, reviews, or publications. Increasing the credibility, validity, and traceability of the risk assessment will enhance the objectivity of the assessment.

5.0 Tailoring the Process

The IA Team Risk Lead is responsible for tailoring the Risk Assessment to fit a respective acquisition. This tailoring involves evaluating the acquisition to determine the effort required, choosing an approach for the Risk Assessment, and selecting the members of the Risk Assessment Team, if necessary.

5.1 Criteria for Choosing the Level of Effort

Determining the level of effort required for a Risk Assessment involves a preliminary evaluation of the information currently available on a certain program or project. This evaluation involves determining the scope, magnitude, complexity, and schedule of the proposed acquisition, as well as the impact it will have on the NAS. Other factors to consider are FAA Advisory Board (FAB) waivers and any congressional mandates that may have been issued. Information in these areas can be obtained from documentation on the proposed technology, or documentation on the proposed program such as an Initial Requirements Document (IRD) or Concept of Operations document. Subject matter experts and program office personnel are also valuable sources of information in this preliminary evaluation.

5.2 Approaches

The IA Team Risk Lead determines the effort required and selects a corresponding approach.

5.2.1 Staff Reviews

When an acquisition is on the low end of scope, cost, and complexity, the IA Team uses staff reviews. This approach involves an internal ASD-400 review of documentation and comments on facet weights in Risk Issue Worksheets (see Appendix B) for any risks they have identified. The IA Team Risk Lead then integrates all risks into a Risk Issue Briefing and an assessment for the IAR.

5.2.2 One-Day Meeting

When an acquisition is thought to be low to mid-range in scope, cost, and complexity, the IA Team uses the one-day meeting approach. The IA Team Risk Lead facilitates this meeting involving the IA Team and representatives from contributing organizations. The IA Team Risk Lead briefs the Risk Assessment Guidelines for the Investment Analysis Process and distributes documentation. The IA Team and representatives from contributing organizations together complete Risk Issue Worksheets for any risks they have identified and come to consensus on facet weights. Risk assessment, based on the risk issues identified, is accomplished as part of this one-day meeting. Results may be available at the end of the meeting when the assessment is computerized. The IA Team Risk Lead then integrates all risks into a Risk Issue Briefing and an assessment for the IAR.

5.2.3 Sub Teams

When an acquisition needs more in-depth expertise to determine individual risks, mitigation strategies, and accompanying probabilities and severities, the IA Team uses the sub team’s approach. Sub teams and sub team leads are chosen for each facet from the IA Team, the sponsoring program office, and representatives from contributing organizations.

The IA Team Risk Lead facilitates an initial risk meeting involving the IA Team, the sponsoring program office, and representatives from contributing organizations. The IA Team Risk Lead briefs the Risk Assessment Guidelines for the Investment Analysis Process and distributes documentation.

The entire group comes to consensus on facet weights. Sub team leads are then chosen and each sub team lead completes a Risk Issue Worksheet for any individual risks that have been identified. The sub team leads make any necessary revisions to the worksheets and submit them to the IA Team Risk Lead. The IA Team Risk Lead then integrates all risks into a Risk Issue Worksheet and an assessment for the IAR.

5.2.4 Full IA Team Meetings

When an acquisition has a large scope, cost, and/or is of high complexity, the IA Team convenes a full IA Team meeting. The IA Team Risk Lead facilitates an initial risk meeting with the IA Team, the program office, representatives from contributing organizations, and subject matter experts. The IA Team Risk Lead briefs the Risk Assessment Guidelines for the Investment Analysis Process and distributes documentation. Sub teams and sub team leads are chosen from contributing organizations, the sponsoring program office, and the IA Team. Each sub team member then completes a Risk Issue Worksheet for any risks they have identified and submits them to the sub team lead. The sub team leads then makes any necessary revisions to the worksheets and submits them to the IA Team Risk Lead. The IA Team Risk Lead integrates all risks into a briefing for the second meeting where all risks, mitigation strategies and accompanying probabilities and severities are agreed to. The IA Team Risk Lead then integrates all risks into a Risk Issue Worksheet and an assessment for the IAR.

5.2.5 Team Membership

The selection of team members for a Risk Assessment should start with the IA Team members, then expand to include the sponsoring program office, contributing organizations, subject matter experts, and any other stakeholder. As previously discussed, the level of effort and approach will help determine the Risk Assessment Team membership.

5.2.6 How to Select and Where to Find Team Members

Risk Assessment Team Members can be selected by contacting major stakeholders in IA and the various divisions responsible for contributing different information and assessments to the IAR. Appendix A is provided for guidance as of April 2002 and is subject to change.

6.0 STEPS IN THE RISK ASSESSMENT PROCESS

The risk assessment process is comprised of the following eight steps:

1. Step 1 - Identify risks

2. Step 2 - Determine facet weights

3. Step 3 - Estimate severity of impact of an adverse event

4. Step 4 - Estimate probability of an adverse event

5. Step 5 - Assign risk ratings

6. Step 6 - Calculate overall alternative risk rating

7. Step 7 - Compare risks among alternatives

8. Step 8 - Transition to joint responsibility with the IPT for JRC-2b

6.1 Step 1 - Identify Risks

Risk identification is an organized and thorough approach to seek out risks. It is not a process of trying to invent highly improbable scenarios of unlikely events to cover every conceivable future possibility.

A Top-Level Risk Matrix (Table 6-1 shows a sample) may be used for each alternative to provide a structured and consistent risk identification process for the thirteen risk facets and to document the results. A Top-Level Matrix is useful for helping frame the development of risks, as the goals, plans, and risks are defined. This step is optional, if a framing context is unnecessary.

• Define Goals Relating to Risk Facets - Goals (for the alternative being assessed) that address potential risks in each risk facet are defined in the Top-Level Risk Matrix. By defining goals, as they relate to mitigating the potential risks in each risk facet, the specific risks that will be important to the alternative can be more easily identified. This information will also aid the process in Steps 2, 3, and 4 to quantify the risks.
  Requirements specified in an IRD should be considered in defining goals. If the requirements are not explicit enough to yield goals related to the risk facets, this is an alert that the requirements might need to be more explicit.

• Define Plans Relating to Risk Facets - Plan(s) for achieving the goals related to each risk facet, and hence mitigating risks, are listed in the Top-Level Risk Matrix. The Top-Level Risk Matrix serves as a forcing function to insure there are plans to address all goals.

• Identify Risks - Risk identification involves identifying the risks pertaining to each risk facet for successfully completing and implementing the alternative. The goals and plans related to each risk facet will aid in identifying the important risks.

  • See Appendix B, Risk Issue Worksheet, for a suggested template for recording risk issues. This worksheet contains: (1) risk issue, one per worksheet, (2) estimates of the probability and severity of the risk for various alternatives, (3) mitigations actions for the risk, (4) estimates for probability and severity for the risk after the mitigations have been carried out, (5) timing of the mitigation, and (6) elements of the work breakdown structure that the mitigation will affect.

The above three steps do not have to be sequentially completed. Iterating among the steps may be helpful. As more risks are listed, the goals and plans may be revised. The risks listed under each risk facet in Table 6-1 provide the basis for the risk quantification in Section 7. The statement of program goals relating to the risk categories and the plans for mitigating the risks will help quantify the risks. Risks cannot be assessed or managed until they are identified and described in an understandable way.

Table 6-1 is presented with sample entries in each box to clarify how the table is used. The sample entries are constructed for a possible alternative related to satellite surveillance.

Table 6-1: Top-Level Risk Matrix (Sample)

RISK FACET	RISK IDENTIFICATION
Technical	Goals: Plans: Risks:	To transition from ground-based radar surveillance to a joint satellite and ground-based surveillance system. Formulate requirements for, develop, and implement new technology to provide joint satellite and ground-based surveillance. ·1 Undue reliance on currently unavailable or unproved technology. ·2 No or minimal prototype testing. ·3 Inaccurate/simplistic modeling.
Operability	Goals: Plans: Risks:	To provide users and the FAA with operational benefits, such as the implementation of free flight. Determine the surveillance requirements of free flight and other advanced automation programs in order to provide a design that fully satisfies these requirements. ·4 Incompatibilities with future NAS systems. ·5 Incompatible or inconsistent operations with existing systems or regulations. ·6 Uncertain operational requirements of the other programs.
Producibility	Goals:   Plans: Risks:	To develop and manufacture ground-based and aircraft-based system components to meet requirements and be within the cost estimates. Use non-developmental items (NDI) and commercial off-the-shelf (COTS) items, and integrated NDI/COTS to the extent possible. ·7 Custom design and manufacture required.
Supportability	Goals: Plans:   Risks:	To provide support for both existing and new surveillance systems during transition to the new system. Closely coordinate with Airway Facilities (AF), including the field, and establish the Project Office within the appropriate Integrated Product Team.   ·8 Satellite support not under FAA control. ·9 Unclear Logistics Center responsibilities. ·10 Existing system may not be maintainable over the implementation period required for new system.
Cost Estimate	Goals: Plans: Risks:	To provide users and FAA with benefits, such as free flight, within estimated program cost. Implement cost control tools that will be used by the program office. ·11 Speculative life cycle costs. ·12 User avionics costs difficult to estimate.
Benefit Estimate	Goals: Plans: Risks:	To provide users and FAA with benefits, such as free flight, within estimated program cost. Implement benefit identification, estimation, and tracking tools that will be used by the program office. ·13 Difficult to identify benefits. ·14 Difficult to estimate benefits.
Schedule	Goals: Plans:   Risks:	To fully implement the new system by the year 20XX according to the schedule for the acquisition. Initiate the acquisition program at the earliest possible time. Implement and maintain a program office with separate staff and budget, and with the authority and responsibility for implementing new system. ·15 Insufficient schedule margin. ·16 Schedule sensitive to technical complexity. ·17 Uncertainties in contracting process. ·18 Excessive task concurrency.
Management	Goals:   Plans:   Risks:	To provide the implementation planning, resources, and controls needed to accomplish the development and implementation, while meeting the requirements, cost, and schedule estimates identified in the program plan. Implement and maintain a program office with separate staff and budget, and with the authority and responsibility for implementing new system. ·19 Inadequate program office staffing. ·20 Inadequate resource allocation. ·21 Inadequate authority. ·22 Undefined integration responsibilities. ·23 Unplanned slips in other programs. ·24 Excessive span of control. ·25 Uncontrolled requirements changes. ·26 Requirements freeze not enforced.
Funding	Goals: Plans: Risks:	To obtain the required development and implementation funding identified in the program plan in a timely manner. Obtain top-management support; reprogram available funding to get an early start on the acquisition alternative. ·27 Unfavorable agency priorities. ·28 Inadequate funding.
Safety	Goals: Plans: Risks:	To ensure that the appropriate safety risk management effort is implemented in the program. Identify safety objectives/ requirements/hazards and the criteria for acceptable risk for all ATM programs. ·29 Safety hazard and the associated safety requirements ambiguous, not fully characterized ·30 Interdependent relationships contributing to system failure not fully considered ·31 Acceptability criteria not fully known or understood for future NAS environments (e.g., free flight) ·32 Safety requirements not validated
Security	Goals: Plans: Risks:	To provide a security infrastructure to protect NAS programs. Formulate plans for the design, procurement, configuration, and maintenance of the security infrastructure. ·33 Severity of system vulnerability ambiguously understood. ·34 Difficulty of threats to system not clearly understood. ·35 Countermeasures have uncertain operational effectiveness.
Human Factors	Goals:   Plans:   Risks:	To ensure that human-in-the-loop system performance and effectiveness maintain or enhance the capability, safety, and capacity of the NAS. To identify human-in-the-loop requirements and objectives to meet benefits goals through effective human-system integration to provide a system design that is usable, operationally suitable, safe, and acceptable to the user community. ·36 Human-in-the-loop benefits or requirements are not fully or adequately defined to ensure usability. ·37 Human-system interfaces fail to provide operational suitability. ·38 Human-in-the-loop interactions with system or other systems contribute to human error or inability to meet task or system performance goals or benefits.
Stakeholder	Goals: Plans: Risks:	To meet the user demands for more flexibility in flight paths. Involve the user/international community in the system design and evaluation process. ·39 Resistance to avionics equipage requirements. ·40 Diverse user community. ·41 Conflicting user demands. ·42 Conflicting user opinions.

As an aid in completing the risk lists in Table 6-1, use Table 6-2, Risk Checklist by Risk Facet. Use this table as a starting point for listing risks for any alternative. Table 6-2 is comprehensive and addresses all program stages; hence, the table may contain risk elements not appropriate to the IA phase or to a particular alternative being assessed.

The relevant items in the checklist should be evaluated to determine whether they apply to the particular alternative. Other potential risks not listed in Table 6-2 should be added to the risk checklist for the particular alternative. The alternative's risk checklist should contain all possible risks that might be related to the alternative. After listing all possible risks, those which are extremely unlikely or where the outcome is irrelevant to program goals should be eliminated from the list. The checklist should be directed towards those that will have a meaningful impact on the program, such as impacts on milestones on the critical path. All meaningful risks should be listed in Table 6-1.

Table 6-2: Risk Checklist by Risk Facet

Technical Risks	Operability Risks	Producibility Risks
Technology ·43 Undue reliance on currently unavailable or unproved technology ·44 Possible better new technology may be available by time alternative is implemented System Engineering ·45 Technically incompatible with NAS Architecture ·46 Inadequate functional analysis ·47 Deficient functional allocation ·48 Incomplete integration ·49 Undefined internal interfaces ·50 Vague operational environment ·51 Insufficient requirements analysis ·52 Unstable requirements ·53 Non-compliant or unvalidated requirements ·54 Weak or non-existent failure modes analysis ·55 Requirements difficult to trace ·56 Unidentified safety/security considerations System Design ·57 Inadequate capacity ·58 Highly complex ·59 Lack of design details ·60 Insufficient design margins ·61 Immature design ·62 Unsatisfactory growth potential ·63 Undefined physical properties ·64 Incomplete hardware design ·65 Incomplete software design ·66 Inadequate software tools ·67 Difficulty of developing real-time, safety critical software ·68 Immature software language ·69 Ineffective fault detection ·70 Inordinate use of unique resources ·71 Complex/incomplete man/machine design ·72 Undefined technical approach System Test ·73 Inaccurate/simplistic modeling ·74 Insufficient simulation ·75 No or minimal prototype testing ·76 Incomplete/inadequate test planning ·77 Unsatisfactory OT&E results Technical Documentation ·78 Inadequate design documentation ·79 Insufficient test documentation ·80 Ambiguous/incomplete requirements documentation ·81 Undocumented technical details	System Operation ·82 Undefined external interfaces ·83 Marginal availability ·84 Insufficient reliability ·85 Inadequate performance ·86 Unsatisfactory OT&E results Systems Inter-operability ·87 Operationally incompatible with NAS Architecture ·88 Incompatibilities with Concept of Operations ·89 Incompatibilities with future NAS systems ·90 Places undue loads on other systems ·91 Incompatible or inconsistent operation with existing systems or regulations ·92 Unspecified operational interfaces ·93 Marginal inter-operability	Design Production ·94 Highly complex design ·95 Undeveloped production requirements ·96 Inadequate built-in test equipment ·97 Non-standard remote maintenance monitoring ·98 Novel/unproved technologies Manufacturing ·99 Deficient manufacturing plan ·100 Novel/unproved manufacturing technologies ·101 Speculative manufacturing strategy ·102 Custom design and manufacture required ·103 Significant special tooling ·104 Undefined tooling requirements ·105 Unclear production requirements ·106 Premature initiation of manufacturing ·107 Unavailable or limited manufacturing facilities ·108 Inadequate quality assurance program ·109 Excessive standards ·110 Unavailable equipment ·111 Inexperienced contractor ·112 Inadequate configuration management process ·113 Insufficient skilled labor ·114 Shallow industrial base Parts and Materials ·115 Undefined long lead items ·116 Unavailable government furnished equipment ·117 Ineffective incoming materials handling ·118 Unidentified hazardous materials ·119 Unavailable parts Testing and Documentation ·120 Inadequate consideration of special test equipment ·121 Insufficient qualification testing ·122 Deficient technical data package ·123 Ineffective factory acceptance test program ·124 Untested design changes

Table 6-2: Risk Checklist by Risk Facet, Cont’d

 

Supportability Risks	Cost Estimate Risks	Benefit Estimate Risks	Schedule Risks
O&M ·125 Inadequate O&M concept ·126 Undeveloped O&M strategy ·127 Specialized O&M equipment ·128 Insufficient maintainability ·129 Unsatisfactory maintenance interfaces ·130 Inadequate maintenance procedures ·131 Undeveloped maintenance plan ·132 Configuration management not enforced ·133 Deficient change process ·134 Logistics ·135 Insufficient spares planning ·136 Spares unavailability ·137 Inaccessible site location ·138 Inadequate training ·139 Unclear Logistics Center responsibilities Testing and Support ·140 Insufficient support equipment ·141 COTS/NDI - Industry refresh rate not likely to be consistent with FAA needs ·142 Undeveloped support requirements ·143 Inadequate automated test equipment (ATE) ·144 Unidentified field support requirements ·145 Poor diagnostics ·146 Insufficient testing and support facilities ·147 Unskilled/insufficient manpower Support Documentation ·148 Deficient technical data ·149 Faulty maintenance plan ·150 Undefined data rights ·151 Inappropriate release cycle System Implementation ·152 Deficient implementation approach ·153 Uncertain transition strategy ·154 Unclear rules and procedures ·155 Insufficient personnel/staffing ·156 Unspecified/inappropriate standards	Cost Estimation ·157 Inadequate cost estimating tools ·158 Estimation errors ·159 Inaccurate discount rate ·160 Faulty BOEs** ·161 Insufficient cost margin ·162 Unrealistic overhead and G&A rates ·163 Relies on scarce resources ·164 Speculative life cycle costs ·165 Sensitivity analysis to cost drivers not undertaken Cost Management ·166 Unsatisfactory cost controls ·167 Insufficient cost monitoring Product Cost ·168 Undefined government furnished equipment ·169 Reliance on unavailable NDI/COTS ·170 Unavailable government facilities ·171 Unavailable contractor facilities ·172 Inadequate budget for tests ·173 Undefined hardware costs ·174 Hidden software costs ·175 Unidentified parts and materials	Benefit Identification ·176 Same benefits claimed by other programs ·177 Unidentified major benefits ·178 Unrealistic identified benefits ·179 Difficult to identify benefits Benefit Estimation ·180 Benefits not quantifiable ·181 Difficult to estimate benefits ·182 Tenuous relationship to projected benefits ·183 External forces may affect achieving benefits ·184 Erroneous benefits estimations ·185 Inaccurate inflation/discount rates ·186 Speculative cost avoidance ·187 Faulty BOEs. Inadequate estimating tools	Schedule Estimation ·188 Inadequate schedule estimating tools ·189 Erroneous estimations ·190 Faulty BOEs ·191 Insufficient schedule margin ·192 Optimistic schedule duration ·193 Inappropriate program schedule Schedule Dependency ·194 Unpredictable labor strikes ·195 Improper test scheduling ·196 Excessive task concurrency ·197 Unidentified need for procedures development ·198 Unidentified need for regulations development ·199 Inordinate number of critical path items ·200 Unidentified need for standards development ·201 Uncertainties in contractor process ·202 Uncertainties in contractor stability ·203 Schedule too ambitious for degree of technical complexity ·204 Unavailable materials ·205 Unavailable parts ·206 Unavailable government furnished information ·207 Unavailable facilities ·208 Unavailable personnel ·209 Unavailable tools ·210 Unavailable contractor Schedule Management ·211 Unsatisfactory schedule controls ·212 Insufficient program schedule monitoring ·213 Improper contractor/subcontractor schedule monitoring

Table 6-2: Risk Checklist by Risk Facet, Cont’d

 

Management Risks	Funding Risks	Stakeholder Risks
Planning ·214 Inadequate program plans ·215 Incomplete contingency plans ·216 Deficient risk management plans ·217 Inadequate management approach ·218 Unplanned slips in other programs ·219 Adverse environmental impacts ·220 Unsubstantiated funding profile ·221 Unsubstantiated manpower requirements ·222 Unidentified personnel skills ·223 Minimal resource alternatives ·224 Excessive dependencies on other system ·225 Unexpected acquisition regulation changes Organizing ·226 Excessive span of control ·227 Inadequate authority ·228 Undefined responsibilities ·229 Unclear communications ·230 Undefined integration responsibilities ·231 Ambiguous organizational interfaces ·232 Inadequate contractor organization Implementing ·233 Insufficient management tools ·234 Inadequate program office staffing ·235 Inadequate resource allocation ·236 Deficient personnel management ·237 Lack of coordination ·238 Tenuous top management support ·239 Cumbersome FAA contracting process ·240 Instability of contractor ·241 Uncertainties in procurement ·242 Unavailable personnel ·243 Deficient change implementation Control ·244 Undefined or ineffective change management ·245 Unsatisfactory configuration management ·246 Insufficient contract evaluation ·247 Inadequate planning for contractor monitoring ·248 Insufficient financial management ·249 Irregular/unscheduled program reviews ·250 Insufficient history/records ·251 Undefined key metrics ·252 Uncontrolled requirements changes ·253 Requirements freeze not enforced ·254 Inadequate tracking systems	Funding Constraint ·255 Unfavorable agency priorities ·256 Inadequate funding ·257 Unavailable funding ·258 Lengthy budget cycle ·259 Inadequate OMB marks ·260 Constraining unique budget scoring rules for lease-purchases and leases per OMB A-11 Funding Support ·261 Inadequate user support ·262 Ambiguous operator support ·263 Unclear political support ·264 Marginal cost/benefits ·265 Inconsistent FAA plans ·266 Lack of alignment of necessary funding profile with agency affordability profile Fiscal Management ·267 Insufficient funding requirements ·268 Insufficient fiscal controls ·269 Insufficient fiscal tools ·270 Insufficient funding plans	Congressional Based ·271 Impact of congressional mandates ·272 Unfavorable congressional hearings on program ·273 Critical GAO report Administration Based ·274 Conflicting FAA priorities ·275 Conflicting DOT priorities Aviation Community ·276 Many different stakeholders ·277 Diverse user community ·278 Conflicting user demands ·279 Conflicting user opinions ·280 Conflicting user priorities ·281 Inordinate pressure from user groups ·282 Marginal user support ·283 Strained relationships with users ·284 Resistance to avionics equipage requirements ·285 Inordinate media attention

Table 6-2: Risk Checklist by Risk Facet, Cont’d

Security Risks	Human Factors Risks	Safety Risks
Vulnerability ·286 Incomplete vulnerability assessment ·287 Security policy and procedures not in place ·288 Easy access to communication ·289 No provision for firewalls between shared networks or Virtual Private Networks Threat ·290 Incomplete threat assessment on intent and capability to exploit vulnerability ·291 No prioritization of threat severity ·292 No provision for penetration testing ·293 Threat difficulty not considered #9; Countermeasures ·294 Few countermeasures defined ·295 Effectiveness of countermeasures on infrastructure not testing ·296 Inadequate configuration audit ·297 Lack of monitoring and enforcement ·298 Insufficient funding tools/controls ·299 Ambiguous funding support	Human-in-the-loop Effectiveness ·300 Inadequate definition of human-in-the-loop operational objectives ·301 Inadequate specification of human-in-the-loop benefits ·302 Inadequate analysis of human-in-the-loop system capability to deliver expected benefits or enhancement ·303 Human error mechanisms or metrics not fully identified ·304 Time required to perform tasks is unknown ·305 Automation does not provide the necessary functionality or information to support effective decision-making/problem-solving Human-in-the-loop Suitability ·306 Lack of consistency, compatibility, or congruity with operational environment or legacy systems ·307 Human-system design/interface induces new/additional human error potential ·308 Inadequate incorporation of functional requirements to support user-system performance goals User Acceptability ·309 New tasks impose excessive attention, memory, or workload demands ·310 Requires new teaming and communication links ·311 Operations interface is unacceptable to user ·312 Maintenance interface is unacceptable to user	Hazards ·313 Hazards and service-level effects not fully identified ·314 Inter-relationship of hazard effects not established ·315 Hazards not classified per common scheme ·316 Hazard class not based on operational environment definition System Safety Interdependence ·317 Hazard interdependence poorly understood ·318 Interoperability of components on system safety not investigated ·319 Systemic approach to safety is lacking one or more components (planning, requirements, procedures, operation, aircraft certification, or user approval) Mitigation Strategies ·320 Mitigation strategies not shared ·321 Operational and safety objective not established ·322 Lack of critical/valid safety information ·323 Mitigation strategies not tied to hazards or safety requirements ·324 Plan for development and operational assurance not in place

6.2 Step 2 - Determine Facet Weights

The IA Team assigns a numerical weight to each risk facet reflecting its importance in the successful implementation and operation of the program. Obviously, a program is successful when it fulfills its mission need. The following are important points to note in assigning facets weights:

• The importance is assigned relative to other facets within the rankings (i.e., a facet's importance to an acquisition completing its life cycle, compared to the importance of the facets directly preceding and following it in the rankings).

• The weights should then be "normalized". To "normalize" a sequence of numbers (1) add the numbers together to obtain a total and (2) divide each number of the original sequence by the total. This will result in a new sequence of "normalized" numbers; each number in the new sequence will be between 0 and 1 and the total of this new sequence will be 1.

• (Note: A weight of zero means that the risk facet does not apply (i.e., it falls below the threshold of what is important compared to the other facets)

• A weight of 1.00 means that only that risk facet applies (i.e., it far exceeds the threshold of what is important compared to the other facets).

• The ranking and assignment of facet weights may be the same for all alternatives and based on a team consensus (i.e., ASD and the sponsoring program office) before other steps of the risk assessment process are conducted.

• The facet weights may remain the same for all alternatives, with the exception of the reference case, where the success of the existing program may have a different weighting of facets.

• Do not exclude a facet from the analysis solely because it has a low facet weight.

Assigning risk facet weights is a way of conveying the importance of each facet in bringing about the success of a program. To achieve objectivity in assigning risk facet weights, the risk team members should determine facet weights in a full team meeting before the evaluation of the individual risks. If the determination of the facet weights were to occur during the risk evaluation or before evaluation of the alternatives, risk team members might be biased. However, team members may want to make small adjustments to the weights after the assessment for sensitivity purposes. Any change in weights at this point should be well documented and explained.

6.3 Step 3 - Estimate Severity of Impact of an Adverse Event

For each risk, the severity of the impact of the adverse event on the alternative (expressed as substantial, moderate, or minor) is determined using Table 6-3, Estimating the Severity of Impact of an Adverse Event, as guidance. The severity from the highest rated risk within each facet is entered in the third column of Table 6-5.

To estimate the severity of impact the IA Team can use expert interviews, analogy comparisons, evaluation of program plans, or the Delphi method. The IA Team can also use multiple methods and approaches other than those listed.

6.4 Step 4 - Estimate the Probability of an Adverse Event

In spite of attempts to be analytic about quantifying risks, considerable subjectivity remains. The degree of risk perceived in a situation is partially a reflection of the personality of the risk assessor(s). A risk-rating scheme built against a set of definitions eliminates some of the ambiguity. Further, the rating scheme should be simple. The following risk rating scheme involves determining a high, medium, or low overall risk rating using the notion that the degree of risk is a judgment reflecting the probability of occurrence of an adverse event and the severity of impact on the alternative should the adverse event occur.

If a particular risk facet does not apply to the alternatives being assessed, then the probability of an adverse event and the severity of the impact of the adverse event do not need to be estimated for that risk facet.

For each risk, the probability of occurrence of an adverse event (expressed as high, medium, or low) is determined using Table 6-4, Estimating the Probability of an Adverse Event, as guidance. The probability from the highest rated risk within each facet is entered in the second column of Table 6-5, Determining Overall Weighted Alternative Risk Score.

Below are four possible methods to estimate the probability of occurrence and severity of impact. More than one method, as well as approaches other than those listed below, can be used.

 

• Expert Interviews - Identify expert(s) and methodically question them about the risks in their area of expertise as related to the alternative. The questioning focuses on extracting information about what the program risks are and their relative magnitude. Data collection sheets can be used to facilitate this process.

• Analogy Comparisons - Assess risks by using data from similar prior programs. The analogy comparisons are based on the idea that no new program, no matter how advanced or unique, represents a new concept or system.

• Evaluation of Program Plans - This technique highlights and isolates risks caused by insufficiencies and disparities in planning. It evaluates program plans for contradictions and voids. The plans do not need to be formal plans, but can include program management plans, acquisition plans, specifications, statements of work, or work breakdown structures. The process assesses the plans for correctness, completeness, currency, and consistency.

• Delphi Technique - The Delphi technique is a structured method to elicit intuitive thinking by a group and produce technological forecasts. It can be used for the systematic collection and collation of informed judgments obtained from a group of experts and for the refinement of these judgments by an iterative process to arrive at a joint judgment or decision. Typically, judgments of the individuals in a group are collected, perhaps integrated as a group response, and fed back to the individuals. Each individual then considers whether to contribute more information or to modify earlier views. This iterative process is continued until a reasonable consensus is obtained. The responses can be fed back anonymously if desired.

 

Table 6-3: Estimating the Severity of an Adverse Event Impacting the Program

Facet	Substantial Severity of Impact	Moderate Severity of Impact	Minor Severity of Impact
Technical	Performance or problem data indicate that with current alternative design margins, full performance would not be met and alternate systems are not available.	Performance or problem data indicate that with current alternative design margins, full performance objectives will only be met by: (1) significant modification to a design of a component or subsystem; or (2) reallocation of design margins among subsystems.	Performance and problem data indicate that only minor hardware/software design changes will be needed to meet full performance objectives.
Operability	No operationally suitable solutions available without major impacts on the overall system performance. Will cause significant impact to existing procedures, causing operational implementation to be unsuccessful.	Technical, operationally suitable solutions partially identified. The solution is not readily available or will have significant impacts on the overall system performance. Will affect several procedures and cause operational implementation to be only partially successful.	Technical operationally suitable solution is identified and readily available. Will affect a few procedures but operational implementation is successful.
Producibility	Manufacturing and production capabilities unavailable.	Manufacturing or production capabilities in state of change, and some capabilities will be unavailable.	Manufacturing and production capabilities known and available.
Supportability	System design characteristics and planned logistics and software support resources do not meet system utilization requirements. Support procedures or technologies will be significantly impacted and prevent suitable transition of support to AF.	System design characteristics and planned logistics and software support resources meet some but not all system utilization requirements. Some support procedures or technologies will be impacted and transition of support to AF will be difficult.	System design characteristics and planned logistics and software support resources meet nearly all system utilization requirements. Only minor support procedures or technologies will be impacted, and transition of support to AF to be successful.
Cost Estimate	Estimated costs are likely to be exceeded by more than 25%.	Estimated costs are likely to be exceeded by 10 - 25%.	Estimated costs are likely to be exceeded by less than 10%.
Benefit Estimate	Less than 75% of the estimated benefits achieved.	75 - 90% of the estimated benefits achieved.	At least 90% of estimated benefits achieved.
Schedule	A schedule slip of more than 25%.	A schedule slip of 10% — 25%.	A schedule slip of less than 10%.
Management, Funding, and Stakeholder	Management, funding and stakeholder aspects, and environments not known, understood, or integrated. Program or project cannot be implemented as planned.	Management, funding and stakeholder facets, and environments in state of change but somewhat known. Program or project can partially be implemented as planned.	Management, funding and stakeholder facets, and environments known and stable, and program or project can be implemented as planned.
Safety	Safety requirements /hazards not identified for future NAS systems.	Safety hazards are identified but requirements for future NAS systems remain to be defined.	Safety requirements/hazards are identified, as well as the risk acceptability criteria.
Security	Security protection at the system perimeter, workstations, and servers are not provided. Security infrastructure and intrusion detection hardware and software is not available or not acceptable.	Some security infrastructure is available for some NAS systems/subsystems including dial-up protection for remote users. Some intrusion detection hardware and software are available.	Complete security infrastructure is available for every NAS system/subsystem including intrusion detection hardware and software.
Human Factors	If one or more of the following conditions are present: 1) Size of the workforce affected by system changes is large and staffing levels and system performance goals are not supported by workload analyses. 2) Analyses indicate personnel skill and ability requirements are changing or unmet by current workforce. 3) Early training analyses are lacking or fail to influence selection of design alternatives for critical tasks such as problem solving and decision-making. 4) Physical and cognitive human-system integration design elements and integration of the system and its components into the user work environment have not been fully analyzed or do not comply with human factors engineering best practices. 5) System changes affect safety critical components and analyses have not yet proven system safety and workforce health are assured.	If one or more of the following conditions are present: 6) Size of the workforce affected by system changes is small and staffing levels and system performance goals are partially supported by workload analyses or by current staffing. 7) Analyses indicate personnel skill and ability requirements are partially met by current workforce. 8) Early training analyses partially identify factors affecting design alternatives for critical tasks. 9) Physical and cognitive human-system integration design elements and integration of the system and its components into the user work environment have been partially analyzed or partially comply with human factors engineering best practices. 10) System changes affect minor safety components or analyses show limited impact on system safety and workforce health.	If all of the following conditions are present: 11) Workload analyses assure that staffing levels support system performance goals. 12) Analyses indicate personnel skill and ability requirements are met by current workforce. 13) Early training analyses influenced alternative analysis and design to ensure ease in performing all critical tasks. 14) A human-centered design approach has been used to design the physical and cognitive human-system integration elements, and the integration of the system and its components into the user work environment. 15) System changes affect no safety critical components and analyses have proven system safety and workforce health are assured.

 

Table 6-4: Estimating the Probability of an Adverse Event Impacting the Program

Facet	High Probability of an Adverse Event	Medium Probability of an Adverse Event	Low Probability of an Adverse Event
Technical	Design unknown. Approach to meet requirements carried only through conceptual design and analysis. Technology is only concept or experimental.	Design is in development or prototype phase. Technology prototype or engineering model tested in relevant environment but not operated in fielded environment.	Design is mature. Technology within state-of-the-art or off the shelf. Performance specifications are known.
Operability	NAS or other interfaces not fully known or documented. Operational concept or implementation of concept has yet to be established. Significant changes to procedures are highly likely.	NAS or other interfaces somewhat known and partially documented. Operational concept has evolved to the point of a design baseline. Changes to procedures are likely.	NAS or other interfaces are known and documented. Design approaches for the operational concept have been demonstrated or implemented. Few changes to procedures are likely to.
Producibility	Manufacturing and production capabilities are likely to be unavailable	Manufacturing or production capabilities are likely to change.	Manufacturing and production capabilities are not likely to change.
Supportability	New support technologies and procedures or substantial modifications to existing support technologies or procedures are likely to be required and could prevent suitable transition of support to AF.	Items similar in concept have been supported as fielded systems or supported during test. Substantial modifications are likely to be required to support technologies or procedures of AF.	Similar items have been fielded and are currently being supported, or similar items have been demonstrated to be supportable during field-testing. Only minor changes to existing support technologies or procedures are likely to be required. Transition of support to AF expected to be straightforward.
Cost Estimate	Basis for cost estimation is inadequate, or major uncertainties exist related to the scope/definition required for estimation. Costs are likely to significantly exceed estimation.	Cost factors not certain, but scope/definition required for estimation is adequate. Costs are likely to exceed estimation.	Cost factors understood and based on or extrapolated from similar items in production. Definition required for estimation is adequate.
Benefit Estimate	Major uncertainties exist related to benefit estimation; extremely tenuous relationship of alternative to projected benefits; or very likely external forces will affect achieving benefits.	Benefits not certain, but scope/definition required for estimation is adequate; slightly tenuous relationship of alternative to projected benefits; or possible external forces may have some affect on achieving benefits.	Benefits understood and based on or extrapolated from similar items in operation. Definition required for estimation is adequate. Direct relationship of alternative to benefits. Little likelihood of external forces affecting the achievement of the benefits.
Schedule	Many schedule interdependencies are likely to develop, for which there is little or no flexibility to absorb delays. Few or no plans to minimize unknowns; difficult or complex system to develop. Knowledge and experience base very limited.	Some schedule interdependencies with little schedule margin are likely to develop. Plans to minimize unknowns are generally complete; some uncertainties exist. Little knowledge and experience in some areas.	Adequate schedule with substantial margins and achievable plans to minimize unknowns. High knowledge and experience base. There are no schedule dependencies beyond the control of the alternative.
Management, Funding, and Stakeholder	Management, funding and stakeholder facets, and environments likely to be unknown or unstable.	Management, funding and stakeholder facets, and environments in state of change but somewhat known.	Management, funding and stakeholder facets, and environments known and stable.
Safety	Hazards and their impact on NAS services likely to be inadequately defined. Interdependency of system components in contributing to system failure likely to be poorly considered. Mitigation strategies not likely to be directly tied to hazards. Mitigation measures likely to be unpalatable.	Process for assessing safety likely to be developed and applied. Mitigation measured are identified and related to hazards. Mitigation strategies are palatable.	Mitigation strategies are funded and applied.
Security	Vulnerability and threat assessments likely to be un planned or conducted. Countermeasures notare likely to be identified or tested.	Vulnerability and threat assessment planned but not likely to be completed or conducted on time. Theoretical countermeasures are not likely to be well identified.	Vulnerability and threat assessments conducted. Countermeasures developed for each threat, and their ability to withstand threats proved.
Human Factors	If one or more of the following conditions are present: 16) System requirements or designs lack human-system performance objectives or are derived without comprehensive human-in-the-loop performance research, studies, or analyses. 17) Human-in-the-loop performance goals are unstated or not achievable within the proposed operational and maintenance concepts or using the proposed design approach. 18) Human interface issues and risk mitigation strategies are not adequately supported by research, funding, technical expertise, or other resources. 19) Proposed automation lacks analyses to ensure full functionality or information to support user tasks. 20) User tasks and skills are not well defined or do not conform to current skill levels. 21) Human-system task performance times are unknown or not quantified. 22) Potential for human error has not been quantitatively analyzed or the impact on human-in-the-loop system capabilities is unknown or changing. 23) Physical or cognitive human-system integration design elements are, individually or in the aggregate, unknown or sufficiently deficient to detract from efficient or effective task performance. 24) Requirements for integration of the system or its components into the user work environment are undetermined or changing. 25) User groups do not contribute to requirements development, design, or analysis.	If one or more of the following conditions are present: 26) System requirements or designs include incomplete human-system performance objectives, or are derived with limited human-in-the-loop performance research, studies, or analyses. 27) Human-in-the-loop performance goals are partially stated or partially achievable within the proposed operational and maintenance concepts or using the proposed design approach. 28) Human interface issues and risk mitigation strategies are partially supported by research, funding, technical expertise, or other resources. 29) Analyses show proposed automation supports partial functionality and information needed to support user tasks. 30) User tasks and skills are defined but changing user roles require reevaluation of skills and training. 31) Human-system task performance times are partially known or partially quantified. 32) Potential for human error has been partially analyzed or impact on human-in-the-loop system capabilities is partially known. 33) Physical or cognitive human-system integration design elements are, individually or taken together, partially known. 34) Some elements for integration of the system or its components into the user work environment are new or changing. 35) User groups partially contribute to requirements development, analysis, and design.	If all of the following conditions are present: 36) System requirements and designs include human-system performance objectives derived from comprehensive human-in-the-loop performance research, studies, and analyses. 37) Analysis indicates that human-in-the-loop performance goals are achievable within the proposed operational and maintenance concepts and using the proposed design approach. 38) Human interface issues and risk mitigation strategies are adequately supported by research, funding, technical expertise, and other resources needed to complete the design within program constraints. 39) Automation provides full functionality to support user decision-making. 40) User tasks and skills are well defined or remain essentially unchanged. 41) Human-system task performance times are known and acceptable. 42) Potential for human error has been quantitatively analyzed and impact on human-in-the-loop system capabilities is known. 43) Physical or cognitive human-system integration design elements are individually and taken together sufficiently mature to assure efficient or effective task performance. 44) Integration of the system or its components into the user work environment is fully compatible with the larger system and operations. 45) User group input is an integral part of requirements development, design, and analysis.

 

Table 6-5: Determining Overall Weighted Alternative Risk Score

Facet	Severity of Impact (Substantial, Moderate, Minor)	Probability of an Adverse Event (High, Medium, Low)	Facet Risk Rating Score (0-10) (from Table 6-7)	Facet Weight (0-1)	Weighted Facet Score (0-10)
Technical
Operability
Producibility
Supportability
Cost Estimate
Benefit Estimate
Schedule
Management
Funding
Security
Human Factors
Safety
Stakeholder
Overall Weighted Alternative Risk Score

6.5 Step 5 - Assign Risk Ratings

Once facet weights are determined, and individual risks are identified for each alternative, a risk rating score is determined for each risk and each facet. The highest rated risk in each facet is then used to represent the facet according to the assignment scheme shown in Table 6-6, Risk Rating Score Assignments.

Table 6-6: Risk Rating Score Assignments

	Severity of Impact
Probability of an Adverse Event	Substantial	Moderate	Minor
High	10	8	5
Medium	8	5	2
Low	5	2	0

 

6.6 Step 6 - Calculate Overall Alternative Risk Rating

The final step in assessing the risk for any alternative is to calculate an overall risk rating for the alternative.

The weights, from Step 2, are entered into the fifth column of Table 6-5. Multiplying the entries in the fourth and fifth columns, and entering the results into the last column of Table 6-5 calculate the weighted facet score for each risk facet. The overall weighted average alternative risk score is entered in the bottom row of Table 6-5 by adding the individual weighted risk facet scores in the last column.

Once the overall weighted alternative risk score is calculated for each alternative (refer to Table 6-5), a descriptive overall alternative risk rating (i.e., high, medium, or low) is determined using Table 6-7. This rating can also be entered into a common table to permit comparison of risk assessment results across alternatives (refer to Table 6-8).

There are metrics in addition to the overall risk level, which represents the risk levels weighted by importance of each facet to the success of the program. Some of these alternative metrics are listed below.

• Analyze the horizontal facets (i.e., analyze one facet across alternatives) between alternatives. Do this after analyzing the vertical facets to evaluate the total overall weighted risk score for each alternative.

• Apply a frequency analysis using a histogram approach by listing the distribution of the facet risk ratings (either numerically by risk rating score and/or by qualitative group such as high, medium, or low) across all facets. This can also be done at the individual risk level for the whole program by citing the number of total individual risks that are high, medium, or low, and/or the number of risks listed by risk rating score.

• A more traditional method is to present a risk matrix 3 x 3 with each cell referring to the number of risks associated with that cell’s risk. This provides information on the total number of risks and their distribution.

• Interdependencies (see Section 6.7) can be shown in an interdependencies matrix 13 x 13.

Table 6-7: Overall Alternative Risk Rating

Overall Rating (Score)	Description
High (7.0 - 10)	Alternatives with high overall risk rating should receive close attention. Risk facets with high-risk ratings should be considered principal risks. Each high risk should have strategies, metrics, a plan of action, and milestones developed by the risk owner. They should be aggressively managed and monitored on a continuous basis until the risk is mitigated to an acceptable level.
Medium (3.0 - 6.99)	Alternatives with a medium overall risk rating require attention. Risk facets should be examined to see if any are rated high and should be placed on the principal risk list and managed as described above. Each medium risk should have candidate strategies, metrics, a plan of action, and milestones developed by the risk owner. They should be frequently managed and reviewed. Any risks on the principal risk list should be aggressively managed and monitored on a continuous basis until the risk is mitigated to an acceptable level.
Low (0 - 2.99)	Alternatives with a low overall risk rating do not normally require attention for risk. However, the risk owner should periodically review status. Any high or medium risk facets should receive attention as described above.

A risk rating for a subset of risk facets can be determined by summing the weighted facet scores (last column of Table 6-5) for risk facets in the subset. For example, the sum of the weighted facet scores for Technical, Operability, and Supportability will yield a score for the risk of the alternative output performing as designed.

These risk ratings reflect a comprehensive review of a set of risk facets relevant to every initial IA. The comprehensive review is conducted at the macro level to be consistent with the level of analysis warranted in the initial investment decision. Often the safety risk management, human factors, or the security organizations of FAA develop more detailed, microscopic assessments. When this information is available, the risk information from these specific assessments should be used as a basis for assigning a risk level for the macro level assessment. When specific information is not available, every effort should be made to use the direct input from the respective offices to ensure that the risk level has been reasonably assigned. In all cases, the macro level assessments and the micro level assessments should be consistent, complementary, and mutually supportive.

6.7 Step 7 - Compare Risks among Alternatives

The risk assessment process is repeated to determine an overall alternative risk rating for each alternative. The individual risk facet ratings and the overall risk rating for all alternatives can be entered into a table, such as Table 6-8, Risk Ratings of Alternatives, to permit comparison of risk assessment results across alternatives.

Table 6-8: Risk Ratings of Alternatives

Risk Assessment Ratings	Alternative 1	Alternative 2	Alternative 3	Alternative . . .
Technical
Operability
Producibility
Supportability
Cost Estimate
Benefit Estimate
Schedule
Management
Funding
Security
Human Factors
Safety Process*
Stakeholder
Overall Weighted Alternative Risk Score
Overall Alternative Risk Rating (H, M, L)

* The risk shown for the "Safety Process" facet is the risk of not performing adequate safety risk management. The risk of this facet must not be confused with the risk associated with operation of the system. The evaluation of operational risk is documented in system safety design analysis reports.

To facilitate cross comparison of alternatives, Table 6-8 cells may be filled with green, yellow, and red shading to represent the alternatives' relative risk rating. Any risk facet receiving a score of 10, using Table 6, shall be shaded red for high risk. The same high-risk designation is reasonable for any risk facet receiving a score of 8. This is consistent with Table 6-7 score ratings. Similarly, any facet receiving a score of five shall be shaded yellow or white to represent medium risk. Lastly, any facet receiving a score of two or zero shall be shaded green for low risk. In this way, the individual risk of each facet can instantly be identified and compared.

The risk assessment results contained in Table 6-8 should be used with the other evaluation factors (i.e., life cycle costs, benefits, schedule, and performance) to narrow the set of alternatives to the most promising one(s) to present to the JRC and to justify those in the subset. The JRC can also use Table 6-8 as part of their decision information.

Interdependencies of Risks

When interdependencies between risks are expected to be significant, the risk analyst may complement the comparison of risks among alternatives by considering the interdependencies of risks for the preferred alternative. These interdependencies affect the different risk categories or facets.

Risks most frequently affect the costs, schedule, technical, benefits/performance, funding, and/or schedule. Entering the risk more than once is double counting, so an option is to place the risk in the category in which it has the highest risk rating. Include the risk in only one category, but describe it in the other risk facets in qualitative text. An interdependency risk matrix as illustrated in Table 6-9 may be used.

Table 6-9: Interdependency Risk Matrix

An interdependency risk matrix consists of a 13 x 13 matrix, 1 row, or column for each risk facet. Each cell represents the interdependencies between each row and each column. Within each cell, are two numbers, with the first number representing the number of interdependent risks between the row and column risk facet. The second number represents the highest-ranking risk level (high, medium, or low) between the two risk facets among all of the interdependent risks. Alternatively, there can be three numbers, which distributes the number of interdependent risks among high, medium, or low, with a number to represent the total number of each risk level. In Table 6-9, the numbers 6/10 would refer to 6 total technical risks that are interdependent with the cost facet, of which the highest rated risk is rated a risk score of 10, corresponding to a high risk level.

6.8 Step 8 - Transition to Joint Responsibility with the IPT for JRC-2b

As Figure 2-1 (in Section 2 above) shows, the product of the IA Team Alternatives Risk Assessment is used to: (1) identify cost and benefit implications of risk and mitigation strategies, and (2) form the joint basis for a detailed IA Team Life Cycle Risk Assessment. In the IA Team Life Cycle Risk Assessment, responsibility is shared between the Integrated Product Team (IPT) and the IA Staff to develop a detailed risk assessment of the four APB metrics and an interdependency metric for the preferred alternative. The final step at the conclusion of JRC-2a, then, is beginning the transition of risk assessment and mitigation from the IA Staff leadership to a joint responsibility with the designated IPT.

The transition is aided by the risk issue database (i.e., the collection of risk issues) and the assessments from the risk sections of the IAR and the APB. The risk issue database contains the following:

• Issue-by-issue description,

• Risk probability,

• Risk severity, and

• Listing of planned mitigation actions.

The risk assessment team compiles the risk issue database as part of the risk assessment process. The database is available to the IPT, and is a basis for further risk assessments.

The risk issue database and the IPT’s own risk identification protocol complement each other to synthesize a detailed IA Team Life Cycle Risk Assessment.

A key aspect for synthesizing the IA Team Alternatives Risk Assessment is the conversion of risk issue ratings from the IA Team Alternatives Risk Assessment into the IPT protocol. Having reviewed the IPT Programmatic Risk ratings criteria for risk likelihood and risk consequences, a conversion can be readily constructed. This suggested conversion is mapped in Table 6-10.

Table 6-10: Alternatives Risk Assessment Mapping to IPT Protocol

IA Rating --Probability	IPT Programmatic Rating -- Likelihood	IA Rating –Severity	IPT Programmatic Rating --Consequences
Low	1	Minor	1
Low	2	Minor	2
Medium	3	Moderate	3
High	4	Substantial	4
High	5	Substantial	5

Figure 6-1 shows the coverage of mapping the risk levels. The IPT matrix is the color matrix shown: red for high risk; yellow for medium; and red for high-risk levels. The IA Team Risk Assessment matrix ratings result in the lettered levels: H for high, and so on. As the figure shows the IA ratings, only cover a portion of the 5 x 5 IPT matrix. In fact, the IA Team Risk Assessment matrix covers a 3 x 3 matrix. Partial coverage is understandable, when one considers that the level of detail available from information in the Initial IA phase of AMS is not very deep. It is natural, then, to work jointly in the Final IA phase of AMS to examine the risks further and develop the detail that allows a finer mapping, and a risk assessment and mitigation that endure through the life cycle of the investment.

Figure 6-1: Conversion Mapping Coverage

 

	5
	4		L	M	H
Likelihood/	3		L	M	M
Probability	2		L	L	M
	1
		1	2	3	4	5
		Consequences/
			Severity

Moreover, there are thirteen facets and four baseline metrics. Each of the facet issues may affect cost, schedule, performance, or the benefit metric. Which metric is impacted may turn out to be a judgment call, and this necessarily requires a joint review and collaborative discussions, hence the need for joint responsibility.

7.0 Risk assessment products and Mitigation Strategies

Each risk identified as part of the IA Risk Assessment must have an accompanying mitigation strategy. A mitigation strategy should adequately describe actions required to mitigate that risk to an acceptable level. Mitigation strategies should also include what schedule, benefit, and cost implications implementing the mitigation strategy will have.

7.1 Products of IA Team Alternatives Risk Assessment

There are generally three essential products that the IA Risk Assessment yields: 1) a section of the IAR, 2) input to the APB, and 3) the Risk Issues Database.

7.1.1 Investment Analysis Report

The IAR contains a separate section for the risk assessment results and findings that will contain the following:

• Risk summary table,

• Risk summary matrix,

• Small paragraph on how the assessment was conducted, and

• Listing of all the high and medium level risks with mitigation.

Risk Summary Table (see Table 6-8) consists of listing of each risk facet with its risk level of high, medium, or low. At the bottom of the table should be the Overall Risk Level for the program as determined by applying the facet weights to the numerical value for each facet’s risk level.

Risk Matrix (similar to Figure 4-1) records the number of individual risks contained in each 3 x 3 cell corresponding to three rows/levels of probability (high, medium, and low) and three columns/levels of severity (substantial, moderate, and minor). The high-risk cells in the table are colored red, the medium risks cells are colored yellow, and the low risks cells are colored green.

Describe the risk assessment process. Topics can include the use of sub teams, experts, or panels, and the overall processes used to list the risks, score the risks (probability and severity), and develop mitigation strategies. Describe how the facet weights were derived; describe, in detail, why the risk was assigned at a particular risk level; and describe mitigation in as much detail as possible.

The cost analysis team will use these risk inputs to develop their high confidence costs. Eventually, the risks and their mitigations will be transferred to the IPT.

Listing the medium and high risks for each alternative is essential to the IAR. If there appears to be a large number of medium and high risks, the author may prefer to list the high risks in the body of the IAR and list the medium risks in an appendix.

7.1.2 Acquisition Program Baseline

The APB is a summary review of the program’s baseline costs, benefits, and risks and is much smaller than the IAR. Therefore, the risk assessment section of the APB will contain only the following:

• Risk summary table,

• Risk matrix, and

• Summary discussion of the overall program risk level.

If the APB needs more detail, describe the highest risks.

7.1.3 Risk Issues Database

The appendixes contain a template for compiling risks into a risk issues database. The risk issues database is the collection of the risk issues identified during the assessment. Eventually, the IA Team will hand over this risk database to the IPT who will incorporate it into their program risk plan and database. The IPT will then become the owner of the risks and be responsible for mitigating them.

7.2 Mitigation Strategies

7.2.1 Mitigation Actions and Cost Range Estimation

After the Risk Assessment Team has identified all risks, the IA Teams Risk Lead will compile all high and medium risk worksheets into a briefing. This briefing will then be used in a meeting with the Cost Estimating Team, the Risk Assessment Team, and with any other major stakeholders to ensure accurate ranges are attached around the elements of the WBS affected by the mitigation strategies for high and medium risks. The objective of this meeting is to confirm that the cost of each mitigation strategy for a high or medium risk is covered in the cost estimate and can be implemented by the program with the money allotted to the different elements of the WBS.

Information gathered during the risk assessment can also be used to attach risk ranges around the different elements of the WBS before Monte Carlo simulations are run to generate cumulative distributions of total possible program costs, allowing areas of identified uncertainty to be accounted for. Using this information at this stage will improve the accuracy of forecasted cost distributions.

It is not always obvious whether an activity is part of an alternative or whether it is a mitigation activity. It is important that the cost of the activity be included in the cost estimate and that there be consistency among the alternatives that the activity be judged part of the alternatives themselves or part of the mitigations to risks.

7.2.2 Mitigation Actions and Benefits Range Estimation

After the Risk Assessment Team has completed identifying all risks, the IA Team Risk Lead will compile all high and medium risk worksheets into a briefing. This briefing will then be used in a meeting with the Benefits Estimating Team, the Risk Assessment Team, and any other major stakeholders to ensure accurate ranges are attached around benefits estimates affected by mitigation strategies for high and medium risks. This meeting is to confirm that the cost and impact of each mitigation strategy for a high or medium risk is covered in the benefits estimates.

7.2.3 Tracking and Evaluation of Mitigations

The program office is responsible for maintaining a risk database to see that all risk mitigation strategies are carried out and all high and medium risks are acceptably mitigated. The program office will use Programmatic Risk Mitigation Plan and schedules ending with Risk Resolution Date.

8.0 Evolution of Risk Assessment

8.1 Aviation Environment Changes

The foregoing Risk Assessment Guidelines describe a standard operating procedure, which can be replicated, traced, and verified. Even though implementing a procedure adds consistency, risks and the risk assessment process still evolve. Not only will risks change throughout the life cycle of a program as more information is known, but mitigations will eliminate some risks. Additionally, the focus of risks may change over time. Risks may be added to the risk facet list and/or combined into others. With the advent of new technologies and procedures, the number of risk facets may grow to incorporate potential risk areas not included in the current set of risk facets.

8.2 Methodological Changes

As risks and the risk assessment processes change, the methods of assessing risk may change. Additional metrics may help to clarify the description, level, and distribution of risks.

The following are examples of changes in methodology that may be applied to the risk assessment:

• Cross tabulating a risk facet across alternatives,

• Adding a risk matrix to the original Guidelines for Risk Assessment,

• Expanding the risk matrix or risk evaluation categories for probability and severity, and

• Changing risk facet definitions to encompass a wider range of risks

APPENDIX A

TEAM PARTICIPANTS

Appendix A: Team Participants

FACET	Contributing Organizations	Possible FAA Contacts	Contact's #
Technical	AUA/AND representative
	ARU-100	Steve Anderson	202-385-7651
	ASD-100	Phil Decara	202-358-5323
	ASD-100	Betty Falato	202-358-5377
Operability	ARU-100	Steve Anderson	202-385-7651
	Air Traffic Contact
Producibility	ASD-100	Phil Decara	202-358-5323
	AUA/AND representative
	Market Survey Author
Supportability	ARU-100	Ralph Taylor	202-385-7661
	AOS-320	Ana Mack	609-485-6629
	AUA/AND representative
Benefits	ASD-430
Cost	ASD-410
Schedule	AUA/AND representative
Management	AUA/AND representative
Funding	ASD-300	Greg Street	202-358-5287
	ARQ-100	Debra Griffith	202-358-7509
	AFZ-400	Cynthia Beck	202-267-3588
Stakeholder	ARU-100	Ralph Taylor	202-385-7661
	ARQ-300	Bob Fitzpatrick	202-358-7608
Information Security	ASD-100	Mary Beth Dormuth	202-358-5381
	ASD-4	Feisel Keblawi	202-358-5317
	AUA/AND representative
	AIS
	ACS
Human Factors	AAR-100	Dino Piccione	202-493-5305
	AAR-100	Glen Hewitt	202-267-7163
Safety	ASD-110	Scott Van Buren	202-358-5326

APPENDIX B

RISK ISSUE WORKSHEET

Appendix B: Risk Issue Worksheet

---