| GUIDANCE | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Investment Analysis > Investment Analysis Standard Guidance > Risk Analysis Methodology for Initial Investment Decision
Guidelines for the Investment Analysis Team’s Alternatives Risk Assessment October 2002 Revised by: Investment Analysis and Operations Research, ASD-400 Federal Aviation Administration 800 Independence Avenue, SW Washington, DC 20591
PREFACE This document presents the guidelines by which the Investment Analysis (IA) Team conducts Alternatives Risk Assessments as part of the Federal Aviation Administration's (FAA's) IA process as proscribed in the Acquisition Management System (AMS). The guidelines were developed to assist IA Teams in their analysis of candidate alternatives/solutions. Risk assessment is a dynamic enterprise. As the National Airspace System (NAS) operations and environment change, we expect that new issues and risks affecting investment analysis will surface. Since the first publication of these Guidelines in July 1997, information security, human factors, and safety issues have gained visibility and prominence as additional risk to be considered. Accordingly, in July 1999, Art Politano, Investment Analysis and Operations Research (ASD-400), and Don Weitzman, Systems Engineering Technical Assistance (SETA), updated the set of risk facets to include their assessment. In April 2002, Art Politano and David Chin (ASD-400), and Christopher Hunt and Darren Wilson (SETA-II), updated the guidelines again. The primary reasons for this update is to: (1) harmonize the IA Team Alternatives Risk Assessment and the other risk analyses in the Acquisition phase, and (2) update the methodology based on what we have learned from numerous applications. This document represents the state of the practice on the IA Team’s Alternatives Risk Assessment. IntroductionRisk Assessments During the Acquisition Phases General Risk at Each Phase of the AMS Extending Across the Phases of the AMS Risk Assessment during Mission Analysis – Portfolio Management 2.3 Risk Assessment during Investment Analysis 2.3.1 The IA Team Alternatives Risk Analysis – JRC-2a (Initial IA Decision) 2.3.2 The IA Team Life Cycle Risk Analysis - JRC-2b (Final IA Decision) 2.4 Risk Assessment during Solution Implementation - IPT Programmatic Risk Management Process 2.5 Risk Assessment during In-Service Management Integration of IA Team Alternatives Risk Assessment with other AssessmentsIA Team Alternatives Risk Assessment Tailoring the Process STEPS IN THE RISK ASSESSMENT PROCESS Risk assessment products and Mitigation Strategies Evolution of Risk Assessment Appendix B: Risk Issue Worksheet Acquisition Management System The Federal Aviation Administration (FAA) Acquisition Management System (AMS) establishes policy and guidance for all aspects of the acquisition life cycle. The AMS simplifies, integrates, and unifies the elements of life cycle acquisition management. It increases the quality, reduces the time, and decreases the cost of delivering needed services to its customers. The AMS has several phases including:
The Investment Analysis (IA) Team Alternatives Risk Assessment is conducted during the Investment analysis phase. Investment Analysis PhaseThe IA phase begins when the FAA determines a potential need to expend funds to meet a mission capability shortfall or to take advantage of a technological opportunity. During the IA phase, the sponsoring and acquiring organizations are partners in ensuring the critical needs of the users are satisfied by an affordable solution. The IA phase ends with a decision on how the requested funds are allocated. Major activities in the IA phase are:
In the IAR, the IA Team analyzes alternatives and candidate solutions. Of the alternatives identified, those determined viable become candidates for detailed analysis. The IA Team evaluates candidate solutions by compiling and analyzing such factors as cost, benefit, schedule, and risk. The IA Team documents the results in an IAR that contains comprehensive, quantitative data for each candidate solution. During the IA phase, the IA Team conducts the Alternatives Risk Assessment, as referenced in AMS policy sections 2.9.13, 2.4.4.1 and 2.4.4.2, which identifies risks and helps to establish realistic baselines that include funding for the risk mitigation. Risk Analysis throughout the AMSThe following describe key elements of risk management at phases in the AMS: Mission Analysis: Identify and characterize risks to the agency’s ability to execute its legislated responsibilities and satisfy customer demands for service. Typically, these risks arise from changes in the operational environment and shortfalls in operational capability. Investment Analysis: Ensure that primary risks associated with candidate solutions are fully identified and evaluated. Sufficient time and money must be included for each candidate solution in the APB to mitigate risk and achieve program success. Program Implementation: Detect and reduce risks early to avoid greater cost of risk consequences later in the life cycle. The Integrated Product Teams (IPTs) manage risks throughout the implementation phase of their products and services. A Risk Management Plan or risk planning section is developed for the Acquisition Strategy Paper (ASP), and risk-mitigations are documented in the Integrated Program Plan (IPP). Risk management requirements and activities are also included in any prime contract for products or services. In-Service Management: Assess and mitigate the risks continually and adjust mitigation plans. The Risk Mitigation Plan contains actions to scan the program’s operational environment to understand any changes in trends that impact the following:
If the risk status of the program worsens, the In-Service Management organization will develop additional mitigations. 2.0 Risk Assessments During the Acquisition Phases GeneralRisk analysis anticipates uncertainties in a potential capital investment, assesses the degree of the risk, and identifies mitigations. Risk mitigation is critical to an investment because it increases the odds of a program being successful. Paragraph 2.9.13 of the Life Cycle AMS states: "Risk management is applied throughout the acquisition management life cycle to identify and mitigate risks associated with achieving agency goals and objectives. Each line of business shall institute risk management processes that: (1) identify and assess risk areas; (2) develop and execute risk mitigation or elimination strategies; (3) track and evaluate mitigations; and (4) continue mitigations until risk is eliminated or its consequences reduced acceptably." Risk at Each Phase of the AMS
. Click image to view Figure 2-1: Integrated Risk Assessment and Management Approach in IA
Risk assessment during the mission analysis is a component of portfolio management. The portfolio view examines the overall range of programs, i.e., the big picture. It allows the managers to pick the optimum set of programs by taking into account the organization’s goals and how the organization can meet those goals. Early in the AMS, at the Joint Resources Council (JRC-1) mission need decision; the JRC approves a project for further development if it is consistent with the FAA mission, and if there is a need for that project. The decision is made with a full awareness of other projects being implemented, and it implies that the project is consistent with the package of programs already underway. Portfolio management identifies the possible benefits of a program or a package of programs and facilitates tracking the delivery of the benefits. A questionnaire, which addresses the following questions, is used to help determine program risk.
Figure 2-2: Typical Value Plot
For the Initial IA Decision at JRC-2a, the critical issue is determining the most viable acquisition alternatives worthy of further analysis. Consequently, the risk assessment for the Initial IA Decision is macroscopic and comprehensive. The process includes the following:
In general, within the IA Team Alternatives Risk Assessment, there are thirteen facets that constitute an exhaustive range of potential issues, which include: technical, operability, producibility, supportability, benefits estimate, cost estimate, schedule, management, funding, stakeholder, security, human factors, and safety process. Discussion with program staff, users, and stakeholders can help identify risk issues. In the course of identifying issues, the IA Team often formulates and records preliminary mitigations, and estimates the likelihood of the issue surfacing and the potential severity of the issue. The next step may be to convene a team of stakeholders, investment and program analysts, union representatives, and other interested colleagues. A team comprised of a broad skill-set helps to form an objective assessment. The team reviews the material and arrives at consensus of each facet’s rating for each alternative by double blind voting (independent votes not visible to others) and justification discussions. The team also affirms or complements issue mitigations. At the end of these discussions, the least-risk alternative emerges. Depending on the complexity of the investment, a full team analysis may not be necessary (see Section 5.0, Tailoring the Process). The analyst then coordinates with the cost, benefit, and other analysts to link the impact of the risks and their mitigation on the cost and/or the benefit estimate of the recommended alternative. This impact contributes to developing ranges around the most likely cost or benefit element in the respective Work Breakdown Structure (WBS). The products of this effort may be a technical memorandum, provided by the Risk Assessment Team Lead, but most often are:
Lastly, the IA Team summarizes the issue ratings in a matrix for the JRC, and the risk issues database forms the foundation for continued analysis in the Solution Implementation phase. The IA Team Life Cycle Risk Analysis - JRC-2b (Final IA Decision)For the Final IA Decision at JRC-2b, the risks for the preferred alternative are identified and costed in detail to ensure successful implementation of the program. The detailed examination includes: (1) providing a technical description of the risk issues, (2) quantifying the impact on cost, schedule, benefit, and performance baselines, and (3) describing the reason for the expected change. The detailed analysis is based on either past performance history or estimates from a simulation or scenario analysis. The Final APB includes the funding needed to mitigate risks threatening the program’s success. Consequently, the risk analysis for the Final Investment Decision is more focused. At this stage, the IPT Programmatic Risk Assessment is complemented with the remaining IA risk issues. 2.4 Risk Assessment during Solution Implementation - IPT Programmatic Risk Management Process The IPT Programmatic Risk Assessment reviews the JRC-2a risk issues and incorporates the relevant issues. In addition, since the APB includes benefit and performance elements, the IPT working with the IA Staff, complements the focus on cost, schedule, and technical elements by adding the benefits and the performance elements. As envisioned, the IPT Programmatic Risk Assessment begins with the information gained for the Initial IA Decision at JRC-2a and examines all issues from the perspective of the Final IA Baseline Metrics. The metrics are cost, schedule, benefit, and performance. For the Final IA at JRC-2b, the focus is on the issues implications to cost, schedule, benefits, and performance. In the Final IA phase, any interdependency risks (i.e., risks linked to the implementation of one program on the success of other programs) will also be examined and mitigated. The overall IPT Programmatic Risk Management Process entails the following steps:
Identifying risks focuses on staffing problems, design changes, operational issues, test changes, negative trends, etc. Assessing risks focuses on understanding the size and impact of the risk. Risk mitigation focuses on what to do with the risk (avoid, transfer, assume, or mitigate). The Risk Mitigation Plan includes necessary changes in budgets, mitigations, and communication with stakeholders to get support for mitigations. If there are insufficient resources to mitigate the risk issues, the program will probably rebaseline and review all the risk issues. Lastly, the groundwork for monitoring and tracking includes expected timing for the mitigation action and expected impact for reducing risk. This allows tracking of mitigation actions, assessment of their impacts, and the additional follow-up remedial actions. The IPT Programmatic Risk Assessment focuses on cost, schedule, performance, and technical aspects of the preferred alternative from the JRC-2a decision. 2.5 Risk Assessment during In-Service Management The Risk Mitigation Plan is carried out during the Solution Implementation and In-Service Management Phase. It is a product of JRC-2b and contains actions to scan the program’s operational environment to understand any changes in trends that impact the following:
3.0 Integration of IA Team Alternatives Risk Assessment with other Assessments The fact that each phase of the AMS has a risk assessment process underscores the importance of risk assessment in the AMS. While each phase’s risk assessment fulfills functional needs, there remains a compelling need to ensure an integrated framework with a smooth, continuous flow of risk and mitigation information. 3.1Connections with Portfolio Risk Assessment Portfolio management, as of this writing, is only beginning to be applied. Typically, a questionnaire is used to interview project knowledgeable experts about potential risks. With little effort, the survey can be expanded to cover all of the thirteen facets of the IA phase, e.g., operability, human factors, benefit estimate, supportability, and producibility. The IA Staff may use the output of portfolio management to determine the more significant risks of a program and focus on the mitigation of these risks. 3.2 Connection with Human Factors, Safety, and Security Assessments The IA Team Alternatives Risk Assessment considers all risk facets, including human factors, safety process, and security risks. Nevertheless, human factors, safety, and security may conduct separate assessments. The consideration of issues in each of those separate assessments should involve those organizations responsible for conducting them, either directly through a special meeting, and/or by their designated representatives participation in the IA Team Alternatives Risk Assessment, or indirectly through guidance materials from their offices. The Risk Team Lead for the IA is responsible for alerting the appropriate human factors, safety, and security offices in the FAA to ensure that their assessments or inputs can be adequately integrated into the IA Team Alternatives Risk Assessment. Human Factors Assessment The Human Factors Assessment for Investment Analysis entails a process that provides essential and inter-related components to the products of the Investment Analysis Team, including: a) the human-system performance contribution to program benefits, b) an assessment of the human-system performance risks, and c) the estimated costs associated with mitigating human factors risks and with conducting the engineering program support. The human factors components related to benefits, risks, and costs are integrated with other program components in the IA products and documentation to ensure that
3.3 Connections with IPT Programmatic Risk Assessment Initial Investment Decision at JRC-2a The IA Team Alternatives Risk Assessment contains a high-level comparative assessment between all alternatives and across all risk facets. The products of the assessment are a risk assessment briefing, technical memorandum, and/or a risk management matrix. The risk management matrix is required for the Initial APB. These products should be comparable in detail with the IPT Programmatic Risk Assessment. The list of major risks and mitigation strategies identified for the preferred alternative are examined in greater detail at the Final IA Decision to form an APB that is stable throughout the Solution Implementation phase of AMS. Final Investment Decision at JRC-2bThe products of the IA Team Alternatives Risk Assessment feed into the IA Team Life Cycle Risk Assessment and gives the IPT a starting point for the IPT Programmatic Risk Assessment. The Office of Architecture and System Engineering and the Office of Investment Analysis and Operations Research, in collaboration with the IA Team, will provide input on the assessment of cost, benefit, schedule, performance, and interdependency risks. The product of the IA Team Life Cycle Risk Assessment is the identification of detailed cost, benefit, schedule, technical, and performance risks, and respective mitigation strategies. These mitigation strategies are discussed with subject matter experts to estimate their impacts on cost, schedule, benefit, and performance baseline that the JRC is to approve in the Final IA Decision. Another product of the IA Team Life Cycle Risk Assessment is the Risk Mitigation or Management Plan, which guides the understanding and mitigation of risks throughout the Solution Implementation and subsequent phases of the AMS. Connections during Solution ImplementationThe Risk Management Plan proactively guides risk monitoring and mitigation during the investment’s implementation. External and internal programmatic events may affect the implementation. Accordingly, it will be useful to anticipate changes and develop contingencies for mitigating them. In some cases, there are such significant changes that the program cannot mitigate the risk consequences because either the impacts exceed expectations or a new unanticipated risk materializes, and must undergo a rebaselining to get the program back on track. In undergoing a rebaselining, the program typically revises the baseline metrics of costs, schedule, performance, and benefits to reflect the current circumstances. When this is necessary, the Program Office, together with the IA Staff, updates the risk assessment reported in the original APB with a traceable record of how and why the risks changed, and expected mitigations for the risks. Reporting of risks for rebaselining follows the same format and documentation as reporting for the original program baseline. However, an in-depth review should be necessary only for those risk facets whose risks have worsened. Any rebaselining of a program must accompany the record of risk changes for original and past rebaselines. 4.0 IA Team Alternatives Risk Assessment Risk FacetsThe life cycle risks are broken down into thirteen facets of risk. These risk facets have been selected to facilitate risk identification and quantification. The thirteen risk facets are defined as follows:
RiskOperability RiskProducibility RiskSupportability RiskBenefit Estimate RiskCost Estimate RiskSchedule RiskManagement RiskFunding RiskStakeholder RiskSecurity RiskHuman Factors RiskSafety If the risk assessment team determines a facet to be inconsequential, the facet can be eliminated, but only if no perceptible risks for the facet can be determined. A risk checklist (see Table 6-2) can be used to assist the analysis in determining whether a facet should be excluded. The rationale for excluding any facet should be documented. The risks associated with each facet should consider the whole life cycle of the investment. Therefore, the thirteen risk facet definitions must be flexible enough to include the development of the investment through the full life cycle. Facet definitions need to be flexible because they must cover all potential risk situations or program evolutionary stages. Some programs may be at such a preliminary stage that the risk facet definitions may be difficult to apply. The risk team may need to adjust and agree to definitions to handle more mature programmatic risks and preliminary investment risks. Clarification of Risk Facets for Benefits and Costs EstimatesSymmetrical Risks Associated with Uncertainty Risk may increase or decrease an estimate for benefit performance and cost values because they may either ameliorate or worsen the behavior of a system. Therefore, risk considerations should include both sides of the uncertainties associated with the estimates. One way to capture this in the IA is to include high and low estimates or a probabilistic distribution around both costs and benefits. If the benefits are underestimated, the risk lies in the denial for requested funding of the program. Similarly, if costs are overestimated, it is possible that the program will not be funded or another alternative will be chosen with fewer capabilities. If both cases occurred simultaneously, benefits were underestimated and costs overestimated, either the alternative will be different (as well as the accompanying capabilities) and/or the project may not be funded at all. 4.3.2 Uncertainty of Cost and Benefit Assumptions, and Qualitative Benefits Risk benefit estimate considers the difficulty in estimating the benefits and in realizing benefits. This risk facet addresses the accuracy of the benefit estimate and issues such as:
Qualitative benefits should be characterized in as much detail as possible, and their uncertainty projected. Often qualitative benefits cannot be measured in financial terms, or in physical numerical terms. This becomes important when there are potentially large unquantifiable qualitative benefits, such as enabling benefits from advanced technology. Therefore, the risks to the accuracy of the benefits calculations should be included in the risk assessment. Cost and Benefit Facet Risk UsesThe IA Team Alternatives Risk Assessment provides more information about the uncertainty in the cost and benefit estimates. This information helps to establish the cost and benefits ranges, which in turn are used to develop the high confidence cost and benefits. The risk mitigation costs can be included in the total cost figures and for traceability and validation should be closely linked to the WBS for costs. Similarly, risk ranges can be developed for the benefits calculations by estimating uncertainties for all benefits assumptions. IA Team Alternatives Risk Assessment ProcessMatrix Table Results of Probability and Severity A frequency analysis using a matrix risk table with probabilities and severities on the X and Y coordinates should be constructed. With three levels of probabilities and three levels of severity, development of the risk matrix will yield a 3-row by 3-column table. According to the Risk Facet Weighting Score Assignment in Section 6, those risks that are high correspond to (high probability with substantial severity); (medium probability with substantial severity); and (high probability with moderate severity). The definitions of medium and low overall ratings are in Section 6. This can be done for individual risks, as well as by risk facet. By applying this technique to all of the individual risks and placing their tallied number in each box for probability and associated severity, the decision-maker can see the total number of individual risks and their distribution through various combinations of both probability and severity. For example, in Figure 4-1, the number 6 in the cell corresponding to high probability and substantial severity denotes that there were 6 total risks with this probability and severity. The sum of all of the numbers in the table represents the total number of risks evaluated. Please see Section 6 for more detail on the definitions of probability and severity associated with each risk facet. Figure 4-1: Example of Risk Facet Rating Matrix Severity of Impact
Alternatively, the matrix could be applied to risk facets, and by tallying the facets for each box in the matrix table, a distribution of the thirteen facets can be clearly evaluated. Furthermore, the risk facets can be given an initial corresponding to the risk facet placed in each box in the table. Additional information will allow the decision-maker to view the highest-ranked risk facets. Objectivity is essential to a successful risk assessment. To ensure objectivity, the risk team evaluators must have an open mind and have appropriate knowledge and experience. Moreover, the IA Team must be balanced by a broad perspective of members (i.e., users, airline representatives, unions, sponsor(s), implementers, and any other stakeholder). Group review of individual risks is important to ensure objectivity as it allows those with broader experience to add a different perspective and provide specific information regarding risk mitigation strategies. Part of the review may include briefing other interested parties or stakeholders throughout the agency on the results. All risks should be verified by either data and/or supporting documentation via risk databases, studies, reviews, or publications. Increasing the credibility, validity, and traceability of the risk assessment will enhance the objectivity of the assessment. The IA Team Risk Lead is responsible for tailoring the Risk Assessment to fit a respective acquisition. This tailoring involves evaluating the acquisition to determine the effort required, choosing an approach for the Risk Assessment, and selecting the members of the Risk Assessment Team, if necessary. 5.1 Criteria for Choosing the Level of Effort Determining the level of effort required for a Risk Assessment involves a preliminary evaluation of the information currently available on a certain program or project. This evaluation involves determining the scope, magnitude, complexity, and schedule of the proposed acquisition, as well as the impact it will have on the NAS. Other factors to consider are FAA Advisory Board (FAB) waivers and any congressional mandates that may have been issued. Information in these areas can be obtained from documentation on the proposed technology, or documentation on the proposed program such as an Initial Requirements Document (IRD) or Concept of Operations document. Subject matter experts and program office personnel are also valuable sources of information in this preliminary evaluation. ApproachesThe IA Team Risk Lead determines the effort required and selects a corresponding approach. When an acquisition is on the low end of scope, cost, and complexity, the IA Team uses staff reviews. This approach involves an internal ASD-400 review of documentation and comments on facet weights in Risk Issue Worksheets (see Appendix B) for any risks they have identified. The IA Team Risk Lead then integrates all risks into a Risk Issue Briefing and an assessment for the IAR. One-Day MeetingWhen an acquisition is thought to be low to mid-range in scope, cost, and complexity, the IA Team uses the one-day meeting approach. The IA Team Risk Lead facilitates this meeting involving the IA Team and representatives from contributing organizations. The IA Team Risk Lead briefs the Risk Assessment Guidelines for the Investment Analysis Process and distributes documentation. The IA Team and representatives from contributing organizations together complete Risk Issue Worksheets for any risks they have identified and come to consensus on facet weights. Risk assessment, based on the risk issues identified, is accomplished as part of this one-day meeting. Results may be available at the end of the meeting when the assessment is computerized. The IA Team Risk Lead then integrates all risks into a Risk Issue Briefing and an assessment for the IAR. When an acquisition needs more in-depth expertise to determine individual risks, mitigation strategies, and accompanying probabilities and severities, the IA Team uses the sub team’s approach. Sub teams and sub team leads are chosen for each facet from the IA Team, the sponsoring program office, and representatives from contributing organizations. The IA Team Risk Lead facilitates an initial risk meeting involving the IA Team, the sponsoring program office, and representatives from contributing organizations. The IA Team Risk Lead briefs the Risk Assessment Guidelines for the Investment Analysis Process and distributes documentation. The entire group comes to consensus on facet weights. Sub team leads are then chosen and each sub team lead completes a Risk Issue Worksheet for any individual risks that have been identified. The sub team leads make any necessary revisions to the worksheets and submit them to the IA Team Risk Lead. The IA Team Risk Lead then integrates all risks into a Risk Issue Worksheet and an assessment for the IAR. Full IA Team MeetingsWhen an acquisition has a large scope, cost, and/or is of high complexity, the IA Team convenes a full IA Team meeting. The IA Team Risk Lead facilitates an initial risk meeting with the IA Team, the program office, representatives from contributing organizations, and subject matter experts. The IA Team Risk Lead briefs the Risk Assessment Guidelines for the Investment Analysis Process and distributes documentation. Sub teams and sub team leads are chosen from contributing organizations, the sponsoring program office, and the IA Team. Each sub team member then completes a Risk Issue Worksheet for any risks they have identified and submits them to the sub team lead. The sub team leads then makes any necessary revisions to the worksheets and submits them to the IA Team Risk Lead. The IA Team Risk Lead integrates all risks into a briefing for the second meeting where all risks, mitigation strategies and accompanying probabilities and severities are agreed to. The IA Team Risk Lead then integrates all risks into a Risk Issue Worksheet and an assessment for the IAR. Team MembershipThe selection of team members for a Risk Assessment should start with the IA Team members, then expand to include the sponsoring program office, contributing organizations, subject matter experts, and any other stakeholder. As previously discussed, the level of effort and approach will help determine the Risk Assessment Team membership. How to Select and Where to Find Team MembersRisk Assessment Team Members can be selected by contacting major stakeholders in IA and the various divisions responsible for contributing different information and assessments to the IAR. Appendix A is provided for guidance as of April 2002 and is subject to change. 6.0 STEPS IN THE RISK ASSESSMENT PROCESS The risk assessment process is comprised of the following eight steps:
Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Risk identification is an organized and thorough approach to seek out risks. It is not a process of trying to invent highly improbable scenarios of unlikely events to cover every conceivable future possibility. A Top-Level Risk Matrix (Table 6-1 shows a sample) may be used for each alternative to provide a structured and consistent risk identification process for the thirteen risk facets and to document the results. A Top-Level Matrix is useful for helping frame the development of risks, as the goals, plans, and risks are defined. This step is optional, if a framing context is unnecessary.
Requirements specified in an IRD should be considered in defining goals. If the requirements are not explicit enough to yield goals related to the risk facets, this is an alert that the requirements might need to be more explicit. Define Plans Relating to Risk Facets Identify Risks
The above three steps do not have to be sequentially completed. Iterating among the steps may be helpful. As more risks are listed, the goals and plans may be revised. The risks listed under each risk facet in Table 6-1 provide the basis for the risk quantification in Section 7. The statement of program goals relating to the risk categories and the plans for mitigating the risks will help quantify the risks. Risks cannot be assessed or managed until they are identified and described in an understandable way. Table 6-1 is presented with sample entries in each box to clarify how the table is used. The sample entries are constructed for a possible alternative related to satellite surveillance. Table 6-1: Top-Level Risk Matrix (Sample)
As an aid in completing the risk lists in Table 6-1, use Table 6-2, Risk Checklist by Risk Facet. Use this table as a starting point for listing risks for any alternative. Table 6-2 is comprehensive and addresses all program stages; hence, the table may contain risk elements not appropriate to the IA phase or to a particular alternative being assessed. The relevant items in the checklist should be evaluated to determine whether they apply to the particular alternative. Other potential risks not listed in Table 6-2 should be added to the risk checklist for the particular alternative. The alternative's risk checklist should contain all possible risks that might be related to the alternative. After listing all possible risks, those which are extremely unlikely or where the outcome is irrelevant to program goals should be eliminated from the list. The checklist should be directed towards those that will have a meaningful impact on the program, such as impacts on milestones on the critical path. All meaningful risks should be listed in Table 6-1. Table 6-2: Risk Checklist by Risk Facet
Table 6-2: Risk Checklist by Risk Facet, Cont’d
Table 6-2: Risk Checklist by Risk Facet, Cont’d
Table 6-2: Risk Checklist by Risk Facet, Cont’d
The IA Team assigns a numerical weight to each risk facet reflecting its importance in the successful implementation and operation of the program. Obviously, a program is successful when it fulfills its mission need. The following are important points to note in assigning facets weights:
Assigning risk facet weights is a way of conveying the importance of each facet in bringing about the success of a program. To achieve objectivity in assigning risk facet weights, the risk team members should determine facet weights in a full team meeting before the evaluation of the individual risks. If the determination of the facet weights were to occur during the risk evaluation or before evaluation of the alternatives, risk team members might be biased. However, team members may want to make small adjustments to the weights after the assessment for sensitivity purposes. Any change in weights at this point should be well documented and explained. Step 3 - Estimate Severity of Impact of an Adverse EventFor each risk, the severity of the impact of the adverse event on the alternative (expressed as substantial, moderate, or minor) is determined using Table 6-3, Estimating the Severity of Impact of an Adverse Event, as guidance. The severity from the highest rated risk within each facet is entered in the third column of Table 6-5. To estimate the severity of impact the IA Team can use expert interviews, analogy comparisons, evaluation of program plans, or the Delphi method. The IA Team can also use multiple methods and approaches other than those listed. 6.4 Step 4 - Estimate the Probability of an Adverse Event In spite of attempts to be analytic about quantifying risks, considerable subjectivity remains. The degree of risk perceived in a situation is partially a reflection of the personality of the risk assessor(s). A risk-rating scheme built against a set of definitions eliminates some of the ambiguity. Further, the rating scheme should be simple. The following risk rating scheme involves determining a high, medium, or low overall risk rating using the notion that the degree of risk is a judgment reflecting the probability of occurrence of an adverse event and the severity of impact on the alternative should the adverse event occur. If a particular risk facet does not apply to the alternatives being assessed, then the probability of an adverse event and the severity of the impact of the adverse event do not need to be estimated for that risk facet. For each risk, the probability of occurrence of an adverse event (expressed as high, medium, or low) is determined using Table 6-4, Estimating the Probability of an Adverse Event, as guidance. The probability from the highest rated risk within each facet is entered in the second column of Table 6-5, Determining Overall Weighted Alternative Risk Score. Below are four possible methods to estimate the probability of occurrence and severity of impact. More than one method, as well as approaches other than those listed below, can be used.
Analogy Comparisons Evaluation of Program Plans Delphi Technique
Table 6-3: Estimating the Severity of an Adv | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
