Title: FAA standard benefits analysis methodology

Version: Final

Type (Standard or Guideline): Guideline

Origination Date: October 29, 2002

Approval Date: November 22, 2002

Purpose: Guidelines for a standard process for preparing benefit estimates are needed to ensure that FAA decision-makers receive the same, consistent, and high quality benefits information that they need to evaluate and decide on the particular question at issue, irrespective of who performs the estimate. Moreover, by adopting a standard process, it should ease the burden and reduce resource requirements for preparing benefits estimates by permitting standardized training, techniques, and data to be used across all benefits estimates efforts in the FAA.

Applicability: This process applies specifically to all benefits estimates generated in support of the initial investment decision (JRC-2a). However, the generic estimating process described in this Guideline is closely related to, if not identical to, benefits estimates prepared in support of many other FAA decision-making needs (e.g., review and approval of a final investment decision). Thus, this generic process is provided as guidance for all FAA benefits estimation analyses, but needs to be tailored to the particular situation. For example, at the JRC-2a stage, more general NAS-level benefits may be generated that are sufficient to differentiate and highlight the relative merits of each alternative. For the JRC-2b final investment decision, the benefits estimate should be more detailed and site-specific, based on a greater definition/ understanding of the proposal and less uncertainty about the quantitative and qualitative outcomes of the proposal. That is, for the JRC-2b final investment decision, the benefits baseline should be in sufficient detail so that progress toward realizing those benefits could be measured. This would have the added value of facilitating a post-implementation assessment. In either case, the same general process should be followed.

Description: The standard benefits estimation process is an eleven-step effort that flows sequentially from the earliest step (Understand the Program) to the last step (Complete Final Report). It will provide the information needed by FAA decision-makers, and will satisfy the guidance documents specified by the Executive Branch (e.g., OMB Circular A-94). It is focused on addressing uncertainty by creating systematically a high confidence, risk-adjusted estimate of the benefits that will accrue over the life-cycle of the proposed program, together with the likely range of estimates that may accrue (i.e., the most pessimistic estimate to the most optimistic estimate). Generally, the pessimistic estimate of benefits will be used for assessing the relative merits of the proposal, and will be baselined and tracked in the Acquisition Program Baseline (APB) of the approved investment program.

Content: The standard process is shown as an attachment. It is stored in FAST at http://fast.faa.gov/TBD. Additionally, a more detailed process for conducting benefits estimates is contained in the ASD-430 Standard Operating Procedure (SOP) (The Art of Benefits Prediction and the Statistical Science of Post Implementation Assessment in Aviation Investment Analysis), which interested practitioners may consult in

http://faa.gov/asd/ia-or/or-reports.htm . The ASD-430 SOP is not intended to be considered as guidance for all FAA personnel, but still it may be useful in understanding in greater detail how a benefits estimate is done for initial investment analysis.

Configuration Management: This Guideline will be maintained by the Investment Analysis and Operations Research Directorate (ASD-400) in the FAA Acquisition System toolset (FAST). It will be reviewed at least annually by ASD-400. If any changes are deemed necessary, they will be proposed by ASD-400 through the normal ASAG/ FAST change process.

1. Context: Benefits estimation is as much an art as a science. The benefits analyst must be creative and imaginative, looking for the most effective way to define and then quantify the benefits to be derived from any proposed new investment program. The analyst needs to work intensively with the most knowledgeable people that will be directly affected by the project; understand fully the economic and operational properties of the proposed program; ask probing, open-ended questions that enable extraction of important data on the investment’s likely risks and benefits; and then synthesize and tailor the best approach for quantifying and monetizing the benefits. One of the major issues is uncertainty and risk. The analyst needs to understand and quantify the range of possible benefits (from pessimistic estimate to optimistic estimate), since there are many uncertainties and risks that will influence the actual achievement of benefits if they come to pass.
2. Process: In Figure 1 below, an eleven (11) step process is defined that enables the analyst to develop a compelling and valuable final report on the estimated benefits of legacy, rebaselined, or new programs. Each of these steps and their products is described briefly below.

Figure 1. Benefits Analysis Methodology

1. Understand the Program: The benefits analysis team must develop and document a good understanding of the proposed new program, including the Reference Case (sometimes called the "do nothing" alternative) and the set of alternative approaches ("do something" alternatives) to be evaluated. Frequently, this is documented in a "Technical Description", which serves as a summary of the characteristics of the program in sufficient detail to enable the subsequent development of benefits estimates for each alternative. In all cases, the Reference Case is considered the "baseline" that is to be compared to all the alternatives. The objective of the entire analysis will be to derive the marginal benefits of each "do something" alternative; i.e., the degree to which the benefits of each alternative exceed the benefits of the Reference Case. The decision-maker needs to know how costs and benefits will behave in the future as a consequence of doing something different from the Reference Case. Therefore, it is not necessary to know the absolute benefits to be derived from the reference case; all that is needed is the marginal or incremental benefits to be derived from each of the alternatives. In all cases, a similar technical description document will be needed to enable cost analysis to be done, so that the benefits analysis team and the cost estimating team should work collaboratively to develop a mutually useful technical description that meets both their needs and minimizes the duplication of effort. The technical description generally will cover these areas:

1. System overview
2. The Reference Case/ "Do Nothing" alternative
3. The set of alternatives to the Reference Case to be evaluated
4. Assumed economic service life
5. System functional relationships
6. System interfaces (including Human- System interfaces)
7. Benefits Breakdown Structure
8. System configuration (hardware/ software)
9. System technical/ operational performance characteristics
10. System quality expectations
11. Major risks threatening the achievement of benefits
12. Predecessor and successor systems
13. System operational concept
14. Quantity requirements/ number of systems
15. Locations/ site-specific system placements
16. Manpower requirements for system operation and maintenance
17. System activity rates (e.g., number of flights handled per day)
18. System implementation/ deployment schedules and milestones
19. Acquisition strategy
20. System development plan
21. System facility requirements

Output Product: A technical memorandum summarizing the data described above.

2. Finalize Initial Planning: This second step develops the Benefits Analysis plan. Like any plan, it describes the tasks needed to complete the benefits estimate; the deliverable products of the analysis; the schedules and resource requirements for each task; and the roles and responsibilities of each organization/ person in completing the effort. New teams normally should consult with previous benefits estimating teams and review their Plans so that they can take greatest advantage of the "lessons learned’. The content of the Benefits Analysis plan should be:

a. The objectives of the effort (e.g., when the benefits estimate must be completed to meet the schedule of FAA decision-makers)

b. A summary of the reference Case and the set of alternatives to be evaluated

c. The benefits estimation tasks required to be completed

d. The deliverables/ products to be completed and their needed completion dates

e. Schedules and resource requirements for each task

f. Roles and responsibilities of each team member/ organization

g. The method for coordinating and controlling the team’s efforts (e.g., how often team meetings for progress reviews will be held and when)

h. How to resolve any uncertainties or disputes that may arise (e.g., how to get executive decisions when there are differences of opinion or uncertainties in data)

Output Product: A Benefits Analysis Plan accepted and agreed by all Team members and the sponsoring executive needing the Final Investment Analysis Report.

3. Evaluate the Enhanced Functionalities of the Alternatives: Once the technical description is completed, the actual benefits analysis can begin. Benefits analysis ultimately leads to an estimate of the marginal economic value of the program in dollars, as well as a summary of the benefits that can not be monetized in dollars. To get at monetary and non-monetary values, it is first necessary to determine the physical or operational value of the goods or services, which the program/ alternatives will yield over the evaluation period. These are usually defined in physical or operational units or terms that represent enhanced functionality (e.g., changes in operations productivity, such as more flights handled per controller, permitted by greater labor-saving automation), which can then be monetized. The initial step is accomplished by understanding how each of the alternatives will look into the future in terms of enhanced functionality (better features and benefits) as compared to the reference case. To understand enhanced functionality, the analyst needs to answer these kinds of questions:

• What will the investment physically or operationally do?
• How can I measure that? What historical data are available for creating a metric? What tools are available for predicting future values of the metric?
• How will this new program permit new FAA or ATC procedures? How will it reduce regulations or permit greater flight efficiency?
• How does the NAS operational concept look with the new capability?

Output Product: A technical memorandum projecting the reference case and each alternative in terms of enhanced functionalities leading to benefits.

4. Identify Relevant Metrics: The analyst, working with the sponsor, identifies and selects those metrics that are expected to improve as a result of the new program or investment. These are driven by the enhanced functionalities that the new program will provide. Normally, the selected metrics should be the same metrics that have been selected by the FAA management team to drive and evaluate the performance of the FAA as a performance-based organization (PBO). The extent of improvement of these metrics represents the benefits of the new program or investment. Depending upon the type of program, these are the kinds of metrics that might be selected:

• Capacity increases (e.g., demand-to-capacity reduction)
• Safety (e.g., accident rate, severity of accidents)
• Efficiency (e.g., passenger delays, operational delays)
• Environment (e.g., emissions reduction)
• Productivity (e.g., number of aircraft handled, avoided staffing costs)
• Security (e.g., number of intrusions, hijackings, severity)

Output Product: Set of metrics to be baselined and projected

5. Establish the Reference Case for Each Metric: This step derives the Reference Case ( the "do nothing" alternative, or how the world will look if no investment is made). The Reference Case consists of a pre-investment baseline statistically derived from the available historical data, and the projection of that baseline into the future if no investment or change is made. This pre-investment baseline is for each metric that will be quantified and monetized in the subsequent steps. It is important to look back as far as possible into the past to derive the pre-investment baseline, so that the trend line from historic data can be extrapolated with greater confidence into the future. The reference case projection assumes no change is made; each alternative’s projection assumes some change is made. In this step, only the reference case projection is developed.

Output Product: The reference case for each metric that will be projected into the future, including the projection of each metric baseline into the future for the Reference Case only.

6.  Define the Benefits Impact, Quantifiable and Non-Quantifiable, for Each Alternative: This step predicts and captures the marginal benefits for each metric between the reference case projection (established in step 4) and all the alternatives. In many cases, benefits can be both measured in physical terms (e.g., reduction in delays) and in monetary terms ($ saved as a result of delay reductions). In other cases, the benefits can be quantified in physical terms (e.g., reductions in harmful emissions), but it is difficult, if not impossible, to monetize the value. In still other cases, the benefits can neither be measured physically nor monetized (e.g., improvement in quality of life due to smoother flights with less turbulence). Still, to the extent possible, all of these benefits should be captured and projected to aid the FAA decision-maker. For qualitative benefits, they should be at least listed and described, even if they cannot be measured or monetized.

Output Product: Benefits impacts (quantified and non-quantified) for each alternative for each metric

7.  Calculate/ Predict Future Metrics Values: For those benefits that will be quantified, it is necessary to estimate metric values for each year of the project life cycle. This must be done for the reference case and each alternative. These estimates are generally made by using deterministic (parametric) models, probabilistic (simulation) models, and/or analytic techniques and are often dependent on other variables. In one example, future air traffic delays may be dependent on predictions of future aviation traffic. Estimates of future traffic have "standard" values obtainable from the latest APO forecast documents. These forecasts, along with assumptions about the effects of a program alternative, may then be used as inputs in a simulation model that predicts delays. In another example, that of an infrastructure program, expert opinion may need to be used to establish values for an independent variable, e.g. mean time to restore (MTTR). In unique circumstances, special purpose models may need to be developed to reflect the singular estimating conditions.

Output Product: Estimation of baseline metrics for each alternative into the future

8.  Convert to Economic Values ($): The analyst must translate the quantified extrapolations of physical benefit improvements into monetary (dollar) estimates, so as to enable comparisons of costs and benefits in the same economic terms. In some cases, this is fairly straightforward, as in translating in-flight time savings into aircraft direct operating costs (ADOC) and passenger time savings. In other cases, it is more difficult, especially when the benefits are shared between two or more interdependent programs and some allocation must be made. One of the major concerns is risk-adjusting the benefits estimates. Usually, this is best accomplished by developing a triangular probability density function (PDF) for the value of the benefits. In combination with Monte Carlo simulation, this produces a cumulative density function (CDF) from which the analyst can select a high-confidence benefits estimate (i.e., an estimate of benefits which is calculated to provide an 80 probability of being overrun in actual program execution).

Output Product: Risk-adjusted benefit estimates for each alternative in constant and then-year dollars.

9. Compare to Reference Case Values: The analyst must derive the marginal benefits of each alternative in comparison to the reference case. This should be done on both a quantitative basis (monetized and non-monetized marginal benefits) and a qualitative basis (pros and cons of each alternative versus the reference case). To justify investment, it must be shown that its marginal benefits are greater than its marginal costs; both being compared to the reference case/ "do nothing" alternative. The analyst must examine carefully the risks to achieving the benefits. Finally, the analyst must develop a recommended alternative to pursue, based only on consideration of benefits and benefits risks. The entire Investment Analysis Team will examine all criteria (e.g., costs, risks, strategic alignment) before deciding on the best option to recommend to the FAA decision-maker.

Output Product: A technical memorandum of the benefits estimation effort. It should contain a summary of all efforts, and a comparison of reference case to all alternatives, considering both quantitative and qualitative benefits aspects. It should include as well a recommended alternative to pursue and a rationale for that recommendation, based only on benefits and benefits risks considerations.

10. Review and Coordinate: The benefits estimate must be reviewed and approved by all the major stakeholders prior to its documentation and presentation to the FAA decision-maker. If necessary, re-estimates may be needed using different data and assumptions to correct any defects that may have been introduced. Similar, additional sensitivity/ "what if" estimates may be generated to test the robustness of the recommended option in the light of different assumptions. The stakeholders begin with the members of the investment analysis team first, and then radiate throughout the FAA at large. Coordination and review are necessary to ensure quality control, to verify that the correct input data has been used, and to forge consensus on the results of the benefits estimation. This will ensure an objective and well-informed understanding of the anticipated benefits by the FAA decision-makers, as well as a lack of most uncertainty at the decision point.

Output Product: Resolution of any issues. Draft technical memorandum on benefits estimation, and develop the benefits portion of the briefing to the FAA decision-maker.

11.  Complete Final Report: This last step records the effort and results of all previous steps, both to prepare for an FAA decision meeting and to populate the historical databases of all investment analyses. The best approach is to drawn on interim technical memorandums to prepare the Final Report. If previous technical memorandums are written very early in the benefits estimation effort so as to ensure accuracy and completeness, then the need for a last-minute Herculean effort to complete the report will be avoided. In all likelihood, the final report will be reviewed and scrutinized subsequently by numerous persons and groups in and out of the FAA (e.g., GAO, OMB, DOT IG, etc.). Therefore, the report must be written in easy-to-understand, layman’s language. Each stakeholder/ reader will have various interests, but all will require a well-documented, high-quality final report so as to understand and to judge the effectiveness of the investment decision. Therefore, the preparation of a very good final report is critical to the credibility of the benefits estimation effort. It must provide an "audit trail" that can be followed easily and understood by inside and outside interest groups/ stakeholders.

Output Product: Final Benefits Estimate Report, including Basis of Estimate.