April, 2003                                       Version 1.0

 Prepared by:

Investment Analysis and Operations Research, ASD-400

Investment Cost Analysis Branch, ASD-410

Federal Aviation Administration

Guidance:

1.  Context: Cost estimation is based on both engineering and economics. The cost analyst must be familiar with the physical and operational characteristics of the proposed program at an engineering level of detail so that the likely costs can be estimated through standard techniques such as analogy or parametric estimation. The analyst needs to work intensively with the most knowledgeable people that will be directly involved in the project (frequently, this is the systems engineers and other technical specialists) that understand the concepts, the physical nature, and operational properties of the proposed program; ask probing and open-ended questions that enable analysis of the program’s likely costs and risks; and then synthesize an accurate cost estimate from all the diverse data.

Cost estimation for the FAA defined here is concerned with deriving the likely costs of a specific future activity or program. The intent is to provide a realistic cost estimate so the decision-maker can judge the relative merits of the proposal against its costs. One of the most difficult cost issues is uncertainty. Since the new investments are unique and have no exact historical antecedents, there can be substantial uncertainty associated with predicting the likely actual costs. The analyst needs to understand and apply these uncertainties and risks to derive probabilistic measures of cost outcomes (i.e., the most optimistic estimate to the most pessimistic estimate of life cycle costs). Most importantly, the analyst must always prepare a risk-adjusted cost estimate, i.e., a cost estimate that reflects the uncertainty associated with what the actual costs are likely to be.

2.  Process: In Figure 1 below, an eight (8) step standard process is defined that enables the analyst to develop a compelling and valuable final report (a Basis of Estimate, see (BOE) guidance at http://fast.faa.gov/investment/iapg.htm on the estimated costs of proposed new programs). Each of these steps and their products is described briefly below. The required depth and extent of each step is dependent upon the relative size and cost uncertainty of the proposed investment itself. Typically, the largest dollar size, most complex, and most uncertain investments will require a full and in-depth cost estimation effort, while lesser scope and levels of investment and uncertainty may be addressed satisfactorily with a lesser depth of analysis.

Figure 1: FAA Standard Cost Estimation Methodology

1.0 Understand The Proposed Program And Existing Estimates: The cost estimating team must develop and document an in-depth understanding of the proposed new program, including the Reference Case (sometimes called the "do nothing" alternative), the Capital Investment Plan (CIP)-constrained alternative, and the set of remaining feasible alternative approaches to be evaluated. A cost analysis to support an initial JRC-2a investment decision always must include life cycle cost estimates (LCCE) for the Reference Case, a CIP-constrained alternative, and other feasible alternatives. For a JRC-2b investment decision, a more detailed life cycle cost analysis for the selected or preferred alternative will be undertaken to establish the program’s baseline cost. For replacement programs, there may be a need to update the costs of the reference system to support a cost-effectiveness analysis to support the development of the business case.

This understanding of the program can be gained by review and analysis of appropriate program documentation (e.g., Requirements Document), by discussions with project subject matter experts (SMEs), and by a review of previous cost estimates relating to the program (if any exist). Frequently, this understanding is documented in a "Technical Description", which serves as a summary of the characteristics of the program and its alternatives in sufficient detail to enable the subsequent development of cost estimates for the Reference Case and each alternative.

The Reference Case is considered the "baseline" that is to be compared to all the alternatives. The objective of the entire analysis is to derive the marginal costs of each "do something" alternative, i.e., the degree to which the costs of each alternative exceed the costs of the Reference Case. The decision-maker needs to know how costs and benefits will behave in the future as a consequence of doing something different from the Reference Case. Like the Reference Case, the CIP-constrained alternative must always be included. It is the program that fits the available CIP funding profile.

In all investment analyses in which cost-benefit studies are needed, a similar technical and operational description document will be needed to enable benefit analysis to be completed. In these cases, the benefits analysis team and the cost estimating team should work collaboratively with the Integrated Product Team to develop a mutually-useful technical description that meets all needs, ensures consistency in input assumptions and data for all teams, and minimizes the duplication of effort.

The technical description generally will cover these areas:

a.   Technical Data

- System overview

- Description of the Reference Case

- Description of the CIP-constrained alternative

- Description of the remaining feasible set of alternatives to be estimated

- System functional relationships

- System configuration (hardware and software)

- System technical and operational performance characteristics

- Concept of operations

- System activity rates

- System requirements

- System facility requirements

- System Interfaces

b.   Programmatic Data

Output Product: A technical memorandum summarizing the data described above.

2.0 Cost Analysis Planning: With an understanding of the program, the cost estimating team then develops a cost analysis plan and a tailored work breakdown structure (WBS) for each alternative that is to be cost-estimated. Each WBS is tailored from, derived from, and consistent with the FAA Standard WBS (available at http://fast.faa.gov/investment/iapg.htm).

Once the WBS is established, a cost estimating methodology must be identified for each element. Numerous methodologies are available and the analyst selects an appropriate one. (For details on cost estimating techniques, the analyst should consult the FAA Life Cycle Cost Estimating Handbook, which can be accessed at http://www.faa.gov/asd/ia-or/lccehb.htm).

Before the actual cost estimating begins, a cost analysis plan should be developed and agreed upon. Like any plan, it describes the tasks needed to complete the cost estimate, the deliverable products of the analysis, the schedules and resource requirements for each task, and the roles and responsibilities of each organization or person in completing the effort. The content of the cost analysis plan should include:

a. The objectives of the effort

b. Identification of the estimates to be developed, including the Reference Case, the CIP-constrained alternative, and the set of remaining feasible alternatives to be evaluated

c. The cost estimation tasks required to be completed

d. The methodologies to be used to derive a cost estimate for each WBS element.

e. The deliverables and products to be completed and the expected completion dates

f. Schedules and resource requirements for each task

g. Roles and responsibilities of each team member/organization

The tailored WBS for each alternative forms the framework for the cost estimation effort. Each tailored WBS is based on the specific work packages that would be needed to implement and operate the proposed alternative. The total life cycle cost is derived by summing the costs of all of the WBS elements.

Normally, the cost estimating team needs to work with project personnel to develop a complete WBS for each alternative representing all needed tasks. The tailored WBS selects those WBS elements from the FAA Standard WBS, which will be appropriate for that particular alternative, and augments the WBS (if needed) to the appropriate level to develop a realistic cost estimate. The level of WBS depth will depend upon the relative maturity and understanding of the new program. Generally, the tailored WBS should be created only to the level that is sufficient to understand risk, to permit accurate cost estimation, and to serve later as the baseline for earned value management. In some cases, several FAA standard WBS elements may not be used at all .

Output Product: A cost analysis plan accepted by all team members. If appropriate, this plan may be included as a part of the Investment Analysis Plan (IAP), rather than published as a separate document. Appended to the plan should be tailored WBS for each alternative, each based on and drawn from the FAA Standard WBS.

3.0 Collect Data For Each WBS Element: The total cost of any WBS element is based on the extent of work required to be performed, which is usually driven by the value of certain work-related physical parameters. For example, for software development, the number of source lines of code (SLOC) to be developed is a strong predictor of the required workload and thus the likely cost of that WBS element. A simpler example is the cost of program management support, which might be estimated as a function of the number of staff hours times the dollar rate per hour.

Cost risk is based on the possible range of physical input parameters associated with a particular WBS element. Since the objective always is to create a risk-adjusted cost estimate, the needed data includes not only the project office point estimate value of each quantifiable physical parameter, but also the possible range from best case to worst case values. In some cases, this range may be very small or even zero, when there is little or no uncertainty as to the required effort. Generally, relatively few WBS elements have the most uncertainty in cost. These typically include WBS elements that relate to such items as software development, system integration (especially integration of commercial off-the-shelf (COTS) products), system performance requirements (e.g., human factors, safety, and security requirements), and test and evaluation.

Therefore, data collection is focused predominantly on the identification and quantification of the possible range of physical parameters for each WBS element. The cost analyst’s primary source for this data is project SMEs. In addition, a comprehensive risk assessment is frequently conducted to identify major risk cost drivers using a model such as the ASD-400 method (See "Guidance for the Investment Analysis Team’s Alternatives Risk Assessment" at http://fast.faa.gov/investment/giatara.htm).

There are several probability distribution techniques available to estimate cost distributions. A common technique that is typically used to derive the probabilistic cost distributions and estimates is a triangular distribution. The triangular distribution method relies on three basic parameters (best case, point estimate, worst case) to establish a probability density function which will be used to statistically quantify the cost risk associated with the program with stochastic models (Contact ASD-410 for more information.). In every case, the rationale for the selected values should be documented in the BOE.

Output Product: Point estimates for all cost-driving physical parameters and the associated Cumulative Distribution Functions (CDF).

4.0 Complete Cost Estimate: With the risk ranges and CDFs of physical input parameters known, the next step is use the methodologies established in the planning phase to develop the statistical distribution of costs for each WBS element and to compute the statistically derived risk-adjusted constant dollars LCCE.

This is normally accomplished by using an analytical tool such as Monte Carlo simulation to randomize the physical parameters according to the range specified by the analyst. Available tools for this purpose provide a CDF as an output, from which the analyst selects a high-confidence total LCCE (i.e., 80% confidence that the cost will not be exceeded in actual investment performance). There is a need to establish high- confidence Facilities and Equipment (F&E) and high-confidence Operations and Maintenance (O&M) cost estimates in order to develop the life cycle cost Acquisition Program Baseline (APB) for the Joint Resources Council (JRC) approval.

Output Product: Risk-adjusted LCCE and WBS elements, including a CDF for the life cycle cost in constant dollars.

5.0 Time-Phase The Life Cycle Cost In Then-Year Dollars: The previous step provides a total, constant dollar LCCE. It is necessary to allocate the total across an annualized fiscal funding profile and inflate the estimate to then-year dollars for the APB and various financial plans. Total WBS costs should be spread across fiscal years according to the risk-adjusted schedule funding requirements for each WBS element. The cost estimating team should get agreement on the proposed spreading of costs, or the process of spreading the costs, from the Product Team that will be responsible for program implementation.

Output Product: Time-phased life cycle cost financial profiles in then-year dollars for APB, Integrated Program Plan (IPP), and financial plans (e.g., CIP).

6.0 Conduct Internal Review: As discussed previously, the cost estimate results are driven by the input data and assumptions on quantifiable parameters. The analyst should conduct a quality assurance review of the input data and the results with other Investment Analysis Team (IAT) members, and coordinate with AFZ and ASD personnel to ensure the data gathered was correct. If the results "don’t look right", there may be a need to re-validate and re-verify the assumptions and input data used to generate the estimate. If needed, the analyst should re-run the tool with corrected assumptions and input data so as to assure the team that the results are correct. Similar, additional sensitivity and "what if" estimates may be generated to test the robustness of the recommended option in the light of different assumptions.

Output Product: Refined WBS life cycle costs, CDF, if iterations are performed.

7.0 Conduct External Review and Coordinate: The cost estimate must be reviewed and approved by all the major stakeholders prior to its finalization and presentation to the FAA decision-makers. If necessary, re-estimates may be needed or required using different data and assumptions to accommodate any comments received during the JRC pre-briefs. The stakeholders begin with the members of the IAT first, and then expand with other stakeholders and impacted parties. Coordination and review is necessary to ensure quality control, to verify that the correct input data has been used, and to forge consensus on the results of the cost estimation.

Output Product: Resolution of any issues.

8.0 Complete Final Report And Documents: This last step records the effort and results of all previous steps, both to prepare for an FAA decision meeting and to populate the historical databases of all investment analyses. The best approach is to begin and to continue preparation of the BOE very early in the cost estimation effort, so as to ensure its accuracy and completeness and to minimize the need for a last-minute effort to complete it. In all likelihood, the BOE will be reviewed and scrutinized subsequently by numerous persons and groups in and out of the FAA (e.g., General Accounting Office (GAO), Office of Management and Budget (OMB), Department of Transportation (DOT) Inspector General (IG), etc.). Therefore, the BOE must be written in easy-to-understand, layman’s language. Each stakeholder/reader will have various interests, but all will require a well-documented, high-quality final report so as to understand and to judge the effectiveness of the investment decision. The preparation of a final report is critical to the credibility of the cost estimation effort. It must provide an "audit trail" that can be followed easily and understood by inside and outside interest groups and stakeholders.

Output Products: Final BOE. Input to the JRC briefing. Cost analysis section of the Investment Analysis Report.