For all indicators, time is shown on the X-axis displayed in a moving twelve month window. Six months of actual (historical data), and six months of projected data. In addition, two months of historical projections are shown enabling management to track performance to projection. The current reporting month is positioned at the center of the X-axis.

• Contractor Earned Value
• Project Milestones
• Requirements Stability
• Product Quality
• Software Progress
• Benefits to Cost Ratio
• Technical Performance

Note that each chart has a Metric Description and an Analysis section. The standard Descriptions are defined in this document, and are reproduced within the ELM tool’s report template. A project may elaborate, as necessary; however, any change in basic descriptions must be reviewed by the Metrics Working Group (MWG). The Analysis section is the key part for each monthly update that explains any significant deviations of actuals from plans - and whether the resulting impact on projections may be different from originally planned. The Analysis should be written clearly and concisely with the goal of providing project personnel, and executive management insight to the chart.

A Project may need additional metrics for management purposes. Developing additional measures is encouraged, and should be available as backup data for the ELM.

A tool has been developed to provide consistency in reporting these metrics. A separate document gives details on using the tool (USEMANXX.DOC -- Where ‘XX’ indicates the version number of the tool).

The following descriptions of the seven metrics are intended to provide purpose, applicability, and identify potential sources for the data to be collected. Each of the metrics is heavily supported by the ‘7-Up’ tool set, a MicroSoft Office based suite. It is strongly suggested that you become familiar with the tools by reading the user manual prior to starting. A working knowledge of Excel and Word are assumed:

1. Contractor Earned Value
1. Purpose: Visibility into the probability of achieving project objectives within cost and schedule - it is an indicator of project health. Earned value is a complex subject in and of itself. It is assumed that project personnel are familiar with its use.
2. Description: This indicator is shown as a line chart with the earned value measures on the left (Y)axis, and time on the lower (X) axis.

Earned Value (EV) - in this context refers to the contractor’s cumulative Cost Performance Report (CPR) data, or equivalent. Since this data is provided by the contractor, it will not be available during acquisition phases which precede or follow the Solution Implementation Phase

Project Cost and Schedule gives management an overview of the value of work completed to date. The budgeted cost of work performed (BCWP), budgeted cost of work scheduled (BCWS), actual cost of work performed (ACWP), estimate at complete (EAC) , and estimated date of completion (EDC) can be projected and compared to planned budget at complete (BAC).

The Cost Performance Index (CPI), Schedule Performance Index (SPI), Estimate at Completion (EAC), Budget at Completion (BAC), Estimated Date of Completion (EDC), Contract Ceiling (K ceiling), and the Project ceiling -- APB (P ceiling) are included in a static display on the chart for this metric.

• K ceiling = Contract Ceiling
• P ceiling = Project Ceiling (APB)
• SV = BCWP-BCWS
• CV = BCWP-ACWP
• VAC = BAC-EAC
• CPI = BCWP/ACWP
• SPI = BCWP/BCWS

The Earned Value terms are well defined in FAA’s EV Management Training course.

3. AMS Phase (Life Cycle):
1. Solution Implementation - EV (contractors); other (non-contractor) major program costs (optional)
2. In-Service Management - EV (contractors); other (non-contractor) major program costs (optional)
3. Service Life Extension - EV (contractors); other (non-contractor) major program costs (optional)
4. Data Sources: Program Master Schedule, Program Plan, CPR.

 

2. Project Milestones
1. Purpose: This measure provides visibility into the project schedule achievements. Key project milestones are identified and tracked for the life cycle of the project. This metric, when analyzed along with the "Contractor Earned Value" metric, will aid in determining the probability of achieving project objectives within cost and schedule.
2. Description: This indicator depicts the cumulative number of project milestones planned versus the cumulative number of milestones achieved. Project milestones are defined as "key" schedule milestones, and should be selected and tracked using the following procedure:
1. Milestones will be selected and documented for a 12 month period. Each subsequent month, additional milestones will be documented and added to the end of the 12 month window. This data represents the project milestone measurement plan. Once documented the milestones should not be changed (i.e., milestones removed and replaced with different ones).
2. Identify project milestones at a level of detail sufficient to average about 3 milestones per month. We recommend that they be selected in accordance with significance to the overall project. Level 1 milestones should be added to the plan first, then level 2, and level 3, etc., until the average number of three milestones per month is achieved.
3. Milestones should NOT include regularly scheduled status meetings or other milestones that will occur independent of schedule progress.
4. A milestone will earn credit (show as completed) when the activity is completed to the satisfaction of the project personnel responsible, and the terms and conditions of the contract, if applicable. A milestone earns credit in the month of actual completion. So it is possible to show 5 milestones completed for a month that had only 3 milestones scheduled, if 2 or more of the milestones were completed late from previous months.
3. AMS Phase (Life Cycle):
1. Mission Needs — indicator applicable for this phase
2. Investment Analysis — indicator applicable for this phase
3. Solution Implementation - indicator applicable for this phase
4. In-Service Management - indicator applicable for this phase
5. Service Life Extension - indicator applicable for this phase
4. Data Sources: Product Team Plan, Project Schedules, Contract.

 

3. Requirements Stability
1. Purpose: Identify the stability of project requirements baselines (Functional and Allocated).
2. Description: This metric depicts the stability of the system requirements baseline (Functional Baseline) referred to as ‘A’ level, and the allocated requirements baseline for software configuration items (Allocated Baseline) referred to as ‘B’ level. Formal baselines need not be established to begin tracking requirements change data. It is intended that data collection commence when the specifications have completed development, and are placed under change control (for purposes of this metric, we refer to this as baselined). They need not be formally approved. This indicator depicts the cumulative percent of system requirements changed on the right Y-axis (line), and the cumulative percent of software requirements changed on the right Y-axis (line). Cumulative percent change = [(#new + #deleted + #modified)/(baseline number)] * 100. Where #new, #deleted, and #modified are cumulative totals. In addition, the baseline number of requirements for each, along with the current month and previous month totals are provided in a static data box.

The analysis should address abnormal volatility in the requirements numbers.

3. AMS Phase (Life Cycle):
1. Mission Needs - indicator applicable for this phase
2. Investment Analysis - indicator applicable for this phase
3. Solution Implementation - indicator applicable for this phase
4. In-Service Management - indicator applicable for this phase
5. Service Life Extension - indicator applicable for this phase (for existing system only).
4. Data Sources: FAA System Level Specification (SLS), System Specification or System Segment Specification, Software Requirements Specification, etc.

 

4. Product Quality
1. Purpose: Track the defect discovery and solution (or closure); these are indicators of current backlog and rework effort, as well as process capability in finding and fixing defects and potential quality of the product.
2. Description: This indicator depicts the cumulative number of problems discovered and closed on the left Y-axis (line). One line identifies the cumulative total number of problem reports of all severity categories discovered. A second line identifies the cumulative total of all problem reports closed. It is intended that this metric depict product quality, therefore, defects should not be counted and tracked until the appropriate time. Software defects should be counted after successful completion of unit level testing to avoid counting normal developmental errors. The defects can be found in any phase by such activities as reviews, inspections, audits, tests, or actual operations. The analysis may note if discovery and/or closure rates appear to be outside of expected limits.
3. AMS Phase (Life Cycle):
1. Mission Needs - N/A
2. Investment Analysis - N/A
3. Solution Implementation - indicator applicable for this phase
4. In-Service Management - indicator applicable for this phase
5. Service Life Extension - indicator applicable for this phase
4. Data Sources: Problem Trouble report, Inspection/review , Audit data bases.

 

5. Software Progress
1. Purpose: Show the progress of the software implementation effort. Progress for both newly developed (or modified), and non developed (COTS or other reuse) software are depicted against the development plan.
2. Description: This indicator depicts the software implementation progress in lines of code (or other measure such as function points) completed on the left Y-axis. In order to standardize the counting practice, completion of code will correspond with successful completion of unit level testing. Since NDI (COTS and reuse) code is considered having already completed developmental testing, it is to be counted immediately. The principle purpose for including NDI code on this metric is to show the variation of NDI usage for the project against the original plan. Size may be in "thousands source lines of code excluding comments" (KSLOC) or in "function/feature points". If both measures are used on a project then function/feature points should be converted to KSLOC [note the conversion factor]. This indicator includes a line for the total software system size estimate, and the baseline size estimate (the contractor’s estimate provided with the proposal) is captured in a static data box. A stacked bar will be used to indicate the current amount of software completed. This data will be shown in contrast to the planned development progress shown as an area line on the chart. The analysis should note reasons for any changes in the software system size estimate, and trends different from planned.
3. AMS Phase (Life Cycle):
1. Mission Needs - N/A
2. Investment Analysis - N/A
3. Solution Implementation - indicator applicable for this phase
4. In-Service Management - indicator may be applicable for this phase
5. Service Life Extension - indicator may be applicable for this phase
4. Data Sources: Function point and/or SLOC analysis, SLOC counter, original/revised estimates; CRs.

 

6. Benefits to Cost Ratio
1. Purpose: Track the program benefits to cost ratio.
2. Description: This metric depicts the benefits to cost ratio for a system or project over its life cycle. An initially-estimated value is declared at the time of the investment decision and will be monitored during solution implementation. The value will change as "actuals" accrue and revised estimates [e.g., EAC vs BAC] are determined for the benefit to cost ratio's components. A significant deterioration in the ratio's value (increase in estimated life-cycle costs and/or decrease in life-cycle benefits) could provide a basis for reassessment of the value of a system or project. Critical factors that, if changed significantly, may impact the benefits to cost ratio include:
1. minimum acceptable rate of return
2. economic service life for the system or project
3. cost profile of the system or project in constant dollars over its total life cycle [convert to Present Value]
4. benefits of the system or project over its total life cycle - expressed in constant-dollar profile [convert to Present Value]

The ASD organization is responsible for the general methodology applied, and the input values of key external drivers (e.g., AR). The project manager will know when project schedule, cost, or functionality may impact these calculations. Both ASD and the project manager will determine if significant changes have occurred which will impact either costs or benefits necessitating a recalculation.

3. AMS Phase (Life Cycle):
1. Mission Needs - N/A
2. Investment Analysis - indicator applicable for this phase
3. Solution Implementation - indicator applicable for this phase
4. In-Service Management - indicator applicable for this phase
5. Service Life Extension - indicator applicable for this phase
4. Data Sources: ASD economic assessments with cost input from the project office.

 

7. Technical Performance
1. Purpose: Track the most critical technical performance risk parameters in terms of probability of occurrence and resulting impact.
2. Description: Each project must identify their most critical technical performance parameters. These parameters have the largest impact on the success of the project in meeting operational requirements. This metric will report on how well the project anticipates satisfying, at most two, critical parameters which the project manager believes are at highest risk [defined in terms of Probability * Impact].

Since technical issues are often phase dependent, the selection of different technical performance issues as the project progresses is an acceptable practice. Each technical parameter should be on a different chart. Time should be on the X-axis, the technical performance parameter on the left Y-axis. Data should be reported for the previous 6 months, if known, as follows:

 

• Baselined value of the technical performance parameter versus time
• Actual value of the technical performance parameter versus time

During early stages of the project, the ‘actual’ value may be the results of running a prototype or a simulation. As the project progresses, the ‘actual value’ will tend to reflect testing or actual implementation and any planned improvements before initial operation. The particular choice of critical parameters is entirely program dependent. Moreover, it may vary over time since it is intended to be the parameters for which the project manager believes face the greatest risk.

3. AMS Phase (Life Cycle):
1. Mission Needs - Dependent upon parameters selected
2. Investment Analysis - Dependent upon parameters selected
3. Solution Implementation - Dependent upon parameters selected
4. In-Service Management - Dependent upon parameters selected
5. Service Life Extension - Dependent upon parameters selected
4. Data Sources: Dependent upon parameters selected