| GUIDANCE |
| The Pricing Handbook
19. Software Pricing 19.9 Intelligent Use of Models
19.9 Intelligent Use of Models As discussed earlier, there is a plethora of software cost and size models available, many of which are very sophisticated. This may lull a manager into an over dependence on models for estimation. According to Robert E. Park, in his article "An Open Letter to Cost Model Evaluators," some managers apparently believe that "models, not estimators, are responsible for estimates". Software models, however, are not magic boxes; they are only as good as the input data used. Also, models inherently have limitations. Furthermore, models are not always useful in analyzing non-cost factors in decision making. The manager, therefore, must recognize the capabilities and limitations of models and use them intelligently. Some of these limitations are now discussed.
The problem of the instability of parametric models, as described by Boehm, is exacerbated by the sensitivity of effort and schedule to changes in input parameters. For example, in most cost models, changes in program size will result in at least an equivalent percentage change in cost or effort. Some inputs have even more dramatic effects. For example, changing personnel capability (analyst and programmer) in COCOMO or REVIC from "very high" to "very low" will result in a 300% increase in effort required. In SEER-SEM, changing security requirements from the lowest to the highest rating will have a similar effect. In PRICE-S, changing the Productivity Factor by 0.1 can result in a 20% change in effort. All models have one or more inputs for which small changes will result in large changes in effort and, perhaps, schedule. The input data problem is further compounded in that some inputs are difficult to obtain, especially early in a program. As discussed in section 19.4, size must be estimated early in a program using one or more sizing models. These models usually have not been validated for a wide range of projects. Some sensitive inputs, such as analyst and programmer capability, are subjective and often difficult to determine. Studies like one performed by Brent L. Barber, Investigative Search of Quality Historical Software Support Cost Data and Software Support Cost-Related Data, show that personnel parameter data are difficult to collect. Figure 19-13, extended from the SEER-SEM User’s Manual shows the relative impact on cost/effort of the different input parameters for that model. Even "objective" inputs like Security Requirements in SEER-SEM may be difficult to confirm early in a program, and later changes may result in substantially different cost and schedule estimates. Some sensitive inputs such as the PRICE-S Productivity Factor and the SLIM PI should be calibrated from past data. If data are not available, or if consistent values of these parameters cannot be calibrated, the model’s usefulness may be questionable.
Figure 19-13. Example of Relative Cost Driver Impacts
A special problem confronts Government managers who must obtain input data from contractors. First of all the Government manager should include a data collection form in the contractual requirements. The recommended minimum input data requirements are shown in Table 19-11 in the software pricing section. However, an analyst may be tempted to run a model using the data provided on the form. This will merely result in a duplication of the contractor’s inputs and proposed cost. A better use of this data would be as a "starting point" to conduct further discussions with the person or group who filled out the form. The form should be as generic as possible so as to collect the type of data needed for a variety of models, not just a specific model. Of course, if the model is known, then a specific input form makes sense. An example of a generic form is the data form included in the Space and Missiles Center Software Database User's Manual. A potential drawback to such generic forms is that they tend to be lengthy and cumbersome to complete and use.
Even if "perfect" input data were available, models may not produce accurate results. Studies have compared model estimates with "known" input data and actual cost and schedule information, and have not found model accuracy to be scintillating. For example, a 1989 study performed by ITT Research Institute for eight Ada programs, Test Case Study: Evaluating the Cost of Ada Software Development, showed that the most accurate models for that type of application (SoftCost-Ada and SASET) were accurate to within 30% of actual costs only 57% of the time. The Ourada study showed even worse results for SEER-SEM, SASET, and REVIC for the 28 military ground programs in an early edition of the Space and Missiles Center database. One limitation of these results is that the models were not calibrated. One of the SEI software cost estimating requisites is that cost models be calibrated/tuned to reflect demonstrated accomplishments on past projects. Calibration can and does improve model accuracy. A 1981 study by Robert Thibodeau entitled An Evaluation of Software Cost Estimating Models showed that calibration could improve model accuracy by up to 400%. However, the average accuracy was still only 30% for an early version of SLIM and 25% for an early version of PRICE-S. Furthermore, this accuracy was only demonstrated for certain types of programs. Likewise, the ITT Research Institute study showed that the accuracy of PRICE-S and System-3 (a forerunner of SEER-SEM) improved with calibration, but only to within 30%, 62% of the time. For development cost or effort estimation, therefore, a manager probably cannot expect the models to be accurate for budgetary purposes (i.e., within 10%), at least for new programs. Software maintenance cost accuracy is in an even worse state. An Air Force study performed by Ferens in 1983 and published in the ISPA Journal of Parametrics, concluded that no software support cost models could be shown to be accurate. Unfortunately, the results of that study have never been refuted. Maintenance cost estimating is further complicated in that traditional estimating practices do not always apply. For example, most models estimate cost (or effort) and schedule given some measure of the work to be done (such as program size). In the block change process used by the Air Force in supporting weapons software, however, the time and number of personnel are fixed, and the amount of work that can be done within those constraints must be estimated. Most cost models cannot address this situation (although the latest editions of SoftCost-OO and SEER-SEM may be useful). The software support estimation problem is further convoluted by lack of quality software support cost data for model development, calibration, and validation. Even if models can be shown to be accurate, another effect must be considered. According to Tarek Abdel-Hamid and Stuart E. Madnick in their book Software Project Dynamics: An Integrated Approach, "a different estimate creates a different project." What a cost model predicts may dictate how a project is managed. For example, if a model predicts an unrealistically high effort and schedule for a program, and management uses the model results to plan and manage the program, then it may actually take longer and cost more than necessary. Abdel-Hamid shows that the use of a "safety factor" (analogous to management reserve) of 25% to 50% in software estimates will result in 15% to 35% increases in actual person-months.
The general requirements and procedures for calibrating and validating a cost model are the same for software cost models. This section provides additional details regarding the software cost model calibration process. NASA’s Handbook defines calibration as the process of determining the deviation from a standard in order to compute the correction factors. For cost estimating models, the standard is considered historical actual costs. The calibration procedure is theoretically very simple. It is simply running the model with normal inputs (known parameters such as software lines of code) against items for which the actual cost is known. These estimates are then compared with the actual costs and the average deviation becomes a correction factor for the model. In essence, the calibration factor obtained is really good only for the type of inputs that were used in the calibration runs. For a general total model calibration, a wide range of components with actual costs need to be used. Better yet, numerous calibrations should be performed with different types of components in order to obtain a set of calibration factors for the various possible expected estimating situations. For instance, the PRICE Software Model addresses this situation using internal productivity factors. These can be modified by the calibration process. According to SEI, organizations that acquire software intensive systems should have a corporate memory or historical database and a process for calibrating their cost models to reflect demonstrated performance of similar past projects. The act of calibration standardizes a model. Many models are developed for specific situations and are, by definition, calibrated to that situation. Such models usually are not useful outside of their particular environment. However, general cost estimating models including commercially available ones such as the PRICE and SEER models (described in Appendix 19D) are designed to be useful as estimating tools for a wide range of situations. The act of calibration is needed to increase the accuracy of one of these general models by making it temporarily a specific model for whatever product it has been calibrated. Calibration is in a sense customizing a generic model. Items which can be calibrated in a model are: product types, operating environments, labor rates and factors, various relationships between functional cost items, and even the method of accounting used by a contractor. All general models should be standardized (i.e. calibrated). Using historical data as a basis for customizing or calibrating a tool is essential. Insure that information for current project development efforts are saved for future reference. At a very minimum, use software life cycle model phases and activities as a basis for collecting and maintaining project envelopment information for future tool use. Once calibration of the model is completed, model validation in accordance with FAA CEH, chapter 12, section 12.4.2 is necessary to establish the model’s accuracy. |
