DOD Parametric Cost Estimating Handbook 2nd Ed

PARAMETRIC

COST

ESTIMATING

HANDBOOK

Joint Government/Industry Initiative

Fall 1995 SPONSORED BY DOD

Parametric Cost Estimating Handbook

PREFACE This Handbook is intended to be used as a general guide for implementing and evaluating parametric based estimating systems, and as the text material for a basic course in parametric estimating techniques. The information contained in this Handbook complements and enhances the guidance provided by DCAA Cost Audit Manual (CAM), FARs/DFARs, and other government regulations or public laws relating to cost estimating. The Handbook is structured so each chapter can be used as a stand alone guide and reference. If the reader is interested in only one subject, then one chapter can be read without reading any other. Expected users of this Handbook include cost model developers, analysts, auditors, and all levels of cost management. Every phase of the government’s procurement process is being reviewed and changed to reflect the realities of today’s cost conscious environment. Among the many aspects of the procurement process under scrutiny is the use of traditional cost estimating tools and processes. These tools and processes, at times, have proven expensive to maintain and operate and did not always provide realistic or timely results. Today’s Acquisition Reform mind-set dictates that the cost estimating community consider a different set of tools and processes which are less labor intensive to operate and result in credible cost estimates. It is partly from the impetus of Acquisition Reform that this document has been created. All of us in the cost estimating community need to find ways to perform our work more cost effectively. Pushed by the common goal of reducing proposal preparation cost, contractor and oversight management are pursuing a cooperative approach to problem solving. This Handbook is an example of such a cooperative effort. If the Handbook succeeds to help produce better estimates, analyses, or audits, then the process has been successful. This Handbook is just one of several by-products of a joint Industry/Government Parametric Cost Estimating Initiative. The Initiative Steering Committee and implementation team includes representation from contractors, buying activities, Defense Contract Management Command and Defense Contract Audit Agency. The Initiative’s far reaching action items include working with pilot contractor sites to test the expanded use of parametric cost estimating techniques and tools, developing parametric cost estimating training, and preparing and distributing this Handbook. This is the first edition of the Handbook. It will be used and tested at pilot contractor sites. The Initiative Steering Committee will be responsible for receiving and incorporating any comments for improving the Handbook from the pilot sites and others, and publishing the second edition of the Handbook. The second edition will be issued in the summer of 1996. A user reply sheet is provided in this Preface to facilitate submission of your suggestions. An Order Form for this Handbook is included at the end of the Preface. The preparation of this Handbook included the review of nearly four thousand pages of documentation. Some of that total is included here. The material has come from many sources, both individual and organizational contributors. There is no way all of the contributors can be acknowledged, since most of them are anonymous. There are, however two who should be

Parametric Cost Estimating Handbook

mentioned at this time, besides those acknowledged later in the test. First, the Parametric Estimating Initiative Steering Committee for authorizing this Handbook and specifically, NAVSEA for funding the Handbook’s development. Thanks to the rest of the Committee for their inputs, reviews and critiques of the draft. And, especially, the Space Systems Cost Analysis Group (SSCAG) who supported the initial concept of the Handbook, and the may individuals within that organization who conceptualized, critiqued, and developed much of the material.

Bill Brundick, General Editor

Parametric Cost Estimating Handbook

PARAMETRIC ESTIMATING INITIATIVE STEERING COMMITTEE Executive Chairman Bob Scott Executive Director of Contract Management Defense Contract Management Command - AQC Cameron Station Alexandria, Virginia, 22304-6190 703-274-0821 phone 703-274-0823 fax

Bob Spiker Controller, Financial and Government Accounting Westinghouse Electric Corporation - ESG P.O. Box 17319, Mail Stop A585 Baltimore, Maryland 21203-7319 410-765-2913 phone 410-765-6038 fax

Michael Thibault Deputy Director DCAA Headquarters Cameron Station Alexandria, Virginia 22304-6178 703-274-7281 phone 703-617-7450 fax

Co-Chairman Jim Collins Parametric Estimating Specialist Westinghouse Electric Corporation D61-ESG P.O. Box 1693, Mail Stop 1293 Baltimore, Maryland 21203 410-765-8033 phone 410-765-5289 fax

David Eck Chief, Policy Formulation Division DCAA Headquarters Cameron Station, Room 4A250 Alexandria, Virginia 22304-6178 703-274-7314 phone 703-617-7452

Members Jim Balinskas Contract Pricing & Finance Division NASA Headquarters, Code HC 300 E Street SW Washington, DC 20546 202-358-0445 phone 202-358-3082 fax

Dean Boyle Deputy Technical Assessment Manager DPRO - Northrup Grumman Defense Logistics Agency, DCMC Bethpage, New York 11714-3593 516-575-9742 phone 516-575-6527

Gary Constantine Manager, Parametric Estimating E-Systems - Greenville P.O. Box 6056, CBN93 Greenville, TX 75403-6056 903-457-5666 phone 903-457-4619 fax

Marty Deutsch Chief, Estimating Policy Review & Compliance Martin-Marietta Astronautics P.O. Box 179, Mail Stop DC 2503 Denver, Colorado 80201-0179 303-971-6060 phone 303-971-5143 fax

Mel Eisman Senior Research Associate Rand Corporation P.O. Box 2138, Mail Stop HMRP/5 Santa Monica, California 90407-2138 310-393-0411 x6704 phone 310-451-7032 fax

Jim Gleason Procurement Analyst US Army Material Command, AMCAQ-E 5001 Eisenhower Avenue, Room 9N15 Alexandria, Virginia 22333-0001 703-274-4437 phone 703-274-3198 fax

Virgil Hertling

Paul Lubell

Parametric Cost Estimating Handbook

Directorate of Pricing and Finance US Air Force Headquarters, AFMC/PKF Building 262, 4375 Chidlaw Road, Suite 6 Wright Patterson AFB, Ohio 45433-5006 513-257-6861 phone 513-476-2435 fax

Senior Engineer, Technical Estimating Westinghouse Electric Corporation - ESG P.O. Box 1693, Mail Stip 1293 Baltimore, Maryland 21203 410-765-6163 phone 410-765-5289 fax

Bernard Rudwick Professor - Financial Management Defense Systems Management College Fort Belvoir, Virginia 22060-5426 703-805-3783 phone 703-805-3184 fax

Marcia Rutledge Contracting Officer, ASW & Mine Systems Branch US Navy, Naval Sea Systems Command 2531 Jefferson Davis Highway, NC-3, Room 5E08 Arlington, Virginia 22242-5160 703-602-0951 x639 phone 703-602-7023 fax

George Salantai Manager, Estimating McDonnell Douglas Aircraft Company Building 305, Post 2E, Room 232, Mail Code 306-5485 St. Louis, Missouri 63166 314-233-8461 phone 314-232-7171 fax

Amir Tarmohamed Proposal Anaysis & Definitization Team Defense Logistics Agency, DCMC-AQCOD Cameron Station, Room 8A454 Alexandria, Virginia 22304-6190 703-274-4130 phone 703-617-0944 fax

Parametric Cost Estimating Handbook

USER REPLY SHEET PARAMETRIC COST ESTIMATING HANDBOOK

Please send comments to: Marcia Rutledge Contracting Officer, ASW & Mine Systems Branch US Navy, Naval Sea Systems Command 2531 Jefferson Davis Highway, NC-3, Room 5E08 Arlington, Virginia 22242-5160 703-602-0951 x639 phone 703-602-7023 fax We would appreciate specific comments such as the suggested alternative language and rationale to support the language, additional information that would add value to the Handbook, or existing information that can be deleted. Specific examples that enhance the Handbook are welcome.

Comment # _______

Chapter _______

Page # _______

Parametric Cost Estimating Handbook

ORDER FORM for the

PARAMETRIC COST ESTIMATING HANDBOOK

YES, please send me _____ copies of the Parametric Cost Estimating Handbook. There is no charge for the Handbook. Requestors are encouraged to limit requests to one copy per organization.

Please send the Handbook to:

Company/Organization

Attention/Individual

Street Address

City, State, Zip Code

Daytime phone number including area code

All orders should be sent to: Naval Sea Systems Command 2351 Jefferson Davis Highway NC-3, Room 5E08 Arlington, VA 22242-5160 Attn: M. Rutledge, SEA 0263 Contracting Officer, ASW & Mine Systems Branch

(Please type or print)

14 September 1995

FOREWORD Early in 1994, a joint Government/Industry Committee was formed to study ways to enhance the use of parametric cost estimating techniques. The Committee found that the lack of training was one of the largest barriers to the use of parametrics. The Committee sponsored this Handbook to provide training and background information on the use and evaluation of parametric tools. The Committee is also working with the Defense Acquisition University to develop classroom training that would be available to both government and industry. The Committee is also sponsoring a pilot program to test the use of parametric tools. As of September 1995, eleven companies are participating in the pilot program. The pilot program is expected to last until the Summer of 1996. This Handbook will be tested at the pilot program sites. The Committee will update the Handbook to incorporate comments, best practices, and additional examples developed at the pilot sites. The Committee also invites your comments on the Handbook and any examples you may have. A User Reply Sheet is provided in the Preface to facilitate your input. The Committee expects to publish the second edition of the Handbook by the Summer of 1996. The Committee hopes, with your help, to make the Handbook the guide people turn to when using and evaluating parametric tools.

Robert Scott

Executive Director of Contract Management Defense Contract Management Command

Robert Spiker

Controller, Financial & Government Accounting Westinghouse Electric Corporation - ESG

Michael Thibault

Deputy Director Defense Contract Audit Agency Executive Chairmen Joint Government/Industry Parametric Cost Estimating Initiative Steering Committee

Parametric Cost Analysis Handbook

PARAMETRIC COST ESTIMATING HANDBOOK TABLE OF CONTENTS FOREWORD PREFACE CHAPTER I -- INTRODUCTION, BACKGROUND AND DEFINITIONS Introduction .............................................................................................................. Background .............................................................................................................. Definitions and Terms .............................................................................................. Cost Realism ............................................................................................................

1 4 9 10

CHAPTER II -- COLLECTION AND NORMALIZATION OF PARAMETRIC DATA Generalizations ......................................................................................................... Significant Adjustments to Parametric Data ............................................................ Consistent Scope .......................................................................................... Anomalies .................................................................................................... Improved Technology .................................................................................. Other Types of Adjustments and Data Normalization ............................................. Inflation ........................................................................................................ Learning Curve ............................................................................................ Production Rate ............................................................................................ Calibration and Validation of Cost Models ............................................................. Some Review and Audit Considerations ................................................................. Data Normalization Process ..................................................................................... Pitfalls to the Use of a Parametric Model ................................................................ Two Illustrations of Typical Data Normalization Problems .................................... Summary ..................................................................................................................

12 14 14 14 15 15 15 16 16 17 18 19 25 27 29

CHAPTER III -- ELEMENTARY STATISTICAL TECHNIQUES AND CER DEVELOPMENT CER and Model Development - Uncertainty and Risk Reduction .......................... Developing Cost Estimating Relationships (CERs) ................................................. Hypothesis Testing of a Logical CER ......................................................... The CER Model ........................................................................................... When to Use a CER ................................................................................................. Strengths and Weaknesses of CERs ......................................................................... Strengths ...................................................................................................... Weaknesses .................................................................................................. Regression Analysis ................................................................................................. Curve Fitting ............................................................................................................ Graphical Method ........................................................................................ LSBF Method ..............................................................................................

32 34 36 36 36 37 38 38 38 41 41 41

i

Parametric Cost Analysis Handbook

Multiple Regression ..................................................................................... Curvilinear Regression ................................................................................ “Goodness” of Fit, R and R2 .................................................................................... Correlation Analysis .................................................................................... Coefficient of Determination ....................................................................... Coefficient of Correlation ............................................................................ The Learning Curve ................................................................................................. Limitations, Errors and Caveats of LSBF Techniques ............................................. Extrapolation Beyond The Range of the Observed Data ............................. Cause and Effect .......................................................................................... Using Past Trends to Estimate Future Trends ............................................. Misinterpreting the Coefficients of Correlation and Determination ............ Summary ...................................................................................................... Examples of CER Use .............................................................................................. Construction ................................................................................................. Weapons Procurement ................................................................................. Electronics ................................................................................................... Cost and Price Analysis ............................................................................... Information and Techniques ........................................................................ Estimate Reliability ...................................................................................... Two Examples of CER Use ......................................................................... Common CERs ............................................................................................ Acceptance Criteria for a Cost Estimating Relationship .......................................... Auditing and Analyzing a CER ............................................................................... CER Analysis ............................................................................................... General Features .......................................................................................... Evaluating and Estimate .............................................................................. Summary of CER Audit and Analysis .........................................................

44 45 45 45 46 47 47 49 49 50 50 50 50 51 51 51 51 52 52 53 54 60 61 63 63 63 64 65

CHAPTER IV -- HARDWARE COST MODELING Introduction .............................................................................................................. Overview of Hardware Cost Modeling .................................................................... Micro-Circuit and Electronic Module Modeling ..................................................... Hardware Operations and Support (O&S) of Life Cycle Models ............................ Deployment and Employment ..................................................................... Hardware Parameters ................................................................................... Program Globals .......................................................................................... Commercial Models ................................................................................................. MicroFASTE ............................................................................................... Price Parametric Models .............................................................................. SEER ............................................................................................................ Regression Based Product Specific Cost Models .................................................... Non-Commercial Cost Models ................................................................................ Non-Statistical Cost Models .................................................................................... Model Calibration ....................................................................................................

68 70 77 79 81 82 82 82 83 87 90 91 93 94 95

ii

Parametric Cost Analysis Handbook

Cost Model Audit and Analysis ............................................................................... Analyzing a Product Specific Cost Model ................................................... Summary of Cost Model Audit and Analysis ..............................................

96 96 99

CHAPTER V -- SOFTWARE PARAMETRIC COST ESTIMATING Software Definition .................................................................................................. The Importance of Software Today ......................................................................... The Software Development Process ........................................................................ The Waterfall Model .................................................................................... The Software Cost Estimating Process .................................................................... Define Project Objectives and Requirements .............................................. Plan the Activities ........................................................................................ Software Estimation Risks ........................................................................... Estimation Methodologies ........................................................................... Software Cost Estimating Standards ............................................................ Benefits ........................................................................................................ Examples of Parametric Software Cost Estimating ................................................. Parametric Software Cost Estimating Tools ............................................................ Desired Functional Capabilities of Parametric Tools .................................. Input Data Collection ................................................................................... Some Commercial Tools ............................................................................. Software Sizing Tools .................................................................................. Glossary of Terms .................................................................................................... Model Calibration .................................................................................................... Trends and Conclusions ........................................................................................... Trends .......................................................................................................... Conclusions ..................................................................................................

101 102 107 107 109 111 111 112 113 116 116 116 119 120 122 122 127 130 130 131 131 132

CHAPTER VI -- AUDITING PARAMETRIC ESTIMATES - A MANAGEMENT VIEWPOINT Cost Estimating Relationships: A DCAA Perspective ........................................... Introduction .................................................................................................. Background .................................................................................................. Parametric Criteria ....................................................................................... Logical Relationships .................................................................................. Significant Statistical Relationships ............................................................ Verifiable Data ............................................................................................. Reasonably Accurate Predictions ................................................................ Proper System Monitoring ........................................................................... Audit Planning and Requirements ............................................................... Observations and Suggestions ..................................................................... Summary ...................................................................................................... Estimating System Reviews ..................................................................................... Forward Pricing Rate Agreement (FPRAs) ............................................................. What to Look For in a Parametric Estimate .............................................................

133 133 134 134 135 135 135 135 136 136 140 142 143 145 148

iii

Parametric Cost Analysis Handbook

Rules of Thumb ........................................................................................... An Example .................................................................................................

150 151

CHAPTER VII -- BUSINESS APPLICATIONS OF PARAMETRIC ESTIMATING Introduction .............................................................................................................. Parametric Estimating in New Business Development ............................................ Estimating Production Buys Using Parametrics ...................................................... Example: Estimating Production Buys Using Parametrics ......................... Estimating Spares and Change Orders .....................................................................

153 153 154 155 158

Appendix A Appendix B Appendix C Appendix D Appendix E Appendix F Appendix G

160 177 198 205 209 218 236

Definitions of Estimating Terminology .................................................... Work Breakdown Structure ....................................................................... DCAA CAM 9-1000 Section 10, Review of Parametric Cost Estimates . More About Statistics ................................................................................ Examples of Other Hardware Estimating Models .................................... Some Currently Available Software Estimation Products ........................ Parametric Estimating System Checklist ..................................................

iv

Parametric Cost Estimating Handbook

CHAPTER 1

INTRODUCTION, BACKGROUND, AND DEFINITIONS

1

Parametric Cost Estimating Handbook

CHAPTER I INTRODUCTION, BACKGROUND, AND DEFINITIONS

INTRODUCTION

This Handbook is intended to be a living document. As advances are made in parametric estimating methodology, they will be introduced into the body of this material.

Changes

suggested from experienced users are solicited, as well as recommendations from other experts in the field.

However, the Handbook is primarily intended for the beginning parametrics

practitioner and to be used to enhance parametric training in the field.

When using the

Handbook, however, we assume that the reader has a basic understanding of algebra and statistics. Defined, a parametric cost estimate is one that uses Cost Estimating Relationships (CER’s) and associated mathematical algorithms (or logic) to establish cost estimates. For example, detailed cost estimates for manufacturing and test of an end item (for instance, a hardware assembly) can be developed using very precise Industrial Engineering standards and analysis. Performed in this manner, the cost estimating process is laborious and time consuming. However, if history has demonstrated that test (as the dependent variance) has normally been valued at about 25% of the manufacturing value (the independent variable), then a detailed test estimate need not be performed and can simply be computed at the 25% (CER) level. It is important, though, that any CER’s used be carefully tested for validity using standard statistical approaches. An exploration of certain statistical approaches relevant to CER development is included later in this Handbook. The need to reengineer business processes and reduce cost in the Department of Defense (DoD) has led to a parametric cost estimating initiative. In every corner of every aspect of defense contracting, Business Process Reengineering (BPR) has become a nineties buzzword. The cumbersome techniques that evolved into the development of the “normal” cost estimating processes of today are beginning to yield to more efficient and less costly approaches to achieve

2

Parametric Cost Estimating Handbook

the same, or superior results. Parametric estimating approaches fit very well into overall BPR methods. The importance of Business Process Reengineering was recently underscored by Lloyd K. Mosemann, II, Deputy Assistant Secretary of the Air Force, in his closing Keynote Address entitled “Predictability,” to the Software Technology Conference in Salt Lake City, Utah, on Thursday, April 14, 1994.

Although addressing the software process, Mr. Mosemann’s

comments are relevant to the cost estimating process in general. In summary, he said, in part: “There seems to be an inability within the software community, in general, to predict how much a software system will cost, when it will become operational, and whether or not it will satisfy user requirements. We need to deliver on our promises. “We have a poor track record regarding predictions. A 1979 GAO report concluded that only two percent of software contracted for was useable exactly as delivered. Twenty different studies have come to the same conclusion. Therefore, we in the DoD are focusing our attention on process improvement:

These include:

specific metrics usage plans, reuse plans, peer

inspections, process controls, proposed architectures in executable code, and government access to contractor on-line development environments. “This emphasis on process will give all of us in the software community a greater confidence that the prospective contractor will deliver the promised product on time and on budget.” Mr. Mosemann’s emphasis on process improvements to improve the quality of predictability of cost and schedule fits nicely with the concept of expanding the use of parametric tools in the cost estimating workplace. Parametrics can play a role in the BPR process as was underscored by Anthony A. DeMarco in his article, CAPE (Computer Aided Parametric Estimating for Business process Re-Engineering, in the PRICE Newsletter, October 1994. In his article, in summary, Mr. DeMarco states that: “Business Processing Reengineering (BPR) is the reengineering of an organization by examining existing processes and then revamping and revising them for incremental improvement. It is doing more with less and sometimes entails “starting over.”

3

Parametric Cost Estimating Handbook

There are five phases to BPR. They are: 1. Create an organization for improvement, 2. Develop an understanding of the process, 3. Streamline the process, 4. Model, implement, measure, and control, and, 5. Design and implement continuous improvement.

“Parametric tools can assist BPR. On one level, they can improve and streamline the BPR phases. On another level, parametric technology is the ‘best practice’ for estimating. Parametric tools bring speed, accuracy and flexibility to estimating processes, processes that are often bogged down in bureaucracy and unnecessary detail.” The need to reengineer the DoD cost estimating process (Acquisition Reform initiatives) became self evident to certain people from both government and industry. A Steering Committee was chartered by government and industry executives to explore the role played by parametrics in the cost estimating process. One goal was to determine what barriers, if any, exist to expanding the role of parametrics, and to develop the action plans to overcome those barriers.

The

committee consists of representatives from all of the Armed Services, the oversite community, selected contractors. This Handbook has been authorized by that Steering Committee. The Handbook is intended to be used by both model developers and model reviewers, their management or oversite, either technical or financial.

Government and industry cost

analysts and auditors who utilize CER’s and/or parametric cost models to develop or evaluate an estimate generated with these parametric tools should find the Handbook useful. It is also intended to be utilized as a source document by trainees within a generic parametric cost estimating training program. This Handbook includes basic information concerning data collection, Cost Estimating Relationship (CER) development, parametric cost models, and statistical techniques. Parametric techniques are a credible cost estimating methodology that can provide accurate and supportable contractor estimates, lower cost proposal processes, and more costeffective estimating systems.

4

Parametric Cost Estimating Handbook

An estimating workbench context model is shown in Figure I-1. The model indicates the tools required within the estimating community of contractors, customers and government agencies. Figure I-2 is a graphical representation of the complete parametric cost estimating process. The figure indicates the process from inputs through modeling and into a post processor phase. The post processor allows for the conversion of parametric output into a cost proposal.

BACKGROUND

The origins of parametric cost estimating date back to World War II. The war caused a demand for military aircraft in numbers and models that far exceeded anything the aircraft industry had manufactured before. While there had been some rudimentary work from time to time to develop parametric techniques for predicting cost, there was no widespread use of any cost estimating technique beyond a laborious buildup of labor-hours and materials. A type of statistical estimating had been suggested in 1936 by T. P. Wright in the Journal of Aeronautical Science. Wright provided equations which could be used to predict the cost of airplanes over long production runs, a theory which came to be called the learning curve. By the time the demand for airplanes had exploded in the early years of World War II, industrial engineers were using Wright’s learning curve to predict the unit cost of airplanes. In the late 1940’s, the DoD, and, especially, the United States Air Force began a study of multiple scenarios concerning how the country should proceed into the age of jet aircraft, missiles and rockets. The Military saw a need for a stable, highly skilled cadre of analysts to help with the evaluation of such alternatives. Around 1950, the military established the Rand Corporation in Santa Monica, California, as a civil “think-tank” for independent analysis. Over the years, Rand’s work represents some of the earliest and most systematic studies of cost estimating in the airplane industry. The first assignments given to Rand concerned studies of first and second generation ICBM’s, jet fighters and jet bombers. While the learning curve technique still proved useful for predicting the behavior of recurring cost, there were still no techniques other than detailed laborhour and material estimating for projecting what the first unit cost might be (a key input to the

5

Parametric Cost Estimating Handbook

ESTIMATING WORKBENCH CONTEXT MODEL PARAMETRIC MODELS/TOOLS CUSTOMERS

REQUEST FOR SUPPORTING DATA

SOLICITATION RESPONSE BUYERS/REQUESTORS, CONTRACTING OFFICERS ESTIMATING WORKBENCH

COST ANALYSTS/ PRICING TEAM/ DPROS

PROGRAM MANAGERS PROPOSAL/COST DATA, • HISTORICAL DATA, • CONTRACTOR DATA, • COST ELEMENT ANALYSIS • CERs • OTHER PARAMETRIC DATA • OTHER NONCOST DATA • EXTERNAL DATA COST COMPARISONS • BACKGROUND DATA HISTORICAL DATA •

OTHER GOV’T AGENCIES/SOURCES FIGURE I-1

6

AUDITORS

ACADEMIA

USERS

Parametric Cost Estimating Handbook

learning curve equation). Worse still, no methods were available for quickly estimating the nonrecurring costs associated with research, development, testing and evaluation (RDT&E). In the defense business in the early to mid 1950’s, RDT&E had suddenly become a much more important consideration. There were two reasons for that fact. First, a shrinking defense budget (between World War II and the Korean War) had cut the number of production units for most military programs, and second, the cost of new technology had greatly magnified the cost of development. The inability to quickly, and accurately, estimate RDT&E and first unit production costs had become a distinct problem.

7

Parametric Cost Estimating Handbook

Fortunately, within Rand, a cost analysis department had been started in 1950. This group proved to be prolific contributors to the art and science of cost analysis -- so much so that the literature of aerospace cost estimating of the 1950’s and 1960’s is dominated by the scores of Rand cost studies that were published during that time. In the mid 1950’s, Rand developed the most basic tool of the cost estimating discipline, the Cost Estimating Relationship (CER), and merged the CER with the learning curve to form the foundation of parametric aerospace estimating. This estimating approach is still used today. By 1951, Rand derived CER’s for aircraft cost as a function of such variables as speed, range, and altitude. Acceptable statistical correlations were observed. When the data was segregated by aircraft types (e.g., fighters, bombers, cargo aircraft, etc.), families of curves were discovered. Each curve corresponded to different levels of product or program complexity. This Parametric stratification especially helped clarify development cost trends. Eventually, a useable set of predictive equations were derived which were quickly put to use in Air Force planning activities. The use of CER’s and data stratification were basic breakthroughs in cost estimating, especially for RDT&E and first unit costs. For the first time, cost analysts saw the promise of being able to estimate relatively quickly and accurately the cost of proposed new systems. Rand extended the methods throughout the 1950’s, and by the early 1960’s, the techniques were being applied to all phases of aerospace systems. Since these rather humble beginnings, the state-of-the-art in parametric estimating has been steadily improving by an explosive growth in the number of practitioners, important methodological improvements, and greatly expanded databases. All of the major aerospace contractors and government aerospace organizations have dedicated staffs of parametricians who maintain and expand databases, develop parametric cost models, and utilize the tools of parametrics to make estimates of new and ongoing programs. NASA and the DoD routinely use parametric estimates to form the basis of new project cost commitments to Congress. The contractor community also routinely uses parametric cost models, especially during product concept definition. These estimates are used for decision making regarding bid strategies and are used as submittals to the government. It is only at the production and full scale development phase that parametrics are not commonly utilized for official proposal submissions (although

8

Parametric Cost Estimating Handbook

contractors still frequently use parametrics to generate target costs and cross-checks on the labor-material/buildup estimates). Over the past several years industry and professional estimating associations (e.g., International Society of Parametric Analyst (ISPA), Society of Cost Estimating and Analysis (SCEA), and the Space Systems Cost Analysis Group (SSCAG)) have been actively working with both Defense Contract Management Command (DCMC) and Defense Contract Audit Agency (DCAA) to explore the expanded opportunities for the use of parametric cost estimating techniques in firm business proposals. ISPA was formed in 1978 when a parametric estimating user’s group evolved into a more generic Society. The Space Systems Cost Analysis Group formed in 1977 under the sponsorship of the U.S. Air Force Space Division, with a mission to: 1.

Promote Cost Analysis Research

2.

Develop new tools to improve cost estimating techniques

3.

Promote sound practices, and

4.

Provide a forum for government and industry cost analysts concerned with the development and production of space-design hardware and software.

Then, in April 1994, a joint Industry and Government workshop on Parametric Cost Estimating occurred at the Defense Contract Audit Institute in Memphis, TN. Under the initiative and leadership of the DCMC, the DCAA, and industry proponents, a group of knowledgeable government and industry executives, policy formulators, and parametric practitioners were assembled to evaluate why there is not greater use of parametric cost estimating in DoD and NASA business proposals; identification of the barriers to expanded use of parametrics; and, action planning to take advantage of identified opportunities. At the conclusion of the workshop, it became clear to the participants that there were no barriers which precluded further implementation and use of parametric cost estimating by contractors in DoD or NASA business proposals.

Rather, barrier analysis and actions

recommended focused on the need for industry leaders to demonstrate that parametric estimating systems can be relied upon by the Government customers, and the need for the Government to train employees so that there exists a clear message that valid parametric estimates are a useful and often cost effective estimating approach.

9

Parametric Cost Estimating Handbook

DEFINITIONS AND TERMS

A complete glossary of parametric terminology, taken from numerous sources, is included in this Handbook as Appendix A. A few of the more important definitions are noted in this chapter. There are several definitions of parametric estimating, but for the purpose of this Handbook, the formal one adopted is as follows: A technique employing one or more CER’S and associated mathematical relationships and logic. The technique is used to measure and/or estimate the cost associated with the development, manufacture, or modification of a specified end item.

The measurement is based on the technical, physical, or other end item

characteristics. This definition establishes the clear linkage between cost and a product’s (or end item) technical parameters. Without this linkage, a product cost cannot be effectively defined. Nonparametric estimating systems generally do not connect technical (parametric) and cost elements with any substantial precision. And, a Parametric Cost Model is defined as: A parametric cost model is a group of cost estimating relationships used together to estimate entire cost proposals or significant portions thereof. These models are often computerized and may include many inter-related CER’s, both cost-to-cost and cost-to-noncost. Some models use a very limited number of independently estimated values and a series of Parametric inter-related cost-to-cost and cost-to-noncost estimating relationships to predict complex proposal cost structures. Parametric cost estimating is a technique used by both contractors and the Government in planning, budgeting, and performance stages of the acquisition process. The technique is used by contractors to expedite the development of cost estimates when discrete estimating techniques would require inordinate amounts of time and resources and would produce similar results. Reliance on properly developed and carefully evaluated CER’s and parametric cost models to produce realistic cost estimates can save both Industry and the Government time and resources in the evaluation and definitization cycle of the proposal or contract. The concept includes the use of cost-to-cost CER’s such as engineering labor overhead rates and material overhead rates which when reviewed using traditional evaluation criteria, are

10

Parametric Cost Estimating Handbook

considered valid estimators by the government. However, the technique also uses cost-tononcost CER’s which require additional analysis to determine their validity and acceptability as estimating tools. Parametric techniques focus on the cost drivers, not the miscellaneous details. The drivers are the controllable system design or planning characteristics and have a predominant effect on system cost. Parametrics uses the few important parameters that have the most significant cost impact on the product(s), hardware or software, being estimated.

COST REALISM

A widely used term today is “cost realism.” Now, no one expects a cost estimate to precisely predict what a hardware or software product or a time and material service will cost. So, cost realism is not about the exact cost estimate.

It’s about the system of logic, the

assumptions about the future, and the reasonableness of the historical basis of the estimate. That is, it’s about the things that make up the foundation of the estimate.

Cost realism analysis answers questions such as: *

Are the assumptions used in the estimating process reasonable?

*

Has the historical data base used been normalized to account for environmental parameters such as inflation?

*

Is the cost estimate logical? Does it make sense in the context of the hardware or software product or service being estimated?

*

Does the estimate display a bias toward being too low or too high? If so, how is this bias displayed in the estimate?

*

Is the cost estimating organization motivated to produce an inordinately high or low estimate in order to serve their own purposes?

*

If the product is fixed price sole source, has the historical basis data been “cherry picked” to insure the cost estimate obtained is unreasonably high (contractor) or unreasonably low (auditor or government customer)?

11

Parametric Cost Estimating Handbook

*

If the program is competitive, has the contractor or government program office created program expectations that are far too optimistic?

The cost estimator or analyst must ensure that they are working toward the goal of cost realism. It doesn’t matter whether or not they are employed by private industry, or the customer as a cost analyst or an auditor. If a contractor chooses to accept a management challenge in a competitive procurement, that’s certainly acceptable.

However, the basis for the challenge

should be clearly identified. There is no easy answer to the cost realism dilemma we all face. Unreasonable biases and expectations from contractor and customer have driven the cost estimating process in the past, and personal and programmatic motivations may continue to drive it in the future. But one thing is certain: the cost estimating process will continue to confront future unknowns. These unknowns are what make the cost estimating job one of the most difficult there is. But sound assumptions, high quality historical data, and unbiased analysts and estimators will improve the process for all.

12

Parametric Cost Estimating Handbook

CHAPTER II

COLLECTION AND NORMALIZATION OF PARAMETRIC DATA

13

Parametric Cost Estimating Handbook

CHAPTER II COLLECTION AND NORMALIZATION OF PARAMETRIC DATA

This chapter provides guidelines concerning data types, data sources, data normalization and adjustment techniques. This chapter also includes utilization recommendations about data to support the development and use of auditable CERs and cost models.

GENERALIZATIONS

A universal format for collecting technical and cost information is the Work Breakdown Structure (WBS). (See Appendix B) The WBS provides for uniform definitions and collection of cost and technical information. MIL-STD-881B provides guidelines for establishing the WBS at DoD, Service and contractor levels. Historical cost and labor hours data are required as a basis for cost estimating, and parametric estimating is no exception. Data should be collected and maintained in a manner that provides a complete audit trail, and expenditure dates should be recorded so that dollar valued costs can be adjusted for inflation. Also required is Technical Non-Cost Data that describes the physical, performance and engineering characteristics of a system, sub-system or individual item. For instance, weight is a common non-cost variable used in CER’s and parametric estimating models. (Other typical examples of cost driver variables include horsepower, materials of construction, watts, thrust, length, etc.) A fundamental requirement for the inclusion of a non-cost variable in a CER is that it be a statistically significant predictor of cost (that is, it should be a cost driver). Relevant program data including development and production schedules, quantities produced, production rates, equivalent units, breaks in production, significant design changes, and anomalies such as strikes, explosions, and other natural disasters are also necessary to fully explain any perturbations in historical data. Such perturbations may exhibit themselves in a profile of

14

Parametric Cost Estimating Handbook

monthly cost accounting data as the labor hour charging may show an unusual "spike" or "depression" in the level of charged hours. Such historical information comes from knowledgeable program personnel or program records (also known as program "memory"). The collecting point for cost and labor hours data is, in most instances, called the general ledger or a company accounting system. All cost and labor hours data, used in parametric CER’s or cost models, must be consistent with, and traceable back to, the original collecting point (the source). The data should also be consistent with accounting procedures and cost accounting standards. Technical non-cost data comes from engineering drawings, engineering specifications, certification documents, or direct experience (i.e., weighing an item). Schedule, quantity and equivalent units, and similar information comes from Industrial Engineering, Operations Departments, program files or other program intelligence. Inflation indices normally combine external and internal information. Examples of external information used in these determinations include the Consumer Price Index (CPI), Producer Price Index (PPI), Commodity Price Indices and other forecasts of inflation from various econometric models. There are other external sources of data including databases containing pooled and normalized information from various places (other companies or public record information). Although such information can often be useful, weaknesses of these sources include: (1)

The inability of the user to have knowledge of the procedures (i.e., accounting) used by the other contributors.

(2)

The treatment of anomalies (how they were handled) in the original data.

(3)

Knowledge of the manufacturing processes used and how they compare to the current scenario being estimated.

(4)

The inability to accurately forecast future indices.

Internal contractor information includes analyses such as private corporate inflation studies, or "market basket" analyses. Such interval information provides data specific to a company's product line(s) (i.e., radar products) that could be relevant to a generic segment of the economy as a whole (i.e., electronics); etc. Such specific analyses would normally be prepared as part of an

15

Parametric Cost Estimating Handbook

exercise to benchmark government provided indices (the CPI), and to compare corporate performance to broader standards. It is important to realize that sources of data can be almost unlimited, and all relevant information should be considered in a parametric analysis, if practical. Although major sources are described above, data sources should not be constrained to a specific list. Any data included in calculating parametric parameters will vary between model developers. However, the way in which parametric models are calculated from historical data and the way they are applied in the estimating process should be consistent within individual estimating systems.

SIGNIFICANT ADJUSTMENTS TO PARAMETRIC DATA

What follows below are some of the more significant adjustments that may have to be made to historical parametric cost data.

Consistent Scope Adjustments are appropriate for differences in program or product scope between the historical data and the estimate being made. For example, if the systems engineering department made a comparison of five similar programs and then realized that only two of the five had design to cost (DTC) requirements. To normalize the data, the DTC hours were deleted from the two programs to create a consistent systems scope and definition for CER development.

Anomalies Historical cost data should be adjusted for anomalies (unusual events), prior to CER analysis, when it is not reasonable to expect these unusual costs to be present in the new projects. The adjustments and judgments used in preparing the historical data for analysis, should be fully documented. For example, a comparison has been made to compare the development test program from five similar programs and then certain observations are made (from history and interviews) that one of the programs experienced a major test failure (e.g., qualification, ground test, flight test).

16

Parametric Cost Estimating Handbook

A considerable amount of labor resources were required to fact find and then determine the root cause of and develop an action plan for a solution. Should the hours be left in or deleted?

Improved Technology Cost changes, due to changes in technology, are a matter of judgment and analysis. All bases for such adjustments should be documented and disclosed. For example, electronic circuitry was originally designed with discreet components, but now the electronics are ASIC technology. A hardware enclosure once was made from aluminum and now is made, for weight constraints, of magnesium. What is the impact on the hours? Perfect historical data may not exist, but judgment and analysis should supply reasonable results. For example, suppose there are four production (manufacturing hours) lots of data that look like this: Lot 1 = 256,000 = 853 hours/unit Lot 2 = 332,000 = 738 hours/unit Lot 3 = 361,760 = 952 hours/unit Lot 4 = 207,000 = 690 hours/unit

Clearly, Lot 3's history should be investigated. It is not acceptable to merely "throw out" Lot 3 and work with the other three lots. A careful analysis should be performed on the data to determine why it behaved the way it did. There may have been a strike, or possibly an unusual and serious engineering problem impacted production costs. In any event, careful analysis is important.

OTHER TYPES OF ADJUSTMENTS AND DATA NORMALIZATION

Inflation There are no fixed ways to establish universal inflation indices (past, present or future) that fit all possible situations. Inflation indices are influenced by internal considerations as well as external inflation rates. Therefore, while generalized inflation indices may be used, it may also be possible to tailor and negotiate indices used on an individual basis to specific labor rate agreements (e.g., FPRAs) and the actual materials used on the project. Inflation indices should be based on the

17

Parametric Cost Estimating Handbook

cost of materials and labor on a unit basis (piece, pounds, hour) and should not include other considerations like changes in manpower loading or the amount of materials used per unit of production. The key to inflation adjustments is consistency. If cost is adjusted to a fixed reference date for calibration purposes, the same type of inflation index must be used in escalating the cost forward or backwards, from the reference date, and then to the date of the estimate.

Learning Curve The learning curve, as originally conceived, analyses labor hours over successive production units of a manufactured item. The curve is defined by the following equation: Hours/Unit = First Unit Hours *Ub or Fixed Year Cost/Unit = First Unit Cost *Ub Where:

U = Unit number b = Slope of the curve

In parametric models, the learning curve is often used to analyze the direct cost of successively manufactured units. Direct Cost equals the cost of both touch labor and direct materials - in fixed year dollars. Sometimes this may be called an improvement curve. The slope is calculated using hours or constant year dollars. A more detailed explanation of learning curve theory is presented in Chapter III.

Production Rate Production rate effects (changes in production rate, i.e., units/months) can be calculated in various ways. For example, by adding another term to the learning or improvement curve equation we would obtain: Hours/Unit = Ub * Rr or, Fixed Yr $/Unit First Unit $ * Ub * Rr Where: U = Unit number b = Learning curve slope

18

Parametric Cost Estimating Handbook

R = Production rate r = Production rate curve slope The net effect of adding the production rate effect equation (Rr) is to adjust First Unit $ for rate. The equation will also yield a different "b" value. Rate effect may be ignored or can be treated in different ways in different models. If possible, rate effects should be derived from historical data program behavior patterns observed as production rates change while holding the learning curve constant. The rate effect can vary considerably depending on what was required to effect the change. For example, were new facilities required or did the change involve only a change in manpower or overtime?

CALIBRATION AND VALIDATION OF COST MODELS

Once data has been collected and normalized, cost models can be developed. Although we will discuss cost models in much more depth in Chapters IV and V, a few comments are relevant here. There are two general types of cost models: internal (contractor developed) and commercially available. Internal, contractor developed models are derived from unique contractor data and generally do not require calibration since they have been calibrated in a defacto manner. On the other hand, commercial models are based on more universal data, and almost always need some form of calibration to be useful. The cost driver equation(s) utilized in a commercial cost model are based on a database external to the specific data being used to support the current estimate. Calibration, then, is the process of computing a multiplier(s), to be used with the general purpose equation(s), such that the combined equation(s) will predict the cost as reflected by the technical and programmatic data being used to support the estimate. Specialized (Internal) cost models are based directly on the data being used to support the estimate. Since the CER’s are derived directly from the supporting data, the model is, by definition, calibrated.

19

Parametric Cost Estimating Handbook

The result of calibrating an item, in a commercial model, is a calibration factor which is used in the commercial model's equations, such that the equations are then made to calculate the value of the item. Cost models need to be calibrated and validated for acceptance. The validation of a cost model is a process which usually includes the following steps: (1) Calibrate the model to historical cost data. (2) Estimate the cost of past completed projects. (3) Compare the estimates with actual costs to demonstrate acceptable accuracy.

It is the combined use of the model with the estimating process that must achieve acceptable results to provide a basis for the validation of the model. It may also involve disclosure of how the model works so that the effects of scaling and heuristic analysis can be evaluated by management, customers or auditors. Validation implies that interested parties have agreed that the model is a valid and acceptable estimating tool, and predicts costs within a reasonable range of accuracy.

SOME REVIEW AND AUDIT CONSIDERATIONS

Almost certainly, any data utilized will have to undergo a review and audit, so proper documentation is a must. Documentation should include: (1) A record of all mathematical formulas and "goodness of fit" and other statistics. (2) A record of adjustments to the original cost data and why those adjustments were made. (3) A statement of work for the historical data; judgment factors should be identified with rationale. (4) An audit trail for the data used for validation that will allow rederivation of the adjusted data and/or CER from the original source(s). This information must be identified and available upon request.

Any CER’s and data used in a cost model will need to be updated periodically and/or as agreed to with the PCO. The updating procedure should satisfy the Truth in Negotiation Act

20

Parametric Cost Estimating Handbook

(TINA) requirements by the time of agreement on price or another time as agreed upon by the parties.

DATA NORMALIZATION PROCESS

Specifying an estimating methodology is an important early step in the estimating process. The basic estimating methodologies (analogy, catalog prices, extrapolation, factors/ratios, grassroots and parametric) are all data-driven. To use any of these methodologies, credible and timely data inputs are required. If data required for a specific approach is not available, then that methodology cannot be used. Given that all methodologies are data-driven, it is critical that the estimator know the best data sources. Here are nine basic sources of data and a description of what specific data can be obtained from each source. Definitions of the differences between primary and secondary sources of data are provided. Finally, there is a review of the type of information that should be available from an accounting system, and a description of how to collect and analyze data is also given. The information presented in this Handbook will help the collection and analysis of the two data types (primary and secondary) required to specify, and apply a parametric estimating methodology. Remember - any data needs to be available, reliable and convincing before an estimating methodology can be chosen that utilizes the foundation data. The two types of data are: 1. Primary data is obtained from the original source. Primary data is considered the best in quality, and ultimately the most useful. 2. Secondary data is derived (possibly "sanitized") from primary data. It is not obtained directly from the source. Since it was derived (actually changed) from the original data, it may be of lower overall quality and usefulness.

When preparing a cost estimate, look for all credible data sources. If at all possible, use primary sources of data. There are nine main sources of data and they are listed in the chart below.

21

Parametric Cost Estimating Handbook

1. 2. 3. 4. 5. 6. 7. 8. 9.

NINE SOURCES OF DATA Basic Accounting Records Primary Cost Reports Either (Primary or Secondary) Historical Databases Either Functional Specialist Either Other Organizations Either Technical Databases Either Other Information Systems Either Contracts Secondary Cost Proposals Secondary

The following normalization process description is not intended to be all inclusive.

Normalizing Cost Data Making Units/Elements of Cost Consistent Making Year of Economics Consistent

Normalizing The Size Data Weight and Density Comparisons Weight Contingency Application (weight reduction programs?) Percent Electronics

Normalizing Products By Mission Application Grouping Vehicles by Complexity Calibrating Like Vehicles

Normalizing End Terms For Homogeneity Account for Absent Cost Items Removing Inapplicable Cost Items

22

Parametric Cost Estimating Handbook

Normalizing Recurring/Non-Recurring Cost Prime Contractors' Estimates Time Phased Costs Flight-Article Equivalent Units

Normalizing State-Of-Development Variables Mission Uniqueness Product Uniqueness

Normalizing Environments (Platform): Manned Space Unmanned Space Aerospace Shipboard Commercial

Collecting the data to produce an estimate, and evaluating the data for reasonableness, is a very critical and time-consuming step of the estimating process. When collecting the data needed to integrate cost, schedule, and technical information for an estimate, it is important to obtain cost information, and also the technical and schedule information. The technical and schedule characteristics of programs are important because they drive cost. They provide the basis for the final cost. For example, assume the cost of another program is available and a program engineer has been asked to relate the cost of the program to that of some other program. If the engineer is not provided with specific technical and schedule information that defines the similar program, the engineer is not going to be able to accurately compare the programs, nor is he or she going to be able to respond to questions a cost estimator may have regarding the product being estimated vis-àvis the historical data.

23

Parametric Cost Estimating Handbook

The bottom line is that the cost analysts and estimators are not solely concerned with cost data. They need to have technical and schedule data available in order to adjust, interpret, and lend credence to the cost data being used for estimating purposes. A cost estimator has to know the standard sources where historical cost data exists. This knowledge comes from experience and from those people, the so-called local experts, that are available to answer key questions. A cost analyst or estimator should be constantly searching out new sources of data. A new source might keep cost and technical data on some item of importance to the current estimate. Do not hesitate to ask anyone who might know or be able to help, since it is critical to have relevant cost, schedule and technical information at all times. The chart below summarizes important points about data collection and evaluation.

DATA COLLECTION, EVALUATION AND NORMALIZATION ∗

Very Critical, Time Consuming Step

∗

Need Actual Historical Cost, Schedule, and Technical Information

∗

Know Standard Sources

∗

Search Out New Sources

∗

Capture Historical Data

In order to develop a parametric model, a necessary requirement is to possess historical cost, schedule and technical data on a set of data points. The idea here is that generally more data is better than less. It is necessary to know what trends exist, and to understand why the trends are as they are. Some models have been found to be based on the opinions of experts instead of historical data. Although the opinions of experts may be germane, sound historical data is preferable for model development, audit and analysis. In addition to the historical data points, information on the cost, technical and quantity drivers needs to be examined to determine which does the best job of predicting cost. A statistical analysis on the data is accomplished to determine the strongest predictor(s) or driver(s) of cost, that is, the independent variable(s). (See further explanations in Chapter III).

24

Parametric Cost Estimating Handbook

It is very important to note that when performing a statistical analysis, be sure that functional specialists can provide realistic and reliable parameters for independent variables, given the stage of the program being estimated. Illustrating this point, suppose a statistical relationship is developed that has very strong correlation, and a potential cost driver has been discovered. However, data for the same independent variable for the estimate is not available. The parametric model would not then help with the estimate. Finally, knowledge of basic statistics, modeling skills and an understanding of analytical techniques is necessary to develop parametric estimating relationships. (See also Chapter III). The above information is summarized on the chart below.

TYPE OF INFORMATION NEEDED TO DEVELOP A PARAMETRIC MODEL ∗

Reliable Historical Cost, Schedule, and Technical Data on a Set of Data Points

*

WBS, WBS Dictionary & Product Tree

*

Analysis to Determine Significant Cost Drivers

*

Knowledge of Basic Statistics, Modeling Skills and CER Development

*

Analysis Techniques

To use a parametric model, the model needs to be well-documented. The documentation of a parametric model should include the source of data used to derive the parameters, and the size and range of the database. Additional information that should be included in the documentation of a parametric model are: how the parameters were derived, what the model's limitations are, the time frame of the database and, how well the parametric tool/model estimates its own database. All of this information should be located in the source document of a parametric model and should be read before deciding to use it in an estimate. By reading the source document, the strengths and weakness of the parametric model can be assessed and a determination can be made about any appropriateness for use. A statistic called the Mean Absolute Deviation (MAD) can be developed for cost models. It is a simple statistic that evaluates and assesses how well a parametric model estimates its own database. For example, if the MAD is 20%, then it means that the parametric equation(s) estimates its own database within plus or minus 20%. This is an important statistic to know, because if a

25

Parametric Cost Estimating Handbook

CER does not estimate its own database well, then its credibility with other data points outside its database would be questionable. To successfully use parametric model methodology, a requirement is to obtain realistic, most-likely range estimates for the independent variable values. Sometimes functional experts are not sure what the real performance requirements are for a new program. In such cases, a most-likely range will provide values that reflect an assessment of the associated uncertainties or unknowns (a Risk Analysis). Again, the top functional experts who know the program should identify and estimate the range of cost driving characteristics. They should also confirm the applicability of the specific parametric cost model from a technical perspective. The information needed to use a parametric data model is listed on the chart that follows.

TYPE OF INFORMATION NEEDED TO USE A PARAMETRIC MODEL Well-Documented Parameters Identifying: Source of data used to derive the parametric model Size of database Time frame of database Range of database How parameters were derived Limitations spelled out How well the parametric model estimates its own database Consistent and well defined WBS dictionary Realistic Estimates of Most Likely Range for Independent Variable Values Top Functional Experts Knowledgeable about The Program You are Estimating To identify most-likely range for cost drivers To confirm applicability of parametric from technical perspective

A parametric estimating methodology can be used at any stage of a program's life cycle. For example, a general parametric model may be utilized in the early, conceptual phase of a program, although the same parametric model could be inappropriate to use in the follow-on

26

Parametric Cost Estimating Handbook

production phase of a program.

However, a detailed parametric model used in production

estimating that is based on the experience and actual historical data of two or three previous production lots, could yield excellent validity. Hence, a parametric methodology can be used at any stage of a program's life cycle as long as the parametric model is based on the level and type of information available at that stage. The methodology can be used for any WBS element, not just hardware and software. Parametrics can be successfully applied to Systems Engineering/Program Management, Test, Training and Data, etc., provided that historical data points are available to develop solid, statistical relationships that provide reliable estimates of independent variables.

PITFALLS TO THE USE OF A PARAMETRIC MODEL

When a parametric model is applied to values outside its database range, the credibility of the resulting estimate becomes questionable. In cost estimating, one rarely finds large, directly applicable databases, and the source document has to be evaluated to determine if the parametric can be applied to the current estimate. However, it is possible to develop parametric tools that relate cost based on generic complexity values or tables. Such generalized parameters, can be related to the task at hand by an experienced modeler that results in a good cost model, but a parametric model always needs to make sense for the present estimate. Additionally and before using, one should validate models based on expert opinion. This is accomplished first by obtaining some actual, historical data points (technical, schedule, and cost) on completed programs similar to the current program. With this data in hand, apply the model to the actual technical and schedule information and see how well the parametric model predicts the known cost. If the model estimated the actual costs with an acceptable margin of error, validate the model for programs that are similar to the historical data point. Careful validation will help insure that cost models are appropriately used. Many times a parametric model needs to be adjusted if the new system has cost drivers and/or requirements that are not reflected in the parametric's database. In some of these cases a combination of parametric methodology with an approach taken from the analogy methodology can

27

Parametric Cost Estimating Handbook

be used to develop an estimate. This is accomplished by adjusting the results of the parametric approach with scaling or complexity factors that reflect any unique requirements. For example, parametrics and analogy approaches could be effectively combined to estimate the costs of a new program for which technology advancement is anticipated. First, either develop or use an existing parametric model, based on similar data points, to estimate the cost of the program, without technology advancement. Second, address the technology advancement by consulting with functional experts to obtain a most-likely range for a relative scaling factor that will reflect any advancements in technology. The relative scaling or complexity factor is applied to the result of the parametric estimate, and adjusts it for the impact of technology advancement. This is a solid and valid estimating approach for assessing the cost impacts of technology advancement, or other "complexity" differences. In such cases, the parametric model has to be adjusted so that it makes sense vis-à-vis the current estimate. If there exist no realistic estimates for the independent variable values for the product or effort being estimated, then parametric models should not be used. The level of uncertainty in the estimate will be considerable, and the validity of the estimate questionable. In cases such as this, parametrics can only be used as a benchmark. As stated previously, it is very difficult for functional specialists to provide a single point estimate.

Moreover, a single point estimate does not reflect the uncertainty inherent in any

functional expert's opinions.

Consider requesting most likely range values rather than point

estimates, if possible. Even after a parametric analysis has been completed it is prudent to follow it with a risk analysis evaluation of the high risk areas and key cost drivers. The chart below displays these points.

PITFALLS TO AVOID IN THE USE OF A PARAMETRIC ESTIMATE *

Using the parametric model outside its database range

*

Using a parametric model not researched or validated

28

Parametric Cost Estimating Handbook

*

Using a parametric model without adjustment when new system requirements are not reflected in the parametric's database

*

Using a parametric model without access to realistic estimates of the independent variables' values for product/effort being estimated

*

Requesting point estimates for independent variable value values versus a most likely range, if possible and practical

TWO ILLUSTRATIONS OF TYPICAL DATA NORMALIZATION PROBLEMS

Illustration 1 You plan to do a parametric estimate of a system using some company history. The system under investigation is similar to a system built several years ago. The two systems compare as follows: Parameter

Historical System

Prospective System

Date of Fabrication

Jul 89-Jun 91

Jul 95-Dec 95

Production Quantity

500

750

Size- Weight

22 lb. external case

20 lb. external case

5 lb. int. chassis

5 lb. int. chassis

8lb. of elec parts

10 lb. elec parts

1 cu ft-roughly cubical

.75 cu ft-rec. solid

12.l x ll.5 x l2.5

8 x lO x l6.2

Manual of oper incl.

Minor chgs to manual

Volume

Other Prog Features

5% Elec parts as spares

Normalization: In this instance, we would require adjusting for inflation factors, the quantity difference, the rate of production effect and the added elements in the original program (the spare parts and manual). The analyst should be careful normalizing these data. General inflation factors are almost certainly not appropriate to most situations. Ideally, the analyst will have a good index of costs which is specific to the industry and will use labor cost adjustments specific to his/her company. The quantity and rate adjustments will have to consider the quantity effects on

29

Parametric Cost Estimating Handbook

the company's vendors and the ratio of overhead and setup to the total production cost. Likewise, with rate factors each labor element will have to be examined to determine how strongly the rate affects labor costs. On the other hand, the physical parameters do not suggest that significant adjustments or normalizations are required. The first order normalization of the historic data would consist of: 1) Material escalation using industry or company material cost history. 2) Labor escalation using company history. 3) Material quantity price breaks using company history. 4) Possible production rate effects on touch labor (if any) and unit overhead costs.

Because both cases are single lot batches, and are within a factor of two in quantity, only a small learning curve or production rate adjustment likely is required.

Illustration 2 You are considering building some equipment for the first time. Relevant labor effort history relates to equipment built some time ago but reliable data is only available from the third lot, beginning at unit 250. The available history indicates a cost improvement curve on this history of 95% from the fourth lot on. This history is considered the best available. Normalization here requires two steps. Unless you believe the early lot cost improvement curve is the same as the historical lot improvement curve of the later lots (unlikely), you will need to identify cost improvement curves which may be applicable. These data points will have to be normalized to some common condition and the resulting improvement curve applied to the case at hand. Suppose the relevant history is as follows:

Case

Date

Quan.

Lots

Case A

Rate of Prod.

Improvement Curve

1985-87

1000

6

5 per day

90%

Case B

1990

400

2

2 per day

83%

Case C

1991

350

3

1.9 per day

91%

30

Parametric Cost Estimating Handbook

Case D

1993-95

5000

10

7 per day

93%

Clearly, the selection of an appropriate learning curve will be a judgment call that will depend upon the case at hand and how it relates to the historical data. For instance, there is some suggestion that the fewer the number of lots, the steeper the learning curve. How does the production rate and quantity match the data? In this situation, the analyst could break up the production history and examine the relationship of the learning curve to the lot sequence, the quantity, and the production rate. The date may reflect the technology involved and any historical comparisons will have to include the impact of relevant technology.

SUMMARY

In summary, let’s explore why there might be a database problem. Given the large amount data available on a hardware system, one might wonder why building a database presents such a problem. Let's consider a few of the more important reasons: (1) Information in the wrong format. While we do indeed possess a great deal of data, in many cases this data is not in an appropriate format to be very useful in developing CER’s. The reason is primarily due to the fact that the information systems which generate required data were originally established to meet organizational objectives that are no longer relevant or are not concerned about cost estimating or analysis. In short, the orientation of a large number of past and existing information systems is toward the input side with little or no provision for making meaningful translations reflecting output data useful in CER development or similar types of analysis. (2) Differences in definitions of categories and inconsistent WBS's and WBS categories. We also have a problem in building a database when the definition of the content of cost categories fails to correspond to the definition of analogous categories in existing databases. For example, some analysts put engineering drawings into the data category while others put engineering drawings into the engineering category. A properly defined WBS product tree and dictionary can avoid or minimize these inconsistencies.

31

Parametric Cost Estimating Handbook

(3) The Influence of Temporal Factors. It goes without saying that historical data is generated over time. This means that numerous dynamic factors will influence data being collected in certain areas. For example, the definition of the content of various cost categories being used to accumulate the historical data may change as a system evolves. Similarly, inflation changes will occur and be reflected in the cost data being collected over time. In addition, as the DoD deals with a rapidly changing technical environment, cost data generated for a given era or class of technology is necessarily limited. In fact, we would consider ourselves especially lucky if we can obtain five to ten good data points for certain types of hardware. Too few data points can lead to problems in our attempts to develop statistically sound CER’s for forecasting costs into the near and distant future. (4) Comparability Problems. Such problems include, but are not limited to, changes to a company's department numbers, accounting systems and disclosure statements, changes from indirect to direct change personnel for a given function, among others. To summarize, when we develop a database, we must normalize that database to be sure we have comparable data. For example, if we are building a database with cost data, we must first remove the effects of inflation so that all costs are displayed in constant dollars. We must also normalize the data for consistency in content; that is, we must ensure that a particular cost category has the same definition in terms of content. Normalizing cost data is a challenging problem, but it is a problem which must be resolved if a good database is to be constructed. Resolving database problems so that an information system exists to meet our needs is not easy. For example cost analysis methodologies typically vary considerably from one analysis or estimate to another. The requirements for CER’s -- hence the data and information requirements -are not constant over time. In short, the analyst who is working in support of long range forecasting needs cannot specify information needs once and for all.

These needs must be reviewed

periodically. The maintenance and update of any database must also be considered. An outdated database may be of very little use in forecasting future acquisition costs. However, we must also consider the costs of maintaining and updating a database. We may find that we simply cannot afford to provide for many current databases. Finally, since today we deal with a set of rapidly changing technology, we may find ourselves limited to developing a CER with a relatively small database.

32

Parametric Cost Estimating Handbook

Resolving the database problem must, at a minimum, focus on being provided with an explicit SOW within the RFP and standardized cost category definitions. Then we must consider how to manage and utilize the database we have within our individual organizations.

33

Parametric Cost Estimating Handbook

CHAPTER III

ELEMENTARY STATISTICAL TECHNIQUES AND

CER DEVELOPMENT

34

Parametric Cost Estimating Handbook

CHAPTER III ELEMENTARY STATISTICAL TECHNIQUES AND CER DEVELOPMENT

Proper CER development and application depends heavily upon understanding of certain mathematical and statistical techniques. Some of the simpler and more widely used techniques are explained in this chapter. Many other techniques are available in standard statistical text books.

CER AND MODEL DEVELOPMENT - UNCERTAINTY AND RISK REDUCTION

CER development and cost modeling are forms of forecasting real world cost events. How much effort will it take to build a bomber or a space sensor? An estimate has a probability of being within a given percentage of the “correct” answer. The better the project cost data and cost model, the closer will be the predicted cost to the final actual cost at project completion. Since a model of the real world involves simplifications, the final actual cost will rarely equal the estimated cost. These modeling uncertainties translate into “risk”” of producing a realistic cost estimate within a given percentage of the final actual project cost. Such uncertainties can be grouped into two major categories: 1.

Uncertainty of any organization to perform as planned due to unexpected resource

or schedulings in the scope of effort to produce the design, prototype or product, and, 2.

Uncertainty associated with the development (and, hence, usefulness) of any cost

model. This item includes: a.

Uncertainty associated with omission of a key cost driver,

b.

Mis-specification of the form of the model equation,

c.

Modeling limitations associated with a lack of data, and,

d.

Data consistency across multiple project databases.

35

Parametric Cost Estimating Handbook

The first type of risk, (1) can be addressed through improved specifications in the scope of work and an improvement in the clarity of understanding between the customer and contractor. This would result in a decrease in the amount of contract and engineering change proposals (CCE/ECPs) and unnecessary rework. One of the second types of risk, (2a), uncertainty associated with omission of key cost drivers, is addressed through development of historical cost data by product line, defined Work Breakdown Structure (WBS), and systematic understanding of the types of cost associated with development, prototype, and production programs. In risk (2b), the avoidance of the mis-specification of the form of the model equation, careful review of in-house and available industry data is required to create the basis of the model. Affirmative answers are required to the following four questions. Does the model make basis economic sense? Is the cost estimate an extrapolation within the scope of the cost and database? Is this type of cost estimate no higher a risk estimate than an interpolation between existing historical data in size and complexity? How well do the outputs compare to history? If poor patterns exist between the model and history, it is likely that an important parameter has been left out. The form of the cost equation should then be re-examined. Risk (2c) deals with the question about the sufficiency of data points that are available to validate the cost model. A lack of data points will increase the uncertainty of the cost model equation. Can more data points be included? The use of additional relevant data can increase the validity of any model. In the absence of substantial data, have experts critique the reasonableness of the model outputs. Usually there is a lack of data in a relatively new technological or programmatic area of cost modeling. How can we model the costs associated with an improvement in processes or teaming arrangements that improve producibility and lower costs relative to the old ways of program management and control? Metric data is now being gathered within many companies, so this data issue should become much less severe in years to come.

In the meantime,

organizational process improvement curves, a variation of the quantity of production (repetition) based on historical cost improvement curves, can be used to evaluate the benefits of process improvements.

36

Parametric Cost Estimating Handbook

Risk (2d), data consistency, can be mitigated by using Work Breakdown Structure (WBS) cost elements which are standardized across projects. (At the end of this Handbook, Appendix B contains a treatise about developing Work Breakdown Structures). If definitions for system engineering, project management, etc., are not consistent, then the resulting cost model and its cost estimating capability for each of the WBS elements may be flawed. Cost analysts need to have at their disposal a correct WBS cost estimating tool, complete with a WBS dictionary. if a cost model is developed for similar hardware items (objects) across programs, then the definition of these items should be standardized as much as possible. Differences in definitions from program to program or between product lines are captured by the complexity of labor, or material cost differences.

DEVELOPING COST ESTIMATING RELATIONS (CERs)

For CERs to be valid, they must be developed based on sound statistical concepts. Once valid CER’s have been developed, then parametric cost modeling can proceed. The following discussion will review some of the more commonly used statistical techniques for CER development. CERs are the key tool used in estimating by the cost analyst. CERs may be used at any time in the estimating process. For example, CERs may be used in the concept or validation phase of estimating program costs because there is insufficient system definition to use anything else. CERs may also be used in later phases of estimating program costs as a cross check of another estimating procedure, or as the primary basis for an estimate. The value of a CER is dependent upon the soundness of the database from which the CER is developed and the appropriateness of that CER to what is to be estimated. Determination of the “goodness” of a CER and its applicability to the system being estimated requires a thorough analysis by the cost analyst. The process of validating and selecting a CER is the subject of this chapter. We will begin by defining CERs and then look at how a CER may be developed. Next, we will look at when to use a CER and note the strengths and weaknesses of CERs. Finally, we will consider the techniques for developing CERs; the linear regression model

37

Parametric Cost Estimating Handbook

of the Least Squares Best Fit (LSBF) model, along with a few other statistical techniques. Appendix D describes additional statistical techniques. We can make a few general observations about CER development. We know that CERs are analytical equations which relate various categories of cost (either in dollars or physical units) to cost drivers or explanatory variables.

We also know that CERs can take numerous

forms,ranging from informal rules of thumb or simple analogies to formal mathematical DEFINITION: Cost Estimating Relationships (CER’s) are mathematical expressions relating cost as the dependent variable to one or more independent cost driving variables.

functions derived from statistical analysis of empirical data. If we are going to develop a CER, however, then we need to focus on assembling and refining the data that constitute the empirical basis for that CER. In deriving a CER, assembling the database is especially important and, generally, a very time-consuming activity (see Chapter II). Deriving valid CERs is a difficult task and the number of really good, valid CERs is significantly fewer than one might expect. While there are many reasons for the lack of valid CERs, the number one reason is the lack of an appropriate database. When developing a CER, the analyst must first hypothesize a logical estimating relationship. For example, does it make sense to expect that costs will increase as aircraft engine thrusts increase? Given that it does make sense that cost and engine thrust has some direct relationship, the analyst will need to refine that hypothesis to determine whether the relationship is linear or curvilinear.

After developing a hypothetical relationship, the analyst needs to

assemble a database. Sometimes, when assembling a database, the analyst discovers that the raw data is at least partially in the wrong format for analytical purposes and displays irregularities and inconsistencies. Adjustments to the raw data, therefore, almost always need to be made to ensure a reasonably consistent and comparable database. It is important to note that no degree of sophistication in the use of advanced mathematical statistics can compensate for a seriously deficient database. Since the data problem is fundamental, typically a considerable amount of time is devoted to collecting data, adjusting that data to help ensure consistency and comparability, and providing

38

Parametric Cost Estimating Handbook

for proper storage or information so that it can be rapidly retrieved when it is needed. In fact, more effort is typically devoted to assembling a quality database than to anything else. Given the appropriate information, however, the analytical task of deriving CER equations is often relatively easy.

Hypothesis Testing Of A Logical CER Complementing the issues of deriving a good database is the need to hypothesize what the CER should be. Some analysts believe the hypothesis comes first, then the data search to build a good database.

Other analysts believe the data search comes first, and given the

availability of data, the subsequent determination of a logical relationship or hypothesis occurs. Regardless of the position taken, the analyst must determine a logical estimating relationship. The analyst must structure the forecasting model and formulate the hypothesis to be tested. The work may take several forms depending upon forecasting needs. It involves discussions with engineers to identify potential cost driving variables, scrutiny of the technical and cost proposals, and identification of cost relationships. Only with an understanding of hardware requirements can an analyst attempt to hypothesize a forecasting model necessary to develop a CER.

The CER Model Once the database is developed and a hypothesis determined, the analyst is ready to mathematically model the CER. While this analysis can take several forms, both linear and curvilinear, we will initially consider one simple model -- the LSBF model. A number of statistical PC packages are available to generate the LSBF equation, but we will manually develop the LSBF equation in the latter part of this chapter. Once established, the database and the hypothesis testing complete the modeling activity and the equations are then relatively easy to derive.

WHEN TO USE A CER

When a CER has been build from an assembled database based on a hypothesized logical statistical relationship, and within an accepted or revised hypothesis, one is ready to apply the

39

Parametric Cost Estimating Handbook

CER. It may be used to forecast future costs or it may be used as a cross check of an estimate done with another estimating technique. For example, one may have generated an estimate using a grassroots approach -- a detailed build up by hours and rates -- then “benchmark” the grassroots estimate with a CER. A CER built for a specific forecast may be used with far more confidence than a generic CER. One must be especially careful in using a generic CER when the characteristics of the forecast universe are, or are likely to be, different from those reflected in the CER. One may need to qualify a generic CER by reviewing the database and the assumptions made for its use. A need to update the database with data appropriate for forecasting may be necessary. One may also find that the generic CER can be used only as a starting point. When using a generic CER as a point-of-departure, enhance or modify the forecast in light of any other available supplementary information. This most likely will involve several iterations before the final forecast is determined. It is important to carefully document the iterations so that an audit trail exists explaining how the generic CER evolved to become the final forecast. In using any CER -- specially built or generic -- the main theme is invariably: Be careful; Use good judgment! CERs are a fundamental estimating tool used by cost analysts. However, we must us them while applying a good deal of judgment. For example, we need to be concerned about the uncertainty of a future system in terms of chance elements in the “real” world; that is, uncertainty about the state of the world in terms of technological, strategic, and political factors. Also, we must carefully document as we go, so that an all important audit trail is maintained.

STRENGTHS AND WEAKNESSES OF CERs

In applying good judgment in the use of CERs, we need to remain mindful of the strengths and weaknesses of CERs. Some of the more common ones are presented below:

Strengths

40

Parametric Cost Estimating Handbook

1.

One of the principle strengths of CERs is that they are quick and easy to use. Given a CER equation and the required input data, one can generally turn out an estimate quickly.

2.

A CER can be used with limited system information.

Consequently, CERs are

especially useful in the RDT&E phase of a program. 3.

A CER is an excellent (statistically sound) predictor if derived from a sound database, and can be relied upon to produce quality estimates.

Weaknesses 1.

CERs are sometimes too simplistic to forecast costs. Generally, if one has detailed information, the detail may be reliably used for estimates.

If available, another

estimating approach may be selected rather than a CER. 2.

Problems with the database may mean that a particular CER should not be used. While the analyst developing a CER should validate that CER, it is the responsibility of any user to validate the CER by reviewing the source documentation. Read what the CER is supposed to estimate, what data were used to build that CER, how old the data are, how they were normalized, etc. Never use a cost model without reviewing its source documentation.

Now that we know what a CER is, how to develop a CER, when to use a CER, and some of a CER’s strengths and weaknesses, we can develop techniques for building CER’s. The LSBF technique is only one mathematical transformation of the database - the linear regression model. Other sophisticated curvilinear models can also be developed, but will not be explored in this Handbook. An analyst should be mindful that little in the estimating world is linear.

REGRESSION ANALYSIS

The purpose of regression analysis is to improve our ability to predict the next “real world” occurrence of our dependent variable.

Regression analysis may be defined as the

mathematical nature of the association between two variables. The association is determined in

41

Parametric Cost Estimating Handbook

the form of a mathematical equation. Such an equation provides the ability to predict one variable on the basis of the knowledge of the other variable. The variable whose value is to be predicted is called the dependent variable. The variable about which knowledge is available or can be obtained is called the independent variable. In other words, the dependent variable is dependent upon he value of independent variables. The relationships between variables may be linear or curvilinear. By linear, we mean that the functional relationship can be described graphically (on a common X-Y coordinate system) by a straight line and mathematically by the common form: y = a + bx where y = (represents) the calculated value of y - the dependent variable x = the independent variable b = the slope of the line, the change in y divided by the corresponding change in x a and b are constants for any value of x and y Looking at the bi-variate regression equation -- the linear relationship of two variables -we find that regression analysis can be described by an equation. The equation consists of two distinctive parts, the functional part and the random part. The equation for a bi-variate regression population is: Y = A + BX + E where A + BX is the functional part (a straight line) and E is the random part. A and B are parameters of the population that exactly describe the intercept and slope of the relationship. E represents the ran or “error” part of the equation. The random part of the equation is always there because the errors of assigning values, the errors of measurement, and errors of observation.

These types of errors are always with us because of our human

limitations, and the limitations associated with real world events.

Since it is practically impossible to capture data for an entire population, we normally work with a sample from that population. We denote that we are working with a sample by adjusting our equation to the form:

42

Parametric Cost Estimating Handbook

y = a + bx + e, where a + bx represents the functional part of the equation and e represents the random part. Our estimate of A and B in the population are represented by a and b, respectively, in the sample equation. In this sense then, a and b are statistics. That is, they are estimates of population parameters. As statistics, they are subject to sampling errors. As such, a good random sampling plan is important. 1.

The values of the dependent variable are distributed by a normal distribution function about the regression line.

2.

The mean value of each distribution lies on the regression line.

3.

The variance of each array of the independent variable is constant.

4.

The error term in any observation is independent of the error term in all other observations. When this assumption is violated, data is said autocorrelated. This assumption fixes the error term to be a truly random variable.

5.

There are no errors in the values of the independent variables. The regression model specifies that the independent variable be a fixed number -- not a random variable. For example, you wish to estimate the cost of a new bomber aircraft at mach 2, then mach 2 is the value of the independent variable. Mach 2 is a fixed number. The regression model will not handle errors in the independent variables.

6.

All causation in the model is one way. This simply means that if causation is built into the model, the causation must go from he independent variable to the dependent variable. Causation, though neither statistical nor a mathematical requirements, is a highly desirable attribute when using the regression model for forecasting. Causation, of course, is what we, as cost analysts, are expected to determine. We do this in our hypothesis of a logical mathematical relationship in either building or reviewing a CER equation.

CURVE FITTING

43

Parametric Cost Estimating Handbook

There are two standard methods of curve fitting. One method has the analyst plot the data and fit a smooth curve to the data. This is known as the graphical method. The other method uses formulas or a “best-fit” approach where an appropriate theoretical curve is assumed and mathematical procedures are used to provide the one “best-fit” curve; this is known as the Least Squares Best Fit (LSBF) method. We are going to work the simplest model to handle, the straight line, which is expressed as: Y = a + bx

Graphical Method To apply the graphical method, the data must first be plotted on graph paper. No attempt should be made to make the smooth curve actually pass through the data points which have been plotted; rather, the curve should pass between the data points leaving approximately an equal number of points on either side of the line. For linear data, a clear ruler or other straightedge may be used to fit the curve. The objective in fitting the curve is to “best-fit” the curve to the data points plotted; that is, each data point plotted is equally important and the curve you fit must consider each and every data point. Although considered a rather outdated technique today, plotting the data is still always a good idea. By plotting the data, we get a picture of the relationship and can easily focus on those points which may require further investigation. Hence, as a first step, we shold plot the data and note any data points which may require further investigations before developing a forecasting graphical curve or mathematical equation.

LSBF Method The LSBF method specifies the one line which best fits the data set we are working with. The method does this by minimizing the sum of the squared deviations of the observed values of Y and calculated values of Y. For example, if the distances: (Y1 - YC1,), (Y2 - YC2), (Y3 - YC3), (Y4 - YC4), etc., parallel to the Y-axis, are measured from the observed data points to the curve, then the LSBF line is the one that minimizes the following equation (see Figure III-1): (Y1 - YC1)2 + (Y2 - YC2)2 + (Y3 - YC3)2 + ... + (Yn - YCn)2

44

Parametric Cost Estimating Handbook

y

• yc4 y3 •

• y4

• yc3 y1 • • yc1

• yc2 • y2 x

Figure III-1. The LSBF Line.

In other words, the sum of the deviations from the observed value of Y, and the calculated value of Y - Yc squared, is a minimum; i.e., (Y - YC)2 is a minimum. This same distance, (Y - YC) is the error term or residual. Therefore, the LSBF line is one that can be defined as: ΣE2 is a minimum because Σ (Y - YC)2 = ΣE2 For a straight line, Y = a + bx and, with N points, we have (X1, Y1,), (X2, Y2), (X3, Y3), ... (Yn , Yn)

The sum of the squares of the distances is a minimum if, ΣY

= AN + BΣX and

ΣXY = AΣX + BΣX2 These two equations are called the normal equations of the LSBF line. Reference to any comprehensive statistical textbook will illustrate that these two equations do meet the requirements of the properties of ordinary LSBF regression. These properties are: 1.

The technique considers all points.

2.

The sum of the deviations between the line and observed points is zero, that is, Σ(Y - Yc)2 = ΣE2 = a minimum. Similarities between these two properties and the arithmetic mean should also be

observed. The arithmetic mean is the use of the values of the independent variable divided by

45

Parametric Cost Estimating Handbook

the number of observations or ΣX/n = x and the sum of the “Ys” divided by the number of observations or Σy/n = y . Now, instead of considering the mean as a point when dealing with only one variable, we are now using a line -- the LSBF regression line. Note that: The parameters, a and b, define a unique line with a Y-intercept of a and a slope of b. To calculate the values needed to solve for a and b, we need a spreadsheet (See Table III1). For example, use the values in Table III-2.

Table III-1. Table Needed to Get Sums, Squares and Cross Products X X1 X2 X3 ΣXn

Y Y1 Y2 Y3 ΣYn

X*Y X1 * Y1 X2 * Y2 X3 * Y3 Σ(Xn * Yn)

X2 X12 X22 X32 ΣXn2

Y2 Y12 Y22 Y32 ΣYn2

X2 16 121 9 81 49 4 280

Y2 100 576 64 144 81 9 974

Table III-2 X 4 11 3 9 7 2 36

Y 10 24 8 12 9 3 66

XY 40 264 24 108 63 6 505

We can substitute the table values into the equations for b and a: where x = ∑ x / n

and, y =

∑y/n

Solving for b, we get: b=

∑ xy − nx y ∑ x − nx 2

2

b = 505 - 6(6)11

46

Parametric Cost Estimating Handbook

280 - 6(6)2 b = 109 64 b = 1.703125 Solving for a yields: a = y − bx

a = 11 − (1703125 . )6 a =.78128 Therefore, the regression equation (calculated y) is Yc =.78128 + 170312 . x

Multiple Regression In simple regression analysis, a single independent variable (X) is used to estimate the dependent variable (Y), and the relationship is assumed to be linear (a straight line). This is the most common form of regression analysis used in contract pricing. However, there are more complex versions of the regression equation that can be used to consider the effects of more than one independent variable on Y. That is, multiple regression analysis assumes that the change in Y can be better explained by using more than one independent variable.

For example,

automobile gasoline consumption may be largely explained by the number of miles driven. However, it might be better explained if we also considered factors such as the weight of the automobile. In this case, the value of Y would be explained by two independent variables. Yc = A + B1X1 + B2X2 where: Yc = the calculated or estimated value for the dependent variable A = the Y intercept, the value of Y when X = 0 X1 = the first independent (explanatory) variable B1 = the slope of the line related to the change in X1, the value by which change when X1 changes by one X2 = the second independent variable

47

Parametric Cost Estimating Handbook

B2 = the slope of the line related to the change in X2, the value by which change when X2 changes by one Multiple regression will not be considered in depth in this Handbook. Consult a statistics text to learn more about multiple regression.

Curvilinear Regression In some cases, the relationship between the independent variable(s) may not be linear. Instead, a graph of the relationship on ordinary graph paper would depict a curve. For example, improvement curve analysis uses a special form of curvilinear regression. As with multiple regression, consult a statistics text to learn more about curvilinear regression. “GOODNESS” OF FIT, R AND R2

Now that we have developed the LSBF regression equations, we need to determine how good the equation is. That is, we would like to know how good a forecast we will get by using our equation. In order to answer this question, we must consider a check for the “goodness” of fit, the coefficient of correlation (R) and the related coefficient of determination (R2). There are a number of other statistics we could check which would expand upon our knowledge of the regression equation and our assurance of its forecasting capability. Some of these will be discussed late.

Correlation Analysis One indicator of the “goodness” of fit of a LSBF regression equation is correlation analysis.

Correlation analysis considers how closely the observed points fall to the LSBF

regression equation we develop. The assumption is that the more closely the observed values are to the regression equation, the better the fit; hence, the more confidence we can expect to have in the forecasting capability of our equation. It is important to note that correlation analysis refers only to the “goodness” of fit or how closely the observed values are to the regression equation. Correlation analysis tells us nothing about cause and effect, however.

48

Parametric Cost Estimating Handbook

Coefficient Of Determination The coefficient of determination (R2) represents the proportion of variation in the dependent variable that has been explained or accounted for by the regression line. The value of the coefficient of determination may vary from zero to one. A coefficient of determination of zero indicates that none of the variation in Y is explained by the regression equation; whereas a coefficient of determination of one indicates that 100 percent of the variation of Y has been explained by the regression equation. Graphically, when the R2 is zero, the observed values appear as in Figure III-2 (bottom) and when the R2 is one, the observed values all fall right on the regression line as in Figure III-2 (top).

y

y

Perfect Positive R2 = 1.0

x

Perfect Negative R2 = 1.0

x

y

No Correlation R2 = 0

x

FIGURE III-2 In order to calculate R2 we need to use the equation: R2 =

( ∑ xy − nx y ) 2 (∑ x 2 − x ∑ x) • (∑ y 2 − y∑ y)

R2 tells us the proportion of total variation that is explained by the regression line. Thus R2 is a relative measure of the “goodness” of fit of the observed data points to the regression line. For example, if we calculate R2 using the formula above and find that R2 = 0.70, this means that 70% of the total variation in the observed values of Y is explained by the observed values of X. Similarly, if R2 = 0.50, then 50% of the variation in Y is explained by X. If the regression line perfectly fits all the observed data points, then all residuals will be zero, which means that R2 =

49

Parametric Cost Estimating Handbook

1.00. In other words, a perfect straight-line fit will always yield R2 = 1. As the level of fit becomes less accurate, less and less of the variation in Y is explained by Y’s relation with X, which means that R2 must decrease. The lowest value of R2 is 0, which means that none of the variation in Y is explained by the observed values of X. Some applications require R2 of at least 0.7 or 0.8. An R2 < 0.25, which corresponds to an R < 0.5, would never be acceptable. Coefficient Of Correlation The coefficient of correlation (R) measures both the strength and direction of the relationship between X and Y. The meaning of the coefficient of correlation is not as explicit as that of the coefficient of determination. We can determine whether R is positive or negative by noting the sign of the scope of the line, or b. In other words, R takes the same sign as the slope; if b is positive, use the positive root of R2 and vice versa. For example, if R2 = 0.81; then R = + 0.9 and we determine whether R takes the positive root (+) or the negative root (-) by noting the sign of b. If b is negative, then we use the negative root of R2 to determine R. So to calculate R we need to know the sign of the slope of the line. It is most important to note that R does not tell us how much of the variation in Y is explained by the regression line. R is only valuable in telling us whether we have a direct or an inverse relationship and as a general indicator of the strength of the association.

THE LEARNING CURVE

Basic form of the “learning curve” equation is, y = a*xb or, Log y = Log a + b Log x where, y = Cost of Unit #x (or average for x units) a = Cost of first unit b = Learning curve coefficient Note that the equation Log y = Log a + b Log x is of precisely the same form as the linear equation y = a + bx. This means that the equation Log y = Log a + b Log x can be

50

Parametric Cost Estimating Handbook

graphed as a straight line on log-log graph paper and all the regression formulae apply to this equation just as they do to the equation y = a + bx. In order to derive a learning curve from cost data (units or lots) the regression equations need to be used, whether or not the calculations are performed manually or using a statistical package for your personal computer. In this sense, the learning curve equation is a special case of the LSBF technique. Since in learning curve methodologies cost is assumed to decrease by a fixed amount each time quantity doubles, then this constant is called the learning curve “slope” or percentage (i.e., 90%). For example, For unit #1 Y1 = A(1)b = A (First Unit Cost) and For unit #2 Y2 = A(2)b = Second Unit Cost So, Y2 = A*(2)b = 2b = a Constant, or “Slope” Y1 A Slope = 2b, and, Log Slope = bLog2 Therefore, b = Log Slope Log2

For a 90% “Slope,” b = Log .9 = -0.152 Log 2 If we assume that A = 1.0, then the relative cost between any units can be computed. Y3 = (3)-0.152 = 0.8462 Y6 = (6)-0.152 = 0.7616

Note that: Y6 = .7616 = 0.9 Y3 .8462

51

Parametric Cost Estimating Handbook

Any good statistical package (for instance StatView) can perform all the calculations (and many others) shown. A quality package will let you customize your results (create presentations) save your work, and calculate all these statistics: frequency distributions, percentiles, t-tests, variance tests, Pearson correlation and covariance, regression, ANOVA, factor analysis and more. Graphics and tables such as scattergrams, line charts, pie charts, bar charts, histograms, percentiles, factors, etc., should be available to the user. Statistical analysis is greatly simplified using these tools.

LIMITATIONS, ERRORS AND CAVEATS OF LSBF TECHNIQUES. When working with the LSBF technique, there are a number of limitations, errors and caveats of which we need to be aware. The following are some of the more obvious.

Extrapolation Beyond The Range Of The Observed Data A LSBF equation is truly valid only over the same range as the one from which the sample data was initially taken. In forecasting into the future, we do not know the shape of the curve; what we do know is that there is more estimating risk involved. We will give less credence to forecasts which exceed the range of the original data. However, this does not mean that we should never extrapolate beyond the relevant range; it may well be that forecasting beyond the relevant range is the only suitable alternative we have. What we must keep in mind is that extrapolation assigns values using a relationship that has been measured for circumstances which may differ from those for the forecast. That is, we are assuming that the past will perfectly predict the future -- and that will not always be true.

Cause And Effect Regression and correlation analysis can in no way determine cause and effect. It is up to the analyst to do a logic check, determine an appropriate hypothesis and analyze the data base such that an assessment can be made regarding cause and effect. For example, an R2 = .95 relates the number of public telephones in a city to liquor sales. Clearly, there is no cause and effect involved here. A deeper variable, population, is the truly independent variable that drives

52

Parametric Cost Estimating Handbook

both the number of public telephones and liquor sales. As analysts, we must ensure that we have chosen approximately related data sets and CERS, and that real cause and effect is at work. Be careful not to find relationships when they do not exist.

Using Past Trends To Estimate Future Trends Conditions can change: if the underlying population changes due to changes in technology, for example, then the LSBF equation would not be suitably used as a forecasting tool. In using a CER, we need to make sure the factors in the forecast still apply to the original historical LSBF equation.

Misinterpreting The Coefficients Of Correlation And Determination R will always be greater than R2. If R2 is 0.64, then R is 0.80. Don’t confuse the two.

Summary In this chapter we have so far discussed the development and use of Cost Estimating Relationships (CERs). We noted that in using or developing a CER, a high quality database is most critical (see Chapter II). Specifically, we highlighted the difficulties of assembling a good database for CER development and indicated several reasons why there could be a problem. We next considered the strengths and weaknesses of a CER. Finally, we developed one simple model for generating a CER -- the LSBF model. We completed our discussion of CER’s by identifying several limitations, errors and caveats when using CERs. Next, we’ll consider the use of CERs.

EXAMPLES OF CER USE Basically, CER’s reflect changes in prices or costs (in constant dollars) as some physical, performance or other cost-driving parameter(s) is changed. The same parameter(s) for a new item or service can be input to the CER model and a new price or cost can be estimated. Such relationships may be practically applied to a wide variety of items and services.

53

Parametric Cost Estimating Handbook

Construction Many construction contractors use a rule of thumb which relates floor space to building cost.

Once a general structural design is determined, the contractor or buyer can use this

relationship to estimate total building price or cost - excluding the cost of land. For example, if we were building a brick two-story house with a basement, we may use $60/square foot (or whatever value is currently reasonable for the application) to estimate the price of the house. $60/sq ft x 2200 sq ft = $132,000 house price

Weapons Procurement In the purchase of an airplane, CERs are often used to estimate the cost of the various parts of the aircraft. One item may be the price for a wing of a certain type of airplane, a supersonic fighter for example. History may enable the analyst to develop a CER relating wing surface area to cost. You may find that there is an estimated $40,000 of wing cost (for instance NRE) not related to surface area and another $1000/square foot that is related to surface area to build one wing. For a wing with 200 square feet of surface area we could estimate a price as: estimated price

=

$40,000 + 200 sq ft x $1,000 per sq ft

=

$40,000 + 200,000

=

$240,000

Electronics Manufacturers of certain electronic items have discovered that the cost of completed items varies directly with the number of total electronic parts in the item. Thus, the sum of the number of resistors, capacitors, inductors, and transistors in a specific circuit design may serve as an independent variable (cost driver) in a predictive CER. Assume a CER analysis indicates that $0.57 a unit is not related to the number of components with another $.11 added per part. If evaluation of the drawing revealed that an item was designed to contain 11 capacitors, 12 resistors, 5 transistors, and 2 inductors, the total part count becomes 30. Substituting the 30 parts into the CER: estimated cost = $0.57 + $.11 per part * number of parts = $0.57 + $.11 (30) = $0.57 + $3.3

54

Parametric Cost Estimating Handbook

= $3.87

Cost And Price Analysis CERs can be used for price analysis in a variety of ways, including aircraft engines, ships, trucks, and passenger autos. Generally, under the Truth in Negotation Act, price analysis is required even when cost analysis is used. CERs provide an extremely useful tool for price analysis after the completion of cost analysis. Like other techniques of price analysis, the CER approach requires the evaluator to determine whether the techniques used for comparison are fair and reasonable. To do this, the evaluator must determine the basis for the estimate (CERs) and its reliability. Some questions for this evaluation relating to how the estimate was made include: 1.

What was the source of information?

2.

What information and techniques were used?

3.

How reliable were the earlier estimates?

The same parametric techniques used for developing estimates can be used in cost and price analysis. Often the estimator uses known CER values as the basis for preparing an estimate. Likewise, estimates can be based on CERs from “similar to” past history.

Information And Techniques Detailed analysis of a requirement requires an in-depth study of specifications and drawings, a physical inspection or “tear down” of the item or a similar item, or an analysis of similar work. From such an analysis, the estimator can get a clear picture of the item or the service to be performed, and develop appropriate information and cost relationships. Once the tasks have been delineated, the user can estimate costs using personal experience, published time standards, component purchase prices, CERs and other data. In using this data for making an estimate, the estimator must consider the effects of such factors as dollar value changes and changes in technology. Quantitative tools, such as use of index numbers, can be used to allow for changes in the value of the dollar. Changes in technology are harder to contend with in cost analysis. Changes in materials and changes in manufacturing procedures occur today at an ever increasing rate. It is estimated that technology

55

Parametric Cost Estimating Handbook

“turns over” today every two years.

Technology is an area requiring user expertise and

knowledge for evaluating price effects of the application of new technology, to the current requirement. The evaluation of the cost impact of technology surges is a difficult task to anticipate and quantify.

Estimate Reliability In considering whether or not to use a specific estimate, we need to examine the “track record” of the cost estimator or the organization providing the estimate. If, in the past, the estimates have been close to actuals, greater reliance may be placed on the estimate. If estimates have been significantly above or below known actuals, then lower reliability should be placed on current estimates. Knowing the reliability of past estimates does not free the cost or price analyst from the obligation to review the estimate and the estimating methodology as they relate to proposal accuracy. Cost and price analysis should be temperred with product value. Knowledge of the product, its functions and its use is essential for sound contract pricing. Value analysis is a systematic and objective evaluation of the function of a product and its related costs. Its purpose is to ensure optimum value. Questions that help in the evaluation are: 1.

What must the product do?

2.

What does it cost now? What does it cost to operate and maintain?

3.

In what other ways can the function be performed?

4.

What will these alternatives cost?

Two Examples Of CER Use One can utilize Consumer Price Index numbers to perform simple value or price analysis. Index numbers are, quite simply, CER’s, and predict, from history the effects of inflation. For example, assume an item of equipment in 1980 cost $28,000 when an appropriate Consumer Price Index (CPI) was 140. If the current index is now 190 and an offer to sell the equipment for $40,000 has been suggested, how much of the price increase is due to inflation? How much of the price increase is due to other factors? Solution:

Px = Ix Py Iy

56

Parametric Cost Estimating Handbook

$28,000 = 1.40 Py 1.90 1.4 Py = 1.9 ($28,000) 1.4 Py = $53,2000 Py = $53,200 = $38,000 1.4

$38,000 now is roughly the equivalent of $28,000 in 1980. Hence the price difference due to inflation is $38,000 - $28,000 = $10,000.

The difference due to other causes is

$2,000

($40,000 - $38,000). The above example would illustrate the use of CPI numbers for a material cost analysis. The steps were: 1.

If we know what the price of an item was in the past, and we know the index numbers for both that time period and today, we can then predict what the price of that item should be now based on inflation alone.

2.

If we have the same information as above, and we have a proposed price, we can compare that price to what it should be based on inflation alone. If the proposed price is higher or lower than we expect with inflation, then we must investigate further to determine why a price or cost is higher or lower.

Consider the purchase of a house as another example that uses the LSBF technique. Historical data for other homes purchased, may be examined during an analysis of proposed prices for a newly designed house.

Using this data, we can demonstrate a procedure for

developing a CER. Step 1: Designate and define the dependent variable. In this case we will attempt to directly estimate the cost of a new house. Step 2: Select item characteristics to be tested for estimating the dependent variable. A variety of home characteristics could be used to estimate cost. These include such characteristics as: square feet of living area, exterior wall surface area, number of baths, and others.

57

Parametric Cost Estimating Handbook

Step 3: Collect data concerning the relationship between the dependent and independent variable. The example, Table III-3, shows data collected on five house plans so that we can determine a fair and reasonable price for a house of 2100 square feet and 2.5 baths. Table III-3. House Model Burger Metro Suburban Executive Ambassador New House

Unit Cost 166,500 165,000 168,000 160,000 157,000 Unknown

Baths 2.5 2.0 3.0 2.0 2.0 2.5

Sq. Feet Living Area 2,800 2,700 2,860 2,440 1,600 2,600

Sq. Feet Exterior Wall Surface 2,170 2,250 2,190 1,990 1,400 2,100

Step 4. Explore the relationship between the independent and dependent variables. As stated earlier, analysis of the relationship between the item characteristics and the dependent variable may be performed using a variety of techniques. Step 5. Determine the relationship that best predicts the dependent variable. Figure III-4 graphically depicts the relationship between the number of baths in the house and the price of the house. From the graph, there appears to be a relationship between the number of baths and house price. The relationship, however, may not be a good estimating tool, since three houses with a nearly $8,000 price difference (12 percent of the most expensive house) have the same number of baths.

$170

$165

$160

$155

$150 1

3

2 58

4

Parametric Cost Estimating Handbook

FIGURE III-4. Number of Bathrooms

Figure III-5 graphically relates square feet of living area to price. In this graph, there appears to be a strong linear relationship between house price and living area.

$170 $165 $160 $155 $150 1400

1600

1800

2000

2200

2400

2600

2800 3000

Figure III-5. Cost vs Square Feet Figure III-6 graphically depicts the relationship between price and exterior wall surface area. Again, there appears to be a linear relationship between house price and this independent variable. $170 $165 $160 $155 $150 1400

1500

1600

1700

1800

1900

2000

2100

Figure III-6. Cost vs Exterior Wall Surface (Sq. Ft.)

59

2200

2300

Parametric Cost Estimating Handbook

Based on this graphic analysis, it appears that square feet of living area and exterior wall surface have the most potential for development of a cost estimating relationship. We may now develop a “line-of-best-fit” graphic relationship by drawing a line through the average of the x values and the average of the y values and minimizing the vertical distance between the data points and the line (see Figure III-7 and Figure III-8).

Figure III-7. Linear Trend of Cost to Living Area (Sq. Ft.)

Viewing both these relationships, we might questions whether the Ambassador model data should be included in developing our CER. In developing a CER, you need not use all available data if all data is not comparable. However, you should not eliminate data just to get a better looking relationship. In this case, we find that the Ambassador’s size is substantially different from the other houses for which we have data and the house for which we are estimating the price.

This substantial difference in size might logically affect the relative

construction cost. The trend relationship in Figure III-8 and Figure III-9, using the data for the four other houses, would be substantially different than relationships using the Ambassador data. Based on this information, you might decided not to consider the Ambassador data in CER development.

60

Parametric Cost Estimating Handbook

Figure III-8. Linear Trend of Cost to Exterior Wall Surface (Sq. Ft.)

If you eliminate the Ambassador data, you find that the fit of a straight line relationship of price to the exterior wall surface is improved. For the relationship of price to square feet of living area, you find a close relationship, i.e., almost a straight line. (See Figure III-9)

$170 $168 $166 (2700: $165,000)

$164 $162 $160

2600 2800 3000 FIGURE III-9. Square Feet (Thousands) 2400

If you had to choose one relationship, you would probably select the one shown in Table III-3 (square feet of living area) over the relationship involving exterior wall surface because

61

Parametric Cost Estimating Handbook

there is so little variance shown about the trend line. If the analysis of these characteristics did not reveal a useful predictive relationship, you might consider combining two or more of the characteristics already discussed, or exploring new characteristics.

However, since the

relationship between living area and price is so close, we may reasonably use it for our CER. In documenting our findings, we can relate the process involved in selecting the living area for price estimation. We can use the graph developed as an estimating tool. The cost of the house could be calculated by using the same regression analysis formula discussed herein:

For square feet of living area:

Y = $117,750 + $17.50 (2600) Y = $117,750 + $45,500 Y = $163,250 estimated price

CERs, like most other tools of cost or price analysis, must be used with judgment. Judgment is required to evaluate the historical relationships in the light of new technology, new design, and other similar factors. Therefore, a knowledge of the factors involved in CER development is essential to proper application of the CER. Blind use of any tool can lead to disaster.

COMMON CERs

Table III-4 lists some common CERs used to predict prices or costs of certain items. In addition to CERs used for estimating total cost and prices, others may be used to estimate and evaluate individual elements of cost. CERs, for example, are frequently used to estimate labor hours. Tooling costs may be related to production labor hours, or some other facet of production. Other direct costs may be directly related to the labor effort involved in a program.

Table III-4. CER Types Product Construction

Independent Variable Floor space, roof surface area, wall surface area

62

Parametric Cost Estimating Handbook

Product Gears

Independent Variable Net weight, percent of scrap, inches of teeth cut, harness, envelope

Trucks

Empty weight, gross weight, horsepower, number of driving axles, loaded cruising speed

Passenger car

Curb weight, wheel base, passenger space, horsepower

Turbine engine

Dry weight, maximum thrust, cruise thrust, specific fuel consumption, by-pass ratio, inlet temperature

Reciprocating engine

Dry weight, piston displacement, compression ratio, horsepower

Sheetmetal

Net weight, percent of scrap, number of holes drilled, number of rivets placed, inches of welding, volume of envelope

Aircraft

Empty weight, speed, useful load, wing area, power, landing speed

Diesel locomotive

Horsepower, weight, cruising speed, maximum load on standard grade at standard speed

ACCEPTANCE CRITERIA FOR A COST ESTIMATING RELATIONSHIP

How good is a CER equation and how good is the CER likely to be for estimating the cost of new projects? What is the confidence level of answers at +/- x% from the number estimated, i.e., how likely is the estimated cost to fall within a specified range of cost outcomes? First, certain necessary conditions for a statistical analysis of a CER need to be stated: 1.

There are more data points than coefficients to be estimated.

2.

Error terms do not form a systematic pattern.

3.

The independent variables are not highly correlated.

4.

The form of the equation to be estimated is linear or has been translated into a linear form using logarithms.

5.

The model makes sense from an economics and technical point of view.

63

Parametric Cost Estimating Handbook

An R2 of equal or greater than 0.80 is desirable in curve fitting. An R2 of 0.50 associated with the CER is as good as tossing a balanced coin. The CER explained 1/2 of the observed cost outcomes. In general, the higher the R2 the better the “explanatory” capability of the cost equation. However, an R2 of 1.0 can indicate an “identity” of the cost variable and explanatory variable.

The data and explanatory variable being used should then be reexamined for

redundancy.

The “F” statistic measures the ratio of “explanation” of the explanatory variables (cost drivers) and the “residual” (error) term. The F statistic should have a value greater than 4.0 or 5.0 to indicate that a good cost driver has been selected for the cost model and that the form of the equation is acceptable. (A value of 1.0 indicates that the cost driver explains only 1/2 of the variation in the cost. This would not be a particularly good cost driver variable). The higher the F value the better the prediction capability of the cost drivers. Also, the higher the “F” statistic, the higher will be the R2 value. “Partial” F statistics can be used to examine the contribution of a single cost driver term. The higher the value, as in the “F” statistic, the better the additional contribution of the particular cost driver. See a statistics text or Appendix D for a more detailed explanation of the “F” statistics.

The “t” statistic can be used to test the validity of adding a particular cost driver variable. First, a “null” hypothesis is made that the cost parameter adds no predictive value to the model (cost equation). That is, the value of the parameter for the cost driver term being reviewed has a value of 0 (zero). The “t” test is used to make a decision to accept or reject the “null” hypothesis for a given cost driver term. Generally, a “t” value greater than five leads to the conclusion to reject the null hypothesis. A “t” value less than five leads to the acceptance of the null hypothesis, that the cost term does not add predictive value to the CER. Each case to accept or reject the null hypothesis depends upon the difference between the hypothesized and estimated coefficients, the confidence interval desired, and the degrees of freedom of the data (number of data observations minus the numbers of parameters being estimated).

64

Parametric Cost Estimating Handbook

The “t” statistic is also used for other applications. For example, in order to determine if two groups of data are from the same population or from two different populations. When then degrees of freedom in a data set approach 30, the statistics of the “t” distribution approach the normal distribution. If it is not known whether a normal distribution is justified, the “law of large numbers” can be invoked that states that for a large enough sample (large enough cost database), the error term involved in estimating cost will approach a normal distribution. That is, the normal distribution can be used instead of the “t” distribution to test the null hypothesis. The “t” is of similar shape to the normal distribution, except that there is a larger probability of lower cost or higher cost (extreme outcomes) associated with the “t” distribution rather than the normal distribution. Also, an analysis of the plot residual values can be useful. If a pattern exists, then correlation may be explained by other factors. If the plot of residuals is a scatter plot with no patterns, then the CER equation may be good if other factors are favorable. Another important statistical measure is the bandwidth or confidence interval associated with the application of the CER cost estimates. The bandwidth of the cost estimate depends upon the confidence interval required or desired, the parameter value, and the degrees of freedom of the data. See Appendix D for a more detailed explanation of the “t” distribution and confidence intervals.

AUDITING AND ANALYZING A CER CER Analysis Cost estimating relationships (CERs) relate cost to some other program element in a definite way. Examples of CERs are per diem rates, “shop supplies”, sales tax estimates, etc. CERs supposedly relate one cost to another or with a well defined parameter. When rolled into an interlocking algorithm, analysts have to probe both the estimate and the underlying data used to develop a CER. What distinguishes a CER from a conventional estimating approach is that CERs define a general relationship based on a set of data rather than a specific relationship based

65

Parametric Cost Estimating Handbook

on a direct precedent. A CER may be less precise than a convential estimating method but the cost savings resulting from the CER approach may be worth the potential loss of precision. Within detailed cost estimates, CER’s may be used for estimating small or derivative cost elements. CER’s are also commonly used for budgetary estimates, “rough order of magnitude” estimates, and simple cost-benefit calculations when the preliminary or uncertain nature of the project discourages a costly estimating effort. However, well built cost models in the hands of a professional can be better cost predictors than detail methods because certain judgement and other biases are more controlled.

General Features A CER can be a functional relationship between one variable and another and may represent a statistical relationship between some well defined program element and some specific cost. Since most company overhead rates are percentages of direct labor expense, these are CERs. Computer and travel costs often show statistical relationships to engineering costs, design is frequently closely correlated to drafting costs and these, in turn, to the number of drawings, parts, size, weight, etc. Many costs can be related to other costs or non-cost variables in some fashion but not all such relationships can be turned into a CER. A CER must have two characteristics. First, it should link in some rational way the underlying variable - the independent variable, and the cost being developed. Second, the CER should have a strong statistical fit, R2, and confidence interval, with the basis element.

Evaluating an Estimate CERs are used in lieu of a direct substantive link between a cost element and some basis of estimate (BOE). Since CERs are developed from collections of cases and represent average values, CERs have uncertainty built into them. (A direct BOE to cost estimate extrapolation is preferred if the cost element is significant, if a good BOE can be found, and if some well defined extrapolation can be postulated between the BOE and the cost element.) The first step in an analysis of a CER influenced estimate is to identify how much of the total estimate is CER driven.

66

Parametric Cost Estimating Handbook

The second step in analyzing a proposal using CERs is to evaluate the CERs themselves. Since CERs supposedly relate an element of cost to some variable or factor, the analyst must determine whether the relationship is truly functional and statistical. If the CER is a factor implied by a functional relationship, the analyst needs an explanation of the function and support for the assertion of a relationship. Both deterministic and statistical support are required. In other words, does the relationship make logical sense and is the pattern of influence regular enough to be helpful?

Base data must be available for examination, preferably original,

“unsmoothed” data. Again, the purpose of a CER is to save time and effort. If the amount of the proposal affected by CERs is not great, the evaluation effort applied to the CER should be an appropriate amount. In a “worse case” situation, the analyst may have to back track to the original data set used to develop a CER. In that case the analyst should attempt to see if all the relevant cases have been included and no “cherry picking” has occurred. In other words, what “risk” is involved by using the CER? Assuming the original data set is available and complete, the developer of the CER must explain the theory of the relationship and the data processing performed. If “outliers” were excluded, the estimator needs to explain why.

If the explanation of the exclusion is

unsatisfactory, the analyst may want to develop a set of CER’s with the outliers included. Ordinarily, outliers affect the deviation of the estimate rather than the value of the CER, but it is useful to check. If several data points have been excluded and if these influence the CER mean and standard deviation significantly, the CER may not be operationally useful even if theoreticaly valid. To illustrate, suppose a relationship is identified between variable K and cost variable C. Suppose the CER is 7 x K = C. If the value 7 is developed from an arithmetic average of a dozen values but three “outliers” have been excluded, then inclusion of the outliers may spread out the sample standard deviation to the point that confidence in the relationship may become suspect. The CER estimator should be able to supply the original data set and his/her analysis. The cost analyst may need to replicate the estimate to verify the calculation. If this is done, the CER statistics should be examined for: 1.

The sample size. Confidence in the estimate will increase as the sample size increases.

67

Parametric Cost Estimating Handbook

2.

The standard error of the mean (or the point estimate) should be shown along with the standard deviation of the calculated mean.

3.

The standard deviation of the sample set. What is the range of the majority of the data points? Confidence in the estimate increases as more and more data points fall within a specified range.

4.

If the CER is developed from a correlation calculation, the cost analyst can examine the coefficient of correlation. Correlation infers a link between the two factors but the relationship may be accidental. Standard statistical tests exist for checking the likelihood that a given correlation coefficient is accidental and should be used if the sample set is small, or if “R” is less than .8 (R2 = .64).

The last step in evaluation of a CER is calculating the effect which reasonable variations on the CER value can have on the estimate. If reasonable variations on the CER impact the estimate greatly, the analyst has to be skeptical of the explanatory power of the CER. The effect of this is to widen the actual range of an estimate.

Summary Of CER Audit And Analysis CER analysis requires addressing the questions: 1.

What is the proportion of the estimate directly affected by CERs?

2.

How much precision is appropriate to the estimate in total and to the part affected by the CERs?

3.

Is there a rational relationship between the individual CER affected variables and the underlying variables?

4.

Is the pattern of relationship functional or purely statistical?

5.

If functional, what is the functional relationship? And why?

6.

If statistical, is the history of the relationship extensive enough to give a confidence that it operates in the given case?

7.

Is the pattern of relationship statistically significant? And at what level of confidence?

8. What is the impact on the estimate of using reasonable variations of the CERs?

68

Parametric Cost Estimating Handbook

If the CERs represent a cost-effective response to an estimating problem and if they are rationally developed and solidly based, the CERs are valid and accurate tools for an estimate. Assuming no “show stopper” problems are uncovered, the analyst can accept the concept of the CERs and apply such margins of variance as seem reasonable. This chapter has presented the concept of Cost Estimating Relationships (CERs) and the statistical underpinnings of CER development and application. The basic mathematical relationships were described and examples showing the use of CERs were also presented. A flow chart of the CER development process is shown in Figure III-10.

69

Parametric Cost Estimating Handbook

Data Collection Company Databank Library Contractors DoD/NASA

Test Relationships Est

Actual

Data Evaluation and Normalization Unit Cost/Quantity Constant Year $ Escalation

Selection of Variables Weight # of Dwg Thrust Materials Range MIPs Impulse SLOC

Regression and Curvefit C = aX C = aXb C = aX + b

Data Analysis and Correlation Correlation Matrix Data Plot Data Subsets Dimensional Analysis

Select CERs C = aX + by

Validation

Approval

Revalidation

CER Database

When Necessary

TO COST MODELS

FIGURE III-10. Typical CER Development Process

70

Parametric Cost Estimating Handbook

CHAPTER IV

HARDWARE COST MODELING

71

Parametric Cost Estimating Handbook

CHAPTER IV HARDWARE COST MODELING

INTRODUCTION

Hardware cost is defined as all program cost, excluding software development, for all components, assemblies, subsystems, and systems for sea, ground, airborne or space applications. Hardware cost includes: mechanical, electromechanical, electrical/electronic, structural or pneumatic equipment, and may also include system test, or system operations. It is defined as the cost to develop, design, build, test, launch, and operate a system. Hardware cost estimating approaches are categorized into five basic methods: (a) parametric, (b) analogy, (c) grassroots, (d) other (methods such as, standard time or expert consensus approaches), and (e) combinations of the other form. The first three are well recognized and clearly defined. The last two "methods" cover everything else. This section will focus on parametric hardware cost modeling techniques. Parametric estimating is the mathematical procedure or process in which product or service descriptors directly yield consistent cost information. A parametric cost model is an estimating system comprised of Cost Estimating Relationships (CERs) and other parametric estimating functions, e.g., cost quantity relationships, inflation factors, staff skills, schedules, etc. Parametric cost models yield product or service costs at designated Cost or Work Breakdown Structure (CBS/WBS) levels and may provide departmentalized breakdowns of generic cost elements. A parametric cost estimate provides a logical and repeatable relationship between input variables and the resultant costs. Hardware parametric cost estimating techniques are widely utilized by industry and government. The use of these techniques expedites the development of price proposals by the contractor, evaluation by the government, and subsequent negotiation. The simplest form of parametric estimating tool is a CER. In this form a relationship between cost and non-cost variables is investigated to determine any possible cause and effect linkages. A statistical analysis is performed on representative samples of a historical database to

72

Parametric Cost Estimating Handbook

validate these linkages. The analysis is usually based on the investigation of the relationship between two or more variables. The known variable (physical parameter, schedule, etc.) is called the independent variable; the variable we are trying to predict, the cost of the item, is the dependent variable. The resultant CER is an expression of the mathematical relationship between parameters characterizing the item to be estimated and its cost. The strength of CER's lie in their relative simplicity and the convenience they afford in predicting costs. The primary limitations of CER's is that they will yield reliable cost estimates only within the limits of the spread of independent variables that were used when the CER was developed. In high technology areas, the limits of input parameters (independent variables) might go outside the boundaries of the historical database. Therefore, either the CER's must be re-examined and updated as the historical database increases, or the use of the CER must be limited, or used appropriately by properly trained professionals. A more sophisticated form of parametric estimating is the computerized Parametric Estimating Model. These models estimate costs on a broader scale and are more versatile and much less restrictive than the acceptable boundaries of CERs. With parametric estimating models, system acquisition costs can be derived that are based upon parameters such as quantity, size, weight, power consumption, environmental specification, complexity and type of electronic packaging, etc.

Some models can also provide schedule

information, or evaluate specified schedules in terms of costs. Parametric estimating models can provide early cost measurement of system concepts with a limited description of physical or other parameters.

They are also frequently utilized for independent assessment of conventionally

prepared cost estimates. Numerous "home grown" parametric cost models exist throughout industry and government to cover a specific range or type of products or systems. There are also universal parametric estimating systems that generate, internal to the model, appropriate expressions of CER's for a broad range of products or systems. These models perform a mathematical extrapolation of past experience in order to predict cost. Computerized parametric cost estimating models often contain many mathematical equations relating the input variables to cost. Each set of input parameters uniquely defines the item of interest for cost modeling.

73

Parametric Cost Estimating Handbook

Universal parametric (i.e, generic) cost estimating models usually require calibration. Considering the variety of products, the differences in accounting methods, etc., the calibration process "fine tunes" these models to emulate the producer's organizational and product characteristics. The calibration process simulates past product history and organizational traits. Parametric estimating systems (models) exist to rapidly compute hardware development and design costs, manufacturing costs of prototypes and production units along with all the manufacturing support costs. Models are also available for computing operation and support costs of hardware in the field. Also, models used for the development and maintenance costs of computer software are available. A model is a representation of a real-life system. In the case of cost estimating, a parametric model is a system of data, equations, and inferences presented in mathematical form to represent or describe the cost of an item. It relates knowns (system descriptions or parameters) to unknowns (cost of systems). A model can be as simple as a single equation (CER), or as complex as an inter-relation of many equations and functions. It can also be designed to estimate items as small as modules or components, or as large as subsystems, systems, total programs, or large development activities.

OVERVIEW OF HARDWARE COST MODELING

A Hardware Cost Model provides estimates of system acquisition costs based upon quantitative parameters such as complexity, quantity, weight, and size; and qualitative parameters such as environmental specification, type of packaging, and level of integration; and schedule parameters such as months to first prototype, manufacturing rate, and amount of new design. Hardware cost parametrics bring speed, accuracy, and flexibility to the cost estimating process. Early cost measurement of concepts is crucial to a new venture since there is little opportunity to change program costs significantly once a design has been detailed. Parametric cost models have been developed to operate with limited concept description so that many alternatives can be costed before designs and bills of material are finalized. Parametrics can also be used extensively as the basis of a cost estimate in preparation of proposals as well as for the independent assessment of conventionally prepared cost estimates.

74

Parametric Cost Estimating Handbook

Hardware cost models may be formulated as a universal system to generate appropriate regressions or CERs for a range of products or systems. In essence, they permit extrapolation of past experience to predict costs. Inputs to hardware parametrics cover a wide range of systems. Since all assemblies must have weight and size, these are often used by the models as principal descriptors. Electronic hardware items are characterized by the electronic application and type of componentry. Mechanical and structural elements can be described in terms of their construction, type of material, functionality, machinability, and manufacturing process. Some uses of Parametric Hardware cost models are: *

Proposal preparation and submittal

*

Estimates of cost to complete

*

Estimates of modifications

*

Should cost analysis

*

Most Probable Cost estimates

*

Evaluations of bids and proposals ("Sanity" Checks)

*

Vendor negotiations

*

Life cycle cost estimates

*

Basis of estimates

*

Independent cost estimates

*

"What ifs"

*

Design cost trade-off analysis

*

Bid/No Bid decisions

*

Estimates of spare parts costs

*

Design to Cost

*

Rough Order of Magnitude (ROM) cost estimates of new or advanced hardware systems

Parametrics are applicable to all aspects of hardware acquisition - development, production, purchased and furnished hardware, hardware modifications, subcontractor liaison, hardware/ software integration, multiple lot production, and hardware integration and test.

A normal

modeling capability should exist at all levels of the Work Breakdown Structure (WBS). Estimates

75

Parametric Cost Estimating Handbook

of integration and test costs which are attributed to each level of integration should also be estimated. When a parametric model calculates a cost for manufacturing, it does not use a parts list and labor resource build up, but rather a parametric representation of the parts and labor costs. Fundamental Input Parameters To Typical Parametric Models Are Listed Below: *

Functional design parameters.

*

Quantities of equipment to be developed, produced, modified, purchased furnished and integrated and tested.

*

Applications (technology of materials and processes) of structural and electronic portions of the hardware.

*

Hardware geometry consisting of size, weight of electronic and structural elements, and electronic packaging density.

*

Amount of new design required and complexity of the development engineering task.

*

Operational environment and specification requirements of the hardware.

*

Schedules for development, production, procurement, modification, and integration and testing.

*

Fabrication process to be used for production.

*

Yield considerations for hardware development.

*

Pertinent escalation rates and mark-ups for General and Administration charges, profit, cost of money, and purchased item handling.

The fundamental characteristic of parametric inputs is inter-relationship. A change in any one parameter is usually not localized to one cost element, but rather, may have a ripple effect on several cost elements. Consider the impact of a change in quantity. Certainly this change would cause a change in the manufacturing cost. It may also effect the fabrication process, and, hence, the cost of tooling and test equipment. In addition, a change in quantity could impact the production schedule, and thus, impact the costs associated with escalation. An impact on integration and test, sustaining engineering, and project management would almost certainly result from a change in quantity. This dynamic effect is characteristic of most input variables, and most parametric models.

76

Parametric Cost Estimating Handbook

The model’s input parameters uniquely define the hardware for cost estimating and modeling. The resultant cost output is determined from the model's mathematical equations alone. Cost may be estimated with a minimal amount of hardware information. This feature makes models a legitimate tool for cost estimation of programs in the conceptual stage of development. Of course, it is always preferable to provide as much information to the models as possible. In this way, statistical uncertainty associated with the input variables is reduced. The modeling activity is basically a conversation between the model and the cost analyst. Parametric data representing the hardware to be costed is formulated, and the analyst calls upon embedded text, tables, and utilities of the model to help create and store the data used to drive the model. In the course of using a parametric model, the analyst controls the output formats, the ability to perform sensitivity and risk analysis, and of course the data that are used to estimate the cost of the hardware. The data elements created during a session usually represent systems or sub-systems composed of many separate sub-assemblies.

For example, an aircraft might be

represented as a system composed of an airframe, propulsion units, avionics, air to ground controls, flight controls, and ordinance. In turn, the flight controls might be a sub-system composed of instrumentation, radar, and various processors. At an even lower level, we might find a digital processor composed of input/output circuits, data storage memory, program memory, logic circuits, power supply, and chassis and housing. The number of hardware elements and relative detail of the parametric information in each is determined by the analyst. There is no limit to the level of detailed data used for an analysis as long as the cost model contains historically verifiable parametric cost relationships at the desired level. Nor is there a requirement that precludes a parametric user from treating the entire aircraft as a basic assembly of the lowest order. Thus, an analysis might be accomplished with one data set representing the aircraft as an assembly, or it might be attained with many elements representing the aircraft as a system of sub-systems, assemblies, and sub-assemblies. The amount of detail provided to the model is purely at the descretion of the analyst. Parametrics can estimate costs for the development and production phases of the acquisition process. Outputs are categorized by such cost elements as: drafting, design, project management, system engineering, data, manufacturing, and tooling and test equipment, etc.

77

Parametric Cost Estimating Handbook

Some parametric models contain the effects of phased interactions between engineering and manufacturing. In addition to considering a normal performance period, estimates of the cost impacts due to accelerated or protracted engineering schedules or due to an operation plan that requires stops and restarts of production effort can also be computed. A comprehensive parametric model should allow an analyst to: *

Calibrate (Automated is best)

*

Estimate the cost of multiple lot production

*

Calculate manufacturing costs of non-homogenous assemblies

*

Determine the cost impact of compressing or extending development or production schedules.

*

Estimate the cost impact of the development schedule (concurrency or lapse) on production.

*

Perform Cost Risk Analysis

A graphic depiction of hardware modeling is shown in Figure IV-1.

HARDWARE MODELING Hardware model applies common parameters in the comparative evaluation of new requirements in light of analogous histories.

• • • • • • • •

Input Parameters Magnitude (quantity) Operating Environment (MIL-SPEC, Airborne, etc.) Amount of new design & design reuse Engineering complexity Manufacturing complexity Schedulde HW & SW integration Weight/Volume

•

⇒

⇒

• •

Output Parameters Cost • Development • Production • Engineering • Manufacturing Schedule risks Unit/System integration cost

FIGURE IV - 1 Hardware modeling normally entails stringing together numerous CERs using an appropriate mathematical logic. The math logic is derived during the calibration process, and allows the math model to emulate the "real world" of program history or to other supportable data or estimates. When the "real world" model is developed, then the new requirements can be

78

Parametric Cost Estimating Handbook

evaluated based on the calibrated model. Modeling offers significant advantages to the estimator in that once CER development and the calibration process is complete, cost estimating is quick, repeatable, and cost effective. For instance, the first unit production missile cost (T1) may be modeled as T1 = [(Structure CER) + (Propulsion CER) + (G&A CER)]*1.15 (integration factor) Then, Total Missile H/W Cost = T1 * Qb, where, Q = quantity, and b is a cost improvement exponent. Also, for systems engineering and program management we may get S/E + P/M = .10*(Missile H/W cost) Models can also be used to estimate non-recurring development costs. These costs would be estimated using parameters such as the number of prototypes, number of tests, the amount of new design and design repeat, design complexity factors, and others. A simplified parametric cost estimating process is shown in Figure IV-2. The process connects the parametric model to a post processor that in turn converts the parametric model outputs into typical cost proposal reports. Ultimately, the customer needs proposal reports in standardized formats so that competing contractors can be conveniently compared. Just as the value of a house, when it is evaluated for a mortgage or a refinancing, is calculated using parametric approaches (square feet, # rooms, zip code, # bathrooms, etc.), the complex products we normally deal with can be evaluated in exactly the same manner. Parametric modeling can greatly simplify the cost estimating and cost analysis processes.

79

Parametric Cost Estimating Handbook

PARAMETRIC COST ESTIMATING CONCEPT OTHER REPORTS

PROGRAM HISTORY Cost

MANPOWER BY FUNCTION

PROGRAM HISTORY Functional Distribution

SUBCONTRACT SUMMARY $ BY HRS BY TIME

Technical Programmatic s

MANPOWER BY WBS COST BY WBS

COST MODELS COST ESTIMATING RELATIONSHIPS

POST PROCESSOR

COST AND MANPOWER REPORTS BY WBS BY LABOR TYPE TIME SPREADS MAKE/BUY

New Program Baseline

Forward Pricing Rates

WBS Lines of Code Make/Buy Weight Power Material Schedules Vendor Quotes

FIGURE IV-2

80

Parametric Cost Estimating Handbook

MICRO-CIRCUIT AND ELECTRONIC MODULE MODELING

Parametric cost models exist that are designed to provide quick and reliable estimates for development and production costs of individual custom micro-circuit chips and/or electronic modules. These models provide the capability to estimate costs and schedules associated with the development and production of modular electronics, custom Application Specific Integrated Circuits (ASICs), hybrid or MMIC packages, common components, and system modules. In view of the significant cumulative expenses for custom chips, these models are invaluable for procurement or product planning. Models can assist analysts in future product planning and incorporate state of the art technology in the cost estimating process. Using a WBS structure, analysts can use models to form cost estimates for individual micro-circuits (chips), the integration of these chips, and additional electronic components on electronic modules. If the situation exists, these models may be integrated onto electronic modules with additional components. The final electronic module may then be integrated into the WBS used by the hardware models. Micro-circuit model input parameters are determined by physically describing the micro-circuit, other electronic components, and the electronic module. A wide range of custom chips can be processed through use of generic input parameters which include: *

Number of pins

*

Number of transistors and/or gates

*

Die size

*

Type of packaging

*

CAD sophistication

*

Amount of new design

*

Manufacturing yield

*

Development/Production schedules

Inputs for electronic modules include: *

Module size

*

Number of sides used for component mounting

*

Technology

*

Function (Memory, Analog, etc.)

*

Quantity produced and/or developed 81

Parametric Cost Estimating Handbook

*

Packaging or interconnect method

*

Board type

*

Development/Production schedules

*

Quantity and types of components

Components as defined for use within these models may be custom chips as well as off-the-shelf items. The amount of detail about the components may be very limited or as detailed as available information will permit. A component database file may be used to catalog standard components that are used on several different electronic modules.

This file may contain

information about the standard components including the cost, if known. Whenever a component from a database file populates a module, it can be referred to by name with only the number of components per module required as additional information. The analyst should be allowed to provide as much cost information as is available. This information may contain component cost, board cost, and/or assembly and test costs. Any costs not provided will then be estimated by the model. Output formats will depend on the item modeled. For a custom micro-circuit, costs might be estimated for: Development engineering costs such as: * Chip specification, * Chip design, * Systems engineering, * Project management, * Data, * Prototype manufacturing

For electronic modules, some of the costs that are estimated are: * Development engineering: * Drafting * Design * Systems engineering * Project management * Data 82

Parametric Cost Estimating Handbook

* Prototype manufacturing * Tooling/Test Equipment

Production costs would be: * Drafting * Design * Project management * Data * Manufacturing * Tooling/Test Equipment.

As with the hardware models, micro-circuit models have the capability to be "fine tuned" to, directly reflect a particular company and/or product through the process of calibration. In this sequence of operation, the model uses actual cost data as inputs and provides development and manufacturing indices directly related to the historical costs and specifications. These calibrated indices can then be used as the basis for estimates of proposed custom micro-circuits and/or modules.

HARDWARE OPERATIONS AND SUPPORT (O&S) OR LIFE CYCLE MODELS

Life cycle cost is defined as the cost associated with all the phases of a program: design, development, prototype, production, and maintenance and operations (M&O or O&S). Models are available to estimate life cycle costs of a variety of hardware programs.

With appropriate

information, the life cycle costs of hardware systems can be evaluated. Through integration with the WBS structure, cost factors for different equipments are calculated in order to estimate life cycle costs. Moreover, the models provide for tailoring our analyses to fit a variety of maintenance concepts and supply systems in order to customize specific programs and user organizations. After the inputs of a set of descriptors, the models produce categories of life cycle cost in each of three phases: development, production, and support. In a typical application, analysts can instruct the model to select the most cost effective of numerous built-in maintenance concepts. Operating features allow the analyst to specify a subset of

83

Parametric Cost Estimating Handbook

the maintenance concepts for model evaluation. In addition, the analyst can tailor a variety of maintenance concepts by mixing the preestablished concepts. Some of these concepts could be: *

Discard CRU at Failure.

*

Replace parts at depot.

*

Replace parts at manufacturer location.

*

Replace mods at equipment level. Scrap bad mods.

The strength of life cycle models are derived from three factors: 1) use of hardware models to rapidly generate all hardware related input parameters -- particularly useful in the decision making stages of early concept development before hardware design details are known; 2) availability of preset maintenance concepts - among which the models can be instructed to determine the most cost effective approach; and, 3) ability to rapidly tailor program parameters to reflect specific support conditions.

The interactive feature of parametrics permit immediate

processing of sensitivity analyses to determine the effect of uncertainties of input data. Some of the sensitivity analyses allow the determination of the effects of changes in: *

Number of equipment level maintenance and/or supply locations.

*

Number of organization level maintenance and/or supply locations.

*

Number of intermediate level maintenance and/or supply locations.

*

Number of depot level maintenance and/or supply locations.

*

Duration of the support period.

*

Cost, size, and weight of units, modules, and parts.

*

Cost of contractor repair.

*

Test equipment costs and loading factor.

*

Shipping costs.

*

Crew size and labor rates.

*

Dedicated vs. non-dedicated crews.

*

Attrition.

Determination of the effects of the following may be addressed using risk analysis features built into some models: *

Equipment operating time.

*

Unit Mean Time Between Failure (MTBF). 84

Parametric Cost Estimating Handbook

*

Unit and Module Mean Time To Repair (MTTR).

*

Authorized stock levels at all supply levels.

*

Safety stock coefficients.

*

Scrap and repair fractions.

Another feature is the capability to model different theaters of deployment and multi year specification of equipment deployment and employment (called gradual force structuring). This capability permits more accurate modeling of remote depots sending work back to a central depot or the manufacturing facility (contractor facility). Force structure data permits variation in force levels for each year, as well as the planned levels of equipment operation for each year. There are three principal categories of data required to estimate costs with Life Cycle models: deployment and employment; hardware parameters; and program globals.

Deployment And Employment This category of inputs pertain to the number of locations at which equipments are installed, the number of locations at which they can be maintained, and the number of locations authorized to stock spares; how often the equipments are operating; and the length of time of the support period. The analyst defines the size of the program in terms of the number of locations at which equipments are installed; the number of organization maintenance locations (for maintenance concepts that involve organization maintenance);

the number of Intermediate maintenance locations (for

concepts that use Intermediate maintenance); and the number of depot maintenance locations (Government or contractor depots). With other inputs, the analyst can specify whether these locations are maintenance only, maintenance and supply, or supply only. To further define the size of the program, the analyst specifies the duration of the support period in years, and the "on-line" of the installed equipments either as hours per month or as a fraction of real time.

Hardware Parameters Hardware dependent parameters should be generated by the hardware parametric models for each element modeled in the WBS structure, and then input into the life cycle model.

Program Globals 85

Parametric Cost Estimating Handbook

This is the largest group of inputs, and they are used to describe the maintenance and supply system to the model. All have preset values that may be changed by the analyst. Unique sets may be created for use in subsequent applications of the model.

Some categories of Program Globals are: Spread Globals: Used to control distribution of resources and/or expenditures across schedules. Supply Administration: To uniquely define the administration as U.S. Air Force or Navy. Discretionary Supply Times: Frequency of unit, module, and part ordering. Input Data Multipliers:

Coefficients to permit uniform variation of hardware input

parameters for cost effect measurement. Miscellaneous: Includes controls for stock administration, programmable test equipment software, test equipment maintenance, attrition, average number of parts consumed per repair, false failure detection scheduled maintenance, and others. Provisioning and Labor: Stock controls for meeting expected demand of all units, modules, and parts at each pertinent supply location, number of hours in a work week, and labor costs at each maintenance location. By Theater:

Theater specific parameters dealing with pipeline times at each

maintenance/supply location, fractions of repairs resulting in scrapping, and supply stocking (inventory) cost factors.

COMMERCIAL MODELS

There are several commercially available hardware cost estimating models. Three typical models (FASTE, PRICE, SEER) that estimate one or more of the hardware types are discussed below. Although there are many others (See Appendix E) this chapter will only discribe these three as representative. The models will allow an analyst to:

*

Model an entire project through Engineering Development, Production, Integration, Installation and Life Cycle Phases.

*

Calibrate and Perform Forward Costing.

*

Project Input Files and Output Files. 86

Parametric Cost Estimating Handbook

*

An Economics File.

*

Input variables to the model.

*

Get needed information through the on-line Help system.

MicroFASTE The MicroFASTE model is used to help the analyst to develop a parametric model to be used to estimate the costs associated with implementing a project. The goal of a particular project may be the production and installation of a Hardware system, a Software system (or a combination of both). It may be the construction of nuclear power stations, radar systems or manned space stations. Even performing cost estimates for the implementation of a financial funding program, the construction and operation of an underground coal or uranium mine, or the design and production of a highly complex module containing very advanced micro-circuitry technology are possible through the techniques of parametric systems analysis. However, the MicroFASTE model (where the "E" on the end of "MicroFASTE" means Equipment), is exclusively for use in performing parametric analyses for Hardware systems. Hardware development projects involve several major phases of implementation. These phases and activities have much in common whether the project involves production and integration of a small hardware device, or a large, complex multi-assembly hardware system. It is these phases in which costing studies are performed. MicroFASTE classifies these common implementation phases into the following categories and sub-categories:

Equipment Acquisition Phases and Life Cycle (O&S) 1.

Engineering Design/Drafting Involves the detail design engineering and drafting effort that implements the governing specification.

Systems Engineering

87

Parametric Cost Estimating Handbook

Establishes the equipment's design, performance and test specifications, predicated on the controlling requirements.

Documentation The recording of engineering activities, preparation of equipment manuals and required management reports. Prototype and Testing Covers all charges connected with the manufacture and testing of engineering prototypes, and includes all brass and bread board models.

Special Engineering Tooling Embodies the special tooling charges affiliated with the prototype efforts. It does not include capital or amortized equipment that may be related to the tooling. When there are no prototypes, there will be no tooling charges.

Project Management Takes in the overall management of all areas connected with the engineering efforts such as planning, budgeting, operations and financial controls.

2.

Production Manufacturing Involves the direct production charges. This is the same cost value as calculated when Total Production is specified without the detail breakdowns.

Engineering Support Embodies the engineering effort that is realted to the manufacturing activity such as material design reviews, corrections of defects, etc.

Documentation The recording of production events as well as changes to equipment manuals as necessitated by design modifications caused by production problems.

88

Parametric Cost Estimating Handbook

Production Tooling Covers special required tooling. It does not include the cost of capital equipment, or tools that are amortized in overhead accounts. Project Management Takes in the management of all areas associated with production such as planning, budgeting, operations and financial control.

3.

Life Cycle (O&S) Life Cycle costs are the charges involved in the maintenance and support of the deployed equipments. The Life Cycle period runs from the first day of the Starting Year until the last day of the Finish Year specified in the Life Cycle procedure. The maintenance concept varies in relation to the equipment's technology and size. The deployed quantities, along with the indicated Mean-Time-Between-Failure and On-Time-Fraction may significantly affect the equipment availability. Although Life Cycle cost studies are performed in association with an Item's acquisition studies, Life Cycle costs may not be accumulated into the project's total costing structure. They must be kept track of separately; the reason for this is that the time frames over which Life Cycle costs would be realized always span a much greater period of time than other facets of the project.

System Integration Integration is a study, as an independent analysis that covers the costs of consolidating various Items into a higher order of assembly. In an Item's Acquisition study, there are variables used to indicate the character of the Item's integration requirements. When an integration study has been completed, the calculated integration costs may be accumulated into the project's total costs.

Equipment Installation Equipment Installation studies are applicable to large energy producing systems such as steam or nuclear power plants, and large hardware systems such as those installed in chemical manufacturing plants. When a significant percentage of the project's total cost will be involved in putting the equipment in place and making necessary mechanical and electrical connections to perform its intended task, that is when it becomes necessary to perform Installation cost analyses. 89

Parametric Cost Estimating Handbook

Installation studies are not generally applicable to relatively small hardware systems, even though the hardware system may indeed have to undergo an actual installation procedure to have it perform its intended function. The costs associated with these activities are accounted for during the System Integration phase of cost analysis. Installation is a study that covers the costs of placing this type of equipment at its work site. Installation studies are performed on an Item-by-Item basis to determine each pertinent Item's installation costs. Installation costs, therefore, are not computed on a total system or project basis. However, after completing an Item's Installation study, the calculated costs may be accumulated into the project's total costs. Normally, the Installation study should follow an Item's Acquisition study. Installation efforts may include production and/or engineering phases, and their costs may be processed in either a total or a detail manner.

Calibration Typically, when a costing study is performed on some new unit of equipment, there are various factors which may not be known about the item, such as how much one unit will weigh after it is produced. The model provides the capability of performing a type of study called a Calibration, which is a costing calculation in reverse, used to generate reference factors. When weight and its complexity factor, traditional key factors in parametric cost calculations, are absent, the model will generate a calculated or "fiducial" weight along with a corresponding weight factor if input costs are given. In performing calibrations for an Item, the analyst supplies as much information in parametric form as is known about the item, and enter the previously known costs of a reasonable, similar item. Then, after the model performs its calculations for the set of calibration data, a calculated weight as well as other calculated reference factor values, including the complexity factor, will be displayed. Those calculated values can be retained, and used as input values when deriving the estimated costs of any new unit of equipment. It may be desirable to perform an iterative series of Calibration and Forward Costing exercises for a particular unit of equipment until reference factor values are derived that seem most reasonable for a given project. The processes of Calibration and Forward Costing can be applied to any or all of program phases: Acquisition, Life Cycle, System Integration and Installation. 90

Parametric Cost Estimating Handbook

Price Parametric Models PRICE models use the parametric approach to cost estimating. Parametric cost modeling is based on cost estimating relationships (CERs) that make use of product characteristics (such as hardware weight and complexity) to estimate costs and schedules. The CERs in the PRICE models have been determined by statistically analyzing thousands of completed projects where the product characteristics and project costs and schedules are known. These projects encompass a wide range of products and companies. The PRICE models contain many hundreds of CERs that work together to compute the costs and schedules for the development and production of hardware and software products. An example of parametric modeling is the technique used for estimating the cost of a house. The actual cost of building a house is, of course, the total cost of the materials and labor. However, defining the required materials and labor for developing this cost estimate are time consuming and expensive. So, a parametric model that considers the characteristics of the house is used to estimate the cost quickly. The characteristics are defined quantitatively (floor area, number of rooms, etc.) and qualitatively (style of architecture, location, etc.). This parametric technique for estimating the cost of building a house is widely used in the insurance business and for taxation assessment. Just as with the building example, determining and costing the material and labor requirements for hardware and software projects are also time-consuming and labor-intensive, and since the product characteristics can be defined long before the material and labor requirements, parametric models permit estimates to be obtained earlier than with other techniques. Early management and engineering decisions can be based on these "up front" estimates. Many business decisions must be made before there is sufficient product definition to develop a conventional estimate.

Using parametrics makes it practical to explore the cost risks associated with

uncertainties in the project, and to get answers to "what if" questions such as: "What if we shorten the development cycle?" "What if we build fewer prototypes?" Predevelopment Cost and Schedule Estimates -- Cost and schedule estimates can be obtained to develop and manufacture a product that is still only a concept. The PRICE models accomplish this by using product characteristics to develop the estimate. PRICE does not require labor hours and a bill of material.

91

Parametric Cost Estimating Handbook

This early-on estimating capability makes PRICE a tool for design engineers, since much of a product's cost is locked in by early engineering decisions. PRICE can provide engineers with cost information needed to develop minimum-cost designs.

Descriptions Of The PRICE Models PRICE H PRICE H is used to estimate the cost of developing and producing hardware. Most manufactured items and assemblies can be estimated using PRICE H. PRICE H uses product characteristics to develop the cost estimate. This makes the model a good tool to use at the product concept stage, when there is insufficient definition to quantify the product labor and material required for a conventional estimate. Key inputs to the PRICE H Model are:

*

Weight -- tells the model the size of the product being estimated.

*

Manufacturing Complexity -- a coded value that characterizes product and process technologies and (optionally) the past performance of the organization.

*

Platform -- a coded value that characterizes the quality, specification level, and reliability requirements of the product application.

*

Quantities -- the number of prototypes and production items to be estimated.

*

Schedule -- the dates for the start and completion of the development and production phases may be specified. The model will compute any dates that are not specified. Only the date for the start of development is required.

*

Development Costs -- effort associated with drafting, design engineering, systems engineering, project management, data, prototype manufacturing, prototype tooling, and test equipment.

*

Production Costs -- effort associated with drafting, design engineering, project management, data, production tooling, manufacturing, and test equipment.

*

All costs are reported at the material, labor, overhead, and dollar level.

It is generally acknowledged that most of the cost of a product is locked in by the early engineering design decisions. This means that the early decision process should consider cost as well as technical alternatives. The lack of product definition early in the concept stage precludes a credible conventional cost estimate. With PRICE H, however, engineers and managers are able to develop cost estimates for each alternative to select minimum cost-product designs. Before any 92

Parametric Cost Estimating Handbook

investment has been made in design engineering, PRICE H can compute the unit production cost of a product. PRICE H can be quickly iterated to obtain estimates under varying assumptions until the value selected for the estimate represents acceptable risk. With PRICE H, cost sensitivity to project uncertainties can be explored, and management's "what if" questions can be answered.

PRICE HL The PRICE HL (Hardware Life Cycle) Model calculates costs, cost effectiveness, and operational availability at the system, subsystem, assembly, and sub-assembly levels. It is used in all phases of the acquisition process and is especially well suited when linked with PRICE H for high-level analyses in the concept and system design phases. This coupling provides a system level trade-off capability to effectively determine hardware life cycle costs during early stages of hardware designs. PRICE HL provides cost outputs for all phases of the hardware life cycle.

PRICE M The PRICE M (Electronic Module and Microcircuit) Model produces estimates of development and production costs for electronic modules and application specific integrated circuits (ASICs). It uses cost estimating relationships based on the number of transistors and/or gates, percentage of new circuit cells and design repeat, specification level, degree of computer-aided design, product familiarity, engineering experience, and calibration factors to estimate ASIC specification and design costs.

Additional relationships provide systems

engineering, project management, and data costs. PRICE M contains a database for frequently used components that can contain predefined costs or that can be calculated from the component input specification.

SEER The SEER family of models include the basic hardware cost estimation model (SEER-H) and a hardware life cycle cost model. The hardware cost model estimates hardware cost and schedules and includes a tool for risk analysis. The hardware model is sensitive to differences in hardware technologies ASIC, MCMS, exotic materials, miniaturization, etc., and to different acquisition scenarios; make, modify, customer-furnished, purchased, off-the-shelf, etc. It is also

93

Parametric Cost Estimating Handbook

sensitive to differences in electronic versus mechanical parameters and makes estimates based on each hardware item's unique design characteristics. The SEER hardware life cycle model (SEER-HLC) evaluates the life cycle cost effects due to variations in: reliability, mean time to repair and repair turnaround times.

SEER-HLC

complements SEER-H, and both models will run on a personal computer. The models are based on actual data, utilize user-friendly graphical interfaces, and possess built-in knowledge bases and databases that allow for estimates from minimal inputs. SEER-IC is an expert system for planning integrated circuit development and production. SEER-IC provides a systematic, expert approach to cost and resource estimation and a capability for Design to Cost (DTC) tradeoffs and risk analysis.

SEER-IC includes an industry-wide

knowledge base. SEER-DFM (Design for Manufacturability) is a tool designed to assist the engineer produce and assemble products with efficiency, in a manner designed to exploit the best practices of an organization. Diverse team members have an idea of what their true interests are and of how they can work together and input to the model accordingly. The fundamental goal of DFM is to make engineering decisions that optimize the manufacturing process. Two fundamental analysis steps are taken in a DFM regime: the gross and detailed tradeoff analysis. *

Gross analysis involves product design decisions, and also fundamental process and tooling decisions. Factors that influence gross analysis include the quantity of the planned product, the rate at which it will need to be produced, and the investment required. There are also machinery, assembly and setup costs to contend with.

*

Detailed analysis takes place once many of the primary production parameters, such as design and basic processes, have been fixed. Factors that can be adjusted for and balanced at the detailed level include tolerances, the proportion of surface finishes, secondary machining options (drilling, reaming, polishing), and the age and degree of process intervention (how soon is a machine recalibrated or rebalanced?).

SEER-DFM explores the DFM effort by developing engineering tradeoffs that are involved in a DFM analysis and presents the results of each as costs. SEER-DFM models the categories of materials and processes used in modern production lines. SEER-DFM integrates the following models: 94

Parametric Cost Estimating Handbook

*

machining (turning, boring milling, shaping, chemical-milling, grinding... ): The model explores the tradeoffs from starting with raw stock vs. sand or investment-casting, etc. The material may also be varied. Tooling, setup and other costs hinge on these choices.

*

sheet metal fabrication (presses, shears, die press).

*

mechanical assembly (spot welding, bolting, bracing).

*

electrical assembly (PC board assembly, parts preparation, soldering & wave soldering, fasteners).

*

injection molding: parameters include weight, cycle time, and cavities in the mold.

*

SEER-DFM performs the analysis from standard work measurements (standard times).

REGRESSION-BASED PRODUCT SPECIFIC COST MODELS

In many cases regression-analysis based cost models are generated by government agencies (or program offices), government support contractors, or hardware/software contractors. In such cases, a sufficient quantity of cost data is usually available to build statistical, i.e., regression based CERS. The flow diagram in Figure IV-3 shows the most commonly used process.

95

Parametric Cost Estimating Handbook

The point of departure is a Work Breakdown Structure (WBS) or a Cost Breakdown Structure (CBS).

There are many WBSs in existence, such as the standard WBS by DoD

(MIL-STD-881B) or contract specific WBS's (See Appendix B). Often, a contract specific WBS is 96

Parametric Cost Estimating Handbook

broken into work packages which are not very suitable for determining component or subsystem costs. Considerable effort is usually required to extract cost drivers from the cost of WBS elements which are meaningful and "uncontaminated" by activities which are applicable to more than one WBS element. Preliminary CERs are conceptualized, and both cost and technical data are collected according to the WBS elements and CER format. The cost data are then normalized with respect to base year constant dollars, quantity of hardware, completeness of cost inclusions, etc., and statistical regressions are performed to correlate cost (dollars or hours) to cost drivers.

Mathematical

equations for the CERs resulting from the regression analyses are first checked for their statistical validity, and next for technical credibility. An extensive review cycle is then entered, complete with Beta testing, checking by technical and cost experts within the government and contractor communities, and modification of the CERs based on revised information or additional data sources. The final step of this process consists of a significant amount of documentation of the CER equations, description of included and excluded costs, data sources, ground rules, rationale, limits, uncertainties and applicable comments. This cost model process leads to parametric CERs which can be used by the government and by contractors.

NON-COMMERCIAL COST MODELS

The Government often sponsors the Development of Parametric Cost Models (e.g., Unmanned Space Craft Model Version 7 by Tecolote Research. And the Missile Analysis and Display Model by EER Systems Corporation which was sponsored by the USA Missile Command). These Cost Models are usually based on historical data from a number of systems and are used by Government and Contractor Cost Analysts alike to develop cost estimates. They are often used as secondary estimating techniques and as crosschecks to the primary estimating methodology.

NON-STATISTICAL COST MODELS

Non-statistical cost models are those which are based on other types of mathematical analysis, and are generally engineering models. In many cases, insufficient data exist to construct a meaningful, regression based CER or model. For example, if only two cost data points are present 97

Parametric Cost Estimating Handbook

for an item (a hardware item like a transmitter, for instance) an approximation of a CER would consist of fitting a power curve with an exceptable exponent through the two data points. The exponent should make sense in light of the analyst's experience. Non-statistical cost models are generally more judgmental than those based on regression and additional actual data. Uncertainties associated with a small number of actual data points could easily lead to large inaccuracies of the cost model. However, a larger amount of data on which a regression-based cost model is based does not necessarily improve its accuracy. It depends on the quality of the majority of the data points (quantity does not always equate to quality). The use of non-statistical cost models puts additional emphasis on the competence of the model originator and will generally increase the skepticism of an auditor. Non-statistical cost models for proposals should generally be used under the following conditions: (1)

An insufficient (to produce statistically based CERs and models) amount of historical data exists;

(2)

The existing database is really not applicable (i.e., new technology or new products are being estimated);

(3)

The cost modeler has sufficient expertise, knowledge and competence to construct the model (e.g., with respect to choice of variables);

(4)

The model can be validated for at least one, preferably more than one, case; and,

(5)

Excellent cost and technical visibility exists for the few data points which form the basis of the model.

The auditor or reviewer needs to scrutinize the cost model with respect to the above mentioned five criteria, and request sufficiently reasonable explanations with respect to the derivation of the cost model. If necessary, the auditor may also require an explanation why the particular set of variables are being used in the model. Non-statistical cost models may not always have weight and complexity as independent variables. Size may be implied by amount of energy, thrust level, tank volume, or parts count. Complexity may be implied by temperature, efficiency, thermodynamic cycle type, or manufacturing processes and packaging density. In many cases, non-statistical cost models can be constructed by equating cost to these directly measurable parameters. Care has to be taken to equate cost to independent technical parameters.

98

Parametric Cost Estimating Handbook

In general, non-statistical cost models are used at a top level, i.e., are often not divisible into the typical WBS labor and material cost components. However, they can also cover simple cost-to-cost ratios such as quality control labor to manufacturing touch labor hours with only one or two data points. In this case, it is up to the contractor to convince the auditor that the historically developed relationship (e.g., labor ratio) for one product will also hold for another (hopefully similar) product.

MODEL CALIBRATION

The act of calibration standardizes a model. Many models are developed for specific situations and are, by definition, calibrated to that situation. Such models usually are not useful outside of their particular environment.

However, general cost estimating models including

commercially available ones such as the FAST, PRICE and SEER Models (described earlier) are designed to be useful as estimating tools for a wide range of situations. The act of calibration is needed to increase the accuracy of one of these general models by making it temporarily a specific model for whatever product it has been calibrated for. Calibration is in a sense customizing a generic model. Items which can be calibrated in a model are: product types, operating environments, labor rates and factors, various relationships between functional cost items, and even the method of accounting used by a contractor. All general models should be standardized (i.e. calibrated), unless used by an experienced modeler with the appropriate education, skills and tools, and experience in the technology being modeled. Calibration is the process of determining if there is any deviation from a standard in order to compute correction factors. For cost estimating models, the standard is considered historical actual costs. The calibration procedure is theoretically very simple. It is simply running the model with normal inputs against items for which the actual costs are known. These estimates are then compared with the actual costs and the average deviation becomes the correction factor for the model. The actual data used for the calibration runs determines what type of calibration is done. In essence, the calibration factor obtained is really good only for the type of inputs that were used in the calibration runs. For a general total model calibration, a wide range of components with actual costs need to be used. Better yet, numerous calibrations should be performed with different types of components in order to obtain a set of calibration factors for the various possible expected 99

Parametric Cost Estimating Handbook

estimating situations. For instance, the PRICE Hardware model addresses this situation using internal complexity factors. These can be modified by calibration to the component level. Board fabrication, parts costs, packaging costs, and the overall design and manufacturing costs have factors that can be tuned to a specific product, product line, or business.

COST MODEL AUDIT AND ANALYSIS

Cost models are significantly more complex subjects than simple CERs. Typically a cost model uses both CERs and mathematical relationships between the CERs. Both cost models and CERs are based on historical data. The analyst reviewing a cost model must review the model, the basis history and the specific application. In addition, models must be evaluated heuristically by test cases and sensitivity analyses. If a product specific model is used in an estimate, the analyst needs access to more information and be able to explore the model's internal relationships and data base. The disadvantage of generic proprietary cost models is that the cost analyst sometimes does not have access to the data base and the CERs embedded in the model's "black boxes."

Analyzing A Product Specific Cost Model A non-standard cost model is usually built by the estimator for the specific purpose of calculating specific product costs. As such, it will have specific company factors, CERs, and relationships. It is not uncommon to have a pricing model as a back end to a proprietary model for purposes of adding pricing factors and to do output formatting. In addition, it is common to have a front end to the product specific to deal with purchased items, ancillary costs and sub-models. The firm submitting an estimate may use a spread sheet which are utilized to adjust one model output or format. See Figure IV-4.

Heuristics

Company Software Model

Specific Factors

Hardware & Project Model 100

Parametric Cost Estimating Handbook

Pricing Factors and Formats

FIGURE IV - 4

When considering calibration, from an auditor's standpoint, basically three things need to be known about an estimate that is prepared using a model. There are: (1) The skill level of the model builder. (2) If the model was calibrated, what kind of data was used to do the calibration? Was it an overall general calibration using several data points or a more specific calibration with actual cost data that represents the project being modeled? A calibration will have more credibility if it is performed with data similar to the project being estimated. (3) The third consideration has to do with the quality of the actual cost and non-cost data that went into the calibration. Historical actual costs are not necessarily "real" costs as far as a calibration activity is concerned (See Chapter II: Normalization of Data). Other sources such as actuals from a database, vendor firm quotes, vendor ROMS, management estimates to completion, industry published costs, and even estimates from other products or other organizations sometimes can be used as inputs for calibration. The appropriateness of such inputs need to be considered. Knowing what went into the calibration (for a specific model) and how it relates to the model's current estimate needs to be taken into consideration when evaluating the appropriateness and quality of the calibration.

When a product specific cost model is used to perform the actual cost estimate, the analyst has to evaluate the model much the same as for single CERs. In addition, however, the relationship between the cost elements has to be explored. Are the cost drivers generating only one set of costs or are the various drivers producing overlapping costs - for example, if a factor is calculating both program management and data, does the program management number exclude data costs? If product specific models are used, the basis cost history is important for checking the calibration of the model. It also has to be examined to see if the source information is clean and well defined. 101

Parametric Cost Estimating Handbook

Regardless of the model type, it is desirable for the model to be calibrated for the specific situation at hand. For proprietary product specific models, the calibration has to include company specific calibration in order to account for company peculiarities, accounting methods and rates. Once the general features of the company are incorporated into the model, the auditor has to check the basis of estimate (sometimes called complexity) factors. Since models usually broaden their effective databases by incorporating functional relationships within parameters, the historical basis data utilized by the model has to be examined for both cost validity and the mathematical relationships between the technical parameters and cost. This step is important since the system to be estimated rarely is exactly like (technically or functionally) the historical case and some normalization is usually required in the value of the independent variables. The analyst must see the documentation associated with the basis case both for the costs and the technical features. In other words, the basis case must show the source data for the estimated engineering, data, prototype costs, etc.

The basis case documentation must also show the technical data - weight, size

complexity etc. Again, if the cost elements are significant, considerable detail may be necessary to verify the basis calibration. For instance, if Printed Wiring Assembly (PWA) costs are a significant driver for the "black box" being estimated, the basis costs and technical descriptions must be available in some detail perhaps down to the board and chip count level. Once the auditor has verified the model validity both for the corporation and for the specific basis case, he or she can then address the application of the model to the case at hand. This involves examining the tasks being estimated and the technical features of those tasks.

In

development cost estimates, the technical detail may be thin, consisting of simply performance requirements. In this case, the basis system performance characteristics become significant. For production estimates, hardware item drawings should exist and can be compared directly to the drawings of the basis case - which might even be a previous version of the item being estimated. In all cases, any extrapolation must be supported by quality information.

Vague references of

"engineering judgment" are not satisfactory; this judgment has to be translated into concrete hardware definitions.

Summary Of Cost Model Audit And Analysis In summary, the analyst reviewing a cost model has to work through several steps in order to evaluate the model-based estimate. These are:

102

Parametric Cost Estimating Handbook

*

Is the model proprietary or non-standard? Does the overall estimate contain interface models or algorithms which transfer data into and between other models?

*

If non-standard, are the model interrelationships well described and well defined, and do they make logical sense?

*

Do the model elements deal with all the costs in the estimate? If not, is there any overlap between the costs derived through the model and the costs developed outside the model?

*

Is the model used without adjustment? If adjusted, what is the justification and source data for the adjustments?

*

Is the basis case used for reference in the estimate a close analog to the case being estimated? If not, how far is the basis case, functionally and physically, from the case to be estimated? What are the functional relationships within the model parameters, and how sensitive is cost to error in specifications? What is the source documentation for the base case?

*

What are the values of the elements in the case to be estimated? How do they compare to the values in the basis case? What are the source data for the case to be estimated?

Given that not all these questions have unequivocal answers, the auditor can examine how variations in the parameter values affect the estimate. If the model is computerized, the auditor can even do "what if" tests to see if some particular cost relationship is especially important. For those elements which show significance and which are unsatisfactorily supported, the auditor can request further information and data. The objective of the analysis is to see whether the estimate is fair and reasonable. In complicated logical structures, this issue of overall reasonableness may evolve as a result of examination of the elements of the estimate.

In addition, softness in data, vagueness in

specification and bias in projection can elevate uncertainty in the cost estimate to the point that this uncertainty becomes a reasonableness issue. In short, the cost analyst evaluating a cost model has to assure him/herself that the model is a good simulation of the true cost influences, that the basis of the estimate is appropriate and well founded, and that the extension of the basis through the model is a true picture of what is proposed in the technical description of the system. If the answer to all these questions is “yes” and if the uncertainty associated with the estimate is reasonable, the analyst is looking at a good cost model and estimate. 103

Parametric Cost Estimating Handbook

Appendix E lists other examples of hardware models and describes in some detail a few of them.

104

Parametric Cost Analysis Handbook

CHAPTER V

SOFTWARE PARAMETRIC COST ESTIMATING

105

Parametric Cost Analysis Handbook

CHAPTER V SOFTWARE PARAMETRIC COST ESTIMATING

SOFTWARE DEFINITION

Software, in general, is a set of programs and accompanying documentation that direct computers to perform desired functions.

In simple terms, a software program is a set of

instructions for a computer. Software is critical, not only in the control of space based systems, but in virtually all current military systems. For example, software controls aircraft engines, drivers and simulators; it directs surveillance systems, handles space shuttle operations, and controls account, inventory and Management Information Systems (MIS). There are three basic types of software. These are: *

System software (also known as the operating system) is a collection of programs that manages all the concurrent tasks being performed by a computer, including the execution of application software programs.

*

Utility software is a set of programs that perform routine day-to-day tasks, such as listing or compressing data, copying files, etc.

*

Application software performs specialized functions like Space Shuttle control and the MIS functions mentioned above, or other useful work not related directly to the operation of the computer itself.

Most analysts are familiar with all three types of software as all are used in personal computers. MS-DOS (Micro Soft Disk Operating System), for example, is a trade name for the operating system for Intel X86 Personal Computers. LOTUS 123 and Micro Soft Excel are examples of application based software. Listing a directory of user files is a utility software function.

The system software procured by the US Government is usually a complex

combination of all three types of software. THE IMPORTANCE OF SOFTWARE TODAY

106

Parametric Cost Analysis Handbook

As discussed earlier, maintenance problems are a driving force behind re-engineering, but a new business strategy now challenges software organizations. It is called business process re-engineering or just business re-engineering. The reader should never confuse business reengineering with software re-engineering. We have included this section to discuss how software re-engineering is being used to support business re-engineering. Business re-engineering is the fundamental rethinking and radical redesign of business processes to achieve dramatic improvements in measures of performance, such as cost, quality, service, and speed. Traditionally, organizations tend to be structured around divisions such as manufacturing, marketing, administrative support, etc.

But this structure is often a barrier to quick

responsiveness to a rapidly changing environment. When confronted with a new objective, client, strategy, etc., every division must be mobilized from the top down. So how can software re-engineering help? Software was originally developed to support business functions within the traditional organizational structure. Thus, we are left with legacy systems that are marketing-oriented, or manufacturing-oriented, etc. Software re-engineering captures the design behind the software. Using new tools and techniques currently available, this design-information can be broken up into “chucks” that are functionally integrated. These “chunks” are then analyzed and regrouped around the newly identified business core processes. In many ways, this seems a lot like the process of translating from structured analysis to object-oriented analysis. Some advocates of business re-engineering admit this is the case. Both involve changing the basic approach to software and business. Software (and organizations) should correspond to a meta-model of the real-world. In the past, we attempted to force our software designs and organizations to conform to a structure that was basically incompatible with the real-world. Users expressed their real-world requirements in terms of familiar objects (e.g. order forms , personnel records, etc.) which were translated into process oriented programs. Maintenance also required a similar translation from the user’s modified needs to the processoriented representation (i.e. program). Now, with object-oriented analysis and business re-

107

Parametric Cost Analysis Handbook

engineering, we are beginning to realign our software and organizational structure so that they correspond to real world objectives and needs. Another area where software re-engineering can aid business re-engineering is the identification of business rules from legacy course code. Embedded within legacy systems are the implied rules that govern how the organization was run at the time the system was developed. By extracting these business rules, an organization can better understand and, later, codify the rules by which they conduct business. Improving software quality and productivity is a major challenge faced by the Department of Defense (DoD) and the non-military community (DOE, NASA) as well.

In addition,

providing affordable weapon system flexibility through software is a specific challenge for the DoD. Improving software quality is basic to how well software meets the requirements and expectations of the users. It also means ensuring that software is adequate, reliable, and efficient. Improving productivity means favorably increasing the ratio between the resources required to develop software and the size and complexity of the developed software. The growth in computer use and computer hardware capabilities has placed demands of increasing magnitude and complexity on computer software. Software development processes, along with attendant methodologies, which may have worked well in the past, often break down when applied to the development of today’s software. For example, studies show that every five years the sizes of software projects, as measured by Source Lines Of Code (SLOC), increase by an order of magnitude, and that the scaling of the development effort, demanded by the order-ofmagnitude increases now require fundamental changes in the development process. As the sizes of software projects have increased, software development processes based on individual programmers have given way to processes based on small teams and, in turn, small teams have given way to larger teams and so on. Higher scaling of software development processes by merely increasing team sizes reaches limits on effective project management and resource availability. Today’s users of software demand software applications of greater size and complexity than before. Advances in computer hardware capabilities are more than adequate to match the demands of users; however, software as it is developed using prevailing processes and

108

Parametric Cost Analysis Handbook

methodologies is not. The challenge is finding software development processes with attendant methodologies and technologies that meet user demands and that improve software quality and productivity. As a percent of total cost, software cost has grown disproportionally more than hardware cost (see Figure V-1).

FIGURE V-1 HARDWARE/SOFTWARE COST TRENDS

The military, like the business world today, sees software providing the versatility and leverage to achieve performance goals. For example, software demonstrated its flexibility to quickly change weapon system capabilities in Desert Storm, the most newsworthy being the rapid development of a new software package for the Patriot Missile system to counter the SCUD. Because of the versatility and leverage provided by software, the DoD’s appetite for software in the future has been described as “virtually insatiable.” Software is an increasingly important element in military systems of all kinds. The capabilities of current and future military systems are dependent on the performance of the systems’ software. As a system is upgraded or improved, much of the additional capability is

109

Parametric Cost Analysis Handbook

achieved through new software.

In fact, many of the functions essential to mission or

organizational success are partly or completely accomplished through the use of software. Unfortunately, software development and maintenance is an error-prone, time-consuming and complex activity. Experience has revealed that many software development efforts falter because the management of these projects fall into several common traps. These problems are: * Lack of adequate requirements/specifications. * Lack of attention to user needs and constraints. * Lack of visibility into the development process. * Lack of control at key points during development. * Lack of standardization. * Lack of attention to cost of ownership considerations. * Lack of adequate documentation. * Lack of adequate training and skills in estimating effort and schedule.

Falling into these traps leads directly to increased development and support costs, schedule slips, and reduced systems capability. DOD-STD-2167A/498, Defense System Software Development, established uniform requirements for software development and is widely applied in all software application areas by all DOD components.

DOD-STD-7935A/498, DOD Automated Information System (AIS)

Documentation Standards provides guidelines for the development and revision of documentation for information systems.

For embedded systems, DOD-STD-2167A/498 is

appropriate. For information systems, both DOD-STD-2167A/498 and DOD-STD-7935A are appropriate. The documentation requirements of DOD-STD-7935A take precedence over those of DOD-STD-2167A/498. In both application areas, tailoring of these standards must be done to reflect the unique needs and constraints of each project. The actual software development tasks will be accomplished by a development organization that is an element of, or subcontractor to, the system development contractor. On occasion, this “contractor” may be a DOD agency.

The development contractor has the

responsibility for delivering a software product that meets all contractual requirements. Unfortunately, at the beginning of a contract’s development efforts, it us usually not possible to

110

Parametric Cost Analysis Handbook

specify precisely and completely all of the characteristics of the final software products and their development processes.

Experience has shown that the difference between successful and

unsuccessful development efforts is the vigor and timeliness of the direction given to the contractor by the Program Manager, supported by the Project Office. Figure V-2 indicates the relative impact of the penalty (cost) for delayed error correction during a software project’s life cycle, from almost no cost during preliminary design to high cost impacts during validation and operative stages. BPR focuses on the later stages (maintenance) and the reduction of errors at those times by re-engineering the early phases. In today’s world of shrinking budgets, providing affordable, flexible software systems requires cost control and predictability that are not found in the traditional software development processes. Increasingly, the DoD demands that software be developed within predictable costs and schedule. Enter, therefore, parametric cost modeling.

FIGURE V-2 RELATIVE PENALTY-ERROR CORRECTION

111

Parametric Cost Analysis Handbook

THE SOFTWARE DEVELOPMENT PROCESS

This section defines the basic process of software development as currently practiced by the majority of government contractors. Major phases and software development activities are defined, as well as key milestones for the measurement of progress.

The Waterfall Model DOD-STD-2167A/498, the current prevailing standard guiding software development, has been interpreted as mandating as specific process for use on all military acquisitions. This process is represented by the “Waterfall” Model, which serves as the conceptual guideline for almost all Air Force and NASA software development. The process described by the model involves development through specific, sequential stages. There are specific objectives to be accomplished in each stage; each activity must be deemed successful for work to proceed to the subsequent phase. The process is usually considered non-iterative. Each phase requires the delivery of particular documentation products (Contract Data Requirements List - CDRL Items). Many of the phases require successful completion of a government review process. Critics of the “Waterfall” Model, in fact, find that the model is geared to recognize documents as a measure of progress rather than actual results. The eight major activities described in 2167A/498 are as follows: * Systems Concept/System Requirements Analysis * Software Requirements Analysis * Software Parametric Cost Estimating * Preliminary Design * Detailed Design * Coding and Computer Software unit (CSU) Testing * Computer Software Component (CSC) Integration and Testing * Computer Software Configuration Item (CSCI) Testing * System Integration and Operational Testing

112

Parametric Cost Analysis Handbook

A schematic overview of the Waterfall Model, representing concurrent hardware and software development, is presented in 2167A/498, and reproduced below as Figure V-3.

FIGURE V-3 WATERFALL MODEL

113

Parametric Cost Analysis Handbook

An alternative approach to software development involves the use of incremental builds. With this approach, software development begins with the design of certain core functions to meet critical requirements, and each successive software “build” provides additional functions or enhances performance. Once system requirements are defined and preliminary system design is complete, each build may follow the waterfall pattern for subsequent development phases. Each successive build will usually have to be integrated with prior builds.

THE SOFTWARE COST ESTIMATING PROCESS

The overall process of developing a cost estimate for software is not different from the process for estimating any other element of cost. There are, however, aspects of the process that are peculiar to software estimating. Some of the unique aspects of software estimating are driven by the nature of software as a product. Other problems are created by the nature of the estimating methodologies. Brooks, in his 1982 collection of essays, referred to large system software programming as a “tar pit.” His description of one such project is typical of Space and Missiles Center and NASA experience with software development. He states:

“The product was late, it took more memory than planned, the costs were several times the estimate, and it did not perform very well until several releases after the first.”

Why is it so difficult to estimate the cost of software development?

Many of the

problems that plague the development effort itself are responsible for the difficulty encountered in estimating that effort. One of the first steps in any estimate is to understand and define the system to be estimated.

Software, however, is intangible, invisible, and intractable.

It is

inherently more difficult to understand and estimate a product or process that cannot be seen and touched.

Software grows and changes as it is written.

When hardware design has been

inadequate, or when hardware fails to perform as expected, the “solution” is often attempted through changes to the software. This change may occur late in the development process, and sometimes results in unanticipated software growth. In this case it is most important to create a picture, since the product can be highly ambiguous at this time.

114

Parametric Cost Analysis Handbook

The software WBS (see Appendix B) is an excellent tool for visualizing the software product. The WBS need not be complex, nor does it need to be highly detailed. A simple product tree line drawing is often adequate for initial software estimates. The hardware WBS can be a useful tool in developing the initial WBS for software. There is usually a software Computer Software Configuration Item (CSCI) or similar module associated with each hardware Line Replaceable Unit (LRU). As the program evolves, the initial or draft WBS should include all software associated with the program regardless of whether it is developed, furnished, or purchased. If furnished or purchased software were omitted, it would not be possible to capture the cost of integrating preexisting or purchased software with the development software. The WBS should depict only major software functions, and major subdivisions. It should not attempt to relate the software to the hardware it controls. Each of the major software functional units can be modeled as a Computer Software Configuration Item (CSCI). Lower level WBS elements can be modeled as a component. Once the WBS is established the next step is to determine which estimating technique should be used for deriving the estimate. One of the most challenging tasks in project management is accurately estimating needed resources and required schedules for software development projects. The software estimation process includes estimating the size of the software product to be produced, determining which functions the software product must perform, estimating the effort required, developing preliminary project schedules, and finally, estimating overall cost of the project. Size and number of functions performed are considered major productivity (“complexity”) factors during the software development process. Effort is divided into labor categories and multiplied by labor rates to determine overall costs.

Therefore, software

estimation is sometimes referred to as software cost estimation. Software life cycle models identify various phases and associated activities required to develop and maintain software, and provide excellent input into the software estimation process. Some of the more common and accepted life cycle models include: (1) Waterfall Model; (2) Rapid Prototyping; (3) Incremental Development Model; (4) Spiral Development Model; (5) Reusable Software Model; and (6) the Transform Model [Boehm and Davis]. These models form a baseline from which to begin the software estimation process and should be reviewed and tailored to the proposed project.

115

Parametric Cost Analysis Handbook

Software identified as mission-critical and developed for the United States government usually must be developed in accordance with DOD-STD-2167A/498. This standard establishes uniform requirements for software development, and does not specify or discuss any particular method of software development. However, it requires the inclusion and documentation of the major software development life cycle activities. The standard also requires that reviews and audits be held in accordance with MIL-STD-1521B, “Technical Reviews and Audits for Systems, Equipment, and Computer Programs.” Additionally, structured approaches to sub-task identification are extremely beneficial in determining tasks and the required effort for each task. The WBS is a method which strongly support this process. The software estimation activity should be approached as a major task and therefore should be well planned, reviewed and continually updated.

The basic steps required to

accomplish software estimation are described in the following paragraphs.

Define Project Objectives and Requirements The objectives and requirements of a software project are usually established by upper management directive or by a contract Statement Of Work (SOW). A clear set of objectives must be established in order to accurately identify project requirements. Project requirements must also include specifications that must be met and applicable standards that must be followed. Project objectives and requirements must be defined as clearly and precisely as possible in order to accomplish the project correctly, as well as identify tasks and ultimately estimate costs as accurately and early as possible.

Plan The Activities As previously mentioned, the software estimation activity should be planned as a major task.

The plan should detail the purpose, products, schedules, responsibilities, procedures,

required resources, and assumptions made.

The plan should include which estimation

methodologies, techniques, and tools will be used.

116

Parametric Cost Analysis Handbook

The project should be organized into a hierarchical set of tasks to the lowest level of detail that available information will allow.

Additionally, a breakdown of documentation

requirements and associated tasks should be defined (the detailed WBS). The WBS helps establish a hierarchical view and organization of the project. The top level is the software system or final software product, and subsequent levels help identify tasks and associated sub-tasks and are used to define and encapsulate system functions. Each of these tasks are divided into software development phases such as design, code and test, and integration. All activities associated with each level must be defined including: project planning and control, configuration management, product assurance and documentation. In addition to early development of detailed knowledge about the project, the WBS provides an excellent methodology for project data collection, tracking, and reporting. During development of the project, each of the WBS tasks can be given a project budget, and a job number which is used for reporting time spent on each project phase or activity. This provides an excellent project tracking and history data collection method. Most government contracts require that such a Cost/Schedule Control System Criteria (C/SCSC) be established. When the data are collected to an established WBS, the information can be placed in a database to be used in refining, tailoring, and customizing the software estimation process.

This information

becomes an invaluable input to the software estimation process for future projects. Software project tasks/subtasks should be defined to the smallest component possible. All technical aspects of the project should be understood as fully as possible since the more details known about the project the more accurate the estimates will be. Newer methodologies are evolving which aid software developers in identifying various functions needed to support the project, such as Object-Oriented Analysis and Design (OOA, OOD). Therefore, organizations should actively keep abreast of evolving technologies.

Software Estimation Risks The effects of inaccurate software estimation and schedule overruns are well known. The problem stems from an inability to accurately assess risks associated with various software development projects. Software estimation errors generally result from four major risk areas, which are:

117

Parametric Cost Analysis Handbook

(1)

The inability to accurately size the software project.

This results in poor

implementations, emergency staffing, and cost overruns caused by underestimating project needs. (2)

The inability to accurately specify a development environment which reflects reality.

This results in defining cost drivers which may be inappropriate,

underestimated or overestimated. (3)

The improper assessment of staff skills. This results in misalignment of skills to tasks and ultimately miscalculations of schedules and level of effort required, as well as either underestimating or overestimating project staffing requirements.

(4)

The lack of well defined objectives, requirements, and specifications, or unconstrained requirements growth during the software development life cycle. This results in forever changing project goals, frustration, customer dissatisfaction, and ultimately, cost overruns.

All potential risks associated with the proposed software development project should be defined and weighed, and impacts to project cost should be determined. This information should always be included in the software estimation process.

Estimation Methodologies Several methods (if possible) should be used during the software estimation process. No one methodology is necessarily better than the other, in fact, their strengths and weaknesses are often complimentary to each other. It is recommended that more than one software estimation methodology be used for comparison and verification purposes. One method may overlook system level activities such as integration, while another method may have included this, but overlooked some key post-processing components.

Five of the methods discussed in Dr.

Boehm’s book Software Engineering Economics are: analogy, bottom-up, top-down, expert judgment, and algorithms (parametrics). These methods are often used in conjunction with each other and have been used for many years by managers of software projects without the use of any formal software estimation

118

Parametric Cost Analysis Handbook

tools. Software estimation tools have only recently been developed which incorporate these methods, and many incorporate multiple methodologies.

Analogy Method Estimating by analogy means comparing the proposed project to previously completed similar projects where project development information is known. Actual data from the completed projects are extrapolated to estimate the proposed project. Estimating by analogy can be done either at the system level or the component level. The main strength of this method is that the estimates are based on actual project data and past experience. Differences between completed projects and the proposed project can be identified and impacts estimated. One problem with this method is in identifying those differences. This method is also limited because similar projects may not exist, or the accuracy of available historical data is suspect. Also, many projects for DOD weapon systems may not have historical precedents. The analogy or comparative technique uses parametric approaches including the use of CER's.

Bottom-Up Method Bottom-up estimation involves identifying and estimating each individual component separately, then combining the results to produce an estimate of the entire project. It is often difficult to perform a bottom-up estimate early in the life cycle process because the necessary information may not be available. This method also tends to be more time consuming and may not be feasible when either time or personnel are limited.

Top-Down Method The top-down method of estimation is based on overall characteristics of the software project. The project is partitioned into lower-level components and life cycle phases beginning at the highest level. This method is more applicable to early cost estimates when only global properties are known. Advantages include consideration of system-level activities (integration, documentation, project control, configuration management, etc.), many of which may be ignored in other estimating methods. The top-down method is usually faster, easier to implement and requires minimal project

119

Parametric Cost Analysis Handbook

detail. However, disadvantages are that it can be less accurate and tends to overlook lower-level components and possible technical problems. It also provides very little detail for justifying decisions or estimates.

Expert Judgment Method Expert judgment involves consulting with human experts to use their experience and understanding of a proposed project to provide an estimate for the cost of the project. The obvious advantage of this method is the expert can factor in differences between past project experiences and requirements of the proposed project. The expert can also factor in project impacts caused by new technologies, applications, and languages.

Expert judgment always

compliments other estimation methodologies. One disadvantage is that the estimates can be no better than the expertise and judgment of the expert. It is also hard to document the factors used by the expert who contributed to the estimate.

Parametric Or Algorithmic Method The algorithmic method involves the use of equations to perform software estimates. The equations are based on research and historical data and use such inputs as Source Lines Of Code (SLOC), number of functions to perform, and other cost drivers such as language, design methodology, skill-levels, risk assessments, etc. Advantages of this method include being able to generate repeatable results, easily modifying input data, easily refining and customizing formulas, and better understanding of the overall estimating methods since the formulas can be analyzed.

However, the results are

questionable when estimating future projects which use new technologies, end equations are generally unable to deal with exceptional conditions such as exceptional personnel in any software cost estimating exercises, exceptional teamwork, and an exceptional match between skill-levels and tasks. However, any estimating approach can be impacted by these drawbacks, and care should be exercised when accounting for such concerns. Additionally, sometimes algorithms are developed within companies for internal use and many be proprietary as well as more reflective of a specific company's performance characteristics.

120

Parametric Cost Analysis Handbook

Software Cost Estimating Standards As stated earlier, very often the government requires software development to follow DODSTD-2167A/498, which is the Department of Defense standard that specifies the overall process for the development and documentation of mission critical software systems. This standard also requires technical reviews and audits to be conducted in accordance with DOD-STD-1521B. Other standards that may affect the estimating process are: MIL-STD-499A, MIL-STD498, Engineering Management; MIL-STD-490A, Specification Practices; MIL-STD-480B, Configuration Control-Engineering Changes, Deviations and Waivers; DOD-STD-1703, Software Products Standards. Software developed in accordance with these standards generally requires more resources and is more time consuming. Therefore, the software estimation process must include and adjust for these requirements where applicable.

Benefits When the software estimation process is performed correctly, the benefits realized far outweigh the cost of doing the estimating. Some of the major benefits include lowering the cost of doing business, increasing the probability of winning new contracts, increasing and broadening the skill-level of key staff members, acquiring a deeper knowledge of the proposed project prior to beginning the software development effort, and understanding, refining, and applying the proper software life cycle mode. As these software estimating components are enhanced, refined, and continually applied, the software estimating process, associated tools, and resulting products attain higher levels of quality and ultimately benefit all.

EXAMPLES OF PARAMETRIC SOFTWARE COST ESTIMATING

Accurately estimating the resources and time needed for a software development project is essential even for the survival of the project. In the great majority of cases, the resources and time actually expended are much more than the initial planning estimates. An approach for estimating the resources and schedule needed for software development is the use of a software cost and

121

Parametric Cost Analysis Handbook

schedule model that calculates the resources and time needed as a function of some other software parameters (such as the size of the program to be developed). The more times an organization has developed software of the same size and complexity for the same type of application, the more accurate the estimates for a new project will be. Unfortunately, when the Program Manager attempts to extrapolate information from small and less complex software development efforts to larger, more complex software in different application areas, the results are often unreliable. The problem results from and exponential relationship between software size and development effort. For example, one very widely used parametric software cost model is the Constructive Cost Model (COCOMO). The basic COCOMO model has a very simple form:

MAN-MONTHS = K1 (Thousands of Delivered Source Instructions) *K2 Where K1 and K2 are two parameters dependent on the application and development environment.

Estimates from the basic COCOMO model can be made more accurate by taking into account other factors concerning the required characteristics of the software to be developed, the qualification and experience of the development team, and the software development environment. Some of these factors are: * Complexity of the software * Required reliability * Size of data base * Required efficiency (memory and execution time) * Analyst and programmer capability * Experience of team in the application area * Experience of team with the programming language and computer * Use of tools and software engineering practices * Required schedule

122

Parametric Cost Analysis Handbook

Many of these factors affect the person months required by an order of magnitude or more. COCOMO assumes that the system and software requirements have already been defined, and that these requirements are stable. This is often not the case. Another popular software cost model is the Putnam model. The form of this model is:

PERSON YEARS= (Lines of Delivered source Instructions) Ck (Development Time in Years)

Where: CK is a parameter dependent on the development environment. CK has a range from-6,500 (poor) to 12,500 (excellent).

The Putnam model is very sensitive to the development time: decreasing the development time can greatly increase the person-months needed for development. One significant problem with the COCOMO, PUTNAM and similar models is that they are based on knowing, or being able to estimate accurately, the size (in lines of code) of the software to be developed. There is often great uncertainty in the software size. Computer programs, several of which are available for PCs, have been developed to implement variations of the COCOMO and other cost models. Another estimating approach, called Function Point Analysis (FPA), is used to estimate both the size of the software to be developed and the effort required for the development. In FPA, you determine the number and complexity of inputs, outputs, user queries, files, and external interfaces of the software to be developed. The sum of these numbers, weighted according to the complexity of each, is the number of function points in the system. This function point number is directly related to the number of end-user business functions performed by the system. Using data from past projects, it is possible to estimate the size of the software needed to implement these function points (typically about 100 source language statements are needed for each function point) and the labor needed to develop the software (typically about 1 to 5 function points per person month). FPA has been developed and applied almost exclusively in Information System applications. A variation, called feature point analysis, has been defined for other application areas. The chief difference between feature point analysis

123

Parametric Cost Analysis Handbook

and FPA is that the number and complexity of the algorithms to be implemented are considered in calculating the number of feature points. Do not depend on a single cost or schedule estimate. Use several estimating techniques or cost models, compare the results, and determine the reasons for any large variations. Document the assumptions made when making the estimates. Monitor the project to detect when assumptions that turn out to be wrong jeopardize the accuracy of the estimate.

PARAMETRIC SOFTWARE COST ESTIMATING TOOLS

As mentioned earlier, one of the critical problems facing software development project managers is determining accurate software estimations for level of effort, schedules, SLOC, and overall costs. Since trends indicate that the cost of producing software products is escalating and consuming an ever increasing percentage of budgets, the need to quickly generate more reliable estimates is becoming even more important. For many years, project managers have relied on software development teams to estimate the cost of producing software products. This has always been a subjective and intuitive process influenced by such factors as personality, opinions, and pressure to win contracts. The process has encouraged low estimates and short schedules, the results of which have been devastating to companies and projects, including projects of national importance. For these reasons, software parametric cost estimating tools have been developed since the late 1970's to provide a better defined and more consistent software estimating process. These tools have been developed from historical data collected from thousands of software projects, as well as research performed to identify key productivity factors. Early tools were hampered by the scarcity of reliable data; however, as more data became available, estimating tools were improved and continued to evolve. Most parametric software estimation tools use algorithms, and some of the more advanced tools are rule-based or knowledge-based as well as interactive. Good software estimating tools do not always guarantee reliable software estimates. If inaccurate software size estimates and attribute ratings are input, then inaccurate estimates will result. Additionally, organizations need to customize the software estimation tools to their own development environment (the calibration process was discussed previously in Chapter IV).

124

Parametric Cost Analysis Handbook

This customization requires collecting and maintaining historical data from current and past projects to provide the necessary inputs required for the software estimating tools. The software development organization should establish a staff that is thoroughly trained in the software estimating process and available estimating tools, and they should perform all the software estimating activities. Experience and existing tools dictate what project development information needs to be maintained.

Desired Functional Capabilities Of Parametric Tools The following section provides an overview of some of the more popular and available software estimating tools, including the major functional capabilities which software estimating tools should perform, and the input data required to support the use of those tools. Major functional capabilities that should be considered when selecting a software estimating tool are listed below. Depending on the organization's needs, the level of significance of these capabilities may differ, and should be considered accordingly. In addition, the organization should analyze their own needs and identify additional desired capabilities specific to them. The organization should then match available tools with overall needs. In general, the tool should: (1)

Allow easy adaptation to an organization's development environment - This means the tool needs to be capable of being customized to fit the using organization's development environment. Customization includes allowing the developer to define applicable inputs, as well as to modify coefficients and exponents of the equations used by the tool. This feature will allow continuous improvement to the estimation capability of the tool since the organization's historical data and current project data will be included in the software estimate generated.

(2)

Be relatively easy to learn and use - The tool should be well documented including methodologies and equations used.

Documentation should be at a level that is

understandable. The tool should include help menus and examples sufficient to assist the support staff in answering questions and providing training. The amount of formal training required to use the tool should be relatively minimal, required inputs should be well defined, and visibility into internal equations and theories should be provided.

125

Parametric Cost Analysis Handbook

(3)

Provide early estimates - The tool should be capable of generating estimates early and quickly in the life cycle process when requirements and development environments are not fully defined. The tool should also allow task detail to be added incrementally as functions, activities, and other information becomes more completely defined. Since there are many unknowns early in the estimating process, the tool should reflect degrees of uncertainty based on the level of detail input (risk analysis). In general, the tool should provide sufficient information to allow initial project resource planning as well as reasonably early "go/no go" decisions.

(4)

Be based on software life cycle phases and activities - The tool should be capable of providing estimates for all phases and activities of the most commonly used software life cycle models. It should allow the organization to decompose and map software development tasks into those phases and activities, as well as support a program WBS. In addition, it should allow for "what if" situations and include factors for design trade-off studies.

(5)

Allow for variations in application languages and application function - It is very important that the tool provide estimates specific to the application of the software project since the associated estimating equations, cost drivers, and life cycle phases should be unique to each application are. General application categories include Information Systems (IS), simulation and modeling systems, real-time systems, accounting systems, and systems based on higher-order languages.

(6)

Provide accurate size estimates - The size of a software development project is a major cost driver in most estimating tools, yet size is one of the most difficult input parameters to estimate accurately. The tool should include the capability to help estimate the size of the software development project, or at least help define a method for estimating the size.

(7)

Provide accurate schedule estimates - As previously mentioned, schedule overruns are common and can be extremely costly. The software estimating tool should be able to provide schedule estimates accurately. The purpose of scheduling is not only to predict task completion given task sequence and available resources, but also to

126

Parametric Cost Analysis Handbook

establish starting and ending dates for the associated work packages and life cycle phases. (8)

Provide maintenance estimates separately - The software estimating tool should be able to provide software maintenance estimates as a separate item.

Software

maintenance includes such activities as correcting errors, modifying the software to accommodate changes in requirements, or extending and enhancing software performance.

Input Data Collection A very important aspect of software estimating is data collection. Data must be collected for inputs to the parametric tool and tool verification, validation, customization, and calibration. Estimates generated by the tools are only as good as the input data used. Careful analysis of all tool inputs are essential since small changes in input values can result in large variations in overall cost and schedules. Inputs vary between tools. Before using a tool, review input requirements and information collected, documentation, and examples provided with the tool. When possible, discuss these with individuals familiar with the tool. Using historical data as a basis for customizing or calibrating a tool is essential. Insure that information for current project development efforts are saved for future reference. At a very minimum, use software life cycle model phases and activities as a basis for collecting and maintaining project development information for future tool use.

Some Commercial Tools There are many software estimating tools on the market today that provide software estimation support. Some tools estimate the size of the software project, while others use size as an input and provide estimates of effort, schedule and cost. The tools presented in this handbook are not being recommended over other tools as each one has unique capabilities and limitations. Therefore, organizations considering using estimating tools should review as many available tools as possible, analyze their software estimating needs, and then determine which tools are most appropriate to their application and development

127

Parametric Cost Analysis Handbook

environment. The list that follows is not all inclusive, and should not be considered complete. The tools discussed are representative only, since there are many others that could be used.

REVIC The Revised Enhanced Version of Intermediate COCOMO (REVIC) model was developed by Mr. Raymond L. Kile formerly of Hughes Aerospace. The Air Force Contract Management Division, Air Force System Command, Kirtland Air Force Base, New Mexico, sponsored the development for use by its contract administrator. The main difference between REVIC and COCOMO is the coefficients used in the effort equations. REVIC changed the coefficients based on a database of recently completed DOD projects. It also uses a different method of distributing effort and schedule to each phase of product development, and applies an automatic calculation of standard deviation for risk assessment. REVIC provides a single "weighted average" distribution for effort and schedule along with the ability to let the user vary the percentages in the system engineering and development test and evaluation phases. REVIC employs a different Ada model than Ada COCOMO. The REVIC model has also been enhanced by using a Program Evaluation and Review Technique (PERT) statistical method for determining the lines of code to be developed. In addition to providing estimates for cost, manpower, and schedule, the program creates estimates for typical

DOD-STD-2167A/498 documentation sizing and long term software

maintenance. REVIC's schedule estimates are often considered lengthy because it assumes that a project's documentation and reviews comply with the full requirements of DOD-STD-2167A/498. REVIC operates on PC compatible systems.

PRICES The PRICES tool is distributed by Lockheed - Martin PRICE Systems. This tool was first developed in 1977, and is considered one of the first complex commercially available tools used for software estimation. The equations used by this tool are proprietary. However, descriptions of the methodology algorithms used can be found in papers published by PRICE Systems. A model describing the PRICES parametric tool is shown in Figure V-4.

128

Parametric Cost Analysis Handbook

FIGURE V - 4

The PRICES tool is based on Cost Estimation Relationships (CERs) that make use of product characteristics in order to generate estimates. CERs were determined by statistically analyzing completed projects where product characteristics and project information were known, or developed with expert judgment. A major input to PRICES is Source Lines of Code (SLOC). Software size may be input directly, or automatically calculated from quantitative descriptions (function point sizing). Other inputs include software function, operating environment, software reuse, complexity factors, productivity factors, and risk analysis factors. Successful use of the PRICES tool depends on the ability of the user to define inputs correctly. It can be customized and calibrated to the needs of the user. It is now available for Windows and UNIX/Motif.

129

Parametric Cost Analysis Handbook

SASET The Software Architecture, Sizing and Estimating Tool (SASET) was developed for DOD by the Martin Marietta Corporation. SASET is a forward-chaining, rule-based expert system utilizing a hierarchiacally structured knowledge database. The database is composed of projects with a wide range of applications. SASET provides functional software sizing values, development schedules, and associated man-loading outputs. development cycle.

It provides estimates for all types of programs and all phases of the It also provides estimates for maintenance support and performs a risk

assessment on sizing, scheduling, and budget data. SASET uses a five-tiered approach for estimating including class of software, source lines of code, software complexity, maintenance staff loading, and risk assessment. The user can either input the program size directly or allow SASET to compute size, based on function-related inputs. The tool also has an extensive customization file in which the user can adjust many parameters. It operates on PC compatible systems.

SEER-SEM System Evaluation and Estimation of Resources - Software Estimation Model (SEER-SEM) is distributed by Galorath Associates and is currently under a five year Air Force wide license agreement. It provides software estimates with knowledge bases developed from many years of completed projects. The knowledge base allows estimates with only minimal high level inputs. The user need only select the platform (i.e. ground, unmanned space, etc.), application (i.e. command and control, diagnostic), development methods (i.e. prototype, incremental), and development standards (i.e. 2167A/498). SEER-SEM is applicable to all types of software projects and considers all phases of software development. SEER-SEM is designed to run on PC compatible systems running Microsoft Windows 3.0/3.1 (Air Force license includes MS-DOS version). It is also available for the Apple MacIntosh running system 6.0.3 and above and the UNIX/SUN work station.

130

Parametric Cost Analysis Handbook

A companion tool called the SEER-Software Sizing Model (SSM) is also distributed by Galorath Associates and is used to estimate the size of the software product.

SLIM The Software Life Cycle Model (SLIM) is marketed by Quantitative Software (QSM). SLIM was developed in 1979 by Mr. Larry Putnam. Originally developed from analyses of groundbased radar programs, the SLIM tool has been expanded to include other types of programs. It can be customized for the user's development environment. SLIM supports all phases of software development, except requirements analysis, as well as all sizes of software projects, but was especially designed to support large projects. Success in using SLIM depends on the user's ability to customize the tool to fit the software development environment and to estimate both a Productivity Index (a measure of the software developer's efficiency) and a Manpower Buildup Index (a measure of the software developer's staffing capability). SLIM also provides a life cycle option which extrapolates development costs into the maintenance phase. A companion tool named SIZE Planner is also distributed by QSM and is used to estimate the size of the software product. QSM provides a training course and leases the tool via a time sharing service. There is also a PC compatible version of SLIM that can be leased for a yearly fee.

SOFTCOST-R SOFTCOST-R is a software estimating tool developed by Reifer Consultants, Inc. (RCI). SOFTCOST-R is based upon the modeling work done by Dr. Robert Tausworthe of the Jet Propulsion Laboratory. It contains a database of over 1500 data processing, scientific and real-time programs. A key input is SLOC, which can be input directly or computed from Function Points. SOFTCOST-R is applicable to all types of programs, however, it was specifically configured to estimate real-time and scientific software systems, and considers all phases of the software development cycle. The tool's primary input is SLOC, however, it also uses the same inputs and provides the same outputs as COCOMO which allows direct comparisons to be made.

131

Parametric Cost Analysis Handbook

SOFTCOST-R has some unique inputs such as user and peer reviews, customer experience, and degree of standardization. It also supports a standard WBS for task planning and scheduling. RCI provides SOFTCOST-Ada, which is a tool to estimate Ada and C++ development costs. Softcost-Ada is a cost estimation tool specifically developed to estimate systems using object-oriented techniques. RCI also has a separate estimating tool called ASSET-R to estimate the size of the software product. SOFTCOST-R, SOFTCOST-Ada, and ASSET-R are leased on an annual license basis, and require a PC compatible running DOS 2.3 or higher.

SYSTEM-4 SYSTEM-4 is marketed by Computer Economics, Inc. (CEI). It contains a proprietary model that is based on the work of Jensen, Boehm, Putnam, and other noted software experts. SYSTEM-4 is applicable to all types of programs and all phases of the software life cycle. Inputs consist of size (SLOC), twenty environmental factors, seven development factors, software type, and constraints. This tool comes with 23 predefined default parameter files. The default sets provide typical values for all parameters except size. There are also seven parameter subset files for various implementations of DOD-STD-1703, DOD-STD-2167A/498, and varying degrees of Ada experience. The user must select one of the default sets and input the SLOC estimate to perform a quick estimate. SYSTEM-4 can accommodate multiple CSCIs or tasks, and each task can be broken down into elements and units. There is a limit of 64 tasks, 64 elements, and 64 units. SYSTEM-4 can be customized to reflect the user's software development environment. CEI has a companion software size estimating tool called Computer Economics Incorporated Sizing (CEIS) System. These tools operate on PC compatible systems.

Software Sizing Tools As discussed previously, a very important factor in estimating software development projects is the ability to estimate the size of the product. Many software estimating tools use size in SLOC or functions performed as the major input. Size is also considered by software development

132

Parametric Cost Analysis Handbook

project managers as a major technical performance or productivity indicator that allows them to track the project during software development. The most commonly used method to estimate the size of a software product is by using both expert judgment and the analogy method. The experts determine the functions the software will perform and estimate the size by comparing the new system to completed projects with similar characteristics. Often, parametric methods are the used to precisely size the software. Estimating tools using analogy methods compare the new program to similar programs of known size. Because past projects are not always exactly like the new project, the estimate is adjusted by a factor determined from experience.

These tools accept characteristics of new

programs as inputs, then search a database for similar programs. The tools either list the similar programs or provide an estimate of size based on an average of the size of the similar programs selected from the database. Expert judgment tools use the opinion of one or more experts to estimate the size of the program, or use structured questions designed to extract judgment from the experts. These are the rule-based or expert system tools. Many tools use the algorithmic method by applying equations to determine size estimates. A technique that is becoming very widely used is Function Point Analysis (FPA). One problem with FPA is that it was developed for IS-oriented programs and does not take into consideration the number or complexity of algorithms in scientific and real-time programs. Many software estimating tools such as REVIC and SLIM use extensions of the Program Evaluation and Review Technique (PERT). PERT is based on a beta distribution of estimates provided by the user and calculates expected size according to the equation: Expected Size = (S + 4(M) + L)/6 where S, M, and L are estimates of the smallest size, most likely size, and the largest size, respectively.

ASSET-R ASSET-R is a function point sizing tool developed to estimate the size of data processing, real-time, and scientific software systems, and is marketed by Reifer Consultants, Inc. It utilizes a knowledge-based system which extends the theory of function points into scientific and real-time

133

Parametric Cost Analysis Handbook

systems by considering issues like concurrence, synchronization, and reuse in its mathematical formulation. The formulas use as many as nine parameters to develop function point counts. It also couples function point and operand/operator counts with architectural, language expansion, and technology factors to generate the size estimate. ASSET-R works with RCI's SOFTCOST-R and SOFTCOST-Ada software estimation tools. It operates on PC compatible systems.

CA-FPXpert CA-FPXpert is distributed by Computer Associates International, Inc. It uses FPA for size estimation of IS type software projects, and conforms to accepted IFPUG standard counting practices. It includes an on-line tutor to help the function point counting process. CA-FPXpert works in conjunction with CA-ESTIMACS to provide software size estimation input, and operates on PC compatible systems.

CEIS CEIS is marketed by Computer Economics, Inc. Estimates are generated by comparing the attributes of the new project to the attributes of three reference projects of known size. The user determines any six attributes that contribute to the number of lines of code and ranks them in order of importance, then selects three reference projects of known size. Separate algorithms are used to produce four independent estimates and to determine a level of confidence. CEIS works in conjunction with SYSTEM-4, and operates on PC compatible systems.

SIZEEXPERT SIZEEXPERT was developed by the Institute for Systems Analysis and is marketed by Technology Application/Engineering Corporation.

This tool is an expert judgment tool that

produces estimates of liens of code based on questions asked by COSTEXPERT. Both tools are packaged and distributed together, and operate on PC compatible systems.

SEER-M SEER-M is marketed by Galorath Associates and is available to government personnel under and Air Force-wide contract. It produces software size estimates in lines of code or function

134

Parametric Cost Analysis Handbook

points. It also provides its own historical database in producing the size estimate. SEER-M works with SEER-SEM software estimating tool, and operates on PC compatible systems. Appendix F includes other currently available cost models, and discusses in more detail the most popular software estimating tools.

GLOSSARY OF TERMS

Appendix A contains definitions of terms commonly used in software estimation technology.

MODEL CALIBRATION

The act of calibration standardizes a model. Many models are developed for specific situations and are, by definition, calibrated to that situation. Such models usually are not useful outside of their particular environment.

However, general cost estimating models including

commercially available ones such as the FAST, PRICE and SEER models (described earlier) are designed to be useful as estimating tools for a wide range of situations. The act of calibration is needed to increase the accuracy of one of these general models by making it temporarily a specific model for whatever product it has been calibrated for. Calibration is in a sense customizing a generic model. Items which can be calibrated in a model are: product types, operating environments, labor rates and factors, various relationships between functional cost items, and even the method of accounting used by a contractor. All general models should be standardized (i.e. calibrated), unless used by an experienced modeler with the appropriate education, skills and tools, and experience in the technology being modeled. Calibration is the process of determining the deviation from a standard in order to compute the correction factors. For cost estimating models, the standard is considered historical actual costs. The calibration procedure is theoretically very simple. It is simply running the model with normal inputs (known parameters such as software lines of code) against items for which the actual cost are known. These estimates are then compared with the actual costs and the average deviation

135

Parametric Cost Analysis Handbook

becomes a correction factor for the model. In essence, the calibration factor obtained is really good only for the type of inputs that were used in the calibration runs. For a general total model calibration, a wide range of components with actual costs need to be used. Better yet, numerous calibrations should be performed with different types of components in order to obtain a set of calibration factors for the various possible expected estimating situations. For instance, the PRICE Software Model addresses this situation using internal productivity factors. These can be modified by the calibration process.

TRENDS AND CONCLUSIONS

Trends Advances in languages, development methodologies, and other areas will have to be addressed by future software cost estimating models and associated methodologies. As software technology matures, changes in development and support concepts occur which will impact software cost estimating. Concepts such as prototyping and spiral development present a challenge to cost estimation since normal software development cycles are altered. Artificial Intelligence (AI) represents a growing area of modern technology. Since AI is software-intensive, proper management of AI software, including cost estimating, will be a challenge for software managers. The development of software for expert system and other AI applications will probably require a different development process. The trend for the future will include better and more accurate ways of developing software estimating methodologies for: * Software size estimates. * Resource Estimates for maintenance and support. * Incorporating the effects of Ada and new paradigms such as rapid prototyping and fourth generation languages. * Modeling the dynamic interaction of variables that affect productivity, cost, and quality. * Object-Oriented Design.

136

Parametric Cost Analysis Handbook

The trend in software estimating tools is to provide a whole family of models which not only estimate cost and effort of software development, but hardware as well. The tools are being upgraded to support higher-order languages such as Ada and C++.

The most significant

improvement is the use of data collected from past software projects to customize the tool to an organization's environment. This is especially true within agencies of DOD and NASA.

Conclusions This chapter has discussed the software estimating process and the various methodologies used in software estimation. The basic software estimating functional capabilities were also discussed. A few different software estimating tools were examined. A review of product literature and user manuals indicate that many tools will perform most of the functional capabilities outlined in this chapter. The users generally agree that the tools they are using satisfy their requirements. The software estimating organization must be able to customize the software estimating tool to their own software development environment. This requires collecting historical data from past completed projects to accurately provide the inputs that the software tool requires. The software development organization should establish a staff that is thoroughly trained in the use of the tools. This staff should do all the software estimating activities and determine what data should be collected to provide a historical database for future reference. The use of two or more software estimating tools using different methodologies is recommended. The software development organization should select a primary tool for software estimating and an alternate tool for comparison and validation. These tools should be used throughout the software development process. Parametric tools are considered to be the best for software estimating for the following reasons: * Equations are based on previous historical development projects. * Outputs are repeatable and formulas can be analyzed. * They can be customized to fit the user's environment. * They require minimal time and effort to use. * They are particularly useful in earlier phases of software development. * They are most frequently used by DOD agencies.

137

Parametric Cost Estimating Handbook

CHAPTER VI

AUDITING PARAMETRIC ESTIMATES A MANAGEMENT VIEWPOINT

138

Parametric Cost Estimating Handbook

CHAPTER VI AUDITING PARAMETRIC ESTIMATES

The following paper written by Michael Thibault, Deputy Director, DCAA, provides background from an auditor's point of view on auditing parametric cost estimating techniques.

COST ESTIMATING RELATIONSHIPS: A DCAA PERSPECTIVE

Introduction The U.S. Defense Contract Audit Agency (DCAA) was established in 1965. DCAA's mission is to perform all necessary contract audits for the Department of Defense and, when requested, to perform contract audit services on a reimbursable basis for other government agencies. In essence, DCAA provides accounting and financial advisory services for procurement and contract administration activities. Contract audit activities include providing professional advice on accounting and financial matters to assist in the negotiation, award, administration, repricing, and settlement of contracts. DCAA audits are conducted in accordance with generally accepted government auditing standards established by the General Accounting Office. This paper discusses the government's expectations when defense contractors use parametric cost-estimating relationships for estimating government contract costs. The paper also emphasizes that companies must meet all the adequacy criteria set out in the Federal Acquisition Regulations (FAR) and applicable supplements to obtain approval for their estimating systems. Companies must apply the same criteria to their parametric cost-estimating relationships to ensure they are acceptable for use in estimating systems. DCAA believes now, as it has always believed, that parametric estimating techniques using cost-estimating relationships are acceptable in the appropriate circumstances for proposing costs on government contracts. DCAA is ready and willing to work with industry in the evolution of parametrics. Operation Desert Shield and Operation Desert Storm dramatically demonstrated that our government must be capable of responding quickly to changing procurement requirements.

139

Parametric Cost Estimating Handbook

Parametric systems can help us do just that. Future estimating systems must be responsive, accurate, and cost effective.

Background DCAA was issuing official guidance on parametric systems as early as 1978. Parametrics was broadly defined as a technique that employs one or most cost-estimating relationships to estimate costs associated with developing, manufacturing, or modifying an end item. In the 1980s, DCAA auditors reported an increase in the number of contractors using parametric cost estimating. DCAA developed and issued audit guidance to assist the field auditors in this new area. Studies conducted by DCAA; the Office of the Secretary of Defense Cost Analysis Improvement Group; Headquarters, Air Force Contract Management Division; Headquarters, Aeronautical Systems Division; and the Space Systems Cost Analysis Group provided the basis for DCAA audit guidance issued in 1982.

Parametric Criteria This guidance was also the subject of an article written for the Spring 1982 issue of Journal of Parametrics published by the International Society of Parametric Analysts. Charles 0. Starret, Jr., then-Director of the Defense Contract Audit Agency, wrote the article entitled "Parametric Cost Estimating -- An Audit Perspective." The guidance contained in that article is essentially the same as the guidance given to DCAA auditors today.

It reiterates DCAA's long-held view that

parametrics is an acceptable estimating technique.

The 1982 article included the criteria a

contractor should apply before submitting a contract price proposal using parametrics. The criteria are still on point today, and they are: 1)

Logical relationships

2)

Significant statistical relationships

3)

Verifiable data

4)

Reasonably accurate predictions

5)

Proper system monitoring

Logical Relationships

140

Parametric Cost Estimating Handbook

Contractors are expected to demonstrate that cost-to-noncost-estimating relationships are logical. "Logical relationship" is often difficult to determine in a finite sense, yet is very important. DCAA's primary concern in this area is that a contractor consider all reasonable logical estimating alternatives and not use only the first apparent set of variables. Contractor analysis may disclose multiple alternatives that appear logical. Statistical testing should be used to help identify the best alternative.

Significant Statistical Relationships Contractors are also expected to demonstrate that a significant statistical relationship exists among the variables used in a parametric cost-estimating relationship. There are several statistical methods such as regression analysis that can be used to validate a cost-estimating relationship; however, no single uniform test can be specified. Statistical testing may vary depending on an overall risk assessment and the unique nature of a contractor's parametric data base and the related estimating system. Proposal documentation should describe the statistical analysis performed, including the contractor's explanation of why the cost-estimating relationship is statistically valid.

Verifiable Data There must be a system in place for verifying data used for parametric cost-estimating relationships. In many instances, the auditor will not have previously evaluated the accuracy of noncost data used in parametric estimates. For monitoring and documenting noncost variables, contractors may have to modify existing information systems or develop new ones. Information that is adequate for day-to-day management needs may not be reliable enough for contract pricing. Data used in parametric estimates must be accurately and consistently available over a period of time, and easily traced to or reconciled with source documentation.

Reasonably Accurate Predictions The contractor's demonstration that the parametric cost-estimating relationships predict costs with a reasonable degree of accuracy is also important. The key is that if the contractor's analysis of historical estimating and cost performance data shows that the parametric estimating

141

Parametric Cost Estimating Handbook

system is as accurate as a discrete estimating system, then the government has increased assurance of receiving a fair and reasonable price. As with any estimating relationship derived from prior history, it is essential for the contractor to document that the work being estimated using parametric cost-estimating relationships is comparable to the prior work from which the parametric data base was developed.

Proper System Monitoring The contractor should also ensure that cost-to-noncost parametric rates and factors will be monitored periodically in the same manner as is expected for cost-to-cost rates and factors. Because of improved technology, production changes, or better pricing alternatives, cost-estimating relationships can and do change. The contractor should be prepared to revalidate any parametric cost-estimating relationship whenever system monitoring discloses that the relationship has changed.

Audit Planning and Requirements The old expression, "The more things change, the more things stay the same," seems to apply. Government procurement procedures may change to accommodate changes in the services and products it buys, but the basic procurement goal of getting products and services for fair and reasonable prices remains the same. And so it goes with auditing. Whether DCAA audits proposed costs for space stations or for missiles, DCAA's basic aim of ensuring that the proposed costs are allowable, allocable, and reasonable and therefore acceptable for government reimbursement is still the same. How DCAA accomplishes its audit objective varies with the sophistication of contractor accounting and estimating systems. Auditors begin the audit by ensuring they have the requisite familiarity with DCAA guidance on estimating systems and techniques. This guidance is contained in DCAA's Contract Audit Manual (CAM), which is available to the general public. The auditor then proceeds to do the following: 1)

Ensure they are familiar with the company's estimating policies and procedures.

2)

Identify the estimating methods used to develop the proposal.

142

Parametric Cost Estimating Handbook

3)

Determine that the supporting cost and pricing data for the individual proposal was derived in accordance with the contractor's estimating system and is in compliance with applicable regulations.

The auditor plans the audit scope using what is known about the contractor. For example, the audit scope will vary depending upon the estimating methods the contractor uses. In addition, the auditor will consider the following types of questions: *

What is the dollar amount and type of contract contemplated?

*

Has the contractor established strong internal controls and sound accounting and estimating systems?

*

What kinds of testing does the contractor do to ensure compliance with these systems?

*

What does our prior audit experience tell us about the contractor's internal controls or estimating practices?

Audit planning requires the auditor to answer all of these questions and to make a determination regarding the government's risk. Judgment is then exercised in deciding the degree of risk that the estimate could be materially misstated. This assessment of risk will be used to decide what audit procedures to employ. The auditor identifies the method of estimating the contractor uses to determine the kind of support that should be available. A contractor could be using any or all of the following methods: 1) Detailed -- also known as the bottoms-up approach. This method divides proposals into their smallest component tasks and are normally supported by detailed bills of material. 2) Comparative -- develops proposed costs using like items produced in the past as a baseline. Allowances are made for product dissimilarities and changes in such things as complexity, scale, design, and materials. 3) Judgmental -- subjective method of estimating costs using estimates of prior experience, judgment, memory, informal notes, and other data. It is typically used during the research and development phase when drawings have not yet been developed.

143

Parametric Cost Estimating Handbook

Parametric estimating techniques may be used in conjunction with any of these methods. Whatever the method selected by the contractor, it must comply with applicable laws and regulations. The laws and regulations most often encountered in dealing with parametrics are: 1)

CAS 401, "Consistency in Estimating, Accumulating, and Reporting Costs"

2)

Truth in Negotiations Act (TINA)

3)

FAR 15.800, Price Negotiation

4)

DFARS 215.811, "Estimating Systems"

Cost Accounting Standards (CAS) provides guidance in accounting for contract costs at larger contractors. CAS 401 requires that a contractor's estimating practices be consistent with those governing the accumulation and reporting of costs during contract performance. Some contractors see parametrics as being inconsistent with CAS 401. Contractors must ensure both cost and noncost information used in estimating is separately accumulated and reported as required by CAS 401. The purpose of the Truth in Negotiations Act, 10 U.S.C. 2306(f), is to provide the government with all facts available to the contractor at the time it certified the cost or pricing data was accurate, complete, and current. Parametric estimates must meet the same basic disclosure requirements under the act as discrete estimates. Although the principles are no different, proposals supported in whole or in part with parametric estimating will have different types of cost or pricing data than traditional discrete cost estimates. Fundamental to the definition of cost or pricing data are "all facts ... which prudent buyers and sellers would reasonably expect to have a significant effect on price negotiations" (FAR 15.801). Reasonable parallels may be drawn between the data examples provided in FAR for discrete estimating approaches and the type of data pertinent to parametric estimating approaches. The contractor is also expected to provide all factual data for the parametric cost estimates. This data must be accurate, complete, and current. Many contractors use parametric cost estimating for supplementary support or validation of estimates developed using other methods. This requires judgment in selecting which data will be used in developing the total cost estimate relied upon for the price proposal. In distinguishing between fact and judgment, FAR states the certificate of cost or pricing data "does not constitute a

144

Parametric Cost Estimating Handbook

representation as to the accuracy of the contractor's judgment on the estimate of future costs or projections. It does apply to the data upon which the contractor's judgment or estimate was based" (FAR 15.804-4b). Thus, if a contractor develops a proposal using both parametric data and discrete estimates, it would be prudent to disclose all pertinent facts to avoid later questions about completeness of the submission. Auditors are also required to evaluate estimating systems of major Department of Defense (DoD) contractors. This includes ensuring that, if parametric estimating procedures are part of the estimating system, they are properly disclosed. Of key concern to the auditor in evaluating the estimating systems' disclosure of parametric procedures are the following: 1)

Do the procedures clearly establish guidelines for when parametric techniques would be appropriate?

2)

Are there guidelines to ensure the consistent application of estimating techniques?

3)

Is there proper identification of sources of data and the estimating methods and the rationale used in developing cost estimates?

4)

Do the procedures ensure that relevant personnel have sufficient training, experience, and guidance to perform estimating tasks in accordance with the contractor's established procedures?

5)

Is there internal review of and accountability for the adequacy of the parametric estimating techniques, including the comparison of projected results to actual results and an analysis of any differences?

DCAA believes parametric estimating approaches are acceptable when they are properly implemented. Auditors encounter it most often as a technique used in conjunction with other estimating methods. For example, parametrics are often used for estimating costs of scrap and other such factors. The majority of the proposals audited are not developed solely based on parametric estimating techniques. The use of parametric estimating is most appropriate in such circumstances where historical data is not available as when the program is at the engineering concept stage, or when no bill of materials exists and the program definition is unclear. Contractors with good parametric cost estimating systems analyze their perspective proposal to determine the appropriate estimating technique for each part of the work breakdown structure.

145

Parametric Cost Estimating Handbook

Observations and Suggestions The contractor can consider some observations made by DCAA auditors as to the pitfalls contractors fall victim to when employing parametric techniques. The first is when a contractor fails to do a cost-benefit analysis before implementing an elaborate parametric estimating model. Key questions for any contractor considering implementing a complex parametric model are: 1) How often can we reasonably expect to use it? 2) How much time can we expect to save? 3) What are the costs of maintaining the model? 4) Will the model produce the necessary precision?

Contractors should be satisfied that implementation and monitoring costs do not outweigh the benefit of reduced estimating costs. Moreover, it is critical that the environment is appropriate for the use of parametrics. It would not be prudent to rely exclusively on parametric techniques to estimate costs when directly applicable historical cost data are available. Such is the case of follow-on production for the same or similar hardware. Contractors manufacturing mature weapon systems already have a record of the actual costs. The use of parametric estimating may be appropriate for certain aspects of follow-on production, however, the contractor should disclose any data that may have a significant impact on cost. The exclusive use of parametrics is generally not appropriate for economic forecasting of such elements as labor and indirect cost rates.

For

parametric estimates, the contractor must ensure that any changes in accounting practices are accounted for in the estimate and that labor and indirect cost rates are appropriately applied. Another problem encountered is contractors failing to properly disclose their parametric estimating practices. Auditors have experienced instances where the first time a parametric model is disclosed is during the evaluation of the proposal. This is often too late. As mentioned earlier, larger DoD contractors have an obligation to disclose in writing their estimating procedures. Making proper and timely disclosure will minimize problems and expedite the negotiation process. Contractors can take the lead in helping to streamline the oversight process. In a few words, they should practice self-governance! DCAA has been a leading proponent of the self-governance program. Self-governance is intended to encourage contractors to establish and maintain good systems of internal control in key areas, including estimating systems. It requires contractors to

146

Parametric Cost Estimating Handbook

provide their own oversight -- to detect system weaknesses and take corrective action as necessary. This initiative recognizes that prudent contractors already have the means in place to ensure their operations are efficient and cost effective. DCAA has another initiative that contractors should consider as a part of streamlining the oversight process. It is called "coordinated audit planning." DCAA defines coordinated audit planning as a voluntary process wherein the DCAA auditor and the contractor's internal and external auditors consider each other's work in determining the nature, timing, and extent of his or her own auditing procedures. Coordinated audit planning considers the extent to which reliance can be placed upon work performed by the other auditor to minimize duplication of audit effort. In addition, this process strengthens the evaluation of internal control systems. Understanding estimating systems controls, assessing risk, and transaction testing are common objectives of DCAA, and the internal and external auditors associated with contractors. This often results in duplicative audit procedures. In the coordinated audit planning conducted to date, DCAA found that the instances of duplication are significant. DCAA auditors are more than willing to rely on the work done by contractor internal and external auditors, providing DCAA has the opportunity to evaluate and test their work. Contractors are expected to establish and maintain reliable estimating systems. Departmental procurement officials, Members of Congress, and the average American citizen hold Defense contractors to high and exacting standards. The funds involved can be enormous. The expectations are equally high for the auditor whose job it is to protect the taxpayer's interest.

The DCAA auditor must comply with the American Institute of Certified Public

Accountants' Statements on Auditing Standards and the GAO's Government Auditing Standards. These standards require that the auditor be independent in fact and in appearance. They also require the auditor exercise a healthy degree of professional skepticism. These requirements, however, should not render the DCAA auditor and the contractor enemies. Such polarized and adversarial relationships are dysfunctional and not in either party's best interest. Both parties are taking significant actions to improve relationships. These changes are producing a culture change that is very positive: positive because contractors are beginning to more fully accept their responsibilities; positive because auditors are more effectively communicating audit plans and objectives.

147

Parametric Cost Estimating Handbook

Defense procurement is taking on a new, streamlined look in the 1990s. Government and industry are both concerned with quicker, less costly means of procuring goods and services. This is one reason why parametric methods continue to stir up so much interest. Parametric techniques, properly applied, can assist contractors and the government alike in streamlining the acquisition process. In addition, the ability of the government to place greater reliance on contractor oversight of contractor systems will also result in meeting procurement needs more timely.

Summary In today's and tomorrow's procurement environment, a great challenge facing us all is the development of a cooperative work climate conducive to quickly acquiring quality products and services at fair and reasonable prices. New estimating techniques such as parametrics can cut estimating costs. Adequate estimating systems, fully supported and self-governed by industry, can cut audit costs. Quick estimating and audit turnaround times can cut procurement costs. All of this, however, requires communication and teamwork. Whether our buying effort is for innovative space equipment or for recurring maintenance, we must all work to meet the challenge. Mike Thibault's message to the parametric community is clear: Parametric techniques are acceptable cost estimating methodologies given that the following five criteria exist: logical relationships, verifiable data, a significant statistical relationship (correlation) exists, techniques produce accurate predictions, and they are easy to monitor and support. He goes on to say that the auditor needs to consider the adequacy of the parametrics cost estimating system and related internal controls, which includes: the audit trail, sufficiency of documentation, currency and sources of data, procedures for calibration and validation, and the appropriateness of parametrics use. Since historical data is normally used as the basis for all estimating, the auditor will use basic auditing techniques to verify that costs are current, accurate, and complete. Mr. Thibault completely dispels the myth that the audit community would not accept the use of appropriate parametric cost estimating techniques for firm business proposals. DCAA's complete audit guidance is included in its Contract Audit Manual, Chapter 9-1000. This chapter is included in its entirety in Appendix C.

148

Parametric Cost Estimating Handbook

ESTIMATING SYSTEM REVIEWS

As part of a regulatory oversight requirement, DCAA will periodically perform contractor estimating system reviews. The intent is to review the requirements delineated in DFARS 215.811 and 252.215-7003. DFARS 215.811 requires all DoD contractors to have adequate estimating systems, requires certain large businesses to disclose their estimating systems in writing, provides guidelines concerning the characteristics of an adequate estimating system, and provides guidance for team estimating system reviews. If a contractor is required to disclose their system, all significant parametric cost estimating techniques need to be disclosed. It is DoD policy that contractors have estimating systems that consistently produce well supported proposals acceptable as a basis for negotiating fair and reasonable prices. Estimating systems should be consistent and integrated with a contractor's related management systems, and be subject to applicable financial control systems. To be considered adequate, an estimating system must be established, maintained, reliable, and consistently applied. It must also produce verifiable, supportable and documented cost estimates. FAR 15.8 provides the criteria for submission of cost or pricing data. Although FAR 15.8 does not mention parametric cost estimating techniques, there are no restrictions that preclude the use of these techniques for firm business proposal submissions. FAR 15.804-6, Table 15-2, does require the offeror to submit any information reasonably required to explain the estimating process. This means that the contractor should clearly describe all parametric cost estimating techniques and provide support for the data used in those techniques. The FAR 15.804-6, Table 15-2, also states that for material cost estimates, the contractor shall provide a consolidated summary of individual material quantities (e.g., bill of material) included in the various tasks being proposed. In some cases it is not feasible for a contractor to provide a consolidated bill of material or parametrics results in a better and more supportable estimate. For example, in the research and development phase of a program, the material requirements often times have not yet been defined, and, as a result, the preparation of a consolidated priced bill of material is often not possible. In these circumstances, parametric cost estimating techniques may provide for a reasonable basis to estimate material costs in lieu of a consolidated priced bill of material.

149

Parametric Cost Estimating Handbook

In any case, when a offeror uses estimating techniques other than a consolidated priced bill of material to estimate material costs, the offeror must adequately describe the techniques being used, provide sufficient support to allow for an independent evaluation, and explain why the techniques used are the best in the circumstances in order to comply with the material cost criteria included in Table 15-2. DFARS 215.811-70 delineates attributes of an adequate estimating system. These are: (1)

Establishes clear responsibility for the preparation, review, and approval of cost estimates.

(2)

Provides a written description of the organization and duties of personnel responsible for ... contributing to the estimating process ...

(3)

Ensures that relevant personnel have sufficient training, experience and guidance ...

(4)

Identifies sources of data and the estimating methods and rationale used in developing cost estimates.

(5)

Provides for appropriate supervision ...

(6)

Provides for consistent application of estimating techniques.

(7)

Provides for detection and timely correction of errors.

(8)

Protects against cost duplication and omissions.

(9)

Provides for the use of historical experience, including vendor pricing information where appropriate.

(10) Requires use of appropriate analytical methods. (11) Integrates information available from other management systems as appropriate. (12) Requires management review [of the estimating system] (13) Provides for internal review of and accountability for the adequacy of the estimating system, including the comparison of projected results to actual results and an analysis of any differences. (14) Provides procedures to update cost estimates in a timely manner. (15) Addresses responsibility for review and analysis ... of subcontract prices.

No well supported and documented parametric estimating system should be threatened by any audit requirements, FAR or DFAR characteristics, or any other compliance (Truth in Negotiation

150

Parametric Cost Estimating Handbook

Act) issue. A well calibrated and validated parametric estimating system will be compliant in all respects. The DFARS goes on to list indicators of or conditions that may cause significant estimating deficiencies (from 215.811-77). Three in particular stand out: (1)

Failure to ensure that relevant historical experience is available to and utilized by cost estimators, as appropriate.

(2)

Consistent absence of analytical support for significant proposed cost amounts.

(3)

Excessive reliance on individual personal judgment where historical experience or commonly used standards are available.

The DFARS emphasis on historical experience is particularly satisfied by well supported parametric systems. After effective calibration activities, parametric estimates are developed that are auditable, compliant with regulations, and suitable for the negotiation of fair and reasonable prices between government and contractor.

FORWARD PRICING RATE AGREEMENTS (FPRAs)

An FPRA, as defined in FAR 15.801, is a written agreement negotiated between a contractor and the government to make certain rates and factors available during a specified period for use in pricing contracts or contract modifications. Such rates and factors represent reasonable projections of specific costs that are not easily estimated for, identified with, or generated by a specific contract, contract end item or task. On the other hand, a Forward Pricing Agreement (FPA) is a written agreement between a DoD contracting office and a large volume contractor which sets forth a methodology that the contractor agrees to follow when pricing items covered by the FPA. It differs from an FPRA in that once established, the FPA may be used to determine the complete final price of individual orders. A typical FPA, for example, may be established to cover and expedite the acquisition of spares. It is clear that any CER or parametric methodology could fall under the umbrella of an FPRA or an FPA. A forward pricing factor is generally represented as a percentage or ratio that is applied to an existing cost determination or estimate in order to arrive at another, usually related, cost determination or estimate. A costing rate used to convert labor hours to dollars is just such a factor

151

Parametric Cost Estimating Handbook

and is truly a CER. Parametric costing is done all the time. An FPRA for a parametric model is a natural next step. Therefore, when repetitive use of single CERs or parametric cost models is envisioned by the contractor, an FPRA should be established. Procedures for establishing such a FPRA should follow the basic guidelines listed in FAR Part 15.809-3 and should include the following: a.

Timely submission by the contractor, at least yearly, of an adequately supported FPRA proposal.

b.

Requirement for clear definitions of all elements of cost (dependent variables) which will be developed by the CERs included in the FPRA and the applicable bases (independent variables).

Additionally, the proposal must indicate the period of

applicability of the proposed CERs and the conditions under which the CERs will be used in price proposals. c.

The proposed CER's or parametric cost models should be evaluated in terms of the ten criteria listed in the Table VI-1. The answer to the majority of these questions should be affirmative - to support the need for a CER or Cost Model.

d.

Tracking of the negotiated CERs and parametric cost models included in the FPRA is required to test their validity as estimating tools. Guidelines for monitoring FPRAs are included in Part 15.809-5 and the DCAA Contract Audit Manual (Vol. 1, Chapter 9-200).

It is the contractor's responsibility to identify the rate(s) to be included under the umbrella of the FPRA or FPA and to specify the latest data already submitted in accordance with the agreement. All data submitted in connection with the agreement, updated as necessary, form a part of the total data that the offeror certifies to be current, accurate, and complete at the time of agreement on price for an initial contract or for a contract modification.

152

Parametric Cost Estimating Handbook

TABLE VI-I CRITERIA

QUESTIONS TO ANSWER

Cost Benefit

Is it more cost beneficial to have a CER for the cost element than to generate a discreet estimate?

Predicts Well

Do you anticipate the CER or parametric cost model to be a good predictor of actual and reasonable cost?

Supportability

Is the data that supports the development of the CER auditable and/or traceable? (This can be from official accounting data or other supporting data (e.g., subcontractor files)

Usability

Can the CER be applied within the bounds of the Estimating System?

Customer Acceptance

Is the CER accepted by the PCO, DPRO, the CER owner, etc.?

Management Acceptance

Does the CER satisfy the contractor's requirements?

Ownership

Can a group/individual be identified who would provide a discrete input if the element of cost was not estimated within the CER?

Applicability

What is the intent for application of this CER or parametric cost model (e.g., changes, restructures, etc.)?

Effective Implementation

Are the persons doing the modeling/ analysis sufficiently trained and experienced to use the tools correctly?

153

Parametric Cost Estimating Handbook

WHAT TO LOOK FOR IN A PARAMETRIC ESTIMATE

With the proliferation of parametric cost estimating models and tools, both commercial models and “home grown” versions, it is impossible to describe what to look for in every model and cost parameter. However, some generalizations can be made. The following is a summary, since most of this has already been covered in the Handbook. As a cost estimator or analyst, knowing and understanding the audit guidance described in this chapter is a good start. If you follow the steps delineated earlier in this chapter, the cost analysis or audit battle is more than half won. If, as an analyst, you are confronted with a parametric cost estimate, you should take a few steps to ensure a fair review. These are : ♦ Understand the cost model used. Do not hesitate to ask questions and to consult with the experts.

Most commercial model builders welcome calls from users, analysts and

auditors. You do not have to have a user’s understanding of the model, just a general knowledge about how it works. Remember, there’s no such thing as a stupid question. ♦ Review the program inputs to the model. Is the schedule correct? Does the WBS adequately describe the product being estimated?

Is anything missing?

Are there

duplications? Does the WBS follow the statement of work? ♦ Review the technical inputs to the model with your own engineers or technical department, and your program office. Check them for reasonableness and benchmark them against your own experience and the experience of the resident experts. ♦ Understand the model’s cost drivers. Generally, there are a few select parameters that are the predominate cost estimating factors that drive cost. Many of the others are just “tweaks” to the model. Concentrate on the cost drivers when performing your analysis. Your may wish to combine the other minor “environmental” factors together as one factor. ♦ Be aware of the assumptions the cost estimator made when he/she built the model. Are they reasonable? Would you have made the same assumptions? If not, why not? ♦ Be knowledgeable of the historical cost basis for the model, if any. Be sure to review the source documentation. Be wary of any model used that has no basis in historical cost.

154

Parametric Cost Estimating Handbook

How was the data “tuned” or normalized? Were data outliers disregarded? If so, why? Was the model calibrated? How was the calibration performed? Were any universal or “standard” cost factors used? Would they be applicable in this case? ♦ Question how the future environment might be different from the historical case. Have these differences been accounted for? Would you account for them differently? If so, how? ♦ Is the estimated program expected to push the state of the technology (state of the art)? How is this planned to be accomplished? How is it accounted for in the estimate? Is it seriously considered, or glossed over? Advances in the state of the art normally have significant impacts on the cost drivers. Does this show up in the parametric estimate, or are the historical factors used without adjustment? It is important to use technical experts on this item. For instance, if a contractor proposes using a technology that has yet to be proven in the laboratory, program trouble may lie ahead and result in inadequate estimates. Consult with your technical people on this and related technical concerns. ♦ Review the track record of the estimator or the estimating organization. What is their past performance? Have their estimates been reliable? ♦ Understand the economics involved with the model. What are the effective costing rates used? Are they reasonable? Do they reflect the organization and skill levels being estimated? ♦ Identify what cost factors have been “tweaked” and why. Focus on the “big ticket” items. Using expert opinion and “rules of thumb,” are any significant cost factors outside the range of reasonableness? For instance, it is very easy to calibrate a software cost model’s cost (hours or dollars) per line of software code. Is the CER reasonable for the estimate? However, different models may define a software line of code differently. It is therefore vital to understand the definitions used in the model you are evaluating. If the model violates your concept of reasonableness, then question it. ♦ Last, don’t be intimidated. Most (especially commercial) models are quite user friendly and the model builders are helpful. Other people have learned how to use them, and you are as smart as they are. The most difficult part is learning the language and gaining the experience over time to develop your skills.

155

Parametric Cost Estimating Handbook

Rules Of Thumb Rules of thumb are general rules, CER’s or other cost factors that are widely used, but do not apply precisely to and specific case. Each estimating organization will have its own peculiar set. Most long time estimators can develop a rapid estimate (a ROM) based on their own personal rules of thumb. These rules come from their own experience. Examples of rules of thumb are the “default” factors that populate many parametric models including the commercial ones. If no factor input is available, the model uses a universally derived factor - one taken from a universal data base or expert opinion. Although there is nothing wrong with the use of these rules per se, the danger is evident. The rules are only norms or benchmarks, and they will never apply to a specific estimate. Their value lies in the comparison of a factor in a model to the universal case. Too much deviation should be investigated. Why is the actual case so different from the rule? Or, stated differently, why doesn’t the rule apply here? For instance, if you believe that a line of software code (for a particular type of software) should take about one-half hour to write (on average), and the parametric model indicates two hours, then some investigation is in order. Although either CER could be the correct one, there is far too much deviation from the norm, and suspicions are aroused. However, if the model indicates six tenths of an hour that compares to your one-half hour, the rule had performed a “sanity” check for you. But the use of the rule does not preclude a thorough cost analysis if you have identified a cost driver. Rules of thumb are important, but they must be used wisely. Develop your own list. This can be accomplished through experience, or by consulting with experts in the field. Many rules exist embedded within commercial parametric models, and are available to the users and modelers. In any event, it is important to have a list of benchmark factors. You’ll probably need a different list for each organization you deal with. Organizational and product differences will demand it. But remember that there are no shortcuts or magic formulas, so use your rules as a quality check, or for quick, rough order of magnitude estimates.

An Example

156

Parametric Cost Estimating Handbook

What follows is an example of some of the above discussion. The example here uses the software cost estimating model, REVIC. The fundamental REVIC equation is: Man Months = A*(KDSI)B*ΠFi, where, A and B are coefficients of several Software Development Modes (for instance, embedded , organic, Ada); KDSI is Thousand Delivered Source Instructions; and; ΠFi is the product of various environmental factors the model uses to adjust the cost estimates. These environmental factors include, for example: analyst and programmer capabilities, applications experience, programming language, storage constraints, requirements volatility, reliability requirements, data base size, product complexity, the use of modern programming practices and software tools, platform (airborne, space, etc.), schedule compression, etc. Each factor is given a factor value from very low through nominal to high.

Suppose you were given the following cost estimate: REVIC = IMS EP 1 = 3.312(DSI/1000)1.2*ΠFi IMS EP 1 = (3.312)(1337/1000)1.2*.874 = 4.1MM

What would you do with it? The basis of estimate (BOE) tells you this: “The Integrated Management System Evaluation Package for Release 1 (IMS EP 1) requires 1337 new lines of embedded source code of new algorithms and uses complex interfaces.

Our superior

programming staff using state-of-the-art software tools, modern programming practices, and possessing significant application experience allows the product of the environmental factors to be less (13%) than nominal.” What do you notice about this estimate? First, this particular estimate is relatively small (4.1MM). You may wish to move on to the more important cost issues. However, you decide to “spot check” this one. You know very little about REVIC, so you turn to the resident expert for advice. He/She confirms that the technical description in the BOE is correct, so the model’s values for A and B are the correct

157

Parametric Cost Estimating Handbook

ones. You make a quick calculation to compare the estimate with a rule of thumb supplied by your resident expert. 4.1MM * 155hrs./MM ÷ 1337DSI = .475hrs/DSI

Your resident expert informs you that he believes this factor to be quite low, possibly two times. He also indicates that the DSI number needs to be verified for accuracy, since it is a major cost driver for this estimate.

He is also suspicious of the “....less (13%) than nominal”

environmental factors product. (Nominal here he believes should equal at least 1.5 due to a partial space platform involved.) Therefore 13% appears to be a mistake of some kind, but it will have to be investigated. He also argues that since this is a competitive proposal that’s almost 80% software, all software estimates may be “aggressive,” and should be carefully reviewed. The model does not appear to have been calibrated except for the vague references “....superior programming staff....” in the BOE. The product of the environmental factors seems to be based upon judgment, although likely an educated one. You decide to carefully review the other REVIC cost estimates in the proposal, so you borrow a REVIC User’s Guide in order to become more proficient. Although this is a relatively easy example, it demonstrates the general idea of what to look for in a parametric estimate.

158

Parametric Cost Estimating Handbook

CHAPTER VII

BUSINESS APPLICATIONS OF PARAMETRIC ESTIMATING

159

Parametric Cost Estimating Handbook

CHAPTER VII BUSINESS APPLICATIONS OF PARAMETRIC ESTIMATING

INTRODUCTION

Recent initiatives affecting government procurement practices have further stressed an already fragile world-wide estimating capability.

Competition, accuracy, flexibility, and

method-proven credibility dictate reformation in the estimating process, especially for new business pursuits. Parametric estimating methods for planning and strategizing new business cost issues is a critical element in the reform of traditional bottoms-up approaches. A paper written by Roy Summers and Bruce Fad summarized here discusses the need for parametric modeling in the new business development process. The arguments are based on experience gained through use of parametrics in this type environment. The role of parametric modeling is examined along with the presentation of a process in which parametrics and bottoms-up estimating are complementary. Organizational conditions leading to effective use of parametrics are also considered. The questions of job responsibilities, management interface, and functional structures are treated from the practitioners' viewpoint. Finally, the issue of credibility includes perspectives from both the using and auditing organizations, with emphasis on calibration and the use of testable databases.

PARAMETRIC ESTIMATING IN NEW BUSINESS DEVELOPMENT

Why do programs die? Why do CEO's, Boards of Directors, DoD, or Congress kill or cancel technically superior projects? There are many examples, and, generally, it is not due to technical viability. Even though millions or even billions are lost, the program graveyard claims many projects. Sommers and Fad suggest that these events happen partly because of an inattention to future cost issues during program development. There is a need to recognize and react to help solve the cost problems. Technical success can often mask future cost issues.

160

Parametric Cost Estimating Handbook

The authors propose that an organization responsible for costing use parametric techniques early on, and that cost and technical objectives be treated together. Parametrics are perfect for cost estimating during a program's conceptual phase, and the tools connect technical and cost parameters together in a very effective way. New business development actions can attack the problem. These actions include: 1) Parallel and coordinated development of technical and programmatic (cost, schedule and quantity) concepts from requirements. 2) Iterative and continuous cost improvement 3) Develop cost targets (bogeys) as the result of 1 & 2, above. 4) A review/reconciliation process that focuses on cost/risk analysis.

The authors go on to suggest that organizations not put off the cost estimating activity, and that cost analysts immediately begin to explore technical/cost alternatives. They also suggest that the parametric function belong to an independent department and not reside within another function outside the organization that owns new business development. To effectively integrate parametrics into an organization, upper management needs to be involved. Upper management involvement will insure the creation of a cooperative team spirit, and more effective management and customer acceptance including estimate credibility.

ESTIMATING PRODUCTION BUYS USING PARAMETRICS

A parametric approach can be used to calibrate production cost and then estimate future production buy(s) and options using a commercial model. The benefit here is a simpler cost proposal with no bill of material and no labor hours roll-up. The model can be calibrated and estimates derived at any level of the WBS, hence useful for spare parts pricing. In either case, the CERs or cost models can be used as the basis of estimate, and delivered to a proposal pricing system via a post processor, or a spread sheet. In either event, the cost estimating process is greatly simplified, and resource economies of scale can be realized.

Example: Estimating Production Buys Using Parametrics

161

Parametric Cost Estimating Handbook

What follows is an example of the comparative or analogous estimating approach.

RFQ And Historical Data PDQ Inc., has received an RFQ for 84 XYZ-L Systems to be built and delivered in 1995. A 120-lot of this same system was delivered in 1993, so recent cost history exists. The cost history yields the following data:

Manufacturing labor (Hands-on) of 42,000 hours divided into: Plant A - 18,000 hours Plant B - 12,000 hours Plant C - 12,000 hours Purchased parts: $9,000,000

Manufacturing Support Labor: 17,280 hours which was generated from 1 Mechanical Engineer, 1 Quality Engineer, and 1 Test Engineer at each of the three plant locations.

Engineering Support Labor: 7,680 hours generated by 2 Engineers at Plant A, and 1 Engineer each at Plants B and C. Tooling Costs: $15,000 Test Equipment Costs: $25,000

The Interview Process How will the proposed 84-lot be different from the historical (baseline) 120-lot? VECP - AB123 has altered the system configuration.

Mechanical Engineering has

estimated the VECP will reduce the assembly time at Plant A by 5 hours per system. Plant B is over-capacity so five of the parts in the system will be resourced to Plant C. These parts are similar in assembly and take 3 hours each to build. Part number XYZ456, previously assembled at Plant A, is being purchased from an outside vendor. Assembly time for this part at Plant A was 4 hours. It is currently being purchased for $200.

162

Parametric Cost Estimating Handbook

Preparation Of The Estimate Manufacturing Labor Calculations: Plant A:

Plant B:

150 Hours

Historical Baseline unitized

-5 Hours

Adjustment for VECP-AB123

-4 Hours

Make to Buy Change for P/N XYZ456

141 Hours

Proposed Unit Hours Estimate for 84-lot

100 Hours

Historical Baseline unitized

-15 Hours* Five parts (@ 3 hours per part) resourced to Plant C 85

Plant C:

*

Hours

100

Hours

Proposed Unit Hours Estimate for 84-lot

Historical Baseline unitized

+17.5 Hours

Five parts resourced from Plant B

117.5 Hours

Proposed Unit Hours Estimate for 84-lot

The 15 hours for the 5 parts were adjusted to account for performance differences between Plants B and C. Performance factor (inverse of efficiency) at Plant B was 1.5. The current performance factor at Plant C is 1.75. Adjustment: 15 hours divided by 1.5 times 1.75 equals 17.5.

Purchased Parts Calculations: $75,000

Baseline Costs unitized (purchased 50% in 1992 and 50% in 1993)

Material for 84-lot will be purchased in 1994. Escalation is 3% per year. $39,784

Baseline (50%) escalated from 1992 to 1994

$38,625

Baseline (50%) escalated from 1993 to 1994

$200 $78,609

Make to Buy Change for P/N XYZ456 (not in baseline) Proposed Unit Base Estimate for 84-lot

Manufacturing Support Labor Calculations: Support labor has historically followed a semi-variable pattern; i.e., when the production quantity is halved, support can be reduced by 25%, or when production is doubled, support cost is increased by 50%.

163

Parametric Cost Estimating Handbook

The proposed 84-lot is 60% of a halving of the baseline 120-lot. Therefore, the support labor adjustment is -15% (60% X 25% = 15%) . 9.00

People Baseline

7.65

People Baseline Adjusted (-15%)

14,688

Hours Proposed support labor @ 1920 hours per person/year

Engineering Support Labor Calculations: 4

People Baseline

3

People Adjusted Baseline (Management Challenge)

5,760 Hours Proposed engineering labor @ 1920 hours per person/year

Tooling And Test Equipment Calculations: $40,000

Baseline costs

$ 1,200

Adjusted for escalation from 1994 to 1995 (3% per year)

$ 4,120

Adjusted for age of equipment

$45,320

Proposed tooling and test equipment costs

Summary Of Direct Costs Manufacturing Labor: Plant A

141.0 Hours X 84 Systems X $75.80 per Hr.

= $ 897,775

Plant B

85.0 Hours X 84 Systems X $81.50 per Hr.

= $ 581,910

Plant C

117.5 Hours X 84 Systems X $77.10 per Hr.

= $ 760,977

Subtotal Manufacturing Labor

= $2,240,662

Purchased Parts: Base

$78,609 X 84 Systems

= $6,603,156

Normal Material Production Allowance (NMPA) 5.0%

= $330,158

164

Parametric Cost Estimating Handbook

Raw Material 2.5%

= $165,079

Material Overhead (Applied to Purchase Parts Only) 8.0%

= $528,252

Sub-Total Purchased Parts:

= $7,626,645

Manufacturing Support: 14,688 Hours X $74.75 per Hr

= $1,097,928

Engineering Support:

= $478,080

5,760 Hours X $83.00 per Hr.

Tooling & Test Equipment:

=

TOTAL DIRECT COSTS:

$45,320

$11,488,635

ESTIMATING SPARES AND CHANGE ORDERS

Parametric techniques can sometimes be used for estimating spares and change order proposals. Using a calibrated system model for spares, select the WBS element that represents the spare part(s) that need to be estimated. Then exercise the model at that point, either a commercial version or one developed for the specific program. The estimate can be exercised very quickly, and the calibrated model will yield accurate results. Engineering Change Proposals (VECPs) or other change proposals have to be accounted for by adjusting the actuals or a delta to the current estimate. The comparative approach is normally well supported by history, as accurate as any other technique. It is much easier to do "what if" analysis and compute option prices, and the proposals are less complex and smaller in volume and more "user friendly", with significantly reduced preparation, review, audit and negotiation time. Table VII-1 indicates the potential parametric applicability for various firm business proposals. Appendix G contains a parametric estimating system checklist.

PARAMETRIC ESTIMATING APPLICATIONS FOR FIRM - BUSINESS PROPOSALS

165

Parametric Cost Estimating Handbook

Type of Proposal

Engineering

Manufacturing

Test

Support

1.

New Business Development

VA

VA

VA

VA

2.

Production

VA

VA

VA

VA

3.

Follow-on Production

VA

VA

VA

VA

4.

Development Contract Change-Order

NVA

NVA

NVA

VA

5.

Production Contract Change-Order

NVA

NVA

NVA

VA

6.

Spares

MBA

MBA

MBA

VA

NOTE: Use of parametric estimating may be very applicable for new business development proposals/RFPs; their application may not be as good for unique development contract change orders with little, if any, relevant history to use for CER development or Cost Model calibration. In addition, change order data is collected at too low a level in the WBS for meaningful utilization.

KEY: VA = Very Applicable; NVA = Not Very Applicable; MBA = May Be Applicable

TABLE VII - 1

166

Parametric Cost Estimating Handbook

APPENDIX A

DEFINITIONS OF ESTIMATING TERMINOLOGY

167

Parametric Cost Estimating Handbook

APPENDIX A DEFINITIONS OF ESTIMATING TERMINOLOGY

Algorithmic Models - (also known as Parametric models) produce a cost estimate using one or more mathematical algorithms using a number of variables considered to be the major cost drivers. These models estimate effort or cost based primarily on the Hardware/Software size, and other productivity factors known as cost driver attributes.

Analogy - A method of estimating developed on the basis of similarities between two or more programs, systems, items, etc.

Analogy Models - use a method of estimating that involves comparing a proposed project with one or more similar and completed projects where costs and schedules are known. Then, extrapolating from the actual costs of completed projects, the model(s) estimates the cost of a proposed project.

Analometric - A recent term meaning a method of combining the analogy and parametric estimating methods to form a cost estimate when only 2 relevant data points are available. It is usually combined with the "D" Factor to adjust (up or down) complexity from the 2 point CER regression line.

Analysis - decision making context involving time horizons extending into the future.

A

concrete set of specifications are not usually available. There could be many major uncertainties, and a wide range of alternatives, each having several configurations.

Analysis is usually

concerned with new equipment proposals and new methods of operation of systems never produced before. Analysis Objectives are to find significant differences in resource requirements among alternatives; and, how will resource requirements for any alternative change as key

168

Parametric Cost Estimating Handbook

configuration characteristics vary over their relevant ranges. It is often a "sensitivity" type of investigation.

Analysis (NES Dictionary) - a systematic approach to problem solving. Complex problems are simplified by separating them into more understandable elements.

Annual Change Traffic (ACT) - the fraction of a software product's source instructions which undergoes change during a year, either through addition or modification. The ACT is the quantity used to determine the product size for software maintenance effort estimation.

Anomalies - variances in cost related data caused by an unusual event(s) not expected to recur in the future.

As Spent Dollars - the cost, in real year dollars, of a project recorded as the dollars were spent without normalization for inflation.

Audit - the systematic examination of records and documents and the securing of evidence by confirmation, physical inspection, or examination.

Audit Trail - information allowing the data being used in an estimate to be tracked back to the original source for verification.

Benefit (NES Dictionary) - Result(s) attained in terms of the goal or objective rather than in terms of output.

Benefit Cost Analysis (NES Dictionary) - an analytical approach to solving problems of choice. It requires: (a) the definition of objectives; (b) identification of alternative ways of achieving each objective; and (c) the identification for each objective or alternative which yields the required level of benefits at the lowest cost. It is often referred to as cost-effectiveness analysis when the benefits of the alternatives cannot be quantified in terms of dollars.

169

Parametric Cost Estimating Handbook

Benefits (NES Dictionary) - advantages which may be quantifiable or non-quantifiable. For example, lowest cost is a benefit, so is the best accuracy and longest range.

Bottom-Up Models - use a method of estimation that estimates each component of the project separately, and the results are combined ("Rolled Up") to produce an estimate of the entire project.

Calibration - in terms of a cost model, a technique used to allow application of a general model to a specific set of data. This is accomplished by calculating adjustment factor(s) to compensate for differences between the referenced historical costs and the costs predicted by the cost model using default values.

Computer-Aided Software Engineering: CASE - identifies a sector of the computer software industry concerned with producing software development environments and tools. The main components of a CASE product are individual tools which aid the software developer or project manager during one or more phases of software development (or maintenance). Other features are a common user interface; interoperability of tools; and a repository or encyclopedia to provide a common tool base and central project database. CASE may also provide for code generation.

Configuration Item - hardware or software, or an aggregate of both, which is designated by the project configuration manager (or contracting agency) for configuration management.

Configuration Management - a discipline applying technical and administrative controls to (1) identification and documentation of physical and functional characteristics of configuration items; (2) any changes to characteristics of those configuration items; and (3) recording and reporting of change processing and implementation of the system.

170

Parametric Cost Estimating Handbook

Constant Dollars - Computed values which remove the effect of price changes over time (inflation). An estimate is said to be in constant dollars if costs for all work are adjusted to reflect the level of prices of a base year.

Contract Work Breakdown Structure - Contract work breakdown structure is defined as the complete work breakdown structure for a contract, i.e., the DoD approved work breakdown structure for reporting purposes and its discretionary extension to the lower levels by the contractor, in accordance with this standard and the contract work statement.

Cost (Fisher) - "Economic costs" are benefits lost. It is for this reason that economic costs are often referred to as "alternative costs" or "opportunity costs." It is in alternatives, it is in foregone opportunities, that the real meaning of "cost" must always be found. The only reason you hesitate to spend a dollar, incidentally, is because of the alternative things that it could buy. Some use the word "cost" when referring to resources. Cost of something is measured by the resources used to attain it. Cost of attaining an objective at some point in time is measured by the resources not available for use in attaining alternative objectives. Costs are a measure of other defense capabilities foregone. Money cost is not necessarily the same as economic cost. "Economic cost" implies the use of resources - manpower, raw materials, etc. Dollars are used merely as a convenient common denominator for aggregating numerous heterogeneous physical quantities into meaningful "packages" for purposes of analysis and decision mating.

Cost (NES Dictionary) - the amount paid or payable for the acquisition of materials, property, or services. In contract and proposal usage, denotes dollars and amounts exclusive of fee or profit. Also used with descriptive adjectives such as “acquisition cost," or "product cost," etc. Although dollars normally are used as the unit of measure, the broad definition of cost equates to economic resources, i.e., manpower, equipment, real facilities, supplies, and all other resources necessary for weapon, project, program, or agency support systems and activities.

Cost Analysis (NES Dictionary) - the accumulation and analysis of actual costs, statistical data, and other information on current and completed contracts or groups of contracts (programs).

171

Parametric Cost Estimating Handbook

Cost analysis also includes the extrapolation of these cost data to completion, comparisons and analyses of these data, and cost extrapolations on a current contract with the cost data in the contract value for reports to customers, program and functional managers, and price estimators. In the procurement organizations of the Department of Defense, cost analysis is the review and evaluation of a contractor's cost or pricing data and of the judgmental factors applied projecting from the data to the estimated costs, in order to form an opinion on the degree to which the contractor's proposed costs represent what performance of the contract should cost, assuming reasonable economy and efficiency.

Cost Analysis (Large) - the primary purpose of cost analysis is comparison - to provide estimates of the comparative or relative costs of competing systems, not to forecast precisely accurate costs suitable for budget administration. In this context consistency of method is just as important, perhaps more so, as accuracy in some absolute sense. In comparing the costs of military systems, we prefer to speak of "cost analysis" rather than "cost estimation," because the identification of the appropriate elements of cost -- the analytical breakdown of many complex interrelated activities and equipment -- is so important a part of the method. Weapon system cost analysis is much more than an estimate of the cost of the weapon itself. Weapon procurement costs may be relatively small compared to other necessary costs, such as base facilities, training of personnel, and operating expenses; and these other costs may vary greatly from system to system.

Cost Benefit Analysis (NES Dictionary) - a technique for assessing the range of costs and benefits associated with a given option, usually to determine feasibility. Costs are generally in monetary terms.

Cost Driver Attributes - productivity factors in the software product development process that include software product attributes, computer attributes, personnel attributes, and project attributes.

172

Parametric Cost Estimating Handbook

Cost Drivers - The controllable system design or planning characteristics that have a predominant effect on the system's costs. Those few items, using Pareto's law, that have the most significant cost impact.

Cost Effectiveness (NES Dictionary) - the measure of the benefits to be derived from a system with cost as a primary or one of the primary measures.

Cost Effectiveness Analysis (NES Dictionary) - a method for examining alternative means of accomplishing a desired military objective/mission for the purpose of selecting weapons and forces which will provide the greatest military effectiveness for the cost.

Cost Estimating (Fisher) - "making an estimate" of the cost of something implies taking a rather detailed set of specifications and "pricing them out."

Cost Estimating (NES Dictionary) - cost and price estimating is defined as the art of predetermining the lowest realistic cost and price of an item or activity which assure a normal profit.

Cost Estimating Relationship - An algorithm relating the cost of an element to physical or functional characteristics of that cost element or a separate cost element; or relating the cost of one cost element to the cost of another element.

Cost Estimating Relationships: CER - a mathematical expression which describes, for predicative purposes, the cost of an item or activity as a function of one or more independent variables.

Cost Factor - a brief arithmetic expression wherein cost is determined by the application of a factor as a proportion.

173

Parametric Cost Estimating Handbook

Cost Model - an estimating tool consisting of one or more cost estimating relationships, estimating methodologies, or estimating techniques used to predict the cost of a system or one of its lower level elements.

Cost/Schedule Control System Criteria: C/SCSC - a set of criteria specified by the Federal Government for reporting project schedule and financial information.

Current Dollars - Level of costs in the year actual cost will be incurred. When prior costs are stated in current dollars, the figures given are the actual amounts paid. When future costs are stated in current dollars, the figures given are the actual amounts expected to be paid including any amount due to future price changes.

Deflators - The de-escalation factors used to adjust current cost/price to an earlier base year for comparison purposes. A deflator is the inverse of an escalator.

Delivered Source Instructions: DSIs - the number of source lines of code developed by the project. The number of DSIs is the primary input to many software cost estimating tools. The term DELIVERED is generally meant to exclude non-delivered support software such as test drivers.

The term SOURCE INSTRUCTIONS includes all program instructions created by

project personnel and proceed into machine code by some combination of preprocessors, compilers, and assemblers. It excludes comments and unmodified utility software. It includes job control language, format statements, and data declarations.

Delphi Technique - a group forecasting technique, generally used for future events such as technological developments, that uses estimates from experts and feedback summaries of these estimates for additional estimates by these experts until a reasonable consensus occurs. It has been used in various software cost-estimating activities, including estimation of factors influencing software costs.

174

Parametric Cost Estimating Handbook

Detail Estimating: Grass Roots, Bottom-Up - The logical buildup of estimated hours and material by use of blue-prints, production planning tickets, or other data whereby each operation is assigned a time value.

Design to Cost: DTC - An acquisition management technique to achieve defense system designs that meet stated cost requirements. Cost is addressed on a continuing basis as part of a system's development and production process. The technique embodies early establishment of realistic but rigorous cost objectives, goals, and olds and thresholds and a determined effort to achieve them. Cost, as a key design parameter, is addressed on a continuing basis and as an inherent part of the development and production process. Cost goals are established early in the development phase, which reflect the best balance among life cycle cost, acceptable performance and schedule.

DOD-STD-2167A/498 - a US Department of Defense standard that specifies the overall process for the development and documentation of mission-critical software systems.

DOD-STD-

2167A/498 has direct connections to five other standards that are not software specific. These five standards are as follows: MIL-STD-1521

Technical Reviews and Audit for Systems, Equipments, and Computer Software

MIL-STD-480

Configuration Control - Engineering Changes,

Deviations,

and

Waivers MIL-STD-481

Configuration Control - Engineering Changes, Deviations, and Waivers (Short Form)

MIL-STD-490

Specification Practices

MIL-STD-499

Engineering Management

MIL-STD-480 and 481 have been superseded by MIL-STD-973.

MIL-STD-973

consolidates several earlier standards for configuration management and resolves inconsistencies and ambiguities that existed between them. When DOD-STD-2167A/498 is applied to contract, appropriate tailoring instructions should be included to indicate that the contract is to comply with the more recent MIL-STD-973.

175

Parametric Cost Estimating Handbook

Domain - a specific phase or area of the software life cycle in which a developer works. Domains define developers and users areas of responsibility and the scope of possible relationships between products.

The work can be organized by domains such as Software

Engineering Environments, Documentation, Project Management, etc.

DTC Goals - A cost bogey, in constant dollars, based upon a specified production quantity and rate, established early during system development as a management objective and design parameter for subsequent phases of the acquisition cycle. (see DTC Target).

DTC Parameter - Approved measurable values for selected cost elements established during system development as design considerations and management objectives for subsequent life cycle phases. A DTC parameter may be an objective, goal, or threshold. Values will be expressed it constant dollars, resources required, or other measurable factor(s) that influences costs.

DTC Targets - Approved DTC parameters divided into smaller. identifiable tasks or areas of responsibility that serve as requisites for contractors or government activities.

DTC Threshold - Costs or values that, if exceeded, will cause a program review.

Economics (Hitch and McRean) - economics is concerned with allocating resources. Economizing always means trying to make the most efficient use of the resources available.

Economics Analysis (Hitch and McRean) - economic analysis - ranging from just straight thinking about alternative courses of action to systematic quantitative comparisons, to identify or achieve near "optimal" solutions.

Economic Analysis (DoDI 7041.3) - a systematic approach to the problem of choosing how to employ scarce resources and an investigation of the full implications of achieving a given objective in the most efficient and effective manner.

176

Parametric Cost Estimating Handbook

Economic Analysis (NES Dictionary) - a systematic approach to a given problem, designed to assist the manager in solving a problem of choice. The full problem is investigated; objectives and alternatives are searched out and compared in light of their benefits and costs through the use of an appropriate analytical framework. Often used to determine the best use of scarce resources.

Effectiveness (DoDI 7041.3) - the performance or output received from an approach or program.

Effort Adjustment Factor: EAF - a term used in COCOMO to calculate the cost driver attribute's effect on the project. It is the product of the effort multipliers corresponding to each of the cost drivers for the project.

Embedded Node - a term used by COCOMO to describe a project development that is characterized by tight, inflexible constraints and interface requirements. The product must operate within (is embedded in) a strongly coupled complex of hardware, software, regulations and operational procedures. An embedded mode project will require a great deal of innovation. An example would be a real-time system with timing constraints and customized hardware.

Equivalent Units - the number, or fraction, of completed units at a given time.

Estimating (NES Dictionary) - to predict costs. Generation of detailed and realistic forecasts of hours, material costs, or other requirements for a task, subtask, operation, or a part or groups thereof - generally in response to a request for proposal or specific statement of work. The art of approximating the probable worth (or cost), extent, quantity, quality, or character of something based on information available at the time. It also covers the generation of forecasted costing rates and factors with which estimate numbers may be converted to costs, and the application of these costing rates and factors to "estimate numbers" to establish forecasted costs.

Expert Judgment Models - use a method of software estimation that is based on consultation with one or more experts that have experience with similar projects. An expert-consensus mechanism such as the DELPHI TECHNIQUE may be used to produce the estimate.

177

Parametric Cost Estimating Handbook

Fixed Year Dollars - dollars that have been adjusted for inflation to a specific year.

Historical Data - a term used to describe a set of data reflecting actual cost or past experience of a product or process.

Improvement Curve - a graphical representation of improvement curve theory that projects resource requirements versus the number of units produced. Examples include labor hours, labor cost, direct cost including labor and materials. When reflecting only labor hours the curve is usually called a learning curve. When depicting cost, it is normally referred to as a Cost Improvement Curve.

Inflation - an increase in the level of prices for the same item(s). Examples of indices that reflect inflation include Consumer Price Index (CPI), Wholesale Price Index, and Producer Price Index. When prices decline it is called deflation.

Knowledge Base - the repository of knowledge in a computer system or organization. The collection of data, rules, and processes that are used to control a system, especially one using artificial intelligence or expert system methods.

Life Cycle - the stages and process through which hardware or software passes during its development and operational use. The useful life of a system. Its length depends on the nature and volatility of the business, as well as the software development tools used to generate the databases and applications.

Management Information Systems - a computer based system of processing and organizing information that provides different levels of management within an organization with accurate and timely information needed for supervising activities, tracking progress, making decisions, and isolating and solving problems.

178

Parametric Cost Estimating Handbook

Metric - Quantitative analysis values calculated according to a precise definition and used to establish comparative aspects of development progress, quality assessment or choice of options.

Normalize - to adjust data (normally cost data) for effects like inflation, anomalies, seasonal patterns, technology changes, accounting system changes, reorganizations, etc.

Operation (Philip Morse: Notes on Operations Research, MIT Press 1962) - an operation is a pattern of activity of men or of men and machines, engaged in carrying out a cooperative and usually repetitive task, with pre-set goals and according to specified rules of operation.

Operational Research (Operational Research Quarterly, Vol. 13, No. 3, 282 (Sept. 1962)) operational research is the attack of modern science on complex problems arising in the direction and management of large systems of men, machines, materials, and money in industry, business, government, and defense. The distinctive approach is to develop a scientific model of the system, incorporating measurements of factors such as chance and risk, with which to predict and compare the outcomes of alternative decisions, strategies, or controls. The purpose is to help management determine its policies and actions scientifically.

Operations Research (Norse, same as above) - the scientific study of operations.

Organic Mode - a term used by COCOMO to describe a project that is developed in a familiar, stable environment. The product is similar to previously developed products. Most people connected with the project have extensive experience in working with related systems and have a thorough understanding of the project. The project contains a minimum of innovative data processing architectures or algorithms. The product requires little innovation and is relatively small, rarely greater than 50,000 DSIs.

Paradigm - a model, example, or pattern. A generally accepted way of thinking.

179

Parametric Cost Estimating Handbook

Parametric Cost Estimate - Estimate derived from statistical correlation of historic system costs with performance and/or physical attributes of the system.

Parametric Cost Estimating Methods - Estimating methods based on physical or performance characteristics and schedules of the end items.

Parametric Cost Model - A mathematical representation of parametric cost estimating relationships that provides a logical and predictable correlation between the physical or functional characteristics of a system, and the resultant cost of the system. A parametric cost model is an estimating system comprising cost estimating relationships (CERs) and other parametric estimating functions, e.g., cost quantity relationships, inflation factors, staff skills, schedules, etc. Parametric cost models yield product or service costs at designated levels and may provide departmentalized breakdown of generic cost elements. A parametric cost model provides a logical and repeatable relationship between input variables and resultant costs.

Parametric Estimating - a mathematical procedure where product or service descriptors (parameters) and cost algorithms directly yield consistent cost information.

Platform - hardware or software architecture of a particular model or family of computers. The term sometimes refers to the hardware and its operating system.

Procedures - manual procedures are human tasks. Machine procedures are lists of routines or programs to be executed, such as described by the Job Control Language (JCL) in a mini or mainframe, or the batch processing language in a personal computer.

Process - the sequence of activities (in software development) described in terms of the user roles, user tasks, rules, events, work products, resource use, and the relationships between them. It may include the specific design methodology, language, documentation standards, etc.

Production Rate - the number of items produced in a given time period (e.g., items/month).

180

Parametric Cost Estimating Handbook

Program Evaluation (DoDI 7041.3) - is the economic analysis of on-going actions to determine how best to improve an approved program/project based on actual performance.

Program

evaluation studies entail a comparison of actual performance with the approved program/project.

Program Evaluation and Review Technique: PERT - a method used to size a software or hardware product and calculate the Standard Deviation (SD) for risk assessment. For example, the PERT equation (beta distribution) estimates the Equivalent Delivered Source Instructions (EDBIs) and the SD based on the analyst's estimates of the lowest possible size, the most likely size, and the highest possible size of each Computer Program Component (CPC).

Program Work Breakdown Structure - A program work breakdown structure covers the entire acquisition cycle of a program, consists of at least the first three levels of a work breakdown structure, as prescribed by MIL-STD-881B, and its extension by the DoD Component and/or contractor(s).

The program work breakdown structure has uniform element terminology,

definition, and placement in the family-tree structure.

Rapid Prototyping - the creation of a working model of a software module to demonstrate the feasibility of the function. The prototype is later refined for inclusion in a final product.

Rayleigh Distribution - a curve that yields a good approximation to the actual labor curves on software projects.

Real-Time - (1) Immediate response. The term may refer to fast transaction processing systems in business; however, it is normally used to refer to process control applications. For example, in avionics and space flight, real-time computers must respond instantly to signals sent to them. (2) Any electronic operation that is performed in the same time frame as its real-world counterpart. For example, it takes a fast computer to simulate complex, solid models moving on screen at the same rate they move in the real world. Real-time video transmission produces a live broadcast.

181

Parametric Cost Estimating Handbook

Re-engineering - process of restructuring and redesigning an operational (or coded) hardware or software system or process in order to make it meet certain style, structure, or performance standards.

Resource Analysis (Fisher) - systematically determining the economic resource impact of alternative proposals for future courses of action.

Resources (Fisher) - raw materials, skilled personnel, scarce materials or chemicals, etc.

Resources (NES Dictionary) - consists of facilities, equipment, management, personnel, laboratories, and scientific, technical, and manufacturing capability.

Reusability - ability to use all or the greater part of the same programming code or system design in another application.

Reuse - software development technique that allows the design and construction of reusable modules, objects, or units, that are stored in a library or database for future use in new applications. Reuse can be applied to any methodology in the construction phase, but is most effective when object oriented design methodologies are used.

Schedule - a time display of the milestone events and activities of a program or project.

Scope - a definition of how, when, where, and what a project is expected to include and accomplish.

Security - the protection from accidental or malicious access, use, modification, destruction, or disclosure. There are two aspects to security, confidentiality and integrity.

Semi-detached Mode - a term used by COCOMO to describe a project that is developed somewhere between organic and embedded. The team members have a mixture of experienced

182

Parametric Cost Estimating Handbook

and inexperienced personnel. The software to be developed has some characteristics of both organic and embedded modes. Semi-detached software can be as large as 300K DSIs.

Should Cost Estimate - An estimate of contract price which reflects reasonably achievable economy and efficiency.

It is generally accomplished by a team of procurement, contract

administration, cost analysis, audit and engineering representatives performing an in-depth analysis of cost and cost effects. Its purpose is to develop realistic cost objectives.

Software Development Life Cycle - the stages and process through which software passes during its development. This includes requirements definition, analysis, design, coding, testing, and maintenance.

Software Development Life Cycle Methodology - application of methods, rules, and postulates to the software development process to establish completeness criteria, assure an efficient process, and develop a high quality product.

Software Method - (or Software Methodology) - focuses on how to navigate through each phase of the software process model (determining data, control, or uses hierarchies; partitioning functions; and allocating requirements) and how to represent phase products (structure charts; stimulus-response threads; and state transition diagrams).

Software Tool - program that aids in the development of other software programs. It may assist the programmer in the design, code, compile, link, edit, or debug phases.

Systems Analysis (Fisher) - Systems analysis may be defined as inquiry to assist decisionmakers in choosing preferred future courses of action by (1) systematically examining and reexamining the relevant objectives and the alternative policies or strategies for achieving them; and (2) comparing quantitatively where possible the economic costs, effectiveness {benefits), and risks of alternatives.

183

Parametric Cost Estimating Handbook

Technical Non-Cost Data - engineering variables that describe the item, subsystem or system. The source of this data includes engineering documents, engineering drawings and works of a technical nature which are specified to be delivered pursuant to the contract.

Top-Down Models - use a method of estimation that estimates the overall cost and effort of the proposed project derived from global properties of the project. The total cost and schedule is partitioned into components for planning purposes.

Update - to update an estimate or CER means to utilize the most recent data to make it current, accurate and complete.

Validation - in terms of a cost model, a process used to determine whether the model selected for a particular estimate is a reliable predictor of costs for the type of system being estimated.

Work Breakdown Structure - A work breakdown structure is a product-oriented family tree, composed of hardware, software, services, data and facilities which results from system engineering efforts during the development and production of a defense material item, and which completely defines the program. A work breakdown structure displays and defines the product(s) to be developed or produced and relates the elements of work to be accomplished to each other and to the end product. MIL-STD 881B is the modern standard for developing a WBS. (See Appendix B for more information.)

Workstation - high-performance, single user microcomputer or minicomputer that has been specialized for graphics, CAD, CAE, or scientific application.

184

Parametric Cost Estimating Handbook

APPENDIX B

WORK BREAKDOWN STRUCTURE

BY: NEIL F. ALBERT

185

Parametric Cost Estimating Handbook

DEVELOPING A USEABLE WORK BREAKDOWN STRUCTURE Neil F. Albert

INTRODUCTION This paper presents a guide for preparing, understanding and presenting a Work Breakdown Structure (WBS). It discusses what determines a good work Breakdown Structure, provides a general understanding for developing a program work breakdown structure, and shows how to develop and implement a contract work breakdown structure. The primary objective of the paper is to achieve consistent application of work breakdown structures using MIL-STD-881 as a background source. This paper is directed primarily at preparation of a defense program work breakdown structure. However, the guidance is also appropriate for use with any work breakdown structure developed at any phase during the acquisition process including concept exploration and definition, demonstration and validation, engineering and manufacturing developments and production phases. The guidelines are directed at both contractors and Government Activities in the development of work breakdown structures for acquisition of defense material items. DEFINITIONS Several definitions are critical to the discussion in the following sections. These definitions are: Program World Breakdown Structure. The Program Work Breakdown Structure is the structure that encompasses an entire program at a summary level. The program work breakdown structure consists of at least three levels of the program with associated definitions and is used by the Government Activity and contractors to develop and extend a Contract Work Breakdown Structure. The upper three levels of the program work breakdown structure would typically be defined as follows: Level 1: Level 1 is the entire defense material item; for example, electronic system refers to an electronics capability such as a command and control system, radar system, communications system, information system, sensor system, navigation/guidance system, electronic warfare system, etc. Level 1 is usually directly identified as a major program or as a project or subprogram within an aggregated program. Level 2: Level 2 elements are major elements of the defense material item; for example, the prime mission product which includes all hardware and software elements, aggregations of system level services (e.g., system test and evaluation, and system engineering/program management) and of data. Level 3: Level 3 elements are elements subordinate to level 7 major elements; for example, radar data processor, signal processor, antenna, type of service (e.g. development test and

186

Parametric Cost Estimating Handbook

evaluation, contractor technical support, training services), or types of data (e.g., technical publications). Lower levels follow the same process. Contract Work Breakdown Structure. Contract work breakdown structure is the Government approved work breakdown structure for reporting purposes and its discretionary extension to the lower levels by the contractor, in accordance with Government direction and the contract work statement. It Includes all the elements for the products (hardware, software, data, or services) which are the responsibility of the contractor. The contract work statement will provide the reporting requirements for each contract world breakdown structure element for which contract status is to be reported to the Government. Normally, the contractor extends each level of the work breakdown structure to the point where manageable size increments of work are defined. The contract work breakdown structure should provide a consistent visible framework that facilitates uniform planning, assignment of responsibility, and reporting of status. Common WBS elements. Common WBS elements are applicable to all types of systems. Common WBS elements include: - Integration, Assembly, Test and Checkout. Integration, assembly, test and checkout element includes all effort required to assemble the level 3 equipment (hardware/software) elements into a level 2 mission equipment (hardware software) as a whole and not directly part of any other individual level 3 element. - System Engineering/Program Management, - System Test and Evaluation, - Training, - Data, - Peculiar Support Equipment, - Operational/Site Activation, - Industrial Facilities, and - Initial Spares and Repair Parts. BACKGROUND When the decision is made to develop and acquire a new or updated system, several factors are considered when planning or monitoring efforts. One of these factors is determining the work breakdown structure to use for the systems. A work breakdown structure is a product-oriented family tree, composed of hardware, software, services, data and facilities, which results from system engineering efforts during the development and production of a defense material item, and which completely defines the program. A work breakdown structure displays and defines the product(s) to be developed or produced and relates the elements of work to be accomplished to each other and to the end product. Therefore, the work breakdown structure plays a significant role in planning and assigning management and technical responsibilities; and monitoring and controlling the progress and status of (a) engineering efforts, (b) resource allocations, (c) cost estimates, (d) expenditures, and (e) Cost and technical performance. Seven defense material items are identified.

187

Parametric Cost Estimating Handbook

These are: Aircraft Systems Electronic/Automated Software Systems Missile Systems, Ordnance Systems Ship Systems, Space Systems, and Surface Vehicle Systems. Work Breakdown Structure Application The work breakdown structure provides a framework for specifying the technical objectives of the program by first defining the program in terms of hierarchically related product oriented element and the work processes required for their completion. Each element of the work breakdown structure provides logical summary points for assessing technical accomplishments, and for measuring the cost and schedule performance accomplished in attaining the specified technical objectives. For each work breakdown structure element, the detailed technical objectives are defined as well as the specific work tasks assigned to each contractor organization element and the resources, materials and processes required to attain the objectives. As resources are employed and work progresses on the task, current technical, schedule, and cost data are reported. The task data may then be summarized to provide successive levels of management with the appropriate report on planned, actual, and current projected status of the elements for which they are responsible. Management will, thus, be better able to maintain visibility of status and to apply their efforts to assure desired performance. a. Technical Management The work breakdown structure provides a framework for defining the technical objectives of the program. Together with the contract statement of work and product specification, the work breakdown structure aids in establishing a specification tree, defining configuration items, and planning support tasks. b. Contract Statement of Work The statement of work (SOW) is the document which describes in clear understandable term what products and services are to be delivered or services to be performed by the contractor. Preparation of an effective SOW requires an understanding of both the products and services that are needed to satisfy a particular requirement. A SOW prepared in explicit terms will facilitate effective contractor evaluation after contract award. The SOW becomes the standard for measuring contractor performance. Therefore, the SOW must clearly define the work to be performed. In preparing the SOW for a system acquisition, the use of a standardized work breakdown structure as a template for constructing the SOW will help streamline the process. Use of the work breakdown structure will also provide the framework and facilitate a logical arrangement of the SOW elements, provide a convenient checklist to ensure all necessary elements of the program are addressed, and direct the contractor to meet specific contract reporting needs. c. Specification Tree A specification tree, developed by system engineering, structures the Performance parameters for the system or systems being developed. It subdivides the system(s) and its elements. The performance characteristics are explicitly identified and quantified. Completed, the specification tree represents a hierarchy of performance requirements for each component element of the system for which design responsibility is assigned. Because specifications may

188

Parametric Cost Estimating Handbook

not be written for each product on the work breakdown structures the specification tree may not match the work breakdown structure completely. d. Configuration Management Configuration management is the process of managing the technical configuration of items being developed. In establishing, the requirement for configuration management on a program, the Government Activity needs to designate which contract deliverable designated for configuration management is called a Configuration Item. For software, this item is called Computer Software Configuration Item (CSCI). Configuration management involves defining the baseline configuration for the configuration items controlling the changes to that baseline, and accounting for all approved changes. The frameworks for designating the configuration items on a program is the work breakdown structure which needs to be extended sufficiently to clearly define all elements subject to configuration management. e. Financial Management The work breakdown structure assists management in measuring cost and schedule performance. By breaking the total product into successively smaller entities, management can ensure that all required products are identified in terms of cost and schedule performance goals. The planning of work by work breakdown structure elements serves as the basis for estimating and scheduling resource requirements. The assignment of performance budgets to scheduled segments of contract work identified to responsible organization units produces a time phased plan against which actual performance can be compared and appropriate corrective action taken when deviations from the plan are identified. This integrated approach to work planning also simplifies the identification of potential cost and schedule impacts of proposed technical changes. f. Contract Budgeting Funds management involves periodic comparison of actual costs with time phased budgets, analysis of performance variances, and follow-up corrective actions as required. When work breakdown structure product elements and the supporting work; are scheduled, a solid base for time phases budgets is made. Assignment of planned resource cost estimates to scheduled activities (tasks) and summarization by work breakdown structure element by time period results in a time phased program/contract budget, which becomes the performance measurement baseline. g. Cost Estimating Use of the work breakdown structure for cost estimating facilitates program and contract management. The work breakdown structure helps the Government program office to plan, coordinate, control and estimate the various program activities that the Government and the contractors are conducting. It provides a common framework for tracking the estimated and actual costs during the performance of each contract. The data from the various program contracts support the Government program manager in evaluating contractor performance, preparing budgets, and preparing program life-cycle costs e.g., as programs move through the various phases of the acquisition process (conceptual design, development, and production), the

189

Parametric Cost Estimating Handbook

actual experience to date and the estimates for the remaining phases provides the basis for reassessment of the total program costs. h. Data Bases Cost information accounted for by work breakdown structure element can be used for pricing and negotiating contracts and contract changes, and for follow-on procurement. Over time, the Government is accumulating a growing cost data base of similar work breakdown structure elements from different programs. Such historical cost can be used to develop learning curves, regression analyses and other techniques to estimate the cost requirements for like elements of new programs. Actual cost data collected by the Government on each program can be compared to the original estimates to identify trends, and establish validity of estimating techniques. Contractors will similarly benefit from such data bases. Since contractors tend to provide similar products on similar programs, the cost history accumulated on their programs can assist them in bidding future contracts and budgeting new work. Relationship To Other Contract Requirements The work breakdown structure is the basis for communication throughout the acquisition process. It provides the common link unifying the planning, scheduling, cost estimating, budgeting, contracting, configuration management, and performance reporting disciplines. It is also the basis used for contracts requiring cost/schedule reports placed on contract. This capability permits the contractor to evaluate progress in terms of contract performance. Contract performance assessments are made at the cost account level since the work breakdown structure facilitates the summarization of data for successively higher levels of management. WORK BREAKDOWN STRUCTURE DEVELOPMENT Work is effort performed by people to transform or create products to solve identified problems in order to verifiably meet specified objectives. Just as the organization hierarchically structures the people who perform work, so the work breakdown structure hierarchically structures the products to be produced and on which the people work. Examples of these products include equipment (hardware/software), data, services and facilities for such systems as missile systems, helicopter systems, automated systems, etc. In order to use the work breakdown structure as a framework for structuring the technical objectives of a program, in addition to its use as a management tool for cost and schedule control, it is important that the work breakdown structure be product-oriented. It’s elements should represent identifiable work products whether they be equipment (hardware, software) data or relatable service products. Because any work breakdown structure is a product structure, not an organization structure, complete definition of the effort encompasses the work to be performed by all participants. Figure l provides an overview of the work breakdown structure development process. Preparing a Program Work Breakdown Structure The Government Activity is responsible for developing and maintaining the program work breakdown structure. The Government Activity will structure a program work breakdown structure for defense material items prior to program initiation. By selecting appropriate elements, the Government Activity will be able to initially map its program work breakdown structure.

190

Parametric Cost Estimating Handbook

Figure 1

191

Parametric Cost Estimating Handbook

The program work breakdown structure should be developed in the conceptual stages of the program. The program work breakdown structure evolves during conceptual design from an iterative analysis of the program objective, functional design criteria, program scope, technical performance requirements, proposed methods of performance, including procurement strategy, as well as drawings, process flow charts, and other technical documentation. Before developing the program work breakdown structure, the Government Activity must know the contractual structure of the contract (e.g., the prime/subcontract relationship, make vs. buy plan, etc.). It is, therefore, important that the development documentation detail the Government plan to build, integrate, and field the system. Through this process the levels of reporting and elements for appropriate Request for Proposal (RFP) selection are determined. a. Program Work Breakdown Structure Element Selection Requirements The program work breakdown structure elements must be selected by the Government Activity and be structured in such a way that products and services may be readily summarized into the program work breakdown structure. The program work breakdown structure and contract work breakdown structure extensions will be used as a framework for technical and management activities. The Government Activity will employ the program work; breakdown structure and its contract work breakdown structure extensions as a coordinating medium in planning for further system engineering, resource allocation, cost estimates contract actions, and work execution. The reporting of progress, performance, and engineering evaluations, as well as financial data, will be based on the program work breakdown structure. Figure II provides an example of a top level program work breakdown structure. b. Levels of Program Work Breakdown Structure The program work breakdown structure contains at least the top three levels expanded to identify elements with a significant degree of technical or cost risk. When program work breakdown structure levels are stipulated to an excessively low level of a program, the contractor's normal method of operation may be hampered; or excessive reporting requirements may result. The SOW is the place to clearly communicate all program requirements. Figure III provides an expanded program work breakdown structure which incorporates elements necessary for contract visibility and control. c. Considerations in Constructing a WBS The following should be kept in mend when constructing a work breakdown: (1) Many elements of a program are not products. A signal processor for example, is clearly a product, as are mockups, and Computer Software Configuration Items (CSCIs). Design engineering, requirements analysis, test engineering labor, aluminum, and direct costs, etc., are not products. Design engineering, test engineering, and requirements analysis are all engineering functional efforts; aluminum is a material resource; and direct cost is an accounting classification. As such, none of these elements are appropriate as work breakdown structure elements. (2) Program-phases (e.g., production), and types of funds (e.g., Research Development, Test and Evaluation) are inappropriate elements of a work breakdown structure.

192

Parametric Cost Estimating Handbook

Figure 2

193

Parametric Cost Estimating Handbook

Figure 3

194

Parametric Cost Estimating Handbook

(3)

Rework, retesting, and refurbishing should be treated as work on the appropriate work breakdown structure element affected, not as separate elements of a work breakdown structure. (4) Non-recurring and recurring classifications are not work breakdown structure elements. The government reporting requirements should require segregation of each work breakdown structure element into its non-recurring and recurring parts. (5) Cost saving efforts such as total quality management initiatives, should cost, warranty, etc., are not work breakdown structure elements. These efforts should be included in the cost of the item they affect and not captured separately. (6) The organization structure of the program office or the contractor should not be the basis for development of a work breakdown structure. The work breakdown structure should always retain its product orientation. (7) Costs for meetings, travel, computer support, etc., arc to be included in the work breakdown structure elements for which they are associated. They are not to be treated as separate work breakdown structure elements. (8) The use of generic term in a work breakdown structure is improper. The system(s) name and/or nomenclature is appropriate. The work breakdown structure elements should be clearly named to indicate the character of the product to avoid semantic confusion. For example, if the level 1 system is Fire Control, then the Level 2 item (prime mission product) is Fire Control Radar. (9) Tooling (e.g., special test equipment, and factory support such as: assembly tools, dies, jigs, fixtures, master forms, handling equipment, etc. should be included in the cost of the equipment being produced. It is a functional cost, not a work breakdown structure element. If the tooling cannot be assigned to an identified subsystem or component, it should be included in the cost of integration, assembly, test, and checkout. Any additional quantities produced for equipment support or maintenance in the field should be included and reported under Peculiar Support Equipment. This same philosophy applies to software. For example, when a software development facility/environment is created to support the development of software, the cost associated with this element is considered part of the CSCI it supports, or if more than one CSCI is involved, it should be included in integration, assembly, test and checkout. (10) The definition of integration, assembly, test, and checkout includes production acceptance testing of R&D (including first article test) and production units, but excludes all system engineering/program management, and system test and evaluation which are associated with the overall system. Figure IV provides and example of both a correct and an incorrect work breakdown structure. e. Software in the Work Breakdown Structure The importance of software in today's Government acquisition environment is growing. As a result, software is identified in two ways for development of a work breakdown structure: The first type of software is that which operates or runs on a specific piece of equipment, and the second type of software is that which may be contracted for separately from the operating equipment or is a stand alone (software intensive system). Software that is being developed to reside on specific equipment must be identified as a subset of that equipment.

195

Parametric Cost Estimating Handbook

Figure 4

196

Parametric Cost Estimating Handbook

Multifunction software will be identified as a subset of the equipment work breakdown structure element which either includes the software in the element specification or exercises the most critical performance constraint. Figure V provides an example of how software should be addressed as part of a specific equipment. In cases where the application of this rule results in a conflict in the selection of the proper elements, the specification relationship will take precedence. For example, an aircraft's electronic equipment typically has software included in each of the subsystem elements. Software that resides and interfaces with more than one equipment, i.e., application software, and overall system software which facilitates the operation and maintenance of the computer systems and associated programs (e.g., operating systems, compilers, and utilities) will be called out at the appropriate level of the program. (Ref. ANSI/IEEE STD 6 10. 19. 1990 for definitions of applications and system software.) It is incorrect to summarize all software on a program or contract in a work breakdown structure (ref. Figure IV). By separating these elements from the hardware they support, performance measurement and management control over each equipment is difficult to obtain. The true cost of each equipment is not readily available for decisions concerning that equipment. Rather than separately summarizing software, it is important to identify software with the hardware it supports. (When needed, contractor's management systems can use an identifier for each software element to produce internal summaries for software management purposes.) A separately contracted or stand alone software will include the software, data, services, and facilities required to develop and produce a software product for a command and control system, radar system, information system, etc. Where software is considered stand alone (i.e., does not reside or support a specific equipment, or is considered a pure software upgrade, etc.), the Government Activity should use the same product-oriented work breakdown structure format. Figure VI provides an example of a work breakdown structure for a stand alone software system. f. WBS Dictionary - When developing a program work breakdown structure, the Government Activity should also develop a WBS Dictionary. The program work breakdown structure dictionary lists and defines the work breakdown structure elements. Although initially prepared for the program work breakdown structure, it can be expanded in greater detail at lower levels by contractors as they develop their contract work breakdown structure. The dictionary lists elements to show their hierarchical relationship to each other, describes each work breakdown structure element and the resources and processes required to produce it, and provides a link to the detailed technical definition documents. The work breakdown structure dictionary should be revised to reflect changes and should be maintained in a current status throughout the life of the program. Preparing a Contract Work Breakdown Structure The individual work breakdown structure elements should be chosen from the program work breakdown structure by the Government Activity for inclusion in a Request for Proposal (RFP). This will be accomplished by selecting the appropriate program work breakdown structure elements for the products that will be required by each contract. Contracts for WBS elements that are at level 3 or below in the Program Work Breakdown Structure will be moved to Level 2 and all

197

Parametric Cost Estimating Handbook

Figure 5

198

Parametric Cost Estimating Handbook

Figure 6

199

Parametric Cost Estimating Handbook

other applicable Level 2 Common WBS elements will be selected. The result is the contract work breakdown structure. Figure VII depicts the development and relationship of the Program Work Breakdown Structure with the Contract Work Breakdown Structure. Each RFP should include the contract work breakdown structure and the initial WBS Dictionary prepared by the Government Activity. The RFP should instruct potential contractors to extend the selected contract work breakdown structure elements to define the complete contract scope. a. RFP Solicitation Requirements The contract line items, configuration item, contract work statement tasks, contract specifications, and contractor responses will be expressed in term of the work; breakdown structure to enhance its effectiveness in satisfying the objectives of the particular acquisition. The Statement of Work; (SOW) tasks in the solicitation and final contract should be structured to align with the contract work breakdown structure. It is important to develop the program work breakdown structure with the development of the SOW so as to form consistency in document structure. When aggregated with the program work breakdown structure, the extended contract work breakdown structure will form a complete work breakdown structure for that program which will be used throughout the acquisition cycle. b. Extended Contract Work Breakdown Structure The contractor extends the contract work breakdown structure in the RFP and submits the complete contract work breakdown structure with its proposal. The proposal submitted should be based on the work breakdown structure in the RFP. Contractors may suggest changes to the selected contract work breakdown structure elements when a change is needed to meet an essential requirement of the RFP or to enhance the effectiveness of the contract work breakdown structure in satisfying the program objective. The contractor should extend the contract work breakdown structure to the appropriate level which satisfies the critical visibility requirements and does not overburden the contractor management system. Implementation of Contract Work Breakdown Structure When the Government selects the contractor, they negotiate the contract and the associated contract work breakdown structure. The proposed contract work breakdown structure included in the successful proposal serves as the basis for negotiating an approved contract work breakdown structure with the Government. The contractor may have proposed alternate approaches to better accomplish the contract objectives. The alternatives, if accepted by the Government Activity, may impact the proposed program work breakdown structures. Revisions will be required to the program work breakdown structure and the contract work breakdown structure to reflect these changes. After adjustments and contract negotiations, the elements selected for the contract will become the basis for contractor extension during the contracted effort. All extensions must sum to the contract work breakdown structure reporting level in the contract. a. Contract Work Breakdown Structure Approval and Contract Award Following Government approval of the negotiated contract, including the contract work breakdown structure, the Government awards the contract. The contract identifies the requirement for providing the WBS Dictionary through the contract data requirements list (CDRL). While strong, efforts should be placed on early and accurate work breakdown

200

Parametric Cost Estimating Handbook

Figure 7

201

Parametric Cost Estimating Handbook

structure planning, work breakdown structure revisions may result from expansion or contraction of program/ contract scope and the movement of a program through its various stages. Normally, changes to the work breakdown structure should not be made after contracts are awarded and work is underway unless major rescoping of the program occurs. NOTE: The sequence shown in the preceding paragraphs may be iterative as the program evolves, contracts are awarded, and the work effort progresses through major program phases. Whenever the work breakdown structure is revised, the ability to crosswalk and track back to the previous work breakdown structure must be maintained. b. Implementation with Subcontractors Contractors may require the use of the work breakdown structure by subcontractors to permit fulfillment of contractual requirements and provide adequate control of the subcontractor. Such subcontractors, whose work accounts for a major segment of the subcontracted portion of the prime contract, will be delineated in contracts at the time of award. It will be the prime or associate contractor's responsibility to incorporate into the contract, with the affected subcontractors, the work breakdown structure requirements. c. Maintain Contract Work Breakdown Structure The contract will indicate the levels of contract work breakdown structure at which costs will be reported to the Government. Traceability of cost accumulations will be required to those extended contract work breakdown structure levels which are used by the contractor for cost control purposes. In the extended contract work breakdown structure, consideration will be given to the specific contractual, technical, and managerial requirements of the defense material item. The contractor has complete flexibility in extending the contract work breakdown structure below the reporting requirement to reflect how work is to be accomplished, assuming lower elements to be meaningful product-oriented lower indentures of a higher-level element. Particular attention will be given to ensure the correlation of lower levels of the contract work breakdown structure to the specification tree, contract line items, data items, and statement of work tasks. Relationship with Contractor Management System As the end product is subdivided into smaller subproducts at lower work breakdown structure levels, the work effort required by each element can be identified with functional organization units. At some point within the work breakdown structure, the contractor will assign management responsibility for technical, schedule, and cost performance. The responsible manager is called cost account manager. The cost management system should provide the necessary visibility of the lower levels of the work breakdown structure as it interfaces with the organization. At the juncture of the work breakdown structure element and organization units cost accounts are usually established and performance is planned, measured, recorded and controlled. To do so, the technical requirements for the work and work product must be specified; the work scheduled, budgeted, and performed; and product attainment of specified technical requirements verified. a. Contractor Organizational Structure People performing work are organized to facilitate effective management. Whether the organization is designed along program, function, natural work teams or matrix lines, the organization structure reflects the way the people who will

202

Parametric Cost Estimating Handbook

accomplish the work have been organized. To assign specific work tasks, the organizational structure must be linked effectively with the work breakdown structure. This linkage can occur at any level of the work breakdown structure. Figure VIII depicts the linkage between the work break-down structure and the contractor's organizational structure. b. Cost Account Level -- To provide the responsible (cost account) manager with the technical, schedule, and cost information needed to manage the organization’s work on the work breakdown structure element for which he is responsible, the management control system must be keyed to the same work breakdown structure element and organization unit. The appropriate work breakdown structure level at which a cost account is established is purely a function of the magnitude of the program and the type of product. The responsible organization level is a function of the management span of control and upper management’s desire to delegate technical, schedule, and cost responsibility for product/contract work breakdown structure elements to lower management levels. In identifying cost accounts, the contractor must be allowed to establish organizational responsibilities at meaningful and appropriate levels, otherwise, the contractor's existing management control systems and responsibility assignments may be affected adversely. For example, when software is a major component of cost and the Government wants it identified separately, care must be taken to not unnecessarily complicate the contractor work breakdown structure and contractor management system. To meet these needs, special reporting requirements are specified in the SOW. In this example, Figure IX shows how the cost management system with job coding (SW) and the work breakdown structure can provide needed detail and visibility without extending the work breakdown structure to extremely low levels. Virtually all aspects of the contractor's management control system, including technical definition, budgets, estimates, schedules, work assignments, accounting, progress assessments, problem identification, and corrective actions, come together at the cost account. Performance visibility is directly relatable to the level and content of the cost account. SUMMARY The development of any work breakdown structure is intended to achieve a clear understanding and statement of the technical objectives and the end item(s) (or end product(s)) of the work to be performed. The process of identifying these objectives assists in structuring the product elements during the work breakdown structure development. Objectives derived from the overall program objective are identified in such a way that identifiable products support economically and technically identifiable subsystems of the program objectives. This process may be repeated until the component level is reached. In this matter, subsystems support a total system capability. When properly implemented, a good work breakdown structure can effectively satisfy the needs of both the Government and contractor program managers. By coordinating the development of the work breakdown structure, the statement of work the contract line item structure and the product specification, cost and technical requirements and performance can be better translated, managed and maintained. Without this generic framework, the acquisition process becomes difficult and complicated.

203

Parametric Cost Estimating Handbook

Figure 8

204

Parametric Cost Estimating Handbook

Figure 9

205

Parametric Cost Estimating Handbook

Neil F. Albert is Director of the Cost Analysis Division at Management Consulting & Research, Inc. (MCR), Falls Church, Virginia. He has over sixteen years of experience in directing and performing cost estimating/analysis tasks. His experience includes engineering and Parametric estimating, cost trade-off analysis, design-to-cost, and lifecycle cost analysis. He has held various corporate positions as a consultant, senior manager and cost analyst including Manager of Cost Analysis/Estimating for Textron Defense Systems. He holds an M.B.A. in Financial Management and a B.A. in Finance. Mr. Albert is the Administrator for MIL-STD-881B Working Group. In this capacity he was responsible for supporting the revision of MIL-STD-881A and for organizing and generating MIL-STD-881B. Mr. Albert was a principal author of the work breakdown structure User Guide which can be found in Appendix I of MIL-STD-881B.

REFERENCES 1. MIL-STD-881B (DRAFT), Work Breakdown Structures for Defense Material Items, 18 February 1992. 2. MIL-STD-881A, Work Breakdown Structures for Defense Material Items, 25 April l975. 3. DOE/MA-0295, Work Breakdown Structure Guide, February 6, 1987. 4. Contractor Cost Data Reporting (CCDR) System, 5 November 1973. 5. Cost/Schedule Control Systems Criteria Joint Implementation Guide , October 1987. 6. DI-A-3023, Contract Work Breakdown Structure, 21 May 1971. 7. MIL-STD-196D. Joint Electronics Type Designation System (JFTDS), l9 January 1985. 8. DOD-STD-2167A, Defense System Software Development, 29 February 1988. 9. ANSI/IEEE STD 610-1990, Software Engineering Terminology, 1990. 10. MIL-HDBK-171 (DRAFT), Work Breakdown Structures for Software Elements, 8 December 1992.

206

Parametric Cost Estimating Handbook

APPENDIX C

DCAA CAM 9-1000 SECTION 10 REVIEW OF PARAMETRIC COST ESTIMATES

NOTE: This is an excerpt from the Defense Contract Audit Agency Contract Audit Manual. The last page of this appendix includes an order form to obtain the entire manual.

207

Parametric Cost Estimating Handbook

APPENDIX C 9-1000 SECTION 10 - REVIEW OF PARAMETRIC COST ESTIMATES

9-1001 INTRODUCTION This section contains an overview and general guidance on reviewing cost-to-noncost estimating relationships, primarily in the context of contractor price proposals. This section also contains guidance on the use of estimating standards in price proposals. This supplementary guidance contains criteria contractors should meet before submitting proposals based on parametric cost estimates. 9-1002 PARAMETRIC COST ESTIMATING TERMINOLOGY 9-1002.1 Definition of Parametric Cost Estimating a. Parametric cost estimating ("parametrics") has been defined as a technique employing one or more cost estimating relationships (CERs) to estimate costs associated with the development manufacture, or modification of an end item. A CER expresses a quantifiable correlation between certain system costs and other system variables either of a cost or technical nature. CERs are said to represent the use of one or more independent variables to predict or estimate a dependent variable (cost). b. Parametrics encompasses even the simplest traditional arithmetic relationships among historical data such as simple factors or ratios used in estimating scrap costs. However, for audit purposes our guidance will limit special consideration of parametrics to more advanced or complex applications. These may involve extensive use of cost-to-noncost CERs, multiple independent variables related to a single cost effect, or independent variables defined in terms of weapon system performance or design characteristics rather than more discrete material requirements or production processes. EDP data bases and/or computer modeling may be used in these types of parametric cost estimating systems. c. Parametric estimating techniques may be used in conjunction with any of the following estimating methods: (1) Detailed—also known as the bottoms-up approach. This method divides proposals into their smallest component tasks and are normally supported by detailed bills of material. (2) Comparative—develops proposed costs using like items produced in the past as a baseline. Allowances are made for product dissimilarities and changes in such things as complexity, scale, design, and materials. (3) Judgmental—subjective method of estimating costs using estimates of prior experience, judgment, memory, informal notes, and other data. It is typically used during the research and development phase when drawings have not yet been developed.

9-1002.2 Distinction Between Cost and Noncost Independent Variables

208

Parametric Cost Estimating Handbook

a. Although the basic criteria for cost-to-cost and cost-to-noncost CERs are generally comparable, the supplementary criteria in this section pertain to cost-to-noncost CERs. Audits of traditional cost-to-cost estimating rates and factors are covered in other sections of this chapter and in referenced appendixes. b. Cost-to-noncost CERs are CERs which use something other than cost or labor hours as the independent variable. Examples of noncost independent variables include end-item weight, performance requirements, density of electronic packaging, number or complexity of engineering drawings, production rates or constraints, and number of tools produced or retooled. CERs involving such variables, when significant, require that the accuracy and currency of the noncost variable data be audited. Special audit considerations are described in 9-1003 and 9-1004. 9-1002.3 Uses of Parametric Cost Estimates a. Parametric cost estimating is used by both contractors and government in planning, budgeting, and executing the acquisition process. Parametric cost models are generally made up of several CERs and can be used to estimate the costs for part of a proposal or the entire proposal. The cost models are often computerized and may be made up of both cost-to-cost and cost-to-noncost interrelated CERs. The guidance contained in this chapter is intended to assist in the review of parametric estimates, CERs, and/or cost models used in developing price proposals for negotiation of government contracts. b. Parametric cost estimates are often used to cross-check the reasonableness of estimates developed using other estimating methods. Generally, it would not be prudent to rely on parametric techniques based on a broad range of data points to estimate costs when directly applicable program or contract specific historical cost data is available, as in the case of followon production for the same hardware in the same plant. Nor would parametric techniques be appropriate for contract pricing of specific elements such as labor and indirect cost rates which require separate forecasting considerations such as time and place of contract performance. The use of a parametric estimating method is considered appropriate, for example, when the program is at the engineering concept stage and the program definition is unclear, or when no bill of materials exists. In such cases, the audit evaluation should determine that: (1) the parametric cost model was based on historical cost data and/or was calibrated to that data, and (2) the contractor has demonstrated that the CER or cost model actually reflects or replicates that data to a reasonable degree of accuracy. 9-1003 PARAMETRIC ESTIMATING CRITERIA FOR PRICE PROPOSALS When a contractor uses parametric cost estimating techniques in a price proposal, the auditor will apply all pertinent criteria applicable to any proposal along with the supplemental criteria provided in 9-1004. 9-1003.1 Disclosure of Parametric Estimating Data a. The purpose of the Truth in Negotiations Act, 10 U.S.C. 2306(a), is to provide the government with all facts available to the contractor at the time of certification and that the cost or pricing data was current, complete, and accurate. Parametric estimates must meet the same basic disclosure requirements under the act as detailed estimates.

209

Parametric Cost Estimating Handbook

b. Although the principles are no different, proposals supported in whole or in part with parametric estimating will present new fact situations concerning cost or pricing data which is required to be submitted. A fundamental part of the definition of cost or pricing data is "all facts ... which prudent buyers and sellers would reasonably expect to have a significant effect on price negotiations" (FAR 15.801). Reasonable parallels may be drawn between the data examples provided in FAR for discrete estimating approaches and the type of data pertinent to parametric estimating approaches. For example, if a contractor uses a cost-to-noncost CER in developing an estimate, the data for the CER should be current, accurate, and complete. c. Many contractors use parametric cost estimating for supplementary support or for crosschecking estimates developed using other methods. Judgment is necessary in selecting the data to be used in developing the total cost estimate relied upon for the price proposal. In distinguishing between fact and judgment, FAR states the certificate of cost or pricing data "does not make representations as to the accuracy of the contractor's judgment on the estimated portion of future costs or projections. It does, however, apply to the data upon which the contractor's judgment is based" (FAR 15.804-4[b]). Therefore, if a contractor develops a proposal using both parametric data and discrete estimates, it would be prudent to disclose all pertinent facts to avoid later questions about full disclosure. d. The auditor should address the following questions during the evaluation of Parametric cost estimates: * Do the procedures clearly establish guidelines for when parametric techniques would be appropriate? * Are there guidelines for the consistent application of estimating techniques? * Is there proper identification of sources of data and the estimating methods and rationale used in developing cost estimates? * Do the procedures ensure that relevant personnel have sufficient training, experience, and guidance to perform estimating tasks in accordance with the contractor's established procedures? * Is there an internal review of and accountability for the adequacy of the estimating system, including the comparison of projected results to actual results and an analysis of any differences? 9-1004 SUPPLEMENTAL ESTIMATING CRITERIA The auditor should also consider the following supplemental criteria when evaluating parametric cost estimates. 9-1004.1 Logical Relationships The contractor should demonstrate that the cost-to-noncost estimating relationships used are the most logical. A contractor should consider all reasonably logical estimating alternatives and not limit the analysis to the first apparent set of variables. When a contractor's analysis discloses multiple alternatives that appear logical, statistical testing (9-1004.3) of selected logical relationships may be used to provide the basis for choosing the best alternative. 9-1004.2 Verifiable Data

210

Parametric Cost Estimating Handbook

The contractor should demonstrate that data used for parametric cost estimating relationships can be verified. In many instances the auditor will not have previously evaluated the accuracy of noncost data used in parametric estimates. For monitoring and documenting noncost variables, contractors may have to modify existing information systems or develop new ones. Information that is adequate for day-to-day management needs may not be reliable enough for contract pricing. Data used in parametric estimates must be accurately and consistently available over a period of time and easily traced to or reconciled with source documentation. 9-1004.3 Statistical Validity The contractor should demonstrate that a significant statistical relationship exists among the variables used in a parametric cost estimating relationship. There are several statistical methods such as regression analysis that can be used to validate a cost estimating relationship, however, no single uniform test can be specified. Statistical testing may vary depending on an overall risk assessment and the unique nature of a contractor's parametric data base and the related estimating system. Proposal documentation should describe the statistical analysis performed and include the contractor's explanation of the CER's statistical validity. 9-1004.4 Cost Prediction Results The contractor should demonstrate that the parametric cost estimating relationships used can predict costs with a reasonable degree of accuracy. As with the use of any estimating relationship derived from prior history, it is essential in the use of parametric CERs for the contractor to document that work being estimated is comparable to the prior work from which the parametric data base was developed. 9-1004.5 System Monitoring The contractor should ensure that cost-to-noncost parametric rates are periodically monitored in the same manner as cost-to-cost rates and factors. If a CER is validated and will only be used in a one-time major new pricing application, rate monitoring capability is not essential (see 9-1004.2). However, if it is expected that the rates should be considered as an ongoing estimating technique, CER monitoring is critical. The contractor should revalidate any CER whenever system monitoring discloses that the relationship has changed. 9-1005 AREAS FOR ESTIMATING

SPECIAL

CONSIDERATION

IN

PARAMETRIC

COST

9-1005.1 Parametric Estimating for Change Orders Change order pricing using parametric cost estimating relationships may need to be considered in a different light than initial contract pricing actions. The contractor may use cost estimating relationships which are unique to change order proposals. In general, contractors do not segregate costs separately for individual change orders Therefore, it is important that the contractor have a system in place to validate, verify, and monitor CERs unique to change orders. However, if the CER was applicable to the basic contract and change orders, the CER could be validated without cost segregation. 9-1005.2 Forward Pricing Rate Agreements

211

Parametric Cost Estimating Handbook

a. Contractors may submit proposals for forward pricing rate agreements (FPRAs) or formula pricing agreements (FPAs) for parametric cost estimating relationships to reduce proposal documentation efforts and enhance government understanding and acceptance of the contractor's system. Government and contractor time can be saved by including the contractor's most commonly used CERs in FPRAs or FPAs. (See FAR 15.809 for basic criteria.) However, such an agreement is not a substitute for contractor compliance at the time of submitting a specific price proposal. FAR requires that the contractor describe any FPRAs in each specific pricing proposal to which the rates apply and identify the latest cost or pricing data already submitted in accordance with the agreement. All data submitted in connection with the agreement is certified as being accurate, complete, and current at the time of agreement on price on each pricing action the rates are used on, not at the time of negotiation of the FPA or FPRA (FAR 15.809 [d]). b. Key considerations in auditing FPRA/FPA proposals for parametric CERs follow: (1) FPRAs/FPAs do not appear practicable for CERs that are intended for use on only one or few proposals. (2) Comparability of the work being estimated to the parametric data base is critical. FPRA proposals for CERs must include documentation clearly describing circumstances when the rates should be used and the data used to estimate the rates must be clearly related to the circumstances. (3) Validation of all the parametric criteria (9-1003) is especially important if a single CER or family of CERs is to be used repetitively on a large number of proposals. 9-1005.3 Subcontract Pricing Considerations a. FAR 15.806-2(a) requires that when a contractor is required to submit certified cost or pricing data, the contractor will also submit to the government accurate, complete, and current cost or pricing data from prospective subcontractors in support of each subcontract cost estimate that is: (1) $1,000,000 or more, (2) both more than $100,000 and more than 10 percent of the prime contractor's proposed price, or (3) considered to be necessary to adequately price the prime contract. Use of parametric CERs does not relieve a contractor of its responsibility to disclose planned subcontract procurements and the related subcontractor cost or pricing data. b. When proposed material costs are based on parametric estimates the contractor must demonstrate that the type of materials required for the proposal are the same as included in the CER data base. The auditor should perform audit procedures to determine if: (1) materials included in the CER data base are not estimated separately in the proposal, and (2) adjustments have been made to the CER data base for those items which were previously manufactured in-house and now are being purchased. If the CER data base has not been adjusted, the contractor should provide a detailed cost estimate for purchased materials. c. The contractor should explain any major differences between parametric estimates of subcontract costs and the subcontractor's quoted price and provide the rationale for using the parametric estimate instead of the quote.

212

Parametric Cost Estimating Handbook

d. Consistency in subcontract cost estimating must be maintained within the contractor's estimating system. Any significant deviations from normal practices in the proposal must be identified and justified by the contractor. 9-1005.4 Parametric Estimating Efficiency a. A primary justification for using parametrics is reduced estimating and negotiation costs. Contractors should perform a cost-benefit analysis before implementing an elaborate parametric estimating model. Their analysis should show that implementation and monitoring costs do not outweigh the benefit of reduced estimating costs. In many instances, new reporting systems may have to be developed to provide reliable noncost independent variables. In addition, the costs of CER validation and monitoring may be substantial. b. When the contractor's cost-benefit analysis indicates that the parametric system implementation costs might outweigh the benefits of reduced estimating costs and/or increased estimating accuracy, the matter should be pursued for potential cost avoidance recommendations. 9-1005.5 Data Base Adjustment Considerations a. One basic criterion (9-1004.4) is that the parametric data base be comparable to work being estimated. However, a contractor may have to adapt a partially comparable data base to its cost history using a "calibration" factor An example would be an adjustment to the data base to estimate the savings as a result of continuous improvement initiatives such as TQM. The utilization of complexity factors and/or adjustments to modify contractor developed in-house CERs is a valid technique. However, the use of such factors or adjustments should be fully documented and disclosed. In addition, this approach increases the contractor's burden to document compliance with the other criteria. b. If a contractor does not support the adjustment factors, the contracting officer should be promptly notified (see 9-1005.7). In addition, the auditor should determine if a qualified or adverse opinion is required. The audit report should disclose the costs associated with the unsupported factors. 9-1005.6 Contract Administration Interface a. Upon receipt of a request to review a price proposal, the auditor will coordinate with the Plant Representative/ACO to make arrangements for any needed technical reviews of the proposal. Because of the special nature of cost-to-noncost estimating relationships, and the possibility of limited cost history and added audit testing, complete coordination is especially important when parametric estimates are involved. b. While the auditor will address special areas of concern as requested by the PCO and/or the Plant Representative/ACO, the audit scope will be established by the auditor in accordance with the auditing standards, unless the PCO only requests a review of part of a price proposal. c. Auditors should be available, on request, to explain applicable price proposal criteria and identify any prospective audit concerns to both government and contractor personnel. An example of such audit advice would be to identify operating reports or records that have not been previously used to forecast costs and would therefore require added contractor support and audit testing. Such advance coordination will help avoid unnecessary contractor system development costs.

213

Parametric Cost Estimating Handbook

9-1005.7 Reporting of Estimating Deficiencies All proposal and estimating deficiencies found during the review of the parametric estimating technique should be immediately reported to the Plant Representative/ACO. These may include incorrect, incomplete, or non-current data and use of inappropriate estimating techniques. When a proposal evaluation discloses estimating system deficiencies, a separate report entitled "Estimating System Deficiency Disclosed during Evaluation of Proposal No. XXX" will be issued immediately after the proposed audit report. 9-1006 ESTIMATING STANDARDS 9-1006.1 Distinction Between Estimating Standards and Parametric Cost Estimating a. In terms of historical evolution and sophistication, the terminology of estimating standards as covered in this paragraph might be viewed as falling between traditional cost-to-cost estimating rates and factors and the more advanced types of parametric estimating systems (see 9-1002). However, a contractor may elect to use any combination of these evaluating methods, perhaps in the same proposal. b. Estimating standards are normally developed through the use of motion timemeasurement studies performed by industrial engineers. Parametrics, on the other hand, are developed by relating historical costs to one or more noncost drivers. While estimating standards usually represent cost-to-noncost relationships they have traditionally been limited to narrower or more discrete elements of estimated cost than may be the case in more complex parametric CERs. Also, the logic of the estimating relationship and the appropriateness of the mathematics in estimating standards will usually be readily apparent. c. Estimating standards will not necessarily require validation under the criteria for parametric cost estimating relationships contained in 9-1003 and 9-1004. Especially when such standards (e.g., hours/pound, hours/drawing, hours/page) have been in place and accepted by government personnel, the evaluation guidance in this paragraph will likely be sufficient. 9-1006.2 Use of Estimating Standards a. Estimating standards may be established by relating engineering and/or production costs (efforts time, and/or materials) to specific characteristics of a product such as composition, weight, size or duration. This approach is designed to save estimating effort and has been used frequently in estimating construction costs and costs of recurring job orders such as printing. Many contractors use the technique in shop-order budgeting and production control. b. Estimating standards may be used to estimate the cost of a single material item required for the work, or the cost of a single labor operation, for example, welding electrodes per ton of structural steel, press operations time per page, or guard service costs per week. More complex, composite standards may be used to estimate costs of groups of components or broader classes of labor operations. c. Use of estimating standards may be appropriate in contract cost estimating situations when there is a close correlation between an amount of production cost and the related product or process characteristic. The data sets being correlated must have been measured in a uniform manner. The cost data used should be verifiable by reasonable means. The units of measure used for base characteristics should be uniform and readily identifiable; the quantity or value of a characteristic should be readily determinable. Standards may be derived from industry-wide

214

Parametric Cost Estimating Handbook

statistics but should be relevant and verifiable to the experience of the particular contractor using them. 9-1006.3 Applicability to Price Proposals Traditionally, estimating standards have been used to estimate costs in lump sums, often including supervision, indirect costs, and occasionally general and administrative expense. To comply with the SF 1411 and cost accounting standards the contractor will normally have to factor the estimate to identify the costs by cost element or function. Alternatively a proposed cost based on an estimating standard might qualify for submission as an "other" cost element if the cost can be tracked as such and is a relatively minor pan of the total proposal. 9-1006.4 Audit Procedures a. Depending on materiality and risk of the costs estimated, the auditor should examine the development and application of estimating standards to determine whether their use is proper in the circumstances. Evaluate all cost and noncost data applicable to each significant estimating standard and determine whether the data has been properly used in the computations. Assure that the measurements and correlation are adequate for the purpose. Determine whether the basis for the standard (for example, the product mix, production rates, and production methods) is sufficiently similar or comparable to that contemplated in the estimate at hand. b. When changes are contemplated in the design or production of an end item or the rate or method of production, the contractor's adjustments of the estimating standards require special scrutiny. Review by government technical specialists may be necessary in this situation. c. During audits of historical costs, sufficient information may be readily available from which the auditor could develop estimating standards to use as one means of appraising recurring contractor estimates. However, this will not substitute for audit review of cost estimates as submitted by the contractor.

215

Parametric Cost Estimating Handbook

Superintendent of Documents Order Form YES, please send me the following publications: subscriptions to DCAA CONTRACT AUDIT MANUAL (DCAAM) for $32.00 per year ($40.00 foreign) The total cost of my order is $________. Price includes regular shipping and handling and is subject to change. International customers please add 25%.

Charge your _______________________________________________________ order. Company or personal name (Please type or print) It’s easy! ________________________________________________________________________ Additional address/attention line ________________________________________________________________________ Street Address ________________________________________________________________________ City, State, Zip Code ________________________________________________________________________ Daytime phone including area code ________________________________________________________________________ Purchase order number (optional)

To fax your orders (202)512-2250

Check method of payment: Check payable to Superintendent of Documents GPO Deposit Account VISA

To phone your orders (202) 512-1800

MasterCard

Thank you for (expiration your order! date) __________________________________________________________________

Authorizing signature

Mail to:

Superintendent of Documents P.O. Box 371954, Pittsburgh, PA 15250-7954

216

Parametric Cost Analysis Handbook

APPENDIX D

MORE ABOUT STATISTICS

217

Parametric Cost Analysis Handbook

APPENDIX D MORE ABOUT STATISTICS

In this Appendix we will discuss, in additional detail, some statistical topics including error, sample size, the Student's "t" distribution, the "f" distribution, and confidence intervals. STATISTICAL INFERENCE In statistical inference we make generalizations based on samples, and, traditionally, such inferences have been divided into problems of estimation and hypothesis testing. In estimation we assign a numerical value to a population parameter on the basis of sample data. In any CER, we are attempting to predict the population behavior(s) from a sample. We need to ask ourselves the question, "How good is our estimation?" When we test a hypothesis, we accept or reject assumptions concerning the parameters or the form of a population. When we use a CER, we need to be confident of the CERs ability to predict the future - without this confidence, the CER should not be used. Z-scores In general, if X is a measurement belonging to a set of data having the mean x (or µ for a population) and the standard deviation s (or σ for a population), then its value in standard units, denoted by Z, is Z=

x−x S

or,

Z=

Χ−µ

σ

depending on whether the data constitute a sample or a population. In these units, Z tells us how many standard deviations a value lies about or below the mean of the set of data to which it belongs. Error An estimate is generally called a point estimate, since it consists of a single number, or a single point on the real number scale. Although this is the most common way in which estimates are expressed, it leaves room for many questions. For instance, it does not tell us on how much information the estimate is based, and it does not tell us anything about the possible size of the error. And, of course, we must expect an error. An estimate's reliability depends upon two things - the size of the sample and the size of the population standard deviation, σ . Any statistics textbook will show that the error term is, E = Z (α

2 )∗

σ n

where,

218

Parametric Cost Analysis Handbook

Z( α /2) represents the number of standard deviations from the mean that we are willing to allow our estimate to be "off" either way by probability of 1 − α . This result applies when n is large and the population is infinite. The two values which are most commonly used for 1 − α are 0.95 and 0.99, with corresponding Z scores, Z ( 0.025) = 1.96 (standard deviations) for 1 − α = 0.95 and Z (0.005) = 2.575 for 1 − α = 0.99, respectively. There is one complication with this result. To be able to judge the size of the error we might make when we use x as an estimate of µ , we must know the value of the population standard deviation, σ . Since this is not the case in most practical situations, we have no choice but to replace σ with an estimate, usually the sample standard deviation, s. In general, this is considered to be reasonable provided the sample is sufficiently large (n ≥ 30). Sample Size The formula for E can also be used to determine the sample size that is needed to attain a desired degree of precision. Suppose that we want to use the mean of a large random sample to estimate the mean of a population, and we want to be able to assert with probability 1 − α that the error of this estimate will be less than some prescribed quantity E. Solving the the previous equation for n, we get,  Z (α / 2 )∗σ  n=  Ε 

2

Confidence Intervals For large random samples from infinite populations, the sampling distribution of the mean is approximately normal with the mean µ and the standard deviation σ / n , namely, that, Z=

x−µ σ n

is a random variable having approximately the standard normal distribution. Since the probability is 1 − α that a random variable having the standard normal distribution will take on a value between -Z( α / 2 ) and Z( α / 2 ), namely, that −Z (α / 2 ) < Z < Z (α / 2 ) , we can substitute into this inequality the foregoing expression for z and it yeilds, − Z (α / 2 ) <

x−µ < Z (α / 2) σ/ n

Using some algebraic manipulation, we get x − Z (α / 2 )∗σ / n < µ < x + z (α / 2 )∗σ / n

219

Parametric Cost Analysis Handbook

and we can assert with probability 1 − α that it will be satisfied for any given sample. In other words, we can assert with ( 1 − α )% confidence that the interval, above, determined on the basis of a large random sample, contains the population mean we are trying to estimate. When σ is unknown and n is at least 30, we replace σ by the sample standard deviation, s. An interval such as this is called a confidence interval, its endpoints are called confidence limits, and the probability 1 − α is called the degree of confidence. Again, the values most commonly used for 1 − α are 0.95 and 0.99, the corresponding values of Z (α / 2 ) are 1.96 and 2.575, and the resulting confidence intervals are referred to as 95% and 99% confidence intervals for µ. Confidence Intervals for Means (Small Samples) To develop corresponding theory which applies also to small samples, it will be necessary to assume that the population we are sampling has roughly the shape of a normal distribution. x−µ We can then base our methods on the statistic t = , whose sampling distribution is a s/ n continuous distribution called the t distribution. This distribution is symetrical and bell-shaped with zero mean. The exact shape of the t distribution depends as a parameter called the number of degrees of freedom, given by n-1, the sample size less one. For the t distribution we define t(α/2) in the same way in which Z(α/2) was defined. However, t(α/2) depends on n-1 (degrees of freedom) and its value must be looked up in a table of values. In the same way as before we can arrive at the following small sample confidence interval for µ: x − t (α / 2 )∗ s / n < µ < x + t (α / 2 )∗ s / n

The degree of confidence is 1 − α and the only difference between this formula and the large sample formula is the t(α/2) takes the place of Z(α/2). Analysis of Variance and the F Statistic The F statistic is a statistic for a test concerning the differences among means. It is defined as:

F=

estimate of σ2 based on the variation among the x 's estimate of σ2 based on the variation within the samples

and is called a variance ratio. The F distribution is a theoretical distribution which depends on two parameters called the numerator and denominator degrees of freedom. When the F statistic is used to compare the means of k samples of size n, the numerator and denominator degrees of freedom are, respectively, k-1 and k(n-1). This is a simple form of an analysis of variance. The basic idea of an analysis of variance is to express a measure of the total variation of a set of data as a sum of terms, which can be attributed to specific sources, or causes of variation. Two such sources of variation could be 1) actual differences, and, 2) chance differences. As a measure of the total variation of an observation consisting of k samples of size n, we use the total sum of squares, 220

Parametric Cost Analysis Handbook

k

SST =

n

∑ ∑(X i =1

ij

− x...) 2 ,

j =1

where Xij is the jth observation of the ith sample, i = 1, 2, ... , k, and j = 1, 2, ... , n, and x , the mean of all the k measurements or observations is called the grand mean. If we divide the total sum of squares by kn-1, we get the variance of the combined data. Letting X i devote the mean of the ith sample, i = 1, 2, ..., k, we can write the following identity: k

k

SST = n∗ ∑ ( X i − X ...) 2 + ∑ i =1

i =1

n

∑ (X

ij

− Xi ) 2

j =1

Looking closely at the two terms into which the total sum of squares SST has been partitioned, we find that the first term is a measure of the variation among the means. Similarly, the second term is a measure of the variation within the individual samples, or chance variation. Dividing the first term by k-1 and the second by k (n-1), we get the numerator and the denominator of the F Statistic as defined, above. The first term is often referred to as the treatment sum of squares, SST and the second term as the error sum of squares, SSE, experimental error, or chance.

Refer to a statistics textbook for a further explanation of these and other statistical subjects.

221

Parametric Cost Analysis Handbook

APPENDIX E

EXAMPLES OF OTHER HARDWARE ESTIMATING MODELS

FROM THE SPACE SYSTEMS COST ANALYSIS GROUP

222

Parametric Cost Analysis Handbook

APPENDIX E EXAMPLES OF OTHER HARDWARE ESTIMATING MODELS

OTHER HARDWARE MODELS

Advance Missions Cost Model

NASA JSC

Aerospace Small Satellite Cost Model

The Aerospace Corp.

ALS/ADP Cost Model

USAF Phillips Lab

EHF/SHF Communications Cost Model

USAF SMC/FMC

Hypercost

NASA JPL

Launch Vehicle Cost Model

TECOLOTE

NASCOM

NASA MARSHALL

Non Nuclear Power

NASA LeRC

Nuclear Space Power

NASA LeRC

Parametric Cost Model

Parametric Consultants/England

Passive Sensor Cost Model

USAF SMC/FMC

SCEEDOS

USAF SMC

Sensat

Owl Wise Laboratory

Solid Rocket Motor Cost Model

TECOLOTE

STACEM

AF Astronautics Lab

TRANSCOST

Deutsche Aerospace

Unmanned Spacecraft Cost Model

USAF SMC

Some examples follow.

223

Parametric Cost Analysis Handbook

COST MODEL DESCRIPTION Title: Purpose:

Advance Missions Cost Model (AMCM) To predict development, recurring and mission operations costs of future space systems, including launch vehicles, spacecraft and human explorations missions.

Applicability:

The AMCM is a system level cost model, therefore it is more appropriate for large scale programs requiring many different systems that will be integrated to perform a complex mission. The methodology is most useful in the pre-conceptual and conceptual design phases of a program when the actual design of the systems is not known and many factors are being traded off.

Model Description:

The AMCM is a two equation, multi-variable cost estimating relationship. The first equation predicts the development and production cost of the system based of various technical and programmatic factors. The second equation predicts the basic mission operations cost of the system. Both equations are fitted to a historical database for AMCM includes ground vehicles, ships, aircraft, helicopters, and missiles as well as launch vehicles and spacecraft. The database spans 50 years of systems development. Most of the systems are US Government developed, but some commercial and European systems are included.

Status/Availability:

The AMCM is complete and is periodically updated. The latest release of the development model is Version 4.1 dated July 1992. The mission operations segment of the model was last updated in November 1986. Papers describing the model are available from the author; Kelley Cyr of NASA JSC. A personal computer version of the model that runs with Microsoft Excel is also being developed.

Input Variables:

-

Output:

Development Quantity Production Quantity Unit Dry Weight Specification value (a numeric value representing the operating type and mission environment) Initial Operating Capability (IOC) year Block Number Difficulty Factor Number of Years of Operations

Development, production and operations costs in millions of dollars. For space systems, the cost estimates are for prime

224

Parametric Cost Analysis Handbook

contractor cost only and do not include fee, program support, or other government costs. For non-space systems, the costs are all inclusive. Data Source:

A large historical 50-year database of land, water, air and space systems. The data includes 54 spacecrafts, 22 space transportation systems, 61 aircraft, 86 missiles, 29 ships, and 18 ground vehicles. All of the data points are from programs that were completed through IOC.

Point of Contact:

Kelley Cyr, National Aeronautics and Space Administration, Johnson Space Center, Mail Code BZ, Houston, Texas 77058

User Community:

Cost Analysts, Project Managers, Systems Engineers

Principle Ground Rules/ Assumption/Limitations:

The AMCM development model is primarily based on raw data from contractors, government reports and published sources. The original data was adjusted to constant year 1991 dollars using various inflation factors. The basic model does not distinguish between non-recurring and recurring cost because of the difficulties associated with separating the costs in development programs with very small production runs. The recurring cost of the system can be computed from the CER’s by calculating the marginal cost. The mean absolute percentage error (MAPE) for the model is approximately 45%.

Software:

The CER’s can be easily programmed in any language or spreadsheet. A Microsoft Excel add-in application is being developed.

Equipment:

The Microsoft Excel add-in application will be able to run on any platform that supports Microsoft Excel.

CER Format:

Cost = a • Qb • Wc • dS • e(1/(IOC-1900)) • Bf •gD Q is the quantity W is the weight s is the Specification IOC is the IOC year B is the Block Number D is the Difficulty Factor a-g are model parameters

225

Parametric Cost Analysis Handbook

Title: Purpose:

Aerospace Small Satellite Cost Model (SSCM) To evaluate cost of designing, building, and testing a modern small satellite.

Model Description:

SSCM estimates the first-unit development and production cost by computing a weighted average of parametric CER’s derived from actual Small-satellite cost and technical databases. Included are production, integration, assembly, and testing. Excluded are any payload development or production costs.

Status/Availability:

SSCM is currently in a test-and-evaluation configuration; a more elaborate program with more user features is forthcoming. The current version is available to government agencies and participating small-satellite contractors.

Input Variables:

Mission type (communications, remote sensing, or space experiments), Procurement environment (military, commercial or NASA), Subsystem masses, performance measures, hardware legacy and production schedule.

Output:

Development cost range and cost sensitivities.

Data Source:

Proprietary data from variety of small-satellite programs such as REX, ALEXIS, STEP, APEX, RADCAL, and LOSAT-X.

Point of Contact:

Erik Burgess, Resource and Requirements Analysis Department, the Aerospace Corporation, (310) 336-4148.

User Community:

Project Managers and Cost Analysts.

Principle Ground Rules/ Assumptions/Limitations

SSCM estimates but costs only; payload development/ production costs must be separately determined.

Software:

SSCM is currently hosted in a stand-alone compiled program that can operate on any IBM-compatible computer. It requires only a 360K-byte driver and DOS 2.0 or higher.

Equipment:

See above.

CER Format:

Cost as explicit functions of input variables.

Test Results/Evaluation:

Validated against several small-satellite development programs.

226

Parametric Cost Analysis Handbook

Title: Purpose:

Aerospace Launch Vehicle Cost Model (LVCM) To evaluate cost of existing, modified, new launch vehicle designs.

Model Description:

LVCM determines RDT&E cost by subsystem, average unit operations cost by subsystem for each stage, total vehicle life-cycle cost, fiscal year funding requirements for overall vehicle program.

Status/Availability:

Internal Aerospace Corporation use only.

Input Variables:

Launch site characteristics (site location, relative number of launchers from each site); propellant type; weight, quantity pre vehicle, design status and previous quantity produced of each of the following subsystems: Structure, Thermal Control, Reentry Protection, Landing System, Electrical Power, Electrical Wiring, Guidance and Control, Data Handling, Instrumentation, Communication, Propulsion, Engines, Reaction Control Propulsion Hardware, Interstage Adapter and Payload Fairing.

Output:

LVCM produces both numerical and graphical results. Major output is life-cycle cost estimate for complete vehicle broken down by principal categories of RDT&E, Investment and Operations, which are further broken down by each stage, fairing, launch operations, flight operations, recovery (and refurbishment) operations, spares, software, facilities, system engineering and technical directions and government support.

Data Source:

Proprietary data from variety of DoD launch vehicle programs.

Point of Contact:

Ronald Hovden, Resource and Requirements Analysis Department, the Aerospace Corporation, (310) 336--5832..

User Community:

Aerospace Corporation Project Managers and Cost Analysts.

Principle Ground Rules/ Assumptions/Limitations

Required detailed knowledge of input parameters.

Software/Equipment:

Written in FORTRAN 77, currently hosted on IBM mainframe. Inputs are selected via an interactive, panel-driven front-end. Lahey PC version is under development, but not yet fully functional.

CER Format:

Cost as explicit functions of input variables.

Test Results/Evaluation:

Validated against a number of existing DoD launch vehicles.

227

Parametric Cost Analysis Handbook

Title:

Unmanned Spacecraft Cost Model, Sixth Edition, Nov 1988 (USCM6)

Purpose:

To estimate total Space Segment Cost including non-recurring and recurring cost of components as well as subsystems for earth orbiting unmanned spacecraft.

Publisher:

USAF, Space Division, Los Angeles AFB

Model Description:

(see attached schematic) By using physical descriptions of components, the model’s CER’s develop component NR and REC cost. Other CER’s develop support costs such as Program Management, Systems Engineering, Data, etc.

Input Variables:

component weights number of sensors volume, design life, BOL power number of solar cells, battery capacity power output, analog electronics weight total spacecraft dry weight

Output:

NR and T1 cost for components NR and T1 cost for integration and assembly NR and T1 cost for ferrite devices NR AGE; launch and orbital ops support NR and T1 for Program Management, Systems Engineering, Systems Test and Data.

Data Source:

The database consists of 9 military, 6 NASA, and 3 commercial satellite programs with start dates between 1964 and 1979

Point of Contact:

Publisher

User Community:

From managers interested in top-level estimates to analysts needing detailed information on CER development.

Principle Ground Rules/ Assumptions/Limitations

Estimates are in 1986 dollars excluding fee. Costs cover all NR and REC Spacecraft cost including launch operations (relating to S/C only) and orbital support.

Equipment:

No equipment other than a calculator is needed. provided for uses on PCs running LOTUS 1-2-3.

228

Macros are

Parametric Cost Analysis Handbook

CER Format:

Both single and multivariate using linear and log/log relationships. An interesting not is the introduction of the “PING” factor with the log/log relationship. Test Results/Evaluation:

229

TBD

Parametric Cost Analysis Handbook

UNMANNED SPACE VEHICLE COST MODEL 6TH EDITION MODEL NONRECURRING COST FLOW INPUTS Weight Sensor Types/Wt Torque Methods/Wt /Vol Wt/BOL Power Wt*BOL Pwr/# cells Wt*BOL Pwr Wt*BOL Pwr Wt*BOL Pwr/Batt Cap Pwr Output/TWTA/Subsys Weight/Design Life Weight Weight/Design Life Weight/Comm Mission Wt/Orbit/Subsys Dry Wt/Stabilization Type

Pwr Output/TWTA/Subsys Weight/Design Life Weight Weight/Design Life Weight/Comm Mission Wt/Orbit/Subsys

S/C •Structure •Attitude Control Subsystem •Attitude Determination •RCS •Thermal Control •Electrical Power Subsystem •Generation •Storage •Distribution •Conditioning •Telemetry, Tracking & Command •Transmitter/Amplifer •Receiver/Exciter •Transponder •Digital Electronics •Analog Electronics •Antenna Propulsion (AKM)

•Communications Payload •Transmitter/Amplifier •Receiver/Exciter •Transponder •Digital Electronics •Analog Electronics •Antennas

S/C $

Total $ Microwave Ferrite Devices

I N T E G R A T I O N

&

Total $

Microwave Ferrite Devices

230

Comm $

A E R O S P A C E

A S S E M B L Y

S/C + Comm + I&A $

G R O U N D

E Q U I P M E N T

Space Program Level Vehicle + Costs AGE $ • • • •

Prog Mgmt Sys Engr Sys Test Data

Parametric Cost Analysis Handbook

UNMANNED SPACE VEHICLE COST MODEL 6TH EDITION MODEL RECURRING COST FLOW INPUTS S/C Space Veh Dry Wt/Mission Weight/Manner Sensor Types, Wt Vol/Design Life/Wt Wt/Manned Area/BOL Pwr/# cells Wt*BOL Pwr Wt*BOL Pwr Wt*BOL Pwr Weight Weight/Sync Orbit Weight/Output Power Suite Weight Weight Total Impulse

•Integration & Assembly •Structure •Attitude Control Subsystem •Attitude Determination •RCS •Thermal Control •Electrical Power Subsystem •Generation •Storage •Distribution •Conditioning •Telemetry, Tracking & Command •Transmitter/Amplifier •Receiver/Exciter •Transponder •Digital Electronics •Analog Electronics •Antenna •Propulsion (AKM)

$

Microwave Ferrite Devices

Program Level Costs

+ Quantity Weight Weight/Sync Orbit Weight/Output Power Component Power Suite Weight Weight

Space Veh Dry Wt

•Communications Payload •Transmitter/Amplifier •Receiver/Exciter •Transponder •Digital Electronics •Analog Electronics •Antennas

Total $

Microwave Ferrite Devices

Launch & Orbital Operations Support (LOOS)

231

$

$

• • • •

Prog Mgmt Sys Engr Sys Test Data

Parametric Cost Analysis Handbook

APPENDIX F

SOME CURRENTLY AVAILABLE SOFTWARE ESTIMATION PRODUCTS

PRODUCED BY THE SOFTWARE SUBGROUP OF THE SPACE SYSTEMS COST ANALYSIS GROUP

232

Parametric Cost Analysis Handbook

APPENDIX F SOME CURRENTLY AVAILABLE SOFTWARE ESTIMATION PRODUCTS

This section incorporates inputs from a variety of sources, primarily from model developers. No attempt has been made to substantiate the information at this point. Comments and updates are welcome, as are full descriptions of other models not discussed here.

Contents: PRICE-S REVIC SASET SEER-SEM SLIM SoftCost-R & SoftCost-Ada Other Models

233

Parametric Cost Analysis Handbook

PRICE-S This model was developed originally by Martin Marietta Price Systems (initially RCA Price, then GE Price) as one of a family of models for hardware and software cost estimation. Developed in 1977 by F. Freiman and Dr. R. Park, it was the first commercially available detailed parametric software cost model to be extensively marketed and used. In 1987, the model was modified and re-validated for modern software development practices. The PRICE-S model is proprietary, it can be leased for yearly use on IBM or compatible PC, and operates within Microsoft windows. It is also available for use on a UNIX workstation. The model is applicable to all types of software projects, and considers all DoD-STD-2167A development phases. Inputs One of the primary inputs for the PRICE-S model is source lines of code (SLOC). This may be input by the user or computed using either object-oriented or function point sizing models. Both sizing models are included in the PRICE-S package. Other key inputs include: 1.

2.

3.

4. 5. 6. 7. 8. 9. 10.

Application: a measure of the type (or types) of software, described by one of seven categories (mathematical, string manipulation, data storage and retrieval, on-line, realtime, interactive, or operating system). Productivity Factor: A calibratable parameter which relates the software program to the productivity, efficiency/inefficiencies, software development practices and management practices of the development organization. Complexities: Three complexity parameters which relate the project to the expected completion time, based on organizational experience, personnel, development tools, hardware characteristics, and other complicating factors. Platform: the operating environment, in terms of specification, structure and reliability requirements. Utilization: Percentage of hardware memory or processing speed utilized by the software. New Design/New Code: Percentage of new design and new code. Integration (Internal): Effort to integrate various software components together to form an integrated and tested CSCI. Integration (Extenal): Effort to integrate various software CSCI’s together to form an integrated and tested software system. Schedule: Software project start and/or end dates. Optional Input Parameters: Financial factors, escalation, risk simulation.

Processing The PRICE-S algorithms are published in the paper entitled "Central Equations of PRICE S" which is available from PRICE Systems. It states that PRICE-S computes a "weight" of software based on the product of instructions and application inputs. The productivity factor and complexity inputs are very sensitive parameters which affect effort and schedule calculations. Platform is known to be an exponential input; hence, it can be very sensitive. A new weighted design and code value are calculated by the model based on the type or category of instructions. Both new design and code affect schedule and cost calculations. Internal

234

Parametric Cost Analysis Handbook

integration input parameters affect the CSCI cost and schedule for integrating and testing the CSCI. The external integration input parameter is used to calculate software to software integration cost and schedule. Outputs PRICE-S computes an estimate in person effort (person hours or months). Effort can be converted to cost in dollars or other currency units using financial factors parameters. Software development schedules are calculated for nine DoD-STD-2167A phases: System Concept through Operational Test and Evaluation. Six elements of costs are calculated and reported for each schedule phase: Design Engineering, Programming, Data, Systems Engineering Project Management, Quality Assurance, and Configuration Management. The PRICE-S model also contains several optional outputs including over thirty graphs, Gantt charts, sensitivity matrices, resource expenditure profiles, schedule reports. In addition, Microsoft Project files, spreadsheet files, and risk analysis reports can be generated. The risk analysis report is a Cumulative Probability Distribution and is generated using either Monte Carlo or Latin Hypercube simulation. Calibration The PRICE-S model can be run in ECIRP (PRICE backwards) mode to calibrate selected parameters. The most common calibration is that of the productivity factor, which, according to the PRICE-S manual, tends to remain constant for a given organization. It is also possible to calibrate platform, application, and selected internal factors. Life Cycle Considerations The PRICE-S life cycle model, included in the PRICE-S package, is a detailed model which computes software support costs. The primary inputs include PRICE-S development inputs, support descriptors which include software support life, number of installations, expected growth, and support productivity factors. The model also has a modification mode which allows up to four modifications per software CSCI. The PRICE-S life cycle model calculates support effort and outputs the cost in three support phases: maintenance, enhancements, and growth. The model allocates effort or cost across six elements of costs for each support phase. Risk Analysis The PRICE-S model contains a robust Monte Carlo simulation utility, which facilitates rigorous risk analysis. Uncertainty can be characterized using probability distributions to define input parameters. Normal, Beta, Triangular and Uniform distributions are among those available. Simulation results are consolidated and reported as a probabilistic estimate. Contact Lockheed-Martin PRICE Systems 700 East Gate Drive, Suite 200 Mt. Laurel, NJ 08054 (800) 437-7423 a.k.a (800) 43PRICE

REVIC

235

Parametric Cost Analysis Handbook

The Revised Intermediate COCOMO (REVIC) model was developed by Ray Kile and the U.S. Air Force Cost Analysis Agency. It is a copyrighted program that runs under DOS on an IBM PC or compatible computer. The model predicts the development costs for software development from requirements analysis through completion of the software acceptance testing and maintenance costs for fifteen years. REVIC uses the intermediate COCOMO set of equations for calculating the effort (man-power in staff-months and staff-hours) and schedule (elapsed time in calendar months) to complete software development projects based on an estimate of the lines of code to be developed and a description of the development environment. The forms of the basic equations are: (I) MM = AB(KDSI)P(Fi) (2) TDEV = CD(MM) Equation (1) predicts the manpower in man-months (MM) based on the estimated lines of code to be developed (KDSI = Delivered Source Instructions in thousands) and the product of a group of environmental factors (Fi). The coefficients (A,C), exponents (B,D) and the factor (Fi) are determined by statistical analysis from a database of completed projects. These variables attempt to account for the variations in the total development environment (such as programmer's capabilities or experience with the hardware or software) that tend to increase or decrease the total effort and schedule. The results from equation (1) are input to equation (2) to determine the schedule (TDEV = Development Time) in months needed to complete the development. REVIC enhancement of the intermediate COCOMO includes: The addition of a fourth mode - Ada. Intermediate COCOMO has three modes of software development: organic, semi-detached, and embedded. These modes describe the overall software development in terms of size, number of interfaces, and complexity. REVIC adds a fourth mode, Ada development, to the model. This mode describes programs developed using an objectoriented analysis methodology or use of the Ada language (with separately cornpilable specifications and body code parts). Each mode provides a different set of coefficients for the basic equations. Addition of the first and last development phases. COCOMO provides a set of tables distributing the effort and schedule to the phases of development (system engineering, preliminary design, critical design, etc.) and activities (system analysis, coding, test plarming, etc.) as a percentage of the total. COCOMO covers four development phases (preliminary design, critical design, code and unit test, and integration and test) in the estimate. REVIC adds two more development phases: software requirements engineering, and integration & test after FQT. REVIC predicts the effort and schedule in the software requirements engineering phase by taking a percentage of the development phases. It provides a default value (12% for effort, 30% for schedule) for this percentage based on average programs, but allows the user to change the percentage.

236

Parametric Cost Analysis Handbook

COCOMO development phase ends at completion of the integration & test phase (after successful FQT). This phase covers the integration of software CSC's into CSCI's and testing of the CSCI's against the test criteria developed during the program. It does not include the systemlevel integration (commonly called software builds) of CSCI's, and the system-level testing to ensure that system-level specification requirements are met. The software to software and software to hardware integration and testing is accounted for in the Development Test and Evaluation (DT&E) phase. REVIC predicts the effort and schedule for this phase by taking a percentage of the development effort. REVIC provides a default percentage of 22% for effort and 26% for schedule based on average programs. It allows the user to modify these percentages if desired. Complete interface between the model and the user. Users are not required to have extensive knowledge about the model or detailed knowledge of algorithms. REVIC contains extensive prompting and help screens. REVIC also removes the typical intimidation factor that prevents analysts from successfully using models. Provide the capability to interactively constrain the schedules and staffing levels. Schedules can be constrained either in the aggregate or by phase of the development effort. Staffing can be constrained by phase. Using these features, the analyst can estimate cost overruns, underruns, and schedule slips at any major milestone by entering the actuals-to-date at any milestone, and letting the program calculate the remaining effort and schedule. Inputs:

Same as COCOMO inputs.

Processing While REVIC processing is mostly the same as Intermediate COCOMO, it provides a single weighted "average" distribution for effort and schedule, along with the ability to allow the user to vary the percentages in the system engineering and DT&E phases. On the other hand, COCOMO provides a table for distributing the effort and schedule over the development phases, depending on the size of the code being developed. REVIC has been enhanced by using statistical methods for determining the lines of code to be developed. Low, high, and most probable estimates for each CSC are used to calculate the effective lines of code and standard deviation. The effective lines of code and standard deviation are then used in the equations, rather than the linear sum of the estimates. This quantifies, and to some extent, reduces the existing uncertainties regarding software size. [see Boehm's Software Engineering Economics for a discussion of effective lines of code]. REVIC automatically performs sensitivity analysis showing the plus and minus three sigma values for effort, and the approximate resultant schedule. Outputs The user is presented with a menu allowing full exercise of the analytical features and displays of the program. All inputs are easily recalled and changed to facilitate analyses and the user can constrain the solution in a variety of ways. Calibration and Accuracy

237

Parametric Cost Analysis Handbook

REVIC's coefficients have been calibrated using recently completed DoD projects (development phase only) by using the techniques in Dr. Boehm's book. On the average, the values predicted by the effort and schedule equations in REVIC are higher than in COCOMO. A study validating REVIC equations using a database different from that used for initial calibration was published by the Air Force's HQ AFCMD/EPR. In terms of accuracy, the model compares favorably with expensive commercial models. The level of accuracy provided by the model is directly proportional to the user's confidence in the lines-of-code (LOC) estimates and a description of the development environment. When little is known about the project or environment, the model can be run leaving all environment parameters at their default (nominal) settings. The only required input is LOC. As the details of the project become known, the data file can be reloaded into the program, and the nominal values can be changed to reflect the new knowledge permitting continual improvement of the accuracy of the model. Maintenance Estimate REVIC provides an estimate for software maintenance over a fifteen year period by using the Boehm's COCOMO equation: (3) MMam = MMnom * ACT P (MFi), where MMnom is the result of equation (1) without multiplying by the environmental factor (Fi); ACT is annual change traffic as a percentage, and Mfi is the environmental factors for maintenance. REVIC provides a default percentage of ACT and allows it to be changed. REVIC also assumes a transition period after delivery of the software, during which residual errors are found before reaching a steady state condition. This provides a declining, positive delta to the ACT during the first three years. Beginning the fourth year, REVIC assumes the maintenance activity consists of both error correction and new software enhancements. Other Reference Materials There is context-sensitive help available on-line while running REVIC, and the information is enough to input data and obtain results in all cases. However, the developers recommend that Boehm's Software Engineering Economics be read to fully understand the implications of parametric modeling. Contact The Air Force Cost Analysis Agency has assumed responsibility for REVIC upkeep and distribution. For suggestions for upgrades or problem reporting, contact: Air Force Analysis Agency AFCSTC/IS (REVIC) 1111 Jefferson Davis Highway, Suite 403 Arlington, VA 22202 [information current as of 5/92] (703) 604-0412

238

Parametric Cost Analysis Handbook

REVIC is available on the Air Force cost bulletin board at (800) 344-3602 or from Washington, D.C. at . (703) 604-0412. Connection should be 1200 or 2400 BAUD with 8 bits, no parity, and one stop bit. [information current as of 5/92] REVIC and its user manual are also available on the Air Force Software Technology Support Center (STSC) bulletin board at autovon 458-7653 or (801) 777-7653. Connection at 2400 BAUD with 8 bits, no parity, and one stop bit. For access problems call Vern Phipps or Rick Dahl at (801) 777-7703. [information current as of 5/92]

239

Parametric Cost Analysis Handbook

SASET SASET is a sophisticated software cost estimating model built by Martin Marietta Astronautics during 1986-1990 under contract with the Naval Center for Cost Analysis under auspices of the Office of Naval research. Enhancements to the model were accomplished under contract with both the Navy and Air Force cost centers. The model combines database information from over 500 successfully completed software projects with "expert" factors to determine software development and maintenance costs, schedules, and sizing numbers used by engineers, estimators, and top-level management. Cost estimating relationships were created, project complexity factors were established, and independent variables were quantified. The result was database-derived software estimating equations for assembly and high-order language software. These equations and resulting software parametric models have been validated by comparing project sizing, labor actuals, and schedules with SASET outputs. During data collection, analysis, and model requirements generation activities, it was decided that the parametric model would include the entire software development life cycle from systems requirements through systems test, with an option to add maintenance, and provide budget and schedule outputs for the engineering development organizations. The database collection approach consists of decomposing software actuals by CSCI class, type and language. Inputs SASET starts from an initial state of no information about the software project, and then proceeds to prompt the user in an orderly fashion for the information necessary to gain insight and emulation of project environment, complexities, and sizing for up to 50 CSCI's, to produce schedule and effort outputs. Processing SASET is a database-derived expert system. The calibrated equations for effort budget and schedule are computed for the system environment, and software complexity, brought together for a common set and then multiplied against the equivalent new HOL SLOC. Outputs The computerized model has the capability of sizing, costing, scheduling and integrating up to 50 multiple CSCI's simultaneously. The model determines a labor estimate expressed in staff months and spread over the phases and subphases of software development life cycle for each individual CSCI and the total project and helps to establish a logical development sequence. The model also identifies an optimal schedule and effort. Calibration All input variables and calibration constants of SASET are open and capable of being changed. The calibration process requires a set of software development records as input. The required data from each record includes staff hours, source lines of code, schedule, and project complexity. Records are sorted by class (manned flight, avionics, ground, etc.) and type of software (system, application, support), then run through multiple regression analysis for productivity constants for each type of software.

240

Parametric Cost Analysis Handbook

Life Cycle Considerations SASET performs detailed environment effort allocation across subphases of the development life cycle. It will also spread calculated maintenance effort across time. The effort included is for engineering organizations only; any other efforts needing coverage need to be added manually. Contact Air Force Analysis Agency AFCSTC/IS (REVIC) 1111 Jefferson Davis Highway, Suite 403 Arlington, VA 22202 [information current as of 5/92] (703) 604-0412

241

Parametric Cost Analysis Handbook

SEER-SEM SEER-SEM is part of a family of software and hardware cost, schedule and risk estimation tools. SEER models run on IBM, Macintosh, and Sun/UNIX platforms with no special hardware requirements. SEER-SEM is used throughout the aerospace and defense industry on two continents. All issues found in today's software environments are addressed. Inputs SEER-SEM accepts source lines of code (SLOC) or function points or both. When selecting function points, the user may use IFPUG standard function points or SEER functionbased inputs which include internal functions. Users follow a Work Breakdown Structure (WBS) describing each CSCI, CSC, and CSU (module or element) to be estimated. Knowledge bases are used to provide fast and consistent inputs describing complexity, personnel capabilities and experience, development support environment, product development requirements, product reusability requirements, development environment complexity, target environment, schedule, staffing and probability. Users can modify all inputs to their specifications at any time. There are four sets of knowledge bases that automatically input environment factors. These knowledge bases cover a wide variety of scenarios and help users produce fast and reliable estimates. Knowledge bases are easily calibrated to user environments to give quick and accurate estimates for the entire life cycle. Users can also change and modify each input at any time. Knowledge bases include the following: Platform describes the primary operating platform. Platform knowledge bases include avionics, business, ground-based, manned space, unmanned space, shipboard, and more. Application describes the overall function of the software under estimation. Application knowledge bases include computer-aided design, command & control, database, MIS, office automation, radar, simulation, and more. Development Method describes the development methods to be used during the development. These knowledge bases include Ada, spiral, prototyping, object oriented design, evolving, traditional waterfall, and more. Development Standard describes the development documentation, quality, test standards and practices which will be followed during the development. These knowledge bases include commercial, ISO-9000, 2167A, 1703, 1679, 7935A and more. Processing SEER-SEM uses proprietary algorithms which are found in the back of the User's Manual. Parameter (input) sensitivities and other insights into the model are also found in the user's documentation. Knowledge bases can be printed out by users. SEER-SEM utilizes a unique process that simulates a 10,000 iteration Monte Carlo for risk analysis.

Outputs

242

Parametric Cost Analysis Handbook

SEER-SEM has almost 30 informative reports and charts covering all aspects of software costs, schedules and risk. The Quick Estimate Report is easily tailored to instantly give the user specific details for trade-off analyses and decision support information. A Detailed Staffing Profile follows SEI suggested staffing categories. Risk reports and graphs based on person months, costs, and schedule are standard features. SEER-SEM gives a minimum schedule output. However, schedules, personnel, and other factors can be changed to give effort and cost tradeoffs. Calibration Calibration of SEER-SEM involves the effort to customize input values to more closely reflect particular program development characteristics. SEER-SEM's Design To Technology and Design To Size functions provide the tools for calibration activities. The SEER knowledge bases are flexible and easy to create and modify, providing the user with a mechanism for building custom calibrated knowledge bases. Life Cycle Considerations SEER-SEM estimates all elements of the life-cycle, beginning with the preliminary design phase and ending with software maintenance. SEER-SEM has many features which support Life Cycle Cost Analysis. Total life cycle cost is reported in the Basic Estimate Report, Activity Report, and the Labor Allocation Reports. The Set Reference feature allows for quick analysis of what happens to both development and maintenance costs with the change of any parameter. Maintenance SEER-SEM baseline maintenance includes all adaptive, perfective and corrective maintenance. Additionally, you may add annual change rate and growth percents to anticipate any functional growth or enhancements over the software maintenance period. Enhancements and block upgrades can also be estimated. Contact Galorath Associates, Inc., SEER Technologies Division P.O. Box 90579 Los Angeles, CA 90009 (3 1 0)670-3404.

243

Parametric Cost Analysis Handbook

SLIM SLIM for Windows 3.1 was developed by Quantitative Software Management, Inc. (QSM). All of the theory behind the model has been published by Prentice Hall in 1992 in the book, Measures for Excellence: Reliable Software, on Time, Within Budget by Lawrence H. Putnam and Ware Meyers. SLIM is based on QSM's Software Equation. This was derived from the Rayleigh-Norden model and has been validated over a 15 year time period with thousands of real, completed projects. The equation takes the conceptual form: (1) Quantity of Function = Process Productivity * Effort * Schedule This means that the product of the time and effort coupled with the process productivity of the development organization determines how much function can be delivered. Extensive empirical study of software data has shown that very strong linearities exist in software behavior. This is taken into account by the calculational form of the software equation and discloses how these non-linearities can be exploited by management. The calculation form of the equation is expressed: (2) Size = Process Productivity Parameter * (Effort/B)(1/3) * Time(4/3) where, 1. 2. 3. 4. 5.

The process Productivity Parameter is the development process proficiency of the organization. It is determined from historical data. Size is the quantity of function created in source lines of code written, function points, objects, or other measures of function. Effort is the development effort required. It includes all categories of labor used on the project. Effort is consistent with the time specified below. B is a complexity adjustment factor. It provides for specialized skills for integration testing, documentation, and management as the size of the system increases. Time is the elapsed calendar development time from the start of detailed design until the product is ready to enter into operational service (frequently this is a 95% reliability level).

SLIM is applicable to all types and sizes of projects. It computes schedule, effort, cost, staffing for all software development phases and reliability for the main construction phase. Because the software equation effectively models design intensive processes and is not methodology dependent, SLIM works well with waterfall, spiral, incremental, and prototyping development methodologies. It works with all languages, and function points as well as other sizing metrics. It is specifically designed to address the concerns of senior management, such as: 1. 2.

What options are available if the schedule is accelerated by four months to meet a tight market window? How many people must be added to get two months schedule compression and how much will it cost?

244

Parametric Cost Analysis Handbook

3. 4. 5.

When will the defect rate be low enough so I can ship a reliable product and have satisfied customers? If the requirements grow or substantially change, what will be the impact on schedule, cost, and reliability? How can I quantify the value of my process improvement program?

Computing Platforms SLIM is available for use on IBM PC or compatible machines running Windows 3.1. SLIM is licensed on an annual basis with full support and free upgrades. Inputs The primary input for SLIM is SLOC, function points, objects, CSCI, or any valid measure of function to be created. The model uses size ranges for input: minimum, most likely, and maximum. Other important inputs include: 1. 2. 3. 4. 5.

6.

7. 8.

Language: Multiple choices and mixes. System Type: One of nine (business, scientific, command & control, real time, etc.). Environmental Information: Tools, methods, practices, database usage; standards in place and adherence and usage of those standards. Experience: Personnel skill and qualifications. Process Productivity Parameter: a macroscopic factor determined by calibration from historical data. It is a reliable tuning factor that accurately reflects application complexity and the efficiency of the organization in building software. This is a secsitive parameter that is capable of measuring real productivity and process improvement. SLIM contains and expert system to determine the Process Productivity Parameter when the user has no historical data. This [non-linear] parameter is dealt with in terms of a linear scale ranging from 0 to 40. Management Constraints: Maximum allowavle schedule, manimum cost, maximum and minimum staff size, required reliability at the time the software goes into service as well as the desired probabilities for each of these constraints. Accounting: Labor rates, inflation rates, and other economic factors. Flexibility: Extensive tailoring for milestones, phase definitions, and fraction of time and effort applied to each phase based on the organization’s own history.

Processing There are three primary modes of operation: building and using an historical database, performing estimating and analysis, and creating presentations and reports. For estimation, SLIM uses the software equation in conjunction with management constraints for schedule, cost, staffing and required reliability to determine an optimal solution with the highest probability of successful completion. Through Monte Carlo simulation techniques, the size range estimates are mapped through the software equation to provide estimates of the uncertainty in schedule, cost staffing and reliability. The solution obtained can be compared with the user's historical data and QSM's historical data to test its reasonableness. This discloses impossible or highly improbable solutions so that expensive mistakes are avoided.

245

Parametric Cost Analysis Handbook

Outputs The primary output of SLIM is the optimal solution, which provides development time, cost, effort and reliability expected at delivery. It also provides comprehensive sensitivity and risk profiles for all key input and output variables, and a consistency check with similar projects. SLIM's graphical interactive user interface makes it easy to explore quickly extensive tradeoff and "what if" scenarios including design to cost, schedule, effort and risk. It has 181 different output tables and graphs from which the user can choose. These outputs constitute a comprehensive set of development plans to measure and control the project while it is underway. Calibration The process productivity parameter for SLIM can (and should) be obtained by calibration using historical data. All that is required are project size, development time and effort. These numbers are input into the software equation to solve for the process productivity. The historical data can also be used to compare with any current solution to compare for reasonableness. Life Cycle Considerations The user can customize his life cycle in terms of phases, sub-phases, staffing profile shapes and milestones. Explicit shapes are provided after delivery to deal with maintenance, ongoing support and enhancement releases. Contact Quantitative Software Management, Inc. 1057 Waverly Way McLean, VA 22102 (703) 790-0055

246

Parametric Cost Analysis Handbook

SOFTCOST-R & SOFTCOST-ADA The SoftCost-R model was developed by Dr. Don Reifer based on the work of Dr. Robert Tausworthe of the NASA Jet Propulsion Laboratory. SoftCost is now marketed by Resource Calculations, Inc. of Englewood, Colorado. It contains a database of over 1500 data processing, scientific and real-time programs. SoftCost-R is applicable to all types of prograrns and considers all phases of the software development cycle. The model is available for lease on IBM PC's. A separate model SoftCost-Ada is available to model Ada language and other objectoriented environments. SoftCost-Ada has been developed to match the new object-oriented and reuse paradigm which are emerging not only in Ada, but also C++ and other object-oriented techniques. It contains a database of over 150 completed projects, primarily Ada. SoftCost-R Inputs A key input of SoftCost-R is size, which can either be directly input in SLOC or computed from function points. SoftCost-R uses a more sophisticated sizing model than COCOMO; besides reused code, sizes of modules added or deleted may be included. The other inputs are in four categories like COCOMO. Some SoftCost-R inputs are similar to COCOMO, but many of the more than thirty inputs are unique. Examples of unique inputs are use of peer reviews, customer experience, and degree of standardization. Each input except size requires a rating ranging from "very low" to "extra high", with "nominal" ratings having no effect on effort calculations. SoftCost-R also uses COCOMO inputs to compare the results of SoftCost-R with those of an updated version of COCOMO. SoftCost-Ada Inputs In the main, the inputs are the same as SoftCost-R, with some changes to reflect the new paradigm. There is no COCOMO comparison. Processing SoftCost-R is not a simple regression model. It uses powerful multivariable differential calculus to develop solutions relying on the T-W probability distribution. This provides the ability for the user to perform "what-if" analysis and look at what would happen to schedule if effort were constrained. Such a capability is not present in COCOMO. SoftCost-R is one of the few models for which the mathematical algorithms are completely described in the user's manual. The SoftCost-R equation is: (1) PM = P0 * Al * A2 * (SLOC)B where, 1. 2. 3. 4. 5.

PM = number of person-months, P0 is a constant factor that may be calibrated, A 1 is the "Reifer cost factor" which is an exponential product of nine inputs, A2 is a productivity factor computed from 22 inputs, B is an exponent which may be calibrated.

247

Parametric Cost Analysis Handbook

The user's manual illustrates values assigned to ratings for all model inputs to help the user understand the effect of each input on effort and schedule. Outputs SoftCost-R computes an estimate in person-months of effort and schedule for each project, plus a productivity value. Other outputs include a side-by-side comparison with a recent version of COCOMO, several "what if" analysis options, a resource allocation summary for any of three development methods (traditional waterfall, incremental development, or Ada objectoriented), and schedule outputs for Gantt and PERT charts. SoftCost-Ada output formats are similar, and can interface with project planning tools in the same way. Calibration The model contains a calibration file which contains values for multiple calibration constants and cost drivers. The user may change these values to better describe the user's unique environment, and store alternative calibration and WBS files for different jobs. SoftCost-Ada and SoftCost-R are similar. Life Cycle Considerations SoftCost-R contains a separate life cycle model for support costs. In addition to SoftCost-R development inputs, life cycle inputs include annual change traffic, length of support period, a sustaining engineering factor, and economic factors. In addition to annual and total support costs, the life cycle model has optional reports for various staffing options, fixed levels of maintenance, and fixed work force levels. Both SoftCost versions are similar, and use the same staff limited approach to life cycle resource allocation. Contact Mr. A.J. (Tony) Collins Resource Calculations, Inc. 7853 East Arapahoe Court, Suite 2500 Englewood, CO 80112-1361 Telephone: (303) 267-0379 Facsimile: (303) 220-5620

248

Parametric Cost Analysis Handbook

OTHER MODELS CHECKPOINT Software Productivity Research, Inc. I New England Park Burlington, MA 01803 (617) 273-0140 COSTAR SoftStar Systems 28 Ponemah Road Arnherst,NH 03031 (603) 672-0987 SOFTST

R Softstar Systems 28 Ponemah Road Arnherst,NH 03031 (603) 672-0987 Sells COSTAR, a COCOMO variant; supports Ada COCOMO and function points IBM PC, VAX/VMS Dan Ligett (personal letter - Jan 16, 1987) [info checked brochure 10/92] SWAN IIT Research Institute 201 Mill Street Rome, NY 13440 (315)336-2359 Fax: (315) 339-7002 Personto Contact: Anthony H. Williarns (315)339-7105 [ok 12/13/93] A COCOMO variant developed for US Arrny, PM Training Devices, 12350 Research Parkway, Orlando, FL. 32826 Written in Ada; runs on IBM AT, MS-DOS 3.0 or higher Includes Ada COCOMO & Function Point Analysis Target Software Dr. George Bozoki 552 Marine Parkway, #1202 Redwood City, CA 94065 (415)592-2560

SSM, the Software Sizing Model

249

Parametric Cost Analysis Handbook

Computer Economics 4560 Admiralty Way, Suite 109 Marina Del Rey, CA 90292 (213) 827-7300 Jensen Model ("System 4"), CEIS (CEI Sizer), training [ref. 1987 IITRI Software Sizing model paper] Mainstay Software Corporation 1099 18th Street, Suite 2920 Denver, COLO 80202 (303) 298-8961 Dan Walkovitz, President Product: Mainstay - an analytical database system [ok 8/92, Setzer] Howard Rubin Associates, Inc. 1666 Massachusetts Avenue, Suite 7 Lexington, MA. 02173 (617)861-7171 Products: FPXpert - an expert system for function point counting, analysis and management; RA-METRICS - a measurement tool and metrics respository; primarily oriented toward information systems, promotes concept of "measurement dashboard" COSTMODL Developed at the Software Technology Branch at NASA/JSC to be used for Space Station Freedom Software Support Environment and will be available from COSMIC.

250

Parametric Cost Estimating Handbook

APPENDIX G

PARAMETRIC ESTIMATING SYSTEM CHECKLIST

251

Parametric Cost Estimating Handbook

APPENDIX G PARAMETRIC ESTIMATING SYSTEM CHECKLIST

Elements of Good Estimating Practice The list that follows identifies elements of good practice that support reliable estimating processes.

Questions: Are there other elements of good estimating practice that should be listed? Are there other items of evidence that indicate good practice that should be listed? Should any of the elements or items of evidence be reworded or eliminated?

Note: The objective here is to identify elements of good practice that are distinguished, if possible, from elements of good estimating processes. This means, among other things, that the elements in the list should be independent of the estimating methods or models used. The element in this list should not tell people how to estimate. They should simply provide guidelines and testable criteria for doing it well.

252

Parametric Cost Estimating Handbook

Why Do We Estimate Size, Cost, and Schedule?

To scope proposed tasks To explore alternative system concepts design to cost/budget To explore alternative design concepts To explore alternative proposals for enhancements and upgrades To identify key design elements To identify key process parameters prioritize needs vs wants To identify key assumptions To identify and quantify uncertainties To identify tasks and their relationships To assess schedule feasibility To identify, allocate and schedule resources To assess an organization’s ability to perform within targeted costs To evaluate the consequences of internal and external constraints To establish achievable objectives To establish a basis for quality service To establish commitments To bound the risk against customer needs To balance levels of risk against customer needs To provide a basis of successful risk management build vs buy analysis To prepare successful proposals To evaluate proposals from competing bidders To establish baselines for project tracking enhance/reuse vs redesign analysis To predict life-cycle costs To predict returns on investments To provide information for establishing business and investment strategies

253

Parametric Cost Estimating Handbook

ELEMENTS OF GOOD ESTIMATING PRACTICE

1.

2.

Element The objective (purpose) of the estimate is stated in writing

Evidence of Maturity Information objectively captured in historic database with a standard structure for ease of analysis

The product to be produced is clearly described

Estimates for product size and content are supported by systematic engineering analyses The terms and parameters that describe the product enable comparisons to be made with other products

3.

The tasks to be estimated are clearly identified

Checklists are used to identify the elements and activities included (and excluded from) the estimate

4.

People from related but different projects or disciplines are involved in the estimating process

Estimating and analysis team members found in historical database

5.

The validity of the estimate is supported by relating it demonstrated performance on completed projects

Values assigned to cost model parameters are supported by comparisons to values used to describe completed projects The reasons for values assigned each cost model parameter are documented in writing

6.

More than one cost model or estimating approach is used

Differences in results are analyzed and accounted for

7.

Potential cost and schedules impacts are estimated for all identified tasks

A structured process is used to identify and scope technical risks Uncertainties in values of descriptive parameters are identified and quantified The effects of uncertainties in descriptive parameters are evaluated

254

Parametric Cost Estimating Handbook

8.

Dictated schedules are analyzed for impacts on cost

Managers (and customers, where appropriate) are informed of potential cost savings associated with alternative schedules

9.

Estimates (or estimates to complete) are updated whenever...

Captured in historical database

- changes to requirements affect cost or schedule

Program manager uses the process to manage the program and associated risks

- constraints change

Process flags management exceeds defined limits

when

risk

- actual values for product or process parameters are found to be significantly different from those on which the plan is based - tracking measures indicate that critical path tasks cannot be completed as planned 10.

The estimating origination has a method (such as a database) for organizing and retaining information on completed projects

Elements in the database are recorded and described in ways that avoid misinterpretation Schedule milestones (start and finish dates) are described in terms of criteria for initiation or completion Effort measures account for unpaid overtime The database is populated with a useful set of completed projects Cost models are used to provide consistent frameworks (standard terms, WBS, parameters, etc.) for recording historical data Inconsistencies in historical data are sought out and explained (this is perhaps best done with the same cost models used for estimating)

255

Parametric Cost Estimating Handbook

11.

Postmortems are held completion of each project

at

the

Technical seminars and training programs incorporate this information in a timely manner

- to ensure that events that affected cost or schedule are described and recorded while still fresh in people’s minds - to ensure that recorded data are valid 12.

The emphasis throughout is on developing consistency in describing completed projects and in relating new work to demonstrated performance on those projects

The consistency achieved when fitting cost models to completed projects is measured and tracked The term “model accuracy” (as in “the model made the estimate”) is never used

256

Parametric Cost Estimating Handbook

INDICATORS OF ESTIMATING CAPABILITY What organizational support is needed to successfully establish and sustain a reliable estimating capability? The following checklist identifies seven indicators of good support. For each indicator, it lists items of evidence one can look for to help evaluate the quality of this support. Questions: Are there other indicators of organizations support that should be listed? Are there other items of evidence that should be listed? Should any of the indicators or items of evidence be reworded or eliminated?

Note: The objective of this checklist is to provide guidance to organizations that are seeking to establish and sustain a successful estimating capability.

257

Parametric Cost Estimating Handbook

Indicators of Estimating Capability - A Checklist for Successful Estimating Environments -

1.

Indicator Management acknowledges its responsibility for developing and sustaining an estimating capability

Evidence of Maturity One or more managers have been assigned responsibility for developing and sustaining the organization’s estimating capability The responsibility has been in place for sufficient time to be effective At least one person in the organization has a standing assignment as an estimator

2.

The estimating function is funded

The funding provides for: -

establishing and sustaining a corporate memory preparing estimates capturing data from ongoing and completed projects acquiring and using estimating models and tools educating and training estimators

The funding is adequate (past, present, and future) The funding is stable 3.

Estimators have been equipped with the tools and training needed for reliable estimating

Estimators have up-to-date desktop facilities (hardware and software)

computing

Cost models and software such as spreadsheets, databases, and statistical programs have been acquired or developed Estimators have received training in the cost, schedule, and size models they use Estimators have received training in estimating and model building Independence/objectivity

4.

The people assigned as estimators are experienced and capable

The estimators have professional experience/ knowledge with the processes whose costs are estimated

258

Parametric Cost Estimating Handbook

Estimators have educational backgrounds that support quantitative analysis as well as qualitative The average years of estimating experience among people assigned as estimators is computed and tracked Consistent methods are used for capturing and recording information from completed projects Cost model calibrations are up-to-date Estimators participate in professional activities or societies related to estimating The people assigned as estimators are viewed as a repository of estimating expertise 5.

6.

The estimating capability of the organization is quantified, tracked, and evaluated

Management tracks and reviews the effectiveness of its estimating capability

Recognition and career paths exists such that qualified people want to become estimators

Estimating is seen by both employees and managers as a career-broadening assignment

- what is working well - what’s not working well - opportunities for improvement

Previous estimators have moved on to positions of equal or higher responsibility People volunteer to be estimators

7.

Estimators help quantify and track progress in software process improvement

Cost models are used to account for factors that make projects different, so that process improvements can be meaningfully measured and compared Trend analyzed derived form cost model calibrations are used to track changes in the organization’s processing and performance parameters Cost models are used as on-going tools in managing program execution

259

Parametric Cost Estimating Handbook

A Manager’s Checklist for Validating Cost and Schedule Estimates When you (or your boss) receive a cost or schedule estimate, what should you look for to determine your willingness to rely on that estimate? The checklist on the following pages identifies key issues that have been identified as important to estimates we can trust. Each issue is associated with elements of evidence that, if presents support the credibility of the estimate. Question: Are there other issues that you (or your boss) look at? Are there other issues that you should be looking at? What evidence relative to any of these issues would support you willingness to rely on an estimate? Should any parts of the present checklist be reworded or eliminated?

Note: The objective of this checklist is to provide operational guidelines that help people evaluate the credibility of cost and schedule estimates for software projects. A secondary objective is to motivate and guide organizations toward improving estimating processes and practices.

260