GUIDANCE
Toolsets/ Human Factors

 

Appendix E Human Factors in the Investment Analysis Process

 

PURPOSE: An investment analysis is conducted to determine the most advantageous solution to an approved mission need. In general, it involves development of operational requirements, conduct of a market survey to determine industry capabilities, analysis of various alternative approaches, and a determination of what the FAA can afford. The purpose of human factors in the investment analysis process is to ensure that:

  • Human-system capabilities and limitations are properly reflected in the system requirements
  • Human-system performance characteristics and their associated cost, benefits, and risks assist in deciding among alternatives (especially since lifecycle operation and support costs are often largely dependent upon personnel-related costs)
  • Human-system performance risks are appropriately addressed in program baselines

HOW TO:

Introduction: The investment analysis must identify for each alternative the full range of human factors and interfaces (e.g., cognitive, organizational, physical, functional, environmental) necessary to achieve an acceptable level of performance for operating, maintaining, and supporting the system in concert with meeting the system’s functional requirements. The analysis should provide information on what is known and unknown about the human-system performance risks in meeting minimum system performance requirements.

Human factors that are relevant to meeting system performance and functional requirements include:

  1. human performance (e.g., human capabilities and limitations, workload, function allocation, hardware and software design, decision aids, environmental constraints, and team versus individual performance)
  2. training (e.g., length of training, training effectiveness, retraining, training devices and facilities, and embedded training)
  3. staffing (e.g., staffing levels, team composition, and organizational structure)
  4. personnel selection (e.g., minimum skill levels, special skills, and experience levels)
  5. safety and health aspects (e.g., hazardous materials or conditions, system or equipment design, operational or procedural constraints, biomedical influences, protective equipment, and required warnings and alarms).

The human factors support to the Investment Analysis process and Team follows the general process flow for Investment Analyses which includes the investment analysis planning, requirements definition, alternative solution identification and analysis, affordability assessment, acquisition program baseline development, and support for the investment analysis reporting, briefing, and decision. Support is provided by a designated, qualified Human Factors Coordinator (HFC). (See Table E-1 for a description of the role of the Human Factors Coordinator.)

Investment Analysis Plan: The Investment Analysis Plan (IAP) provides the planning information necessary for conducting the particular investment analysis in a timely and efficient manner. It must be completed early in the investment analysis phase. The IAP provides:

  • the composition of the Investment Analysis (IA) Team:
  • a schedule for completing the various activities within the investment analysis process
  • the assignment of roles and responsibilities for accomplishing the necessary activities
  • a list of all alternatives identified and the end set chosen for further analysis as candidate solutions.

The inclusion of human factors in the investment analysis process is dependent upon the groundwork that is laid in the IAP. Human factors inputs to the IAP include (as available) information about salient human factors issues, how human factors engineering and these specific issues will be assessed, and human factors activities needed to support the investment analysis process. Other information about schedules, costs, assessment criteria, roles and responsibilities may be addressed as appropriate.

Requirements Definition Activities: Requirements are developed early in the investment analysis process by the sponsoring organization. Capability shortfalls or technological opportunities identified in the MNS are translated into essential top level operational and functional requirements. An Initial Requirements Document (iRD) is prepared and continuously updated throughout the Investment Analysis process. Requirements evolve into greater specificity throughout the process to support detailed market, investment, and affordability analyses.

The iRD establishes the baseline criteria for selecting candidate solutions, conducting market analyses, analyzing alternatives, and performing affordability assessments to provide the best overall approach for satisfying the mission need.

Throughout the alternatives and affordability assessment phases of the investment analysis process, requirements are evaluated against cost, benefit, schedule, and performance considerations.

Requirements that are descriptive enough of what is being asked of industry to satisfy (via a contract or other government vehicle) will be provided to the IA Team to conduct the market analysis.

Human factors inputs to the Requirements Document identify requirements for human performance factors that may impact system design. Broad cognitive, physical, and sensory requirements for the operator, maintainer, and support personnel that contribute to or constrain total system performance are established. Any safety, health hazards, or critical errors that reduce job performance or system effectiveness are defined. The staffing and training concepts are also described.

Alternative Solution Identification and Analysis: Conducting the alternative solution identification and analysis entails the following human factors activities:

  1. Alternative Identification: The HFC on the IA Team assists in identifying alternative solutions to meet the desired capability. Having identified the requirements (in the previous step), the HFC will assist in identifying each alternative’s human factors approach for the various types of acquisition and system upgrade solutions (e.g., NDI, COTS), as well as for those that may not entail a material solution. (Non-material solutions include procedural, training, staffing of special skills or abilities, or job or organizational design changes that will achieve the mission need.)


  2. Issue Identification: Using the alternatives identified, the initial or refined requirements, the predecessor system performance, and the critical operational issues, the HFC establishes a list of the human factors issues (explicit and implied) that potentially have an effect on the performance of the system. Initially these issues may be concerns with broad categories of human-system performance such as manpower requirements, training requirements, human-system effectiveness, and suitability. As the IA Team continues to refine their work, these issues will become more defined and refined. (See Table E-2: Human Factors Issues in Investment Analyses.)


  3. Evaluation Criteria Selection: Using the results of the mission analysis and based on the initial or refined human factors requirements and issues for the alternative solutions, the HFC identifies the human factors criteria that may be used to help select a preferred alternative. The HFC begins to identify the human factors criteria, measures, thresholds, and data needed to assess the issues and alternatives from a human performance and human resource perspective. These criteria include quantitative and qualitative information about the operation and maintenance of the alternative solutions. Criteria relevant to the solution selection includes human factors components of cost, benefits, schedule, and performance parameters:

    - Cost and benefit criteria may include funding for research, acquisition, and life-cycle support related to manpower levels; cost or savings related to type and skill of required personnel; training costs or savings; and equipment costs or savings necessary to achieve the appropriate level of human (-system) performance.

    - Schedule criteria may include the amount of estimated time necessary to identify and resolve human factors issues or the amount of risk associated with resolving human factors issues.

    - Performance criteria may include human-system measures of effectiveness, human-system measures of suitability, workload, usability, personnel and staffing requirements, and considerations of performance payoffs from training. Measures of these criteria may address the nature of operator tasks involved, accuracy and error rates, training time, CHI complexity, design guideline compliance, or other measures.

  4. Market Analysis Participation: The HFC participation in the market analysis provides support for the assessment of candidate solutions from the human-system performance and ergonomic perspective. The HFC provides a list of issues that should be explored and information that should be collected during the conduct of the market analysis. These issues and information requirements will be derived from the issue identification step discussed above. Issues to be addressed during market analysis may include:

    - special skills and training required

    - special tools and software required

    - complexity of system hardware and software designs

    - human-system performance demonstrations, testing results, or guarantees

    - operator and maintenance performance records on fielded systems

  5. Alternatives Analysis: Using the IA Team’s selection criteria (e.g., constraints, limitations, costs, benefits, risks), the HFC provides human factors input to the analysis of the alternatives from a human factors perspective. The IA Team compiles a comparative assessment of the alternatives (from the human factors perspective) that will enable the program to establish the importance of the human factors criteria relative to other solution characteristics or functional assessments (e.g., use terms such as current dollars, system throughput, or program schedule impacts).

    6. - Human Factors Impact: For all alternatives identified (using the criteria developed, the market survey results, and subsequent analyses), determine the human factors implications of each alternative (in absolute terms or in terms that are relative to the other alternatives). That is, determine the sensitivity of the alternative solution to the range of human factors implications and concerns in view of the alternative’s complexity, human-system interface, technology reach for operators and maintainers, and schedule. In order to assess the total human factors impact, it may be necessary first to determine the impact on each controller/maintainer, each site, or each component of the system. For Example:
    Issue/ Criteria Alternative #1 Alternative #2
    Training 20 hours/controller 60 hours/controller
    Staffing 3 additional personnel/shift 1 additional personnel/shift
    Organizational Design Major structural changes No structural changes

    - Risk Assessment: For each of the alternatives presented, identify the risks associated with the human factors component of the proposed solution. Perform sensitivity analysis of the alternative solutions to potential or probable program outcomes. Identify and assess all risks (known and unknown) to include salient research requirements, key schedule considerations, and human-system performance parameters for the human factors dimensions (e.g., staffing, training, usability, CHI, human-system performance, staffing levels, organizational design considerations, health and safety issues).

    - Human Factors Evaluation: Identify and summarize the human factors impact on cost, schedule, priority, and resources associated with each candidate solution. Use a risk assessment tool or technique such as the Human Factors Risk Assessment Guide for Investment Analysis prepared by ASD-430 to assist in structuring the approach to estimating risk. Risk assessments should consider, at a minimum:

    - How much the cost of each alternative is affected by human factors considerations (e.g., staffing, training, CHI design)

    - How much it will cost to conduct the human factors activities associated with each solution

    - The probable or necessary schedule for the human factors activities to be properly conducted for each candidate solution

    - How the human factors component may affect the schedule of other activities for each candidate solution

    - How the human factors component of each candidate solution may affect the priority of the capability being acquired

    - How sensitive the human factors planned activities and issues are to the available funding/resources.

    For each alternative, the IAT considers (as appropriate) the cost, benefit, schedule, and performance impact of critical human factors issues. (See Table E-3 for a brief discussion on estimating human factors costs/benefits.) A notional display format is at figure E-1.1.

    Affordability Assessment: The human factors coordinator participates in the affordability assessment through the previous activities that provided quantitative and qualitative resource and schedule information about the alternatives. Additional support is rendered as required.

    Acquisition Program Baseline Development: The HFC provides input to the acquisition program baseline by conducting the following activities:

      - Determine the human factors cost, benefit, schedule, and performance baselines for each candidate solution

      - Identify the human factors and human performance measures and thresholds to be achieved (e.g., for the equipment, software, environment, support concepts, and configurations expected for the solution)

      - Determine the human factors activities to be undertaken during the program, the schedule for conducting them, their relative priority, and the expected costs to be incurred

      - Calculate or estimate the relative or absolute benefits of the human factors component of each solution in terms of decision criteria (e.g., cost, schedule, human-system performance)

    Investment Analysis Report, Briefing, and Decision: The HFC provides summary human factors information to support the reporting, briefing, and decision making for the Investment Analysis Joint Resources Council and the Investment Analysis Report. Included are any assumptions, constraints, and limitations that impact on the recommendation. The HFC provides a discussion of the recommendations from a human factors perspective using the results of the analysis of the alternatives, assessment of the risks, affordability assessment, and baseline development. The HFC also provides a list of the human factors issues and evaluation criteria, and a discussion of the importance of them relative to other issues and criteria to be used in solution selection.

     

    CHECKLIST: In conducting the human factors activities in support of the investment analysis phase, the following items should be considered:

    • Has responsibility been clearly designated for the collection of human factors information and for the conduct of human factors supporting activities?
    • Has early human factors participation in Requirements and Investment Analysis teams been organized (especially for plans and schedules of support activities)?
    • Has a foundation been established for collection of human factors data (e.g. parsing of human factors elements from critical operational issues, human factors issue and criteria identification, operational concepts analysis, human-system performance expectations, identification of critical tasks and interfaces)?
    • Have human factors requirements been developed?
    • Is time allocated to scope the human-system performance issues (e.g., during market analysis)?
    • Are human performance criteria and metrics being used (such as staffing requirements, errors rates, satisfactory performance levels, training time, operations and support costs, trouble reports)?
    • Is information to be obtained on how much system training, staffing, and performance is needed to achieve program objectives?
    • Is a controlled repository established for human factors data to be used across programs and projects?
    • Is funding provided in the budget to conduct the IA human factors tasks and analyses, and is a separate line item used for human factors IA tasks?
    • Are human performance analyses (e.g. functional, task, preliminary training analyses; use of scripted action sequences) to be conducted, including those to identify (root) causal information?
    • Have procedures been instituted to manage the flow of human factors information from disparate program components?
    • Do cost, schedule, and performance baselines reflect the detail necessary to cause the identification or resolution of human-system performance issues/risks?

     

    Table E-1: General Role of the Human Factors Coordinator in Investment Analyses

    The Human Factors Coordinator on the Investment Analysis Team provides the support for the integration of human factors engineering in the investment analysis phase of system development and acquisition. The HFC helps the Investment Analysis Team to initiate, structure, direct, and monitor their human factors efforts. The HFC serves with IA Team to identify, define, analyze, and report on human performance and human factors engineering considerations to ensure they are incorporated in investment decisions. Typical human-system performance and human factors engineering studies and analyses conducted, sponsored, or supported by the HFC include requirements analyses, baselines performance studies, trade-off determinations, alternative analyses, lifecycle cost estimates, cost-benefit analyses, risk assessments, supportability assessments, and operational suitability assessments. The HFC helps identify system specific and aggregate technical human factors engineering problems and issues that might otherwise go undetected for their obscurity, complexity, or elaborate inter-relationships. The human performance considerations are developed for staffing levels, operator and maintainer skills, training strategies, human-computer interface, human engineering design features, safety and health issues, and workload and operational performance considerations in procedures and other human-system interfaces. The HFC facilitates the establishment of the necessary tools, techniques, methods, databases, metrics, measures, criteria, and lessons learned to conduct human factors analyses in investment analysis activities. The HFC provides technical quality control of human factors products to the IA Team, participates in special working groups, assists in team reviews, helps prepare IA documentation, and collaborates on technical exchanges among government and contractor personnel.

    Table E-2: Human Factors Issues in Investment Analyses

    During the conduct of the Investment Analysis, the following issues may need to be assessed:

    • Workload: Operator and maintainer task performance and workload
    • Training: Minimized need for operator and maintainer training
    • Functional Design: Equipment design for simplicity, consistency with the desired human-system interface functions, and compatibility with the expected operation and maintenance concepts
    • CHI: Standardization of computer-human interface (to address common functions employ similar user dialogues, interfaces, and procedures)
    • Staffing: Accommodation of constraints and opportunities on staffing levels and organizational structures
    • Safety and Health: Prevention of operator and maintainer exposure to safety and health hazards
    • Special Skills and Tools: Considerations to minimize the need for special or unique operator or maintainer skills, abilities, tools, or characteristics
    • Work Space: Adequacy of work space for personnel and their tools and equipment, and sufficient space for the movements and actions they perform during operational and maintenance tasks under normal, adverse, and emergency conditions
    • Displays and Controls: Design and arrangement of displays and controls (to be consistent with the operator’s and maintainer’s natural sequence of operational actions)
    • Information Requirements: Availability of information needed by the operator and maintainer for a specific task when it is needed and in the appropriate sequence
    • Display Presentation: Ability of labels, symbols, colors, terms, acronyms, abbreviations, formats, and data fields to be consistent across the display sets, and enhance operator and maintainer performance
    • Visual/Aural Alerts: Design of visual and auditory alerts (including error messages) to invoke the necessary operator and maintainer response
    • I/O Devices: Capability of input and output devices and methods for performing the task quickly and accurately, especially critical tasks
    • Communications: System design considerations to enhance required user communications and teamwork
    • Procedures: Design of operation and maintenance procedures for simplicity and consistency with the desired human-system interface functions
    • Anthropometrics: System design accommodation of personnel (e.g., from the 5th through 95th percentile levels of the human physical characteristics) represented in the user population
    • Documentation: Preparation of user documentation and technical manuals (including any electronic HELP functions) in a suitable format of information presentation, at the appropriate reading level, and with the required degree of technical sophistication and clarity
    • Environment: Accommodation of environmental factors (including extremes) to which it will be subjected and their effects on human-system performance

    Table E-3: Brief Description of Human Factors Cost/Benefit Estimates

    The costs/benefits of human factors must be analyzed within the context of the capability being acquired to meet the mission need. Like other attributes of the alternatives, the human factors contribution to the system costs/benefits should be assessed in both qualitative and quantitative terms, especially as they relate to the measures and criteria established for the alternatives’ evaluation.

    The type of acquisition will also affect the approach. Thus, the analysis approach taken for NDI/COTS will likely differ from the analysis approach for developmental acquisitions. For example, the former assesses the relative costs/benefits among solution alternatives or vendor products, while the latter assesses the costs/benefits among alternative operational and maintenance concepts. Also, the activity timing (when the human factors activity is conducted) and type of data collected may also differ between an NDI/COTS and a developmental program. For example, in the former, data may be collected during investment analysis (e.g., via market surveys) on the cost and effectiveness of training programs that implement vendor alternatives, whereas, in the latter, data may be collected during solution implementation (e.g., via task analyses) on the critical tasks to be trained.

    If it is not possible to collect definitive cost/benefit and human performance data, heuristics and rules of thumb may be employed to provide gross estimates. For example, the funding necessary to conduct a comprehensive human factors engineering program for a system has been estimated to be between 1% and 5% of system developmental costs (depending upon the sensitivity of the solution to human factors issues). The benefit from conducting a comprehensive human factors program has been estimated at between 20% to 30% of total acquisition costs. Such rules of thumb may be useful for gross approximations but are a weak substitute for more through analyses and data collection.