Design and Evaluation Methods

Human Factors PSYC 2200 Department of Cognitive Science

Michael J. Kalsher

Design and Evaluation: An Overview

• Designers tend to focus primarily on the product and its functions without fully considering the use of the product from a human point of view •

Cost-Benefit?

– HF can save companies time and money if applied early in the design process

What

’

s the problem?

Human Factors specialists must often settle for workable solutions that are not always the best solutions.

Limited Resources - Limited Time - Constraints on what can be done - HF Specialists are the last ones to be called in!

Potential Benefits

Increased

– Sales – User productivity – Quality of service

Decreased

– Costs of training – Customer support costs – Maintenance costs – Development costs – Training time – User turnover – User error

Health and Safety Benefits

Decreased

– sick leave/time off – number of accidents/injuries – medical/rehabilitation expenses – number of citations/fines – decreased numbers of lawsuits

Increased employee satisfaction

– Lower turnover rate

HF in the Product Design Life Cycle

User Centered Design

– Center design

process

around the user • Determine the needs of the user • Involve the user at all stages of the design process • Elicit their needs and preferences • Ask for the insight and design ideas

The Goal?

– Create a system that supports users ’ needs rather than making a system to which users must adapt

User Centered Design

A sub-field of

usability engineering

Four general approaches

– Early focus on user and tasks – Empirical measurement –

Iterative design

using prototypes – Participatory design (involving the user as part of the design team)

Sources for Design Work

Data Compendiums

– Condensed and categorized databases that contain tables and formulas concerning human capabilities and limitations

HF Design Standards

– Precise recommendations that relate to very specific areas or topics

Human Factors Principles and Guidelines

Front End Analysis

Considerations before design solutions are generated in the concept design stage: 1.Who are the product/system users?

2. What are the major functions to be performed by the system? What tasks must be performed?

3. What are the environmental conditions under which the product/system will be used? 4. What are the user ’ s preferences or requirements for the product?

5. Are there any existing constraints with respect to design of the system?

6. What are the human factors criteria for design solutions?

User Analysis

Identify potential users Create complete description of user population

’

s characteristics

• Age, Gender • Education Level • Reading Ability / Physical Abilities • Physical Size • Familiarity with type of product

Function and Task Analysis

Function Analysis

– List of basic functions performed by the system Human/Machine, Human/Software, Human/Environment, etc.

Task Analysis

– Specifies jobs, duties, and tasks that the person will be doing – More complex systems require more detailed analysis

Preliminary Task Analysis

Extensive interactions with multiple users

– Users can be asked individually to describe general activities they perform with respect to the system – Alternatively, effective

focus groups

can be more cost • Pros and Cons of focus groups?

Preliminary Task Analysis:

Surveys, Direct Observation, Questionnaires, Environment Analysis

• Watch users perform activities with existing versions of product/system (direct observation) • Evaluate how users go about activities and also their preferences using surveys and questionnaires • Analyze where users fail to achieve goals, make errors, show lack of understanding, or seem frustrated/uncomfortable • Evaluate characteristics of the environment that may constrain user ’ s activities

Identify User Preferences & Requirements

• Determine key needs and preferences that correspond to each of the major activities users will perform (closely related to a market analysis) • Conduct brainstorming sessions with focus groups to: 1. Consider sources of resistance and ways to overcome resistance 2. Evaluate existing products 3. Develop ideas for new or different products

Providing Input for System Specifications

System specifications include:

1. Overall objectives for system 2. System performance requirements 3. Design constraints Objectives at this point should be written in global/abstract terms to avoid premature design decisions

Establish Performance Requirements

• Operationally define the requirements by which the product will accomplish its functions • State what system will be able to do and under what conditions • List design constraints – For example: weight, speed, cost, etc.

Conceptual Design Activities

•

System design approach

– Analyze entire human machine system to determine the best configuration of characteristics •

Function allocation

– Determine whether each function is to be performed by system (automatic), by person (manual), or some combination

Function Allocation

•

Traditional View

: Allocate function to more capable system • Ultimate Function Allocation using “ Capability Criterion ” • Feasibility Studies • Human preferences

Iterative Design and Testing

Purpose of Task Analysis is to identify: – Major user goals and associated activities – Tasks/Subtasks required to achieve goals – Conditions in which tasks are performed – Results of performing tasks – Information needed to perform tasks – Communications with others for performing tasks – Equipment needed to perform tasks – Secondary factors associated with each task, including: • Frequency, importance, difficulty, minimum expectations for task performance, severity of consequences should task be performed incorrectly, whether tasks will be performed concurrently

Cognitive Task Analysis

Focus on analyzing the mental processes, skills, strategies and types of information required for task performance

When is task analysis important?

– In jobs/situations requiring complex decision making, problem solving, diagnosis, or reasoning from

incomplete

data – When large amounts of conceptual knowledge must be used to perform tasks – In the context of large and complex rule structures that are highly dependent on situational characteristics

Techniques of Task Analysis

• Direct Observation • Structured/Unstructured Interviews • Think Aloud Protocols • Task Performance with Questioning ----------------------------------------------- • Documentation

Structured vs. Unstructured Interviews

Structured Interviews

– Use of specific types of questions and/or methods • How do you perform task x?

• What do you do before you perform task x?

• What happens after you perform task x?

– Prepared ahead of time – Answers are sometimes video- or audio-taped

Unstructured Interviews

– Asks user to describe tasks and activities, but does not have a method for structuring the conversation – Not particularly effective for task analysis

Think Aloud Verbal Protocol

• Users think out loud as they perform various tasks • Yields insight to goals, strategies and decisions • Three Types of –

Concurrent

Obtained during performing the task –

Retrospective

Obtained after task was performed via memory or videotape –

Prospective

User given hypothetical situation and then asked to think aloud as they imagine performing the task

Task Performance with Questioning

Users perform the tasks while answering question “ probes ” – Advantage • may cue users to verbalize underlying goals/strategies more frequently – Disadvantage • can be disruptive

Documentation

Common ways to organize task analysis data: 1. Lists, outlines, matrices 2. Hierarchies or networks 3. Flow charts

Lists, Outlines, Matrices

Part of Task Analysis for Using a Lawnmower shown in outline form Step 1: Examine Lawn

a. Make sure grass is dry b. Look for any object laying in the grass

Step 2: Inspect lawnmower

a. Check components for tightness 1. Check to see that grass bag handle is securely fastened to the support 2. Make sure grass bag connector is securely fastened to bag adaptor 3. Make sure that deck cover is in place 4. Check for any loose parts (such as oil cap) 5. Check to make sure blade is attached securely b. Check engine oil level 1. Remove oil fill cap and dipstick 2. Wipe dipstick 3. Replace dipstick completely in lawnmower 4. Remove dipstick 5. Check to make sure oil is past mark on dipstick

Major Disadvantage:

tasks tend to have a hierarchical organization that is hard to show in an outline

Hierarchies and Networks

Graphical Notation Method

: organizes tasks as sets of actions needed to accomplish higher level of goals

GOMS

; goals, operations, methods & selection rules

Flow Charts

• Capture chronological sequences of sub-tasks as they are normally performed • Depicts the decision points for alternative pathways

Operational Sequence Diagram

—

shows sequence of activity and categorizes operations into various behavior elements – Disadvantage: cognitive groupings or branches are not evident – Evidence suggests that people mentally represent goals and tasks in clusters and hierarchies

HF General Principles

From: Donald Norman,

The Psychology of Everyday Things

Provide a good

mental model Make things visible

“ Hidden ” controls tend to be difficult to use. How to use the conference call feature of a telephone.

HF General Principles

From: Donald Norman,

The Psychology of Everyday Things

Use natural mapping

• Mapping refers to relationship between input to or output from a system and the associated system state or event • Example: Stove on left makes use of natural mapping from controls to burners on a stovetop. The one on the right does not.

HF General Principles

From: Donald Norman,

The Psychology of Everyday Things

Provide feedback

Products should be designed so that users know that an action has actually been done and what the results were within the system Example: hour glass on the computer while task is being performed

HF General Principles

From: Donald Norman,

The Psychology of Everyday Things

• Simplify task structure • Make it easy to determine what actions are possible at any moment • Make it easy to evaluate current state of system • Exploit the power of constraints (can only be done one way) • Design to allow for easy recovery • Standardize … if need be

Use of Prototypes

Advantages

– Support design team in making ideas concrete – Provides communication medium for design team – Support for heuristic evaluation – Gives users something to react to and use • Computer interface design allows for

rapid prototyping

– Making extremely quick changes in the interface so that many design iterations can be performed in a short period of time

Heuristic Evaluation

• In usability engineering, heuristic evaluation means making sure that it meets usability standards • Evaluating characteristics of a product or system design to determine whether they meet human factors criteria • Ideally should be performed by multiple independent evaluators

Additional Evaluative Studies & Analyses

• Comparison of different design features and their implications • Most common method is decision matrix – Design alternative on left side – Each feature is give weight – Each design alternative is assigned a # representing where it stands with respect to the feature – Each design alternative is given a total score by multiplying the individual score by the feature weights and adding the scores together

Trade Off Analysis

• Small scale study conducted to determine which design alternative results in the best performance • When multiple factors are considered, design tradeoffs might revolve around design with greatest number of advantages and smallest number of disadvantages

Work Load Analysis

• Evaluate if it is going to place excessive mental work load on the user • TAWL—one tool for estimating mental work load. Evaluates total attention demand placed on users performing task

Simulations or Models

• Used to test HF characteristics of design configurations in the abstract before undergoing manufacturing costs • Simulation tools can help predict human errors by estimating the mismatch between cognitive resources and demands of particular problem solving tasks

Safety Analysis

• Analysis should be conducted to identify potential hazards or likelihood of human errors • Much more on this later!

Usability Testing

• •

Usability

— degree to which the system is easy to use or “ user friendly ”

Usability testing

system to identify human factors design flaws overlooked by designers — process of having users interact with •

Usability Considerations

– Learnability—system should be easy to learn – Efficiency—high level of productivity is possible – Memorability—easy to remember so that user is able to return to system after period of inactivity without having to relearn everything – Errors—should have low error rate and errors should be easy to recover from – Satisfaction—should be pleasant to use

Support Materials

• Manuals, assembly instructions, owner ’ s manuals, training programs, etc.

• HF specialists make sure that materials are compatible with characteristics and limitations of the human user – maximizes likelihood that user will read, understand and comply with instructions

Final Test & Evaluation

Evaluation Design

– Comparison between new product & some other condition (old product) – 2 most common designs: • Between-Subjects Design • Within-Subjects Design

Test Participants

– Representative of final user population – Critical characteristics include: age, physical characteristics, education level, knowledge, skills, job-related ability, etc.

Final Test & Evaluation Measures

Proximal Measures:

Ones directly associated with person ’ s preference or thought – User satisfaction – Usability – Task performance levels – Number of performance errors related to safety issues

Distal Measures:

Capture impact of product on more global factors pertaining to company or organization as a whole – Manufacturing costs, efficiency, waste, etc.

– Personal costs – Number of accidents & injuries – Number of Disability Claims – Sick Leave and other health indices

Programs of Change

• HF may give recommendations to manufacturing plants on the whole; re-engineering a whole organization • Examples: – Safety procedures & policies – Efficiency of plant layout, jobs & tasks – Adequacy of employee training – Reward/Incentive Programs – Information exchange and communication •

Macroergonomics

—global approach to system redesign