Evaluate—Qualitative Methods October 2, 2007 NEEDS DESIGN EVALUATE IMPLEMENT Evaluation     A little out of sequence due to scheduling Will get more implementation over next two weeks Imagine you’ve implemented your application These are.

Download Report

Transcript Evaluate—Qualitative Methods October 2, 2007 NEEDS DESIGN EVALUATE IMPLEMENT Evaluation     A little out of sequence due to scheduling Will get more implementation over next two weeks Imagine you’ve implemented your application These are.

Evaluate—Qualitative Methods

October 2, 2007

NEEDS DESIGN EVALUATE IMPLEMENT

1

Evaluation

2     A little out of sequence due to scheduling Will get more implementation over next two weeks Imagine you’ve implemented your application These are techniques you will need to design user study (end of project)

3

Methods for evaluating system

  Qualitative – Rich, subjective – Exploratoring concepts – More useful for earlier input Quantitative – Precise, objective, repeatable – Demonstrating claims – More useful at documenting improvement – Can be expensive

For your project

4   Will require aspects of both qualitative and quantitative methods – Qualitative  How users react to project, perceptions?

– Quantitative  How users perform on project?

What would you improve on next iteration?

– Perhaps users’ perceptions of performance more important than actual values – Elevator waiting story

5

Design

evaluation methods!

  Most important aspect of evaluation is upfront design!

– Expensive to line up users, collect data – Design to collect right information Pick appropriate method for what you want to learn

6

Applying an evaluation method

         Determine the activity to observe Develop the method Human subjects review approval Pilot test the method Recruit participants Collect the data Inspect & analyze the data Draw conclusions to resolve design problems, reflect on what you learned Redesign and implement the revised interface

Demographic information

7   Demographic data – Age, gender, culture – Task expertise, experience – Motivation – Frequency of use – Education, literacy, training No matter what method, collect demographic data

8

Environmental information

 Besides info on the user, may also need info on the operating environment – Windows, Mac, Linux?

– Firefox, Internet Explorer, Safari?

– Wired ethernet, wireless, modem – Morning, afternoon, night – Office, mobile, home

Qualitative methods

9       “Discount” usability methods – Hueristic Evaluation – Cognitive Walkthrough Questionnaire / Survey Think aloud protocol Co-discovery Semi-structured interview Deploy and observe in use

10

“Discount” usability methods

   Enable evaluation at early stage, before prototype implemented Conducted quickly, inexpensively Early evaluation investment saves downstream development costs – Hueristic evaluation – Cognitive walkthrough

11

Heuristic Evaluation

   Fancy way to describe expert review – HCI expert – Domain expert Expert review identifies usability issues before implementation Our grades on your homework are form of heuristic evaluation

12

Evaluation hueristics

Visibility of system status Match between system and the real world User control and freedom Consistency and standards Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from errors Help and documentation

13

Heuristic evaluation method

    Multiple experts individually review (around 5 experts get 75% problems) Observer records issues, answers questions, gives hints Conduct using low fidelity prototype or task analysis with storyboards and scenarios Generate list of usability problems according to hueristic compromised

14

Hueristic Evaluation analysis

  After created list of problems – Rank severity – Estimate fixability – Suggest possible fixes Analysis may involve larger team

15

Hueristic Evaluation as rigorous design review

  You can make a living out of doing Hueristic Evaluation – Substantial consulting market for conducting Heuristic Evaluation You may pay a consultant to do a Heuristic Evaluation – Know what you’re paying for – Especially the Severity, Fixability, Potential Fix aspects

16

Learning more about Hueristic Evaluation

 You can learn to do a Hueristic Evaluation – http://www.useit.com/papers/heuristic/

Cognitive Walkthrough

17   Have user imagine walking through the process of using system Can use low-fidelity prototyping, partially implemented prototype  Can use target user rather than expert – Pluralistic walkthrough uses experts, users, developers  Like a code walkthrough C. Wharton et. al. "The cognitive walkthrough method: a practitioner's guide" in J. Nielsen & R. Mack "Usability Inspection Methods" pp. 105-140.

Walkthrough procedure

18   Give user representation of interface and task – Can they discover how to accomplish goal with description of interface?

– Can ask “From here, how would you like to accomplish…?” Step through interface – User takes action, system provides response – Describe actions not depicted in interface representation – Somewhat like Wizard of Oz

19

Stepping through interface

    Will user try to achieve the right goal? – Conceptual model of goals and tasks Will user notice correct action is available? – Visibility – Understandability Will user associate correct action with the goal to be achieved? – Aligning goals with sequence of actions If correct action performed, will user see progress toward solution? – Feedback

20

Next assignment

 Testing storyboard with one user – Effectively, this is a cognitive walkthrough – Create storyboard – Define task – Step through with one user

Questionnaires & surveys

21    User responses to specific questions Preparation is expensive, administration relatively cheap Oral vs. written – Oral provides interaction, followup, but takes more time – Written more efficient, can provide quantitative data

22

Designing questions

 Design questions with analysis in mind – Closed format more precise, easier to analyze  Convert qualitative  quantitative measures  You give categories to users – Open-ended questions provide richer feedback, longer to analyze  Users give you categories

23

Designing survey questions

   Multiple choice – Collecting information Ordinal ranking – Expressing relative preferences Likert scales – Expressing personal reactions

24

Closed format styles

Multiple choice Which social networking systems do you use?

facebook MySpace LinkedIn Orkut Other_____________ Ordinal ranking Rank frequency of use from 5 – Most frequent 1- Least frequent 0 - Unused ___ facebook ___ MySpace ___ LinkedIn ___ Orkut ___ Other__________

25

Likert scales

    Ask users to rate on a numeric scale Odd number scale allows a neutral midpoint (5- or 7-point scale) Even number scale forces taking a position (4- or 6-point scale) “Anchors” give examples of points along the scale

Example question

 How important is the Berkeley Stanford Big Game?

Not Important Very Important Could not care less Maybe I’ll go if my friends go Most important event this Fall

26

27

Closed Format

 Advantages – Clarify among alternatives – Easily quantifiable – Eliminate useless answers – Relatively quick to administer  Disadvantages – Must cover whole range – All choices should be similarly likely – Don’t get interesting, “different” reactions

28

Questions people can answer about themselves

      What they do How they do it Opinions about current activities Complaints about current activites Comparing one thing with another How often they have done something in the recent past

29

Questions people cannot answer about themselves

    Predicting what they would do / like / want Imagining a hypothetical scenario Whether they would like a certain feature or product Estimating how often they do things

30

What’s most important?

31

Web-based survey tools

    Surveymonkey – http://www.surveymonkey.com/ Zoomerang – http://info.zoomerang.com/ Allows free basic analysis, more advanced features for fee Can extend reach to large number of respondents

Thinking aloud protocol

32     Have subject “think out loud” while performing task Psychology to elicit cognition Requires training task Facilitator actively prompts if subject falls silent for more then 10 secondss – “What are you thinking now?” – “So, you are trying to…?” – “And now you are…?”

33

Exercise: Volunteer

 Never used Photoshop before

34

Co-discovery

  Have two people work on a task together (even though the task is normally done by one person) Coordination with each other naturally elicits cognition

35

Exercise: Two volunteers

 Never used Photoshop before

36

Think aloud and co discovery

    Valuable to evaluate tasks that require cognition Time intensive Rich feedback Think aloud requires training

37

Semi-structured interviews

    Interactively asking questions (face-to face, telephone) Give users chance to explain “why” to complement “what” they did, subjective user’s viewpoint Can help with design questions – “What improvements would you suggest?” Can be done individually or in groups

38

Semi-structured interviews

   Begin with list of open-ended questions – Ask all users these questions – Let users elaborate – Flexibility to ask follow-up questions Must audio-record Interviewer should attend to user (not notepad or laptop), use audio record for data (note timestamps)

Questionnaire Issues

39    Language – Beware terminology, jargon Clarity – “How effective was the system?” (ambiguous) Avoid leading questions – Phrase neutrally rather than positive or negative  “How easy or hard was it to accomplish the task?”

Questionnaire Issues (2)

40     Prestige bias – People answer a certain way because they want you to think that way about them Embarrassing questions – “What did you have the most problems with?” Hypothetical questions “Halo effect” – When estimate of one feature affects estimate of another (e.g. intelligence/looks) – Aesthetics & usability, one example in HCI

41

Interviews

 Disadvantages – Subjective view – Interviewer(s) can bias the interview – Problem of inter-rater or inter experimenter reliability (agreement) – Time-consuming – Hard to quantify

42

Pilot test observation method

  Pilot test method with some target users – Debug the questions, methods – Also debug logistics – Don’t count pilot data in analysis Make changes now before collecting data (want method for collecting data to be consistent)

43

Methods used in combination

  Mix of closed format, open-ended questions Surveys, questionnaires often used with quantitative performance measures to assess how users feel about interactions

44

Mechanics of user testing

  Readings give more detailed nuts and bolts Common sense structuring of the experience to help it run smoothly

45

Analyzing qualitative data

   Rich, open-ended data Goal: Structure to characterize, describe, summarize data Sounds harder than it is

Analyzing qualitative data

46    Exercise to immerse in data – Develop categories to count  Range  Average – Identify common patterns Allows identifying the interesting, unusual, exceptions Also look for correlations

47

Exercise: Analyzing conceptual map of Berkeley

   Example of rich, qualitative data See if we can detect some patterns Characterize set of qualitative data

Berkeley map

48       Number of features?

Format of map Common features – Landmarks – Roadways Unusual features Assessments Correlations

49

Qualitative analysis

     Start with things you can count – Average, range, median Look for patterns that are in common Recognize features that are unusual, interesting Look for correlations Reflect on what the data is saying

50

Qualitative study of your project

 What do you want to learn – User reactions, perceptions – Conceptual model problems – Areas to improve design – Does the design work?

Next time

51    Quantitative methods Readings – "A face(book) in the crowd: social

Searching vs. social browsing"

– "iPod distraction: effects of portable

music-player use on driver performance"

Questions on Project Proposal assignment?