Transcript How to Validate a Taxonomy: Workshop

How to Validate a Taxonomy: Workshop
Joseph A. Busch, Founder & Principal, Taxonomy Strategies LLC
Dave Cooksey, Founder & Principal, saturdave UX Consulting
Today’s agenda
>
1:00-1:30
30 min Introduction
1:30-2:00
30 min Taxonomy & validation overview
2:00-2:15
15 min Closed card sorting
2:15-2:30
15 min Coffee Break
2:30-2:45
15 min Delphi card sorting
2:45-3:00
15 Min Task-based validation methods
3:00-3:15
15 min Search analysis
3:15-3:45
30 min Hands-on exercise
3:45-4:00
15 min Q&A, Closing
Joseph Busch
• Over 25 years in the business of organized information.
–
–
–
–
–
Founder, Taxonomy Strategies LLC
Director, Solutions Architecture, Interwoven
VP, Infoware, Metacode Technologies
Program Manager, Getty Foundation
Manager, Pricewaterhouse
• Metadata and taxonomies community leadership.
– President, American Society for Information Science & Technology
– Director, Dublin Core Metadata Initiative
– Adviser, National Research Council Computer Science and Telecommunications
Board
– Reviewer, National Science Foundation Division of Information and Intelligent
Systems
– Founder, Networked Knowledge Organization Systems/Services
Dave Cooksey
• 10 years in the business of organized information.
–
–
–
–
Founder & Principal, saturdave UX Consulting
User Experience Lead, GSI Commerce
Information Architect, GSI Commerce
Web Application Developer, LandAmerica Financial Group
• User experience community leadership.
– Chair, PhillyCHI – Philadelphia region’s chapter of ACM SIGCHI, 2007-2009
– Information Architecture Institute
– American Society for Information Science & Technology
What we do
What we do
Organize Stuff
Who are you? What sectors do you work in?
Your Role
–
–
–
–
–
–
–
–
–
–
–
Content Manager
Editor
Information Architect
Usability Expert
Librarian
Records Manager
Knowledge Engineer
Ontologist
Chief Information Officer
Communications
Administration
Industrial Sector
–
–
–
–
Consumer Products, etc.
Education
Financial Services (Banking & Insurance)
High Tech (Computers, Software &
Telecommunications)
– Heavy Manufacturing (Steel,
Automobiles, Aircraft, etc.)
– Government (Federal, State or local)
– Medical & Health Care
– Mining & Refining (Petrochemicals, Oil
& Gas)
– Pharmaceuticals (Drugs, Biotech)
Today’s agenda
>
1:00-1:30
30 min Introduction
1:30-2:00
30 min Taxonomy & validation overview
2:00-2:15
15 min Closed card sorting
2:15-2:30
15 min Coffee Break
2:30-2:45
15 min Delphi card sorting
2:45-3:00
15 Min Task-based validation methods
3:00-3:15
15 min Search analysis
3:15-3:45
30 min Hands-on exercise
3:45-4:00
15 min Q&A, Closing
What is a Taxonomy?
• A categorization framework agreed upon by business
and content owners (with the help of subject matter
experts) that will be used to tag content.
–
–
–
–
–
6 broad, discrete divisions (called facets)
2-3 levels deep.
Up to 15 terms at each level.
1200 terms total.
With some logic—hierarchical, equivalent and associative
relationships between terms.
Why taxonomy is important
• Easier information management
• Flexibility to respond to changing needs
• Foundation for findability and usability
Taxonomy
• Is an expert activity.
• Comes from a privileged view.
• Is more closely aligned to business & technology goals
than user goals.
Taxonomies & Metadata
• By their very nature, hold implicit meanings:
– Hierarchy & relationships.
– Labeling & naming conventions.
• Are interpreted from the user’s experience, context, &
goals.
The experience, context, & goals of users
are rarely our own!
Taxonomy Validation
• Ensures information organization
is user-centric
• Vets untried ideas with real users
But it also...
• Justifies the cost of taxonomy
research & analysis
• Reassures the business that
project focus is on success
• Helps alleviate organizational
conflict
You’ve been warned: Princeton Record Exchange.
Types of taxonomy testing
• Formative Testing Techniques
– Explore the concepts that users employ to describe the real world.
– Focus on the way folks think about something (mental models)
– Encourage open discussion & brainstorming.
– Inform your design.
• Summative Testing Techniques
– Examine the way people use an existing design.
– Focus on the way folks do something (work flow)
– Encourage feedback & critique.
– Inform design strategy.
Testing Requires Neutrality
• When testing, you’re not a taxonomist anymore - you’re a fly on the
wall.
• Good test moderation is hard and takes practice.
How to Moderate Effectively ...
• Reassure the participant that you will not be upset by
comments or critique: “I’m not the designer so I won’t be
offended by what you say.”
• Avoid leading questions – try to say as little as possible
• Don’t ask a participant if they would like something, a feature,
category, label – she will almost always say, “yes.”
• When asked specific questions of how things work, avoid
explaining and simply say something like, “I don’t know. What
do you think?”
Consider Test Structure & Effects
• Environmental effects: in-lab, in the mall, moderated
session online.
• Precedent effects: order of activities, transparency of
previous results.
• Compensation effects: stipend up front, afterwards, or
none at all.
• Moderation effects: interview during, after, or none at all.
• Methodological effects: think aloud, observation, artifact
creation, etc.
• Medium effects: in-person, on the phone, online.
How you structure the test will affect the
results so pay attention to details.
Taxonomy Validation Techniques
Taxonomy design validation methods
Method
Process
Who
Requires
Validation
Web tool
Open Card
sort
 Representative  Rough taxonomy  Card sort analysis (emergent
users
patterns)
Cards &
markup
Delphi card
sort
 Representative  Rough taxonomy  End result (consensus)
users
Walk-thru
Show &
explain
 Stakeholders
Check
conformance
to editorial
rules
 Taxonomist
Matrix or
cards
Closed card
sort
 End users (or
surrogates)
 Rough taxonomy  Consistent results
User
satisfaction
Survey
 End users (or
surrogates)
 Draft taxonomy
 Reaction to taxonomy
Tagging
samples
Tag sample
content with
taxonomy
 Taxonomist
 Sample content
 Content “fit”
 Indexers
 Draft taxonomy
(or better)
 Fills out content inventory
Walk-thru
 Rough taxonomy  Approach
 Appropriateness to task
 Draft taxonomy
 Consistent look & feel
 Editorial rules
 Top queries, etc.
 Training materials for people &
algorithms
 Basis for quantitative methods
Usability validation methods
Method
Usability
Testing
Process
Requires
Validation
Task-based
 End users (or
scenario
surrogates)
testing (Find
it, Tag it, etc.)
 Draft taxonomy
 Tasks are completed successfully
 Tasks & answers
 Time to complete task is reduced
 End users (or
surrogates)
 Draft taxonomy
 Reaction to taxonomy
 UI mockup,
Search
prototype
 Reaction to new interface,
Reaction to search results
User
Survey
Satisfaction
Who
Collection analysis validation methods
Method
Distribution
Process
Statistical
analysis
Who
 Taxonomist
 Analyst
Requires
 Tagged
collection
Validation
 Do categories follow Zipf
distribution?
 Do % of categories equal % of
documents?
 Identify candidates for merging &
splitting.
Query log &
click trail
examination
Clustering &
statistical
analysis
 Taxonomist
 Search logs
 Analyst
 Web analytics
 Do (clustered) queries follow Zipf
distribution?
 Are top (clustered) queries in
taxonomy?
Today’s agenda
>
1:00-1:30
30 min Introduction
1:30-2:00
30 min Taxonomy & validation overview
2:00-2:15
15 min Closed card sorting
2:15-2:30
15 min Coffee Break
2:30-2:45
15 min Delphi card sorting
2:45-3:00
15 Min Task-based validation methods
3:00-3:15
15 min Search analysis
3:15-3:45
30 min Hands-on exercise
3:45-4:00
15 min Q&A, Closing
Closed card sorting overview
• Key characteristics: Qualitative, small sample.
• Based on usability method for testing top-level
website navigation.
• Used to confirm high-level taxonomy facet design.
• May be conducted using 3x5 cards, matrix, or webbased tool such as websort.net.
• Run as warm-up for other validation tests.
• Conduct 15-20 sessions.
Step-by-step closed card sorting
1. Normalize/group terms from the top words and phrases
in search logs.
2. Select 25 terms, place them in a matrix and agree on
the “correct” answer key.
3. Interview participant
a. Walk-through high-level taxonomy with subjects and explain
exercise.
b. Participant organizes and comments.
c. Watch and interview.
d. Survey subjects on task difficulty using a 3 or 5 point Likert
scale.
4. Analyze data.
Performing closed card sorting
• Subject chooses the one best place to sort a term.
• Answer the question: “In which facet would I look to
find content about … ?”
• Set a time limit of 20 minutes to sort 25 terms.
• Don’t forget to survey the subject at the end of the
sorting exercise.
Closed card sorting case study
• Project: Develop Substance Abuse and Mental
Health Services Administration (SAMHSA) taxonomy
to be the backbone of a unified agency information
service that collects, integrates, and reports on
customer interaction data in ways that help the
agency plan health communications products,
messages and outreach.
• Task: Sort popular queries (words and phrases) from
search logs into the most likely Taxonomy facet.
Term sorting data collection form
Term
Alcohol
Anger
Anxiety
Autism
Binge Eating
Bipolar Disorder
Change Management
Child
Co-occurring Disorders
Counseling
Depression
Fulton County
Health Literacy
Immigrant Adolescents
Inhalants
Low Income
Methadone
Native American Culture
Ohio
Opium
Posters
PTSD
Residential Treatment
Rohypnol
Smoking
Stress
Substance Abuse
Suicide
University Students
Vicodin
Webcast
Content
Type
Audience
Population
Groups
Substances
Conditions &
Disorders
Intervention &
Treatment
Topics
Professional &
Research Topics
Geographic
Locations
Summary of term sorting results
Correct category
Term
Alcohol
Anger
Anxiety
Autism
Binge Eating
Bipolar Disorder
Change Management
Child
Co-occurring Disorders
Counseling
Depression
Fulton County
Health Literacy
Immigrant Adolescents
Inhalants
Low Income
Methadone
Native American Culture
Ohio
Opium
Posters
PTSD
Residential Treatment
Rohypnol
Smoking
Stress
Substance Abuse
Suicide
University Students
Vicodin
Webcast
Frequently chosen related category
Content
Type
Audience
1
1
9
1
2
1
9
1
Frequently chosen incorrect category
Population
Conditions & Intervention &
Professional & Geographic
Groups
Substances Disorders Treatment Topics Research Topics Locations None
1
10
1
9
1
2
12
12
10
1
1
12
2
9
1
12
11
1
9
3
12
12
12
12
10
1
1
9
2
10
2
7
4
12
12
1
1
12
1
11
10
1
1
5
3
1
2
10
1
1
1
1
4
1
3
7
1
3
12
11
1
1
2
Percentage of popular search terms
sorted correctly
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Blind sorting of popular search terms
Results: Excellent
(n=12)
50-60%
(7%)
25-50%
(6%)
< 25%
(3%)
84% of terms
were correctly
sorted 60-100%
of the time.
Difficulties
 For Methadone, confusion when, in this case, a substance is a treatment.
 For general terms such as Smoking, Substance Abuse and Suicide, confusion
about whether these are Conditions or Research topics.
Search terms sorting task user rating
(n=12)
Medium
33%
Easy/Medium
25%
No one rated the task difficult!
Easy
42%
Today’s agenda
>
1:00-1:30
30 min Introduction
1:30-2:00
30 min Taxonomy & validation overview
2:00-2:15
15 min Closed card sorting
2:15-2:30
15 min Coffee Break
2:30-2:45
15 min Delphi card sorting
2:45-3:00
15 Min Task-based validation methods
3:00-3:15
15 min Search analysis
3:15-3:45
30 min Hands-on exercise
3:45-4:00
15 min Q&A, Closing
Today’s agenda
>
1:00-1:30
30 min Introduction
1:30-2:00
30 min Taxonomy & validation overview
2:00-2:15
15 min Closed card sorting
2:15-2:30
15 min Coffee Break
2:30-2:45
15 min Delphi card sorting
2:45-3:00
15 Min Task-based validation methods
3:00-3:15
15 min Search analysis
3:15-3:45
30 min Hands-on exercise
3:45-4:00
15 min Q&A, Closing
Delphi card sorting overview
• Key characteristics: Qualitative; Small Sample.
• Based on the Delphi Method (RAND Corporation)
• Developed by Professor Kathryn Summers & Celeste Lyn Paul
at University of Baltimore.
• Hierarchy is laid out in cards – seeded or not.
• Test for both categories & individual items.
• Participants modify the structure one at a time
• Runs for 15-20 participants until hierarchy “stabilizes.”
• Watch and learn – Interview for detail.
• Can also present navigation schema, images for labeling, filters
or facets.
Step-by-step Delphi card sorting
1.
2.
3.
4.
5.
Decide to seed deck or let first user create seed.
Think of other pertinent test points: navigation, facets, images for labeling.
Put categories on cards (You or first participant)
Put representative items on cards | script (You)
Interview participant
a. Explain exercise.
b. Participant organizes and comments.
c. Watch and interview.
6. Repeat with participants until you are satisfied.
7. Analyze data.
Performing Delphi card sorting
1. User starts with deck from previous participant.
2. Participant modifies card deck
a. Removing category - turning the card over.
b. Changing labels - creating new card & place over old one.
c. Moving category - turn card over & create new card in new place.
3. Participant examines changes made to card deck
a. Looks at existing cards & placement.
b. Turns cards over to see previous changes to labels & placement.
4. Moderator interviews informally during session
Differences from traditional card sorting
•
•
•
•
•
•
Targets group understanding, not individual understandings.
Participants pick up where previous participant left off.
Assumes some level of “expertise.”
The end result is the “final” hierarchy.
“Round Robin” approach reduces analysis time.
Informs the “black box” (Thanks, Michael)
Delphi sorting method summary
• Delphi Method designed for use with groups of experts.
• Each expert contributes her knowledge to building a consensus view.
• Outliers are removed through iteration.
Consider...
•
•
•
•
What kind of users are you testing?
Do they have domain expertise?
Do you have multiple user types? Then multiple studies perhaps?
Do you have a general population (e.g. public website)?
Tips for Delphi Card Sorting
•
•
•
•
•
•
•
•
Plan for lots of table space for cards.
Allow for plenty of time between participants.
Create a “starter” sheet to capture notes.
Take pictures of table after each participant.
Detail is great, but keep your eye on patterns.
Video recording is essential.
Use questions to explore what the user is doing.
Employ multiple methods to solicit feedback.
Today’s agenda
>
1:00-1:30
30 min Introduction
1:30-2:00
30 min Taxonomy & validation overview
2:00-2:15
15 min Closed card sorting
2:15-2:30
15 min Coffee Break
2:30-2:45
15 min Delphi card sorting
2:45-3:00
15 Min Task-based validation methods
3:00-3:15
15 min Search analysis
3:15-3:45
30 min Hands-on exercise
3:45-4:00
15 min Q&A, Closing
Task-based testing overview
• Key characteristics: Qualitative, small sample.
• Based on Dona Maura’s task-based usability method.
• Tests how easy it is to work with a taxonomy to
complete tasks such as finding, or tagging content.
• May be conducted using paper or low-fidelity
prototypes.
• Run with other validation tests.
• Conduct 15-20 sessions.
• Test with the same items in each session.
Step-by-step task-based testing
1. Work with client to identify a set of most popular, most
important or problematic content items.
2. Agree on the “correct” outcome for each item in the
test set; then select 5 items to use in the test.
3. Interview participant
a.
b.
c.
d.
Walk-through taxonomy with subjects and explain exercise.
Participant navigates (or tags) and comments.
Watch and interview.
Survey subjects on task difficulty using a 3 or 5 point Likert
scale.
4. Analyze data.
Performing task-based testing
• Make the subject handle the taxonomy pages (you
may have to handle it for them at first).
• Answer the question: “In which facet, and in which
category would I look for this content item?”
• Set a time limit of 20 minutes to find (or tag) 5 items.
• Don’t forget to survey the subject at the end of the
exercise.
Task-based testing case study
• Project: Develop American Society of Civil Engineers
(ASCE) taxonomy to draw from existing databases
and content sources, and provision the next
generation websites being built to support this nonprofit, membership association’s activities.
• Task: Find specific content (web pages)
Find web pages
ASCE Continuing Education
http://www.asce.org/conted/
T Topics
A Audiences
C Content Types
E
Event Types
L
Locations
T.1
T.2
T.3
T.4
T.5
T.6
O Organizations
T
Topics
T.7
T.8
T.9
T.10
T.11
T.12
T.13
T.14
T.15
T.16
T.17
Architectural Engineering
Coasts & waterways
Construction
Cross-Cutting Topics
Disaster & Hazard
Management
Education & Career
Development
Engineering Mechanics
Energy
Environment
Geotechnical Engineering
People, Projects & Heritage
Planning & Development
Professional Issues
Project Management
Structural Engineering
Transportation
Water & Wastewater
T.6 Education & Career
Development
T.6.1 Continuing Education
T.6.2 Engineering Education
T.6.3 Management &
Professional Development
T.6.4 Scholarships, Internships
& Competitions
Summary of navigation results trial
Correct category
Trial
Frequently chosen related category
Frequently chosen incorrect category
JF
KSS
MJ
KS
IM
VM
KSM
GSB
KCH
1
2
3
4
5
6
7
8
9
C>A>T
C>E
Key
Navigation
ASCE Continuing Education
A>C>T
Audience
A>O>C
A.27
Content Type
C.12
Event Type
E.4
E.4
E.4
E.4
Location
Organization
Topic
Difficulty
T.6.1
Difficult
T.6.1
Easy
Difficult
Easy
Gave up
Easy
Difficult Medium
T.6.1
Easy
Medium Difficulty
Overall navigation task performance
(n=54)
Incorrect
9%
Gave Up
4%
Correct
41%
Alternative
46%
 87% navigated as predicted or used a reasonable alternative.
 In only 4% of the trials, did the subject give up.
Overall user rating of navigation task
(n=9)
Easy
33%
Medium
67%
No one rated the overall task Difficult!
Today’s agenda
>
1:00-1:30
30 min Introduction
1:30-2:00
30 min Taxonomy & validation overview
2:00-2:15
15 min Closed card sorting
2:15-2:30
15 min Coffee Break
2:30-2:45
15 min Delphi card sorting
2:45-3:00
15 Min Task-based validation methods
3:00-3:15
15 min Search analysis
3:15-3:45
30 min Hands-on exercise
3:45-4:00
15 min Q&A, Closing
Search analysis overview
• Key characteristics: Quantitative, Interpretive, & Focused on Patterns
• Uses existing reporting.
• In your hands becomes a barometer of user satisfaction, requests, &
complaints.
• Easily shows you:
– What folks want from the system (search results)
– What folks need and are not getting (no search results)
– The vocabulary folks use to describe things.
• Great for pointing out opportunities for business & technology.
• Easy but time consuming.
Step-by-step search analysis
1. Find the owner of analytics and ask for advice.
2. Determine what’s available to you: search results, no search results, etc.
3. Use spreadsheet to filter and order results in order to highlight patterns.
4. Identify: like terms & misspellings, adjectives, frequent terms both
returned & not.
5. Create summary.
6. Discuss summary with business to get contextual detail.
7. Create recommendations brief: changes to data, process, & functionality.
Tips for search analysis
•
•
•
•
•
•
Know the data: what do the numbers really mean?
Know the system.
Keep your interpretations simple.
Plan extra time for analysis – it’s time consuming.
When delivering results, create 2 views: overview & detail.
Pass results to teams that can benefit: business, search, technology,
marketing.
• Works best when used with other methods / techniques.
Using multiple methods will increase the
impact of validation testing.
Today’s agenda
>
1:00-1:30
30 min Introduction
1:30-2:00
30 min Taxonomy & validation overview
2:00-2:15
15 min Closed card sorting
2:15-2:30
15 min Coffee Break
2:30-2:45
15 min Delphi card sorting
2:45-3:00
15 Min Task-based validation methods
3:00-3:15
15 min Search analysis
3:15-3:45
30 min Hands-on exercise
3:45-4:00
15 min Q&A, Closing
Today’s agenda
>
1:00-1:30
30 min Introduction
1:30-2:00
30 min Taxonomy & validation overview
2:00-2:15
15 min Closed card sorting
2:15-2:30
15 min Coffee Break
2:30-2:45
15 min Delphi card sorting
2:45-3:00
15 Min Task-based validation methods
3:00-3:15
15 min Search analysis
3:15-3:45
30 min Hands-on exercise
3:45-4:00
15 min Q&A, Closing
Joseph Busch
Taxonomy Strategies
42 Bonview St
San Francisco, CA 94110
☎ +1.415.377.7912
Twitter: @joebusch
[email protected]
That’s All, Folks!
Dave Cooksey
saturdave UX Consulting
713 Pine Street 1R
Philadelphia, PA 19106
☎ +1.215.219.8960
Twitter: @saturdave
[email protected]
© 2009 saturdave and Taxonomy Strategies LLC