Transcript Spatial Hypertext in the Digital Library

Understanding Users: Building
Usable Digital Library Systems
George Buchanan
Future Interaction Technology Lab,
Swansea University
Not in New Zealand
• First “Area of Outstanding Natural
Beauty” in the U.K.
• Still wet…
Libraries and Readers
• Libraries have users (readers)
• Digital libraries are no exception
• There has long been study of how to
help library users find the
information they need
• This tutorial addresses this for digital
libraries
Human Computer
Interaction
• HCI (CHI in USA) is the study of how
to design and create effective, usable
computer systems
• Informed by psychology’s methods
and understandings of human
thought
• Long established field
HCI and Libraries
• Today’s tutorial is really about the
HCI of Digital Libraries
• Study the highlights of the area
• Point you at the key literature
• Introduce you to main ideas
Tutorial
• It will not tell you what to do just to
make your users happy
• There are no universal answers
• But there is always a route to
improving your system
– Or understanding how to design better
ones!
Agenda
•
•
•
•
•
•
Background: three disciplines
State-of-the-art review
Case Study 1: Social DLs
Case Study 2: User Search
Case Study 3: Reading
Conclusions
Background
• Three disciplines
– Library and Information Science
– Computer Science
– Psychology and Interaction Design
• Often little or no overlap
• Biggest joint area is HCI (Human
Computer Interaction)
– Computer Science and Psychology
Studying Libraries
• Librarians: (conceptual) storage,
retrieval and organisation
• Computer scientists: technology,
performance
• Interaction design: people, cognition
Studying Libraries
• Everyone both possesses and lacks
necessary skills
• Many people mistakenly think other
skills are trivial
• Need to develop individuals who
excel at all three
Studying Libraries
• This tutorial will
– Help you start to understand other areas
– Give you examples of good practice
• This tutorial will not:
– Make you an overnight expert
– Tell you how to make a ‘perfect’ library
State-of-the-Art
Part 1: Introduction and
Social Studies
State-of-the-Art
• Earliest studies of DL usability
appear mid-90’s
• Increasing activity 1998 onwards
• Main areas:
– Social factors in DL use
– Usability of DL software and systems
– Use of DLs in a wider information context
Social Studies of DLs
• Ann Peterson Bishop
– Mostly mid-to-late 90’s, DL use from a
sociological perspective
• Anne Adams
– Early to mid 2000’s, DLs in medical
institutions
• Ann Blandford
– DLs in legal and clinical domains
Social Studies of DLs
• Qualitatively driven research
– Finding out why things happen
• Data capture methods:
– Ethnographic, Anthropological,
Sociological
– Interviews, Action research
• Evaluation methods:
– Grounded theory
Social Studies of DLs
• Ann Peterson Bishop
– ACM DL 2000
• Studied both academic and lowincome community use of DLs
• Found that understanding
community practices were key
factors impacting DL design
Ann Peterson Bishop
• Libraries need to consider
– Uses perceived by their community
– Norms and knowledge assumed by their
community members
– Practices followed by their users in
handling and using information
Anne Adams
• DLs often assume “open access” to
a community
• In medical domains, controlling and
restricting information is sensitive
• Doctors limit information given to
nurses, radiographers etc.
• Senior doctors limit information for
their juniors, etc.
Anne Adams
• Open-access terminals placed in
hospital wards for all staff to use
• Senior staff had these removed to
inhibit information access
• Senior staff threatened by juniors’
apparently better electronic search
skills and awareness
Ann Blandford
• Examines the role the wider
information environment
• Fit of DL use with the user’s task
• Example domains:
– Clinical (with Anne Adams)
– Humanities (with Buchanan and Warwick)
– Law (with Makri and Attfield)
Ann Blandford
• Access and availability vary with
work location (e.g. on/off campus)
• Time is precious
• Users adopt conservative search
strategies to “save time”
– Impedes development of skills across long
term
Social DL Use
• The wider community impacts on
user behaviour, needs and tasks
• “Free availability” of information is
not always accepted
• Learning new search skills a low
priority
Case Study 1: Social DL
• Study of humanities researchers
• Conducted 2004-2005
• Interviewed 19 academics and PhD
students at Waikato University
• Objective: to understand their
information seeking behaviour and
needs
Case Study 1: Social DL
• Participants demonstrated a search
that they had attempted recently
• Were interviewed on their search
strategies and tactics
– How they used communication with
colleagues
– How they learnt these methods
Case Study 1: Social DL
• Humanities academics use simple
search terms and syntax
• Chose where to search: often
smaller, specialist collections
• Extensively use colleagues as a
source of information
– Particularly more senior researchers
Case Study 1: Social DL
• Often seek a wide range of sources
when engaged in research
• Do not mind long lists of semirelevant material
– They want to decide for themselves!
– Serendipity often had significant value
State-of-the-Art
Part 2: Evaluating DL Systems
Evaluating DL Systems
• Usually quantitative emphasis
• Data gathering methods
– Questionnaires, surveys
– Laboratory studies
• Evaluation methods
– Primarily statistical analysis
– Some supporting qualitative work
Key Work
• Ann Blandford
– Comparative evaluation
• Dave Nichols
– Task-based laboratory- and web-studies
• George Buchanan, Steve Jones
– Evaluating novel interfaces
Ann Blandford
• Main study periods 1999-2003
• Identified many basic problems in
user interface designs for DLs
– Users do many things at once
– Satisfice rather than optimise
– Many basic features suffer low usability
Ann Blandford
• Comparative studies revealed that
– Specific features of a DL often overlooked
– Difficulties due to different syntax &
semantics
– e.g. query syntax, ranking of results
– Differences may encourage simple search
– Different access requirements also confuse
• On-campus versus off-campus
• Different logon procedures
David (Dave) Nichols
• Studies of Greenstone use
• Also of Ajax and web interfaces
• A lot of related work on CSCW,
collaborative work, etc.
Dave Nichols
• Study of Greenstone installation
• Developers thought installation was
simple/straightforward
• In-use study revealed many small
difficulties due to:
– Ambiguous messages
– Limited knowledge of OS platform
– Varying user privileges
Dave Nichols
• Study of Greenstone Librarian
Interface (GLI) demonstrated:
– Confusion of terminology and concepts
• Collection versus server or classification
• File versus document
– Limitations from Java environment (GLI
implemented in Java/Swing)
– Underlying OS/setup problems
Dave Nichols
• …and Greenstone is very robust and
developed compared to other DL
systems
– Bigger development team than DSpace,
Fedora
– Longer lifespan than either
– Already evaluated in many user tests
• Commercial and bespoke systems
are often much worse!
George Buchanan
• Mostly evaluation of new DL
interfaces and technologies
• Study both librarian/cataloguer and
user/reader interfaces
• Often based on standards:
– FRBR, Aggregates, METS, etc.
• Also interface technologies
– mobile, hypertext
George Buchanan
• Studying aggregates:
• Aggregates = journals, series, etc.
– Multi-part works with a connection
• On paper:
– users manage aggregates easily,
– but organisation is hard
• Electronically:
– Vice versa!
George Buchanan
• Studying mobiles:
• Greenstone implemented on simple
mobile interface
• Users able to read
• Navigation really hard
• South African users (and other
cultures) do not understand
hierarchies!
George Buchanan
• Beware of unexpected problems!
– Culture and community can catch you out!
• Paper and electronic worlds are not
the same
– Problems can be quite different
– Ditto the solutions!
– Research on physical behaviours not
necessariy a good basis for designing
digital interfaces
Summary
• You cannot be too simple
• Assume as little as possible
• Test, test, and test again
– And even then problems may remain!
• If developing a working DL, interface
design cannot be tested too early!
Drawing Lessons
“Google-isation”
• Phenomenon of users using Google
tactics for all searchers
– Not using fielded search
– Expecting rank ordering only
• Has lead to simple “Google-like”
interfaces for library catalogues
• But this does have problems!
“Google-isation”
• Google emphasises getting good
matches (precision) rather than all
matches (recall)
• This conflicts with knowledge of
particular user communities
– e.g. Humanities and law users need good
recall
Fielded Search
• Users resist using “non-Google”
tactics
– Boolean logic
– Fielded search
• However, these are necessary tools
for satisfying many needs
• A major research challenge is
uncovering how to bridge this gap
Case Study 2: User Search
• Users do not use even simple query
options:
– Quote marks
– Boolean logic
• Have problems diagnosing results
– Why no hits?
Case Study 2: User Search
• Stelmaszewska et al, 2004 tried a set
of improvements
– Provide tips on how to improve queries
– Displayed to the user in interface
• Used analysis of the DL content
– Co-occurrence of terms
– Fields tested for matches
Case Study 2: User Search
• Example 1:
– Query = ‘mediaeval economy’
– Suggestion = ‘ “mediaeval economy” ’
– Two (relatively) common terms too vague
• Example 2:
– Query = ‘bison AND forestry’
– Suggestion = ‘bison OR forestry’
– No document contained both terms
Case Study 2: User Search
•
•
•
•
•
•
Results were mixed
Users still reluctant to experiment
Some found tips intrusive
Some tips were followed directly
Generally query complexity rose
Precision of result sets improved
Other Studies
• Topi and Lucas (2004)
– examined the effect of training versus an
“intelligent” interface
– for boolean logic queries
– found that training users was equalled or
narrowly bettered in effectiveness by a
good interface
Methodology
How to Do A User Study
• Decide what your goal is
– Precise goals: e.g. “identify common errors
made in query syntax”
• Decide on a comparative or
evaluative study
– Evaluative = one system
– Comparative = multiple systems
How to do a User Study
• Recruit representative users
– Not who just happens to be passing/to
hand
• Have plenty of users
– 24+ for moderate effects (most issues)
– 12+ for very large effects (few cases)
• Use a latin-square design
– Balance for task order, interfaces (if
multiple)
How to do a User Study
• Decide what to measure
– And how!
• Ensure measurements are relevant to
your goal
– And can be measured!
• Evaluate data systematically
– Student’s T-test, Chi-Squared, ANOVA
– Grounded theory for qualitative data
How to do a User Study
• Capture as much information as
possible:
– Software logs
– Video capture (cheap with software)
– Interview pre- and post- study
– Questionnaires to capture basic
information
– Video user (small details can emerge)
– Don’t rely on memory – capture it!
How to do a User Study
• Read up on good HCI methods
• Paul Cairns, Anna Cox “Research
Methods for Human-Computer
Interaction”, Cambridge University
Press, 2008
Common Study Flaws
• Measurements unrelated to goals
• Descriptive (mean, average) data, not
statistically rigorous
• Overtesting of data (too many
statistical tests – unreliable!)
• Undertesting of data (is there another
explanation?)
Learning by Doing
• Try a pilot study first
– To find out what you should be doing
– Only three or four users
– And throw those results away!
• Don’t worry about “bad” results
– They will tell you something!
– Is it your experiment or the goal?
– Big results often follow a bad one!
State-of-the-Art
Part 3: Use in context
Use in Context
• A combination of social and
evaluative DL studies
• Studies how (particular) issues
related to information behaviour
affect use of DL features
Use in Context
• Evaluative studies typically look at
standard DL features
– Search, browse, thesaurus, etc.
• However, other elements of
information use will affect these
– What a user reads, work practices etc.
• Use-in-context combines these,
emphasising the system
Use in Context
• A relatively recent emerging field
• Leading figures include:
– Catherine Marshall
– George Buchanan
• A lot of work focuses on reading
– And how it affects a user’s interaction
Use in Context
• Use-in-context means
– We do not look only for one outcome of an
information seeking activity
– Each information seeking act is affected by
the last action performed by a user
– e.g. a user’s behaviour in studying one set
of search results is affected by their last
search, and this result will influence the
next
Use in Context
• The ACM/IEEE JCDL 2008 Best Paper
Award: Catherine Marshall
• A study of personal archiving
– How researchers stored their own research
materials for later use/reference
– Uncovered that sources are often
distributed
– e.g. collaboration via email; home working
– So archiving services need to support this
Methodology
• Emerging: borrows from both
existing traditions at present
• For comparing systems, new
methods are emerging:
– Complex tasks, indirect effects, multiple
logging, difficult analysis
Case Study 3: Reading
• Study of how quick initial reading of
found documents is affected by
medium
– Paper, PDF, PDF via result list
• 30 participants, 20 documents
• Video capture, observer note-taking,
interviews
Case Study 3: Reading
• Demonstrated that media biases user
preferences:
– Electronic makes shorter documents
subjectively more relevant
– Users read less of a digital document than
a printed one; often just the beginning
Case Study 3: Reading
• Now being followed up with deeper
instrumentation
– PDF reader with fine-grained logging
• Preparing structured tasks designed
to expose fine differences
• Altering the understanding of “the
information seeking process”
Tutorial: Summary
Overview
• We’ve studied:
– Social studies
– Evaluative studies
– Use in context studies
• Only limited detail – get out and read
lots of literature!
Recommended Reading
• “Information Seeking in Electronic
Environments”, Gary Marchionini,
Cambridge University Press, 1995
• Paul Cairns, Anna Cox “Research
Methods for Human-Computer
Interaction”, Cambridge University
Press, 2008
Recommended Reading
• Christine Borgman “From Gutenberg
to the Global Information
Infrastructure”, MIT Press, 2000
• Christine Borgman “Scholarship in
the Digital Age”, MIT Press, 2007
– Good on academic use of DLs
Recommended Reading
• John Seely Brown “The Social Life of
Information”, Harvard, 2002
• Ian Witten “How to Build a Digital
Library” Morgan Kaufmann, 2002
• Articles by any of the researchers
mentioned today
Conclusions
•
•
•
•
Users seek after simplicity
Information needs are complex
Making progress is hard!
Focus on simple factors and
simple ideas
• Be relentless in finding the truth
• There are years of research ahead!
Questions?
Thank you
George Buchanan
http://www.cs.swan.ac.uk/~csgeorge/
[email protected]
Humanities Users
• Have wider needs: “relevance” hard
to capture in a few keywords
• Emphasise recall (capture all
possibly relevant documents)
• Not (particularly) time-pressured
• Have a large repertoire of familiar
content to build from
Journalists
• Seek highly relevant information
• Value precise search results
• Have very strong time pressures
– Often < 1 hour to cover a story
• Little extant knowledge of topic
IR Terminology
• Relevance: whether a document
addresses an information needs
• Precision: out of ‘n’ documents, how
many are relevant? (e.g. 5/10)
• Recall: out of all documents relevant
to a topic, how many were found (e.g.
5/31)
Document Triage
• Document triage: the decision made
by a user about the potential
relevance of a document
– Not relevant/Part relevant/Relevant
– But also re. genre, length, timeliness
• Triaging a document has many
consequences
– Identify new/satisfy old/refine existing info.
need
Document Triage
• “Triage” introduced by Catherine
Marshall in studying “Spatial
Hypertext”
– See “Spatial Hypertext and the Practice of
Information Triage”, ACM Hypertext 1997