Transcript The Price of Experience - Chartered Institute of Library
The Price of Experience: evidence for the value of the experienced health librarian above that of the novice
Andrew Booth
, Reader in Evidence Based Information Practice, ScHARR, University of Sheffield.
Me and My Big Mouth!
• ABo: “As you know, Paul, I am interested in Evidence Based Library and Information Practice, “To what extent do you think that the finding that performance gets worse with duration of practice applies equally to Librarians?”” • PG: “I am not aware of any studies that look at this issue in other professions. I would imagine that someone like yourself would be best placed to look into that particular issue” • [ABo: {Sotto voce}
with less “Assuming, of course, that the job wouldn’t be better done by a librarian experience!”]
The Trade-off!
• Limitations of trading “Precise answers to the wrong questions at the expense of approximate answers to the right questions” John Tukey (1915-2000) • What is experience?
What is experience?
• Age?
• Number of years in practice?
• Number of years post qualification?
• Number of years in generic context (e.g. health librarianship)?
• Number of years in specific context (e.g. online searching)?
• Domain versus System • Relationship to volume of experience (how much/how often)?
• Problem with “expertise” – comparing “experts” versus “novices” can be self-fulfilling!
• Objective evaluation versus self-assessed?
• Outcome measures – skill, knowledge etcetera?
From the Wider Literature - 1
• Repertoire of 10 years experience generally accepted to qualify person as an expert (Klein & Calderwood, 1986) • Not only length of experience important; also quality of experience. • For fire-fighters, 10 years on-the-job experience not as valuable for skill development as one year in decaying inner city (Klein et al. 1986) • Mental structure of novices more chaotic and disorganized. Experts have acquired great deal of prior knowledge; more able to impose structure on content (Spires and Donley, 1998).
From the Wider Literature - 2
Lipshitz and Shaul (1997) found that: • Experts collect more information on situation before they make a decision because they know what questions to ask.
• Experts engage in more efficient information search because they know what information to keep track of.
• Experts read situation more accurately because they are able to identify the difference between relevant and surplus information.
• Experts make fewer bad decisions related to reading situation correctly.
• Experts communicate more frequently and elaborately with friendly units.
• Experts more likely to consider other people’s perspectives when making their decisions.
From the Wider Literature 3
• Glaser and Chi (1988) extracted list of themes commonly associated with expert performance.
• Expertise is domain specific.
• Experts see patterns.
• Experts are faster and make fewer errors.
• Experts have superior memory in their domain (environmental cues are an aid to recall).
• Experts see and represent problem at a deeper level.
• Experts spend more time trying to understand problem but have more efficient path to solving it.
• Experts have strong self-monitoring skills.
• Experts have refined perceptual abilities.
Focused Question
• • • • •
P
opulation: Librarians (health librarians) or non librarians engaging in “typical” library-related activities
E
xposure: Extensive exposure (hours, years) OR arbitrary identification as “expert”. Ideally duration of exposure is principal measure (not one of many variables); other confounders controlled for.
C
omparison: Less Extensive exposure (hours, years) OR arbitrary identification as “novice”.
O
utcomes: Performance (not Attitudes)
In English, Published 1980-2010 [Only one study excluded]
Indicative Search results
• LISA (285), LISTA (370), Google Scholar (16,900*), ERIC (76), Pubmed (17), Web of Knowledge (77) • Started with Very Sensitive Subject Searches • (Experience* OR Expert* OR Advanced) AND (Novice* OR Beginner* OR Newbie*) {AND Librar*} • Follow-up of References • Email to Discussion Lists • Citation Searching
Number of Studies Retrieved
• In Health Librarians • In Librarians (non-health) • Skills Commonly Practised by Librarians 10 13 56 Sample comparisons • Libns vs Libns; Libns vs Students, Libns vs Clinicians; Students vs Students etc
Findings – 1
• 96 studies for review • Review Articles • 79 studies included • Information Retrieval • Teaching & Learning • Reference/Enquiry Work • Knowledge Management • Marketing ( 3) (67) ( 6) ( 3) ( 2) ( 1)
More Experience > Less Experience
•
Web of Science:
Experienced females best for success score/error rates; novice males worst (Ahmed et al, 2004) • • •
Database Searching:
Experts significantly faster than novices (Dillon & Song, 1997). Novice librarians twice as long as experienced librarians (Fenichel, 1981) Librarians significantly better precision. Novice clinicians lower recall and precision than librarians/expert clinicians (Haynes et al, 1990) Clinicians recall slightly lower than for librarians, precision significantly less (Haynes et al, 1990). Experienced librarians statistically significant advantage in recall over physicians (Hersh et al, 1994)
• • • • •
More Experience > Less Experience
Clinical Experts retrieved significantly more references than Librarians or Novices; Librarians retrieved significantly more relevant references. (McKibbon et al, 1988) Librarians significantly better for both recall and precision than novices (McKibbon et al, 1990). Undergraduates poorer relevance than librarians (Penhale & Taylor 1986) Librarians
better precision
(McKibbon et al, 1990).
than experienced end-users.
Use of Boolean operators [OPAC] of thesaurus terms
(Dinet et al, 2004); (Fenichel, 1981)
Use Greater Number of terms
,
Less errors
1981) . in use of nonsupported operators (Lucas & Topi 2002). Experts used six times as many ORs (Rudner, 2000) 1.3 non-typographical errors/search for experienced subjects and about twice as many for novices (Fenichel,
Cost effectiveness:
searchers performed briefest, most cost effective searches (Fenichel, 1981) moderately ERIC-experienced
More Experience > Less Experience
• • • •
Web Searching :
Task Completion
sites Speed
(Aula et al, 2006) Information experts capture
higher quality web
compared with doctors (Groot et al, 2001) Experts
faster
, completed
more tasks efficient and effective
. Significantly more proficient in
locating Web sites
2000) ,
more
(Lazonder et al Experts used
more keywords
and evaluated sites based on
established criteria
(wider knowledge base). Novices used “Back” key more often, used fewer numbers of search engines, and missed some highly relevant sites (Tabatabai& Luconi, 1998).
More Experience = Less Experience
• • • •
Searching:
Precision of novice/experienced librarians nearly same (Fenichel, 1981); Experienced clinicians/librarians achieved comparable recall (Haynes et al, 1990; McKibbon et al, 1990) For locating information on specific Web sites, performance of experienced/novices equivalent (Lazonder et al 2000) No significant performance differences between experience levels on any of four Web search tasks (Thatcher 2008 ) Experts outperform novice users when system interface organized semantically, not when organized randomly (Salmeron et al 2005)
• • •
More Experience = Less Experience
Enquiries:
NS Difference: Accuracy of executives (
experts
)/frontline employees (
novices
). Executives demonstrate overconfidence in judgements; frontliners demonstrate underconfidence. (Hallin et al, 2009)
Reference work:
2007). Students’ answers had same level of quality as responses supplied by professional, experienced librarians working on Ask The Librarian service (Sveum
Bibliographic Instruction:
undergraduate learning gains (methods more important). (Stec 2006) NS difference in
More Experience < Less Experience
• Novice clinicians
much higher recall
than expert searchers, paid price in precision (Hersh & Hickam, 1994) • Skilled students
more errors
than novices when using physically inconsistent e learning system (Rhee et al 2006) • More experience results in more passive methods of curriculum implementation; less experienced librarians display
more active methods
(Pittman 2003)
Findings – 2
• •
Knowledge Management
• Older, more experienced librarians higher level of collaboration, more familiar with organizations, more secure in workplace, understand importance and essence of KM and ready to share knowledge with colleagues (Aharony, nd).
Computer Learning
• Novices best with offline manual, and worst with visual online. Experienced best with online training and worst with visual online. • Experienced users reasonably good perceptions of what kinds of training best contribute to own learning. Novices largely erroneous in their perceptions (Hsu & Turoff 2007)
Findings – 3
• • •
How Knowledge is Organised
Experienced archivists'
knowledge of institution
significant compared with novices'. Experienced archivists directly access primary resources whereas novices often needed to perform meta-searches. Experienced archivists’ decisions based on
evaluations
of quality of primary resources for form and format, comprehensiveness and accuracy of content to be as efficient as possible while still providing 'good' information.
Evaluation
extended to use of access tools. Experienced archivists used
cues in environment
and
episodic memory of prior events
to locate relevant information without referring to labels, texts, etc.: ability to intellectually and physically access information was intertwined Yakel & Antony (2006)
How good are we?
• Experienced reference librarians typically retrieve about twice as many citations as less experienced users (Hersh & Hickam, 1993). • With minimal training, physicians could improve their performance to level of experienced searchers by their fourth online search (Haynes et al, 1993).
• “After about eight searches they [clinicians] got as many relevant citations from the literature as the experienced librarians.
“ (Computers a cornerstone of evidence-based care, conference told. F Lowry -
CMAJ: Canadian Medical Association Journal
, 1995)
Other elements in the mix
Domain & System Interplay (Jenkins et al, 2003)
Web Novice Web Expert Domain Expert
Searched
breadth first
but evaluated information more thoroughly using subject knowledge.
Domain Novice
Searched
breadth first
little/no evaluation of results
Depth-first
searches, following deep trails of information. Evaluated information based on most varied and sophisticated criteria.
Mixed, breadth first/depth-first
pattern attempted to evaluate information using general criteria.
Discussion
• • Performance relies on mix of
Technical and Communication Skills
(E.g.
Reference Interview Searching)
plus
Online
• Perform better when we
use interfaces that play to our strengths
(semantic and hierarchical)
It ain’t what you do it’s the way that you do it
(e.g. instruction/curriculum development)
Interim Conclusions
• Overplaying Technical Skills (easily acquired) • • Should stress
superior mental models
(see “whole picture”) and
communication abilities Self awareness
,
evaluation
•
Structure
can reduce differences (checklists/metaphors/maps/models) • Experience is
red herring
cp. • “Poor searchers either gave up too quickly, employed few search terms, used only simple queries, or used wrong search terms. Good searchers persisted longer, used larger, richer set of terms, constructed more complex queries, more diligent in evaluating retrieved results” (
Performance
!). (Sutcliffe et al, 2000)
The Way Forward
• Categorise Outcome Tools and Measures (Subjective/Objective; Surrogate Outcomes/Final Outcomes) • • Analyse Qualitative Data (The How, not just the What) • Article in
Evidence Based Library and Information Practice
or
HILJ? [References available on request]
Acknowledgements
• Dr Alison Brettle, University of Salford • Lori Kloda, McGill University • Kate Misso, Information Specialist, Kleijnen Systematic Reviews Ltd.
• Professor Dianne Oberg, University of Alberta