The Price of Experience: evidence for the value of the experienced health librarian above that of the novice

Andrew Booth

, Reader in Evidence Based Information Practice, ScHARR, University of Sheffield.

Me and My Big Mouth!

• ABo: “As you know, Paul, I am interested in Evidence Based Library and Information Practice, “To what extent do you think that the finding that performance gets worse with duration of practice applies equally to Librarians?”” • PG: “I am not aware of any studies that look at this issue in other professions. I would imagine that someone like yourself would be best placed to look into that particular issue” • [ABo: {Sotto voce}

with less “Assuming, of course, that the job wouldn’t be better done by a librarian experience!”]

The Trade-off!

• Limitations of trading “Precise answers to the wrong questions at the expense of approximate answers to the right questions” John Tukey (1915-2000) • What is experience?

What is experience?

• Age?

• Number of years in practice?

• Number of years post qualification?

• Number of years in generic context (e.g. health librarianship)?

• Number of years in specific context (e.g. online searching)?

• Domain versus System • Relationship to volume of experience (how much/how often)?

• Problem with “expertise” – comparing “experts” versus “novices” can be self-fulfilling!

• Objective evaluation versus self-assessed?

• Outcome measures – skill, knowledge etcetera?

From the Wider Literature - 1

• Repertoire of 10 years experience generally accepted to qualify person as an expert (Klein & Calderwood, 1986) • Not only length of experience important; also quality of experience. • For fire-fighters, 10 years on-the-job experience not as valuable for skill development as one year in decaying inner city (Klein et al. 1986) • Mental structure of novices more chaotic and disorganized. Experts have acquired great deal of prior knowledge; more able to impose structure on content (Spires and Donley, 1998).

From the Wider Literature - 2

Lipshitz and Shaul (1997) found that: • Experts collect more information on situation before they make a decision because they know what questions to ask.

• Experts engage in more efficient information search because they know what information to keep track of.

• Experts read situation more accurately because they are able to identify the difference between relevant and surplus information.

• Experts make fewer bad decisions related to reading situation correctly.

• Experts communicate more frequently and elaborately with friendly units.

• Experts more likely to consider other people’s perspectives when making their decisions.

From the Wider Literature 3

• Glaser and Chi (1988) extracted list of themes commonly associated with expert performance.

• Expertise is domain specific.

• Experts see patterns.

• Experts are faster and make fewer errors.

• Experts have superior memory in their domain (environmental cues are an aid to recall).

• Experts see and represent problem at a deeper level.

• Experts spend more time trying to understand problem but have more efficient path to solving it.

• Experts have strong self-monitoring skills.

• Experts have refined perceptual abilities.

Focused Question

• • • • •

P

opulation: Librarians (health librarians) or non librarians engaging in “typical” library-related activities

E

xposure: Extensive exposure (hours, years) OR arbitrary identification as “expert”. Ideally duration of exposure is principal measure (not one of many variables); other confounders controlled for.

C

omparison: Less Extensive exposure (hours, years) OR arbitrary identification as “novice”.

O

utcomes: Performance (not Attitudes)

In English, Published 1980-2010 [Only one study excluded]

Indicative Search results

• LISA (285), LISTA (370), Google Scholar (16,900*), ERIC (76), Pubmed (17), Web of Knowledge (77) • Started with Very Sensitive Subject Searches • (Experience* OR Expert* OR Advanced) AND (Novice* OR Beginner* OR Newbie*) {AND Librar*} • Follow-up of References • Email to Discussion Lists • Citation Searching

Number of Studies Retrieved

• In Health Librarians • In Librarians (non-health) • Skills Commonly Practised by Librarians 10 13 56 Sample comparisons • Libns vs Libns; Libns vs Students, Libns vs Clinicians; Students vs Students etc

Findings – 1

• 96 studies for review • Review Articles • 79 studies included • Information Retrieval • Teaching & Learning • Reference/Enquiry Work • Knowledge Management • Marketing ( 3) (67) ( 6) ( 3) ( 2) ( 1)

More Experience > Less Experience

•

Web of Science:

Experienced females best for success score/error rates; novice males worst (Ahmed et al, 2004) • • •

Database Searching:

Experts significantly faster than novices (Dillon & Song, 1997). Novice librarians twice as long as experienced librarians (Fenichel, 1981) Librarians significantly better precision. Novice clinicians lower recall and precision than librarians/expert clinicians (Haynes et al, 1990) Clinicians recall slightly lower than for librarians, precision significantly less (Haynes et al, 1990). Experienced librarians statistically significant advantage in recall over physicians (Hersh et al, 1994)

• • • • •

More Experience > Less Experience

Clinical Experts retrieved significantly more references than Librarians or Novices; Librarians retrieved significantly more relevant references. (McKibbon et al, 1988) Librarians significantly better for both recall and precision than novices (McKibbon et al, 1990). Undergraduates poorer relevance than librarians (Penhale & Taylor 1986) Librarians

better precision

(McKibbon et al, 1990).

than experienced end-users.

Use of Boolean operators [OPAC] of thesaurus terms

(Dinet et al, 2004); (Fenichel, 1981)

Use Greater Number of terms

,

Less errors

1981) . in use of nonsupported operators (Lucas & Topi 2002). Experts used six times as many ORs (Rudner, 2000) 1.3 non-typographical errors/search for experienced subjects and about twice as many for novices (Fenichel,

Cost effectiveness:

searchers performed briefest, most cost effective searches (Fenichel, 1981) moderately ERIC-experienced

More Experience > Less Experience

• • • •

Web Searching :

Task Completion

sites Speed

(Aula et al, 2006) Information experts capture

higher quality web

compared with doctors (Groot et al, 2001) Experts

faster

, completed

more tasks efficient and effective

. Significantly more proficient in

locating Web sites

2000) ,

more

(Lazonder et al Experts used

more keywords

and evaluated sites based on

established criteria

(wider knowledge base). Novices used “Back” key more often, used fewer numbers of search engines, and missed some highly relevant sites (Tabatabai& Luconi, 1998).

More Experience = Less Experience

• • • •

Searching:

Precision of novice/experienced librarians nearly same (Fenichel, 1981); Experienced clinicians/librarians achieved comparable recall (Haynes et al, 1990; McKibbon et al, 1990) For locating information on specific Web sites, performance of experienced/novices equivalent (Lazonder et al 2000) No significant performance differences between experience levels on any of four Web search tasks (Thatcher 2008 ) Experts outperform novice users when system interface organized semantically, not when organized randomly (Salmeron et al 2005)

• • •

More Experience = Less Experience

Enquiries:

NS Difference: Accuracy of executives (

experts

)/frontline employees (

novices

). Executives demonstrate overconfidence in judgements; frontliners demonstrate underconfidence. (Hallin et al, 2009)

Reference work:

2007). Students’ answers had same level of quality as responses supplied by professional, experienced librarians working on Ask The Librarian service (Sveum

Bibliographic Instruction:

undergraduate learning gains (methods more important). (Stec 2006) NS difference in

More Experience < Less Experience

• Novice clinicians

much higher recall

than expert searchers, paid price in precision (Hersh & Hickam, 1994) • Skilled students

more errors

than novices when using physically inconsistent e learning system (Rhee et al 2006) • More experience results in more passive methods of curriculum implementation; less experienced librarians display

more active methods

(Pittman 2003)

Findings – 2

• •

Knowledge Management

• Older, more experienced librarians higher level of collaboration, more familiar with organizations, more secure in workplace, understand importance and essence of KM and ready to share knowledge with colleagues (Aharony, nd).

Computer Learning

• Novices best with offline manual, and worst with visual online. Experienced best with online training and worst with visual online. • Experienced users reasonably good perceptions of what kinds of training best contribute to own learning. Novices largely erroneous in their perceptions (Hsu & Turoff 2007)

Findings – 3

• • •

How Knowledge is Organised

Experienced archivists'

knowledge of institution

significant compared with novices'. Experienced archivists directly access primary resources whereas novices often needed to perform meta-searches. Experienced archivists’ decisions based on

evaluations

of quality of primary resources for form and format, comprehensiveness and accuracy of content to be as efficient as possible while still providing 'good' information.

Evaluation

extended to use of access tools. Experienced archivists used

cues in environment

and

episodic memory of prior events

to locate relevant information without referring to labels, texts, etc.: ability to intellectually and physically access information was intertwined Yakel & Antony (2006)

How good are we?

• Experienced reference librarians typically retrieve about twice as many citations as less experienced users (Hersh & Hickam, 1993). • With minimal training, physicians could improve their performance to level of experienced searchers by their fourth online search (Haynes et al, 1993).

• “After about eight searches they [clinicians] got as many relevant citations from the literature as the experienced librarians.

“ (Computers a cornerstone of evidence-based care, conference told. F Lowry -

CMAJ: Canadian Medical Association Journal

, 1995)

Other elements in the mix

Domain & System Interplay (Jenkins et al, 2003)

Web Novice Web Expert Domain Expert

Searched

breadth first

but evaluated information more thoroughly using subject knowledge.

Domain Novice

Searched

breadth first

little/no evaluation of results

Depth-first

searches, following deep trails of information. Evaluated information based on most varied and sophisticated criteria.

Mixed, breadth first/depth-first

pattern attempted to evaluate information using general criteria.

Discussion

• • Performance relies on mix of

Technical and Communication Skills

(E.g.

Reference Interview Searching)

plus

Online

• Perform better when we

use interfaces that play to our strengths

(semantic and hierarchical)

It ain’t what you do it’s the way that you do it

(e.g. instruction/curriculum development)

Interim Conclusions

• Overplaying Technical Skills (easily acquired) • • Should stress

superior mental models

(see “whole picture”) and

communication abilities Self awareness

,

evaluation

•

Structure

can reduce differences (checklists/metaphors/maps/models) • Experience is

red herring

cp. • “Poor searchers either gave up too quickly, employed few search terms, used only simple queries, or used wrong search terms. Good searchers persisted longer, used larger, richer set of terms, constructed more complex queries, more diligent in evaluating retrieved results” (

Performance

!). (Sutcliffe et al, 2000)

The Way Forward

• Categorise Outcome Tools and Measures (Subjective/Objective; Surrogate Outcomes/Final Outcomes) • • Analyse Qualitative Data (The How, not just the What) • Article in

Evidence Based Library and Information Practice

or

HILJ? [References available on request]

Acknowledgements

• Dr Alison Brettle, University of Salford • Lori Kloda, McGill University • Kate Misso, Information Specialist, Kleijnen Systematic Reviews Ltd.

• Professor Dianne Oberg, University of Alberta