Transcript Evaluation Chat Reference Service from User’s perspective
Evaluation of a Chat Reference Service from the User’s Perspective
Lili Luo Jeffrey Pomerantz School of Information & Library Science UNC Chapel Hill
Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931
Brief Outline
Research Context: NCknows Research Questions & Theoretic Framework Study Design & Methodology Results Discussion & Future Work
NCknows
Collaborative state-wide chat service in North Carolina (www.ncknows.org) 18-month pilot phase launched in February 2004, with 19 libraries participating.
•Librarians in all 19 libraries staffed the service during the pilot.
•Participants: public, research university, community college, government libraries; urban and rural, large and small.
Pilot phase is completed. Service had its final launch in July 2005.
Evaluation
4 Components to the Evaluation •Baseline: Statistics from the 24/7 reference chat application •Quality control: Peer reviews of chat transcripts •Administration: Interviews with librarians •Use & user satisfaction: Exit surveys & follow-up interviews
Research Questions
Motivation: What motivates users to use a chat reference service?
Use: How do users use the information provided by the chat reference service?
Satisfaction: What is the users’ level of satisfaction with chat reference service?
Dervin’s Sense-Making Theory
situation
motivation to seek assistance from chat reference service gap bridging
fulfillment
use of information provided by chat reference service
Study Design
Exit Survey an exit survey popped up for the user at the conclusion of a chat session Email & Telephone Interviews semi-structured interviews were conducted two to three weeks after users’ chat sessions
Results
Prior to Use of the Service •Use of Other Reference Services •Discovery of NCknows •Motivation for Using the Service •Motivation for Asking the Question The Point of Service •Satisfaction Levels •Role of User Use of the Information Provided by the Service
How did you find out about the NCknows chat service?
(Follow up interview − 68 responses) Four routes by which users learned about the service emerged:
1. The local library, or library-produced publicity materials (69%).
2. Online searching (12%).
3. School system (12%).
4. Friends and colleagues (7%).
Have you ever used any of your library’s other reference services? Which?
(Follow up interview − 67 responses)
Reference Services
desk reference service
Respondents
48% e-mail reference service telephone reference service 19% 33% never used any other reference services 19%
did not answer the question directly librarians or library staff 7% 7%
Why did you use the chat service to get this question answered rather than any of these other services?
(Follow up interview − 68 responses)
•Convenience (47%).
•Other means of seeking information were not helpful (15%).
•Curiosity (13%).
•Serendipity (12%).
•Recommended by others (7%).
•Personal characteristics/habits (7%).
•Other reference services were not available (1.5%).
Can you tell me about what motivated the question that you asked the NCknows chat service?
(Follow up interview − 72 responses)
To satisfy an inquiry about the library itself, 3% To help others look for information, 3% Others, 3% To locate a known item, 8% To answer a question arising from the user’s personal life, 32% To answer a work/school related question, 51%
Business-related:
“I’m a small business owner and I need the information to negotiate with other type of business people to do a joint venture.”
School-related:
“I was doing a research paper on homeschooling. Want some hard stats numbers from the website about how homeschool students do academically.”
Genealogy:
“Wanted to find passenger ships of 1800 that had been lost at sea, trying to find one that may have been carrying wife and children of my great grandfather.”
Vocational or academic change:
“doing research on school that meets the criteria that I’m looking for. I want to go back to school.”
Known-item search:
“I was looking for an audio book and couldn’t understand the new system.”
On behalf of others:
“It was my son’s schoolwork questions in Geography class. He had found the answers to many of his geography questions but we had searched and searched using Google and also used Ask Jeeves but were not successful in finding the answers to a couple of questions.”
How satisfied were you with the …?
(Follow up interview − 72 responses) Very satisfied Satisfied Completeness of the answer 68.5% Speed that the librarian answered your question 67.7% Helpfulness of the librarian 81.2% Ease of use of the software 82.7% 23.55% 24.9% 13.1% 14.2% Dissatisfied 5.4% 5.6% 3.4% 1.8% Very dissatisfied Total: n = 2.6% 387 2.1% 381 2.4
382 1.3% 387
Would you recommend this chat service to a friend?
(Exit survey − 385 responses) Recommend Very likely Maybe Unlikely Never Total Percentage 87.5
10.1
1.6
0.8
100%
You asked this question in your role as a(n): (Exit survey − 379 responses) Role Other Student: Undergraduate Student: Graduate Student: K-12 Parent Librarian Medical Professional Teacher: K-12 Higher Ed Faculty Adult Educator Administrator Teacher: Pre-school Policymaker Politician 3.6
3.5
3.1
2.7
0.8
0.0
0.0
Percentage 29.7
23.2
13.5
10.3
7.8
7.7
6.3
Have you had a chance to use the information that NCknows provided to you?
(Follow up interview − 67 responses)
70% 63% 60% 50% 40% 30% 20% 10% 0% 12% 25% Use Partial Use No use
Use:
“It was utilized as background information for my lecture.” “I used the information in the negotiation with other business owners. Through the knowledge gained, we have (hopefully) brought another business to Fayetteville. At least we are talking now.”
Partial Use:
“NCknows told me the person to call at my local library who said they can't provide that information. I was going to use it in my graduate school thesis. Alas…” “I am still working on this matter but was provided good leads from your service.”
No use:
“The librarian did give me a website for on loan tapes, but they took to long to receive.” “I plan to call the number I got from the service in a couple of time.”
Discussion
Target User Groups
Campaigns to market chat reference services should be targeted to user groups that are likely to have a need for the service: for example, in grade schools, colleges, and universities; businesses and other corporate settings; and social organizations.
Expand Service Hours
The availability of the service at all hours of the day and night is one important factor in the convenience of the service.
Identify Appropriate Reference Medium for Users
Encourage users to use a reference service in the most appropriate medium.
Research question: How to identify the most appropriate medium for a user / for a question?
Limitations
• Non-response bias • Self-selection bias
Future Work
• •
Methodological enhancements
: alternative subject recruiting methods
Further the exploration of service value
: what users gain from using the information provided by the service