ARL

Global Developments in Assessment: LibQUAL+

®

and Beyond…

Duane Webster

Executive Director Association of Research Libraries

Selena Lock

Research and Development Cranfield University

6th International JISC/CNI Conference York, U.K.

July 6-7, 2006 www.arl.org

Library Assessment in an Electronic Era

What are some of the current developments with library assessments efforts?

ARL StatsQUAL ™ E-Metrics LibQUAL+ ® DigiQUAL ™ MINES for Libraries ™

Where are the most critical needs and opportunities?

What are the lessons learned?

Duane Webster, Executive Director www.arl.org

Mission:

Shaping the future of research libraries in the changing environment of public policy and scholarly communication.

Members: Ratios:

123 major research libraries in North America.

4 percent of the higher education institutions providing 40 percent of the information resources.

Three million students and faculty served.

Users: Expenditures:

$3.4 billion annually, $1.1 billion for acquisitions of which 31 percent is invested in access to electronic resources.

Duane Webster, Executive Director www.arl.org

Thinking Strategically About Library Futures

• What is the central work of the library and how can we do more, differently, and at less cost?

• What important set of services does the library provide that others can’t? What new roles are needed?

• What advantages does the research library possess?

• What will be the most needed by our community of users in the next decade? How is user behavior changing?

• What should our libraries aspire to be ten years from now? What are the implications of technology driven change?

• What are the essential factors responsible for the success of the library?

Duane Webster, Executive Director www.arl.org

Defining Success in a Digital Environment

• Crafting new measures of success.

• Moving from measuring inputs to outputs.

• Understanding impact of library roles and services. • Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals.

• Reallocating and managing capabilities to focus on new definitions of success.

Duane Webster, Executive Director www.arl.org

ARL Statistics and Measurement

Duane Webster, Executive Director www.arl.org/stats/

StatsQUAL

™

.

www.statsqual.org

www.libqual.org

StatsQUAL

™ Stats Home LibQUAL+ ARL Statistics (/arlstats) DigiQUAL (/digiqual) MINES (/mines) SAILS (/sails) E-Metrics (/emetrics)

Login User Profile Institution Profile

Duane Webster, Executive Director Interactive Statistics (/interactive)?

Survey Management www.statsqual.org

Updating the Traditional ARL Statistics

• E-Metrics = ARL Supplementary Statistics – On going efforts to update and refine core data.

– Exploring feasibility of collecting e-metrics. • ARL Task Force on New Ways of Measuring Collections : – Growing concern with utility of membership index.

– Study ARL statistics to determine relevance.

– Develop Profile of Emerging Research Libraries.

Duane Webster, Executive Director www.statsqual.org

E-Metrics: Recommended Statistics



Patron Accessible Electronic Resources (R1-3)



Use of Networked Resources and Services (U1-5)



Expenditures for Networked Resources and Related Infrastructure (C1-3)



Library Digitization Activities (D1-3) Duane Webster, Executive Director www.arl.org/stats/newmeas/emetrics/index.html

E-Metrics: Recommended Statistics



Patron Accessible Electronic Resources (R1-3)

• R1 – Number of electronic full-text journals • R2 – Number of electronic reference sources • R3 – Number of electronic books

Duane Webster, Executive Director www.arl.org/stats/newmeas/emetrics/index.html

E-Metrics: Recommended Statistics



Use of Networked Resources and Services (U1-5)

• U1 – Number of electronic reference transactions • U2 – Number of logins (sessions) to electronic databases • U3 – Number of queries (searches) in electronic databases • U4 – Items requested in electronic databases • U5 – Virtual visits to library’s website and catalog

Duane Webster, Executive Director www.arl.org/stats/newmeas/emetrics/index.html

E-Metrics: Recommended Statistics



Expenditures for Networked Resources and Related Infrastructure (C1-3)

• C1  Cost of electronic full-text journals • C2  Cost of electronic reference sources • C3  Cost of electronic books • C4  Library expenditures for bibliographic utilities, networks and consortia • C5  External expenditures for bibliographic utilities, networks and consortia

Duane Webster, Executive Director www.arl.org/stats/newmeas/emetrics/index.html

Library Digitization Activities



Library Digitization Activities (D1-3)

• D1 – Size of library digital collection • D2 – Use of library digital collection • D3 – Cost of digital collection construction and management environment.)

Duane Webster, Executive Director www.arl.org/stats/newmeas/emetrics/index.html

A LibQUAL+

®

Update

• The LibQUAL+ ® methodology premise, dimensions, and • LibQUAL+ ® results • LibQUAL+ in action

Duane Webster, Executive Director www.libqual.org

A LibQUAL+

®

Update

LibQUAL+™

Presented by:

Selena Lock Cranfield University

6th International JISC/CNI Conference York, U.K. July 6-7, 2006

http://www.libqual.org

A LibQUAL+

®

Update

  

The LibQUAL+

®

premise, dimensions, and methodology LibQUAL+

®

results LibQUAL+

®

in action

http://www.libqual.org

LibQUAL+™

The LibQUAL+

®

premise, dimensions, and methodology

http://www.libqual.org

The need for LibQUAL+

®

 

Underlying need to demonstrate our worth The reallocation of resources from traditional services and functions



Rapid shifts in information-seeking behavior

• Need to keep abreast of customer demands 

Increasing user demands

• 37% of UK 16 – 18 year olds expect better libraries in return for their top-up fees

http://www.libqual.org

The LibQUAL+

®

Premise

PERCEPTIONS SERVICE

“….

only

customers judge quality; all other judgments are essentially irrelevant”

Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.

http://www.libqual.org

Multiple Methods of Listening to Customers

           Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys * Employee field reporting Employee surveys Service operating data capture

*A SERVQUAL-type instrument is most suitable for these methods

Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C

.

http://www.libqual.org

LibQUAL+

®

Development

    An ARL/Texas A&M University joint developmental effort based on SERVQUAL LibQUAL+ ® initially supported by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) Initial project established a expert team, re grounded SERVQUAL concepts, and designed survey methodology Survey conducted at over 700 libraries resulting in a data base of over a million user responses

http://www.libqual.org

13 Libraries English LibQUAL+® Version 4000 Respondents

PURPOSE Emergent 2000 QUAL

Describe library environment; build theory of library service quality from user perspective

DATA LibQUAL+® Project

Unstructured interviews at 8 ARL institutions

ANALYSIS

Content analysis: (cards & Atlas TI)

PRODUCT/RESULT QUAN

Test LibQUAL+® instrument Web-delivered survey Reliability/validity analyses: Cronbachs Alpha, factor analysis, SEM, descriptive statistics Case studies 1 Valid LibQUAL+® protocol Scalable process Enhanced understanding of user-centered views of service quality in the library environment 2

QUAL

Refine theory of service quality Unstructured interviews at Health Sciences and the Smithsonian libraries Content analysis

QUAL

Refine LibQUAL+® instrument E-mail to survey administrators Content analysis

QUAN

Test LibQUAL+ ® instrument Web-delivered survey Focus groups Reliability/validity analyses including Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis

QUAL

Refine theory

Iterative 2005

700 Libraries English, Dutch, Swedish, German LibQUAL+® Versions 160,000 anticipated respondents Vignette Re-tooling Cultural perspective 3 Refined survey delivery process and theory of service quality 4 Refined LibQUAL+® instrument 5 Local contextual understanding of LibQUAL+® survey responses 6

76 Interviews Conducted

      York University University of Arizona   Arizona State University of Connecticut  University of Houston  University of Kansas  University of Minnesota University of Pennsylvania University of Washington

Smithsonian Northwestern Medical http://www.libqual.org

LoadedPT:P1:01xxxxxxxxxxxxxx.txt,S:\Admin\Colleen\ServQual Interviews\TEXT Only\01xxxxxxxxx.txt (redirected: c:\zz\atlasti\fred

Dimensions of Library Service Quality

Affect of Service Empathy Responsiveness Assurance Reliability Library Service Quality Library as Place Utilitarian Space Symbol Refuge http://www.libqual.org

Model 3 Information Control Scope of Content Convenience Ease of Navigation Timeliness Equipment Self-Reliance

LibQUAL+

®

Participants

350 300 250 200 150 100 50 0 12 2000 164 316 207 43 2001 2002 2003 http://www.libqual.org

2004 255 2005

Rapid Growth in Other Areas

  

Languages

• American English • British English • • • French Dutch Swedish

In development

• Chinese • Greek • • Spanish German

Consortia

• Each may create 5 local questions to add to their survey  

http://www.libqual.org

• • • • •

Types of Institutions

• Academic Health Sciences • Academic Law • • Academic Military College or University Community College European Business Hospital Public State

Countries

• U.S., U.K., Canada, the Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia

World LibQUAL+

®

Survey 2005

Participating Libraries

LibQUAL+

®

Languages

American English French Canadian Swedish Continental French Swedish (British English) British English German Afrikaans Norwegian Dutch English Finnish

Over 700 institutions 1,000,000 respondents

http://www.libqual.org

Dutch Danish

Survey Instrument – “22 items…

http://www.libqual.org

…and a Box”



Why the Box is so Important:

• About 40% of participants provide open ended comments, and these are linked to demographics and quantitative data • • Users elaborate the details of their concerns Users feel the need to be constructive in their criticisms, and offer specific suggestions for action

http://www.libqual.org

LibQUAL+™

LibQUAL+

®

results

http://www.libqual.org

Understanding LibQUAL+

®

Results

   For the 22 items LibQUAL+ asks users’ to rate their: • Minimum service level • Desired service level • Perceived service performance This gives us a ‘Zone of Tolerance’ for each question; the distance between minimally acceptable and desired service ratings Perception ratings ideally fall within the Zone of Tolerance

http://www.libqual.org

Key to Bar Charts Key to Radar Charts

http://www.libqual.org

LibQUAL+

®

2005 Summary Colleges or Universities (American English) http://www.libqual.org

LibQUAL+

®

2005 Summary Colleges or Universities (British English) http://www.libqual.org

Overall

8.50

8.00

7.50

7.00

6.50

6.00

5.50

2003 2004 2005

Longitudinal Analysis for Colleges or Universities (American English)

8.50

8.00

7.50

7.00

6.50

6.00

5.50

2003

Affect of Service

2004 8.50

8.00

7.50

7.00

6.50

6.00

5.50

2005 2003

Information Control

8.50

8.00

7.50

7.00

6.50

6.00

Library as Place

2004 5.50

2005 2003 Minimum Mean Desired Mean Perceived Mean

http://www.libqual.org

2004 8.50

8.00

7.50

7.00

6.50

6.00

5.50

2005 2003

Overall

2004 2005 Minimum Mean Desired Mean Perceived Mean Minimum Mean Desired Mean Perceived Mean

Overall

8.50

8.00

7.50

7.00

6.50

6.00

5.50

2003 2004 2005

Longitudinal Analysis for Colleges or Universities (British English)

8.50

8.00

7.50

7.00

6.50

6.00

5.50

2003

Affect of Service

8.50

8.00

7.50

7.00

6.50

6.00

2004 5.50

2005 2003

Information Control

8.50

8.00

2004 7.50

7.00

6.50

6.00

5.50

2005 2003

Library as Place

Minimum Mean Desired Mean Perceived Mean

http://www.libqual.org

2004 8.50

8.00

7.50

7.00

6.50

6.00

5.50

2005 2003

Overall

2004 2005 Minimum Mean Desired Mean Perceived Mean Minimum Mean Desired Mean Perceived Mean

Overall

8.50

8.00

7.50

7.00

6.50

6.00

5.50

2003 2004 2005

Comparisons by user group

College or University (British English)

8.50

Undergraduates Overall

8.50

Postgraduates Overall Academic Staff Overall

8.50

8.50

Library Staff Overall

8.00

7.50

7.00

6.50

6.00

8.00

7.50

7.00

6.50

6.00

8.00

7.50

Minimum Mean Desired Mean Perceived Mean 7.00

6.50

6.00

8.00

7.50

Minimum Mean Desired Mean Perceived Mean 7.00

6.50

6.00

Minimum Mean Desired Mean Perceived Mean 5.50

2003 2004 5.50

2005 2003 2004 5.50

2005 2003 Minimum Mean Desired Mean Perceived Mean

http://www.libqual.org

2004 5.50

2005 2003 2004 2005 Minimum Mean Desired Mean Perceived Mean

General findings

 Highly desired • Making electronic resources accessible from my home or office • • Print and/or electronic journals I require for my work A haven for study, learning or research  Lowest • • • Library staff who instil confidence in users Giving users individual attention Space for group learning and group study

http://www.libqual.org

LibQUAL+™

LibQUAL+

®

in Action

http://www.libqual.org

UK University Case Study – Using LibQUAL+

®

Results

   Strategic Service Developments • Data to support service development • • Ability to identify where not meeting expectations Measure if change has met need Budget Discussions • Data to support bid for increased funding • Data to support case for change in emphasis (towards e-provision) Marketing Position • Status of the library within the University • Importance of national & international benchmarking

http://www.libqual.org

LibQUAL+

®

Outcomes

Funding received for:



New Web Services Administrator



Increased opening hours

• Now providing 222,578 seat hours per week 

Library refurbishment programme reinstated at costs in excess of £8 million

http://www.libqual.org

From:

Library Refurbishment

To:

http://www.libqual.org

Why use LibQUAL+

®

?

Feedback from LibQUAL+

®

Users

  “Why did you choose to use LibQUAL+

®

?”  LibQUAL+

®

was recommended to us as offering a well designed, thoroughly Library-focused set of survey tools  Opportunity to benchmark Cost-effectiveness Automated processing & fast delivery of results  Respectability and comparability (with others and historically)

http://www.libqual.org

The benefits of LibQUAL+

®

LibQUAL+

®

has enabled us to find out what a broad range of our users thought of the services we offer; what level of service delivery quality we had achieved in their eyes, and to get a clear picture of what they actually wanted the Library to deliver (as opposed to what we thought they wanted).

UK HE Institution, 2006

http://www.libqual.org

In Closing LibQUAL+

®       Focuses on success from the users’ point of view (outcomes) Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense Requires limited local survey expertise and resources Analysis available at local, national and inter-institutional levels Offers opportunities for highlighting and improving your status within the institution Can help in securing funding for the Library

http://www.libqual.org

LibQUAL+

®

Resources

 LibQUAL+ ® Website: http://www.libqual.org

 Publications: http://www.libqual.org/publications  Events and Training: http://www.libqual.org/events  LibQUAL+ ® Bibliography: http://www.coe.tamu.edu/~bthompson/servqbib  LibQUAL+ ® Procedures Manual: http://www.libqual.org/Information/Manual/index.cfm

http://www.libqual.org

Acknowledgements

    Fred Heath, Bruce Thompson, Colleen Cook, Martha Kyrillidou and the rest of the LibQUAL+ ® team Stephen Town, Cranfield University Helen Durndell and Jacqui Dowd, Glasgow University All SCONUL LibQUAL+ SCONUL Working Group on Performance Improvement ® Participants and the

http://www.libqual.org

LibQUAL +

®

Contact Information

 Martha Kyrillidou • • Director, ARL Statistics and Measurement Program [email protected]

 Selena Lock • • Research and Development, Cranfield University [email protected]

http://www.libqual.org

• • •

DigiQUAL™

Developing the DigiQUAL

™

Protocol for Digital Library Evaluation

NSF Funding Building on the LibQUAL+ ® experience Secures feedback on user’s perceptions of library’s web site • • Five questions on services, functionality, and content Goal is to determine utility, reliability, and trustworthiness

www.digiqual.org

Developing DigiQUAL

™

Survey Items

Background:

ServQUAL  LibQUAL+ ®  DigiQUAL ™ LibQUAL+ ® Dimensions of Service Quality: • Affect of Service • • Information Control

Library as Place

DigiQUAL™ 12 themes of service quality: • Accessibility • Navigability • Interoperability • Collection building • Resource Use • Evaluating collections • DL as community for users • DL as community for developers • DL as community for reviewers • Copyright • Role of Federations • DL Sustainability

Duane Webster, Executive Director www.digiqual.org

Pilot Testing Survey Items and Implementation

Duane Webster, Executive Director www.digiqual.org

Building a Survey

• Review and select items • Issue: aligning items to individual DL needs & users – vocabulary and content

Duane Webster, Executive Director www.digiqual.org

Customize Survey

Building a Survey, cont.

Duane Webster, Executive Director

Issue: Flexibility vs. Standardization

www.digiqual.org

Newsletters

Implementing Survey – Notification Methods

Links on site Next to resources

Duane Webster, Executive Director

Issues: no pop-ups, no individual emails

www.digiqual.org

Implementing Survey Incentives

Issues: must be easily transferable, requires email address - clear IRB

Duane Webster, Executive Director www.digiqual.org

Analysis and Reporting Issues

(mis)Interpreting results from individual digital libraries in the context of other sites Sites reluctant to share data and results

Duane Webster, Executive Director www.digiqual.org

Outstanding Issues and Challenges

• Unique DLs: niche market, critical mass, both?

• Balance: – custom vs. generic content  – results flexible vs. standard implementation  scaling • Mixed methods – Preserving user privacy – Collecting truly useful data • Moving target: digital libraries as… it depends.

Duane Webster, Executive Director www.digiqual.org

Assessing the Value of Networked Electronic Services: The MINES survey

Measuring the Impact of Networked Electronic Services (MINES) MINES for Libraries™

Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

What is MINES?

• A research methodology consisting of a web-based survey form and a sampling plan. • Measures who is using electronic resources, where users are located at the time of use, and their purpose of use.

• Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003.

• Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, E Metrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+ ® , DigiQUAL ™ ).

Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

Questions Addressed

• How extensively do sponsored researc hers use the new digital information environmen t?

• Are researchers more likely to use net worked electronic resources from inside or outside the lib rary?

• Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on campus, but not in the library; or off-campus)?

• What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?

• Are particular network configurations more conducive to studies of digital libraries patron use?

Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

MINES Methodological Considerations

• A representative sampling plan, including sample size, is determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library.

• Random moment/web-based surveys are employed at each site.

• Participation is usually mandatory, negating non respondent bias, and is based on actual use in real-time.

• Libraries with database-to-web gateways or proxy re writers offer the most comprehensive networking solution for surveying all networked services users during survey periods.

Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

Web Survey Design Guidelines

• Simple text for different browsers – no graphics.

– Different browsers render web pages differently.

• Few questions per screen or simply few questions.

• Easy to navigate.

• Short and plain.

• No scrolling.

• Clear and encouraging error or warning messages.

• Every question answered in a similar way - consistent.

– Radio buttons, drop downs.

• ADA compliant.

• Introduction page or paragraph.

• Easy to read.

– Must see definitions of sponsored research. • Can present questions in response to answers.

Dillman, D.A. 2000 (December).

Mail and Internet Surveys, The Tailored Design Method

.

Duane Webster, Executive Director

2nd Ed. New York: John Wiley & Sons.

www.arl.org/stats/newmeas/mines.html

Library User Survey Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

Recent Data Collection Activities

Data was collected at seven main campus libraries and seven academic health science libraries in the U.S., between January, 2003 and January, 2005.

Main University Libraries

University of Colorado University of Connecticut University of North Carolina Oregon State University University of Utah University of Virginia Washington University

Academic Medical Libraries

University of Connecticut Health Center University of North Carolina University of Texas Medical Branch University of Texas Southwestern University of Utah University of Virginia Washington University

Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

Recent Data Collection Activities

• More than 45,000 networked electronic services uses were surveyed.

• At each library, the MINES survey was one component of a comprehensive cost analysis study that assigned all library costs to sponsored research, instruction/education/non-sponsored research, patient care, other sponsored activities and other activities.

• MINES is also being conducted by the Ontario Council of University Libraries (OCUL) in 2004-2005 in conjunction with ARL.

Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

Purpose of Use By Location Main Campus Libraries 2003 – 2005

66% In the Library n = 7,618 n = 5,012 26% Off-Campus 33% 3% 5% 2% 6% On-Campus, not in the Library n = 6,641 14% 2% Sponsored Research Instruction Other Other Sponsored Activities 63% All Usage n = 19,271 24% 21% 2% 11% 63% 59% •

72% of sponsored research usage of electronic resources occurred outside the library; 83% of this took place on campus.

Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

Location of Users 2003-2005

19,798 Medical Library 6,312 In Library Users Remote Users Main Library 11,653 60% 40% 7,618

Duane Webster, Executive Director

31,451 All Libraries 70% 30% 13,930

www.arl.org/stats/newmeas/mines.html

Demographics by Location of User Main Libraries

45% Inside the Library n = 7,064 8% 20% 39% Off-Campus n = 4,953 27% 22% 27% 12% On Campus, Not in the Library n = 6,391 3% 40% Graduate Students 24% Faculty, Staff, Research Fellows Undergraduate Students All Other Users 33% Total Users n = 18,408 38% In the Library Outside the Library 62%

Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

Electronic Services Sponsored Research Use Compared to Print Journal and Total Library Use

Main Libraries 50% 40% 30% 20% 10% E-Resources Print Journals Total Use 0% Li bra ry 1 Li bra ry 2 Li bra ry 3 Li bra ry 4 Li bra ry 5 Li bra ry 6 Li bra ry 7 Av era ge

Duane Webster, Executive Director www.arl.org/stats/newmeas/mines.html

Where are the most critical assessment needs and opportunities?

• Complementing LibQUAL+ ® with additional measures.

• Developing impact studies on user success, economic value, and community return on investment.

• Moving target: what is a digital library?

• E-Resources: understanding usage.

• Gaining acceptance and use of standard measures for e-resources.

• Building a climate of assessment throughout library.

Duane Webster, Executive Director

What are the lessons learned?

• Understanding changes in users approach to information resources.

• Service quality improvement is a key factor.

• Understanding the impact of e-resources on library services TRL.

• Learning how to compete with Google.

• Upfront investment in design and development.

• Making the assessment service affordable, practical, & effective.

• Assessment needs to be satisfying and fun.

Duane Webster, Executive Director

In Closing

• As higher education is challenged on accountability and effectiveness issues so will libraries.

• A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work.

• Basic questions of role, vision, and impact must be answered by library community.

Duane Webster, Executive Director

• Kyrillidou, Martha and Sarah Giersch. “Developing the DigiQUAL Protocol for Digital Library Evaluation.” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, 2005. [Available at http://www.libqual.org/documents/admin/digiqual-jcdl05-v5.pdf

] • Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report.” Presented at the 6th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, 2005. [Available at http://www.libqual.org/documents/admin/FINAL%20REPORT_Jan26mk.pdf

] • Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment"

Information Research

, 9(4) paper 187 (2004). [Available at http://InformationR.net/ir/9-4/paper187.html] • Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services."

ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC,

230/231 (2003): 20-21. [Available at http://www.arl.org/newsltr/230/usage.html

].

• Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) LibQUAL+™ Protocol: An E-service for Assessing the Library of the 21st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2-3, 2003. [Available at http://www.libqual.org/documents/admin/NSDL_workshop_web1.pdf

] • Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections.” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, 2004. [Available at http://www.libqual.org/documents/admin/MERLOT%20Paper2_final.pdf

] • Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and LibQUAL+™ Data Bases.” 7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6-8, 2005. [Available at http://www.libqual.org/documents/admin/ISKO.PDF

]

Duane Webster, Executive Director