Transcript Slide 1
ARL Statistics and Measurement Program Effective, Sustainable and Practical Assessment Martha Kyrillidou Director, ARL Statistics and Service Quality Programs Association of Research Libraries Steve Hiller Director, Assessment & Planning, University of Washington Libraries ARL Visiting Program Officer Jim Self Director, Management Information Services, University of Virginia Library ARL Visiting Program Officer University of Cape Town Libraries 16 August 2007 ARL www.arl.org ARL Mission • Non-profit organization of the libraries of research institutions in North America • Forum for exchange of ideas • Agent for collective action Mission: Shaping the future of research libraries in the changing environment of public policy and scholarly communication. Members: 123 major research libraries in North America. Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources. Users: Three million students and faculty served. Expenditures: $3.4 billion annually, $1.1 billion for acquisitions of which 37 percent is invested in access to electronic resources. www.arl.org What makes a research Library? • • • • • • • • Breadth and quality of collections and services Sustained institutional commitment to the library Distinctive resources in a variety of media Services to the scholarly community Preservation of research resources Contributions of staff to the profession Effective and innovative use of technology Engagement of the library in academic planning --from ARL ‘Principles of Membership’ The Beauty and the Beast • What makes a library? quality research “Quality much like beauty is in the eye of the beholder” The Beauty and the Beast • What makes a quality research library? “Quality much like beauty is in the eye of the beholder” The Beauty and the Beast • What makes a quality research library? “Quality much like beauty is in the eye of the beholder” Assessment at ARL A gateway to assessment tools: ARL StatsQUAL™: ARL Statistics -- E-Metrics LibQUAL+® DigiQUAL™ MINES for Libraries™ •Library Assessment Conferences •Service Quality Evaluation Academy •Library Assessment blog •Making Library Assessment Work •ESP Assessment – Effective, Sustainable, Practical www.arl.org StatsQUAL™ . Duane Webster, Executive Director www.statsqual.org StatsQUAL™ Stats Home www.libqual.org LibQUAL+ ARL Statistics (/arlstats) DigiQUAL (/digiqual) MINES (/mines) Interactive Statistics (/interactive)? Survey Management SAILS (/sails) E-Metrics (/emetrics) Login User Profile Institution Profile www.statsqual.org Updating the Traditional ARL Statistics • E-Metrics = ARL Supplementary Statistics – On going efforts to update and refine core data. – Exploring feasibility of collecting e-metrics. • ARL Task Force on New Ways of Measuring Collections : – Growing concern with utility of membership index. – Study ARL statistics to determine relevance. – Develop Profile of Emerging Research Libraries. www.statsqual.org Tradition • ARL membership criteria index: – Volumes held – Volumes Added Gross – Current Serials – Total Expenditures – Professional plus support staff Task Force Recommendations • Reserve use of the current membership criteria index to those occasions when it is needed for consideration of membership issues • Implement an expenditure-focused index • Use the new expenditure-focused index for any public reports, such as the Chronicle of Higher Education • Begin to develop a services-based index that combines the following three factors: collections, services, and collaborative relationships • Revise definitions for collections-related data categories, such as serials, and experiment with a variety of new measures, including usage data, strength of collections, and service quality measures to develop a richer set of variables for potential inclusion in the three factor alternative index • Collect qualitative data to develop a profile of ARL member libraries The power of narrative … (Peter Brophy, Yvonna Lincoln, Colleen Cook …..) The focus on economics • ARL Expenditures focused index: – Total library expenditures – Expenditures for Professional staff – Expenditures for Library materials – Professional plus support staff 100 75 50 25 0 0 Rank of expind03 Rank of expind03 = 3.24 + 0.94 * rindx03 R-Square = 0.89 25 50 Rank of index03 75 100 Linear Regression Rank of expind05 = R-Square = 0.89 100 Rank of expind05 25 0 0 50 75 2.50 + 0.94 * rindx05 25 50 Rank of index05 75 100 Linear Regression Thinking Strategically About Library Futures • What is the central work of the library and how can we do more, differently, and at less cost? • What important set of services does the library provide that others can’t? What new roles are needed? • What advantages does the research library possess? • What will be the most needed by our community of users in the next decade? How is user behavior changing? • What should our libraries aspire to be ten years from now? What are the implications of technology driven change? • What are the essential factors responsible for the success of the library? www.arl.org Defining Success in a Digital Environment • Crafting new measures of success. • Moving from measuring inputs to outputs. • Understanding impact of library roles and services. • Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals. • Reallocating and managing capabilities to focus on new definitions of success. www.arl.org • "You're going to be able to go back and forth from Google Book Search into Harvard, or from Harvard's catalog into Google,“ reminding users that it was important to keep the HOLLIS catalog the "starting point for their research," since only HOLLIS can give a full picture of the 15.5 million books in Harvard's collection, the nation's largest …Harvard would look to scan all of its-out-of copyright books, about one million, over time but that ultimately "the ideal" is one big digital library of books held in libraries around the world. ”It shouldn't matter whether it came from Harvard or Michigan" - - Sidney Verba, Harvard library director, Library Journal Academic Newswire – April 24, 2007 A LibQUAL+® Update The LibQUAL+® premise, dimensions, and methodology LibQUAL+® results LibQUAL+® in action old.libqual.org Dimensions of Library Service Quality Library Service Quality Affect of Service Information Control Empathy Scope of Content Responsiveness Library as Place Assurance Convenience Ease of Navigation Utilitarian Space Reliability Timeliness Symbol Equipment Refuge Self-Reliance old.libqual.org Model 3 Survey Instrument – “22 items… old.libqual.org …and a Box” Why the Box is so Important: • About 40% of participants provide openended comments, and these are linked to demographics and quantitative data • Users elaborate the details of their concerns • Users feel the need to be constructive in their criticisms, and offer specific suggestions for action old.libqual.org Developing DigiQUAL™ Survey Items Background: ServQUAL LibQUAL+® DigiQUAL™ LibQUAL+® Dimensions of Service Quality: DigiQUAL™ 12 themes of service quality: • Affect of Service • Information Control • Library as Place • • • • • • • • • • • • Accessibility Navigability Interoperability Collection building Resource Use Evaluating collections DL as community for users DL as community for developers DL as community for reviewers Copyright Role of Federations DL Sustainability www.digiqual.org Assessing the Value of Networked Electronic Services: The MINES survey Measuring the Impact of Networked Electronic Services (MINES) MINES for Libraries™ www.arl.org/stats/newmeas/mines.html What is MINES? • A research methodology consisting of a web-based survey form and a sampling plan. • Measures who is using electronic resources, where users are located at the time of use, and their purpose of use. • Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003. • Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, EMetrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+®, DigiQUAL™). Questions Addressed • How extensively do sponsored researchers use the new digital information environment? • Are researchers more likely to use networked electronic resources from inside or outside the library? • Are there differences in usage of electronic information based on the user’s location (e.g., in the library; oncampus, but not in the library; or off-campus)? • What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys? • Are particular network configurations more conducive to studies of digital libraries patron use? Where are the most critical assessment needs and opportunities? • Complementing LibQUAL+® with additional measures. • Developing impact studies on user success, economic value, and community return on investment. • Moving target: what is a digital library? • E-Resources: understanding usage. • Gaining acceptance and use of standard measures for e-resources. • Building a climate of assessment throughout library. What are the lessons learned? • Understanding changes in users approach to information resources. • Service quality improvement is a key factor. • Understanding the impact of e-resources on library services TRL. • Learning how to compete and/or collaborate with Google. • Upfront investment in design and development. • Making the assessment service affordable, practical, & effective. • Assessment needs to be satisfying and fun. In Closing • As higher education is challenged on accountability and effectiveness issues so will libraries. • A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work. • Basic questions of role, vision, and impact must be answered by library community. What’s in a “Library” • A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. – --Justice Oliver Wendell Holmes Selected References Selected References • Kyrillidou, Martha and Sarah Giersch. “Developing the DigiQUAL Protocol for Digital Library Evaluation.” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, 2005. [Available at http://www.libqual.org/documents/admin/digiqual-jcdl05-v5.pdf] • Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report.” Presented at the 6th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, 2005. [Available at http://www.libqual.org/documents/admin/FINAL%20REPORT_Jan26mk.pdf] • Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment" Information Research, 9(4) paper 187 (2004). [Available at http://InformationR.net/ir/9-4/paper187.html] • Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services." ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 230/231 (2003): 20-21. [Available at http://www.arl.org/newsltr/230/usage.html]. • Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) LibQUAL+™ Protocol: An E-service for Assessing the Library of the 21st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2-3, 2003. [Available at http://www.libqual.org/documents/admin/NSDL_workshop_web1.pdf] • Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections.” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, 2004. [Available at http://www.libqual.org/documents/admin/MERLOT%20Paper2_final.pdf] • Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and LibQUAL+™ Data Bases.” 7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6-8, 2005. [Available at http://www.libqual.org/documents/admin/ISKO.PDF]