Ten years after:

„Measuring Quality“ revised

Roswitha Poll Münster, Germany

Measuring Quality 1st edition 1996 English, German, French, Russian, Italian only academic libraries 17 indicators

2/21

Measuring Quality 2nd revised edition 2007 academic and public libraries 40 indicators Authors: Peter te Boekhorst, Roswitha Poll

3/21

Measuring Quality: Why?

So what are you doing at the moment?

Writing a book about performance indicators for libraries

What’s that, performance indicators?

Methods for seeing whether the library is good things right

For example?

Okay, sounds easy

But why do you need a book for that?

That all the books are always in their right place

Yes, that’s important. And such things show you whether you are good?

Well, not quite, but I can compare with other libraries

And if they are worse, you know that you are good?

That’s about it 4/21

Changes in the library world

electronic services predominant the teaching library as new role cost-effectiveness crucial competition of the institutions library as physical place widens to meeting and communication 5/21

Trends in performance measurement

stakeholders‘ views quantitative and qualitative measures standardisation has proceeded - ISO 11620 will comprise e-measures user surveys find broad acceptance impact/outcome measures 6/21

More use of performance measures?

More data available -

For electronic services

- resources - expenditure usage

In national statistics

- reference transactions - data for library space But: • Visits still only physical visits • Digitisation missing - attendances at user training and events 7/21

More use of performance measures?

Joint benchmarking projects

BIX –Library Index

German public and academic libraries CASL (Council of Australian State Libraries) public libraries

Swedish Quality Handbook

all types of libraries HELMS (UK Higher Education Library Management Statistics) academic libraries

Netherlands University Libraries

8/21

More use of performance measures?

Joint benchmarking projects Projects:

5

Indicators:

55

Corresponding to ISO 11620:

25

Used in more than one project:

20

9/21

More use of performance measures?

Interactive national statistics allow combinations of data (e.g. per capita) Less reports from individual libraries But: service level agreements with performance indicators on websites Performance measures have been integrated 10/21

The handbook: structure

Balanced Scorecard chapter on impact/outcome bibliography for each indicator examples for possible scores 11/21

The handbook: How to choose indicators

The set of indicators should cover the full range of library services consider traditional and electronic services if possible „merged“ indicators include indicators for „potentials and development“ consider the view of different stakeholders 12/21

Library quality: Stakeholder views

Users

Access to information worlwide Delivery of information to the desktop Speed and accuracy of delivery Good in-library working conditions Responsiveness of staff Reliability of services 13/21

Library quality: Stakeholder views

Financing authorities

Cost-effectiveness Clear planning, effective organisation Positive outcome on users Benefits for the institution’s goals Effective cooperation High reputation of the library 14/21

Library quality: Stakeholder views

Staff

Good working conditions Clear planning, straight processes Systematic staff development High reputation of the library 15/21

The handbook: How to choose indicators

Only one indicator for one question?

Example: How intensively is the collection used?

Indicators: collection turnover loans per capita percentage of stock not used Each indicator gives a different picture and might serve different purposes of the evaluating library 16/21

The indicators

A. Resources, infrastructure: What does the library offer?

Library as place for learning and research

1. User area per capita

Collections

2. Seats per capita 3. Opening hours compared to demand 4. Expenditure on information provision per capita 5. Availability of required titles 6. Percentage of rejected sessions 7. Ratio of requests received to requests sent out in interlibrary lending 8. Immediate availability

Staff Website

9. Staff per capita 10. Direct access from the homepage

17/21

Opening hours compared to demand

Definition

The actual number and time of opening hours compared to opening hours as desired by users Method: Survey Possible formula: A/B A = the number of present opening hours B = the number of hours which the users state that they need (present hours + additional hours) If a library opens 60 hours per week and in the survey users ask for 10 hours more, the score would be 60:70 = 0.86

18/21

Immediate availability

Definition

The percentage of immediate loans of total loans

Method:

Immediate loans = loans minus reservations Total loans = loans plus ILL loans received

Question:

The probability that a user’s loan request will be fulfilled immediately 19/21

Direct access from the homepage

Definition

The availability of the most frequently used resources and services via the homepage of the library’s website, measured by the number of clicks necessary and the comprehensibility of the terms used

Method

Cognitive walk-through: A small group of experts simulates user behaviour

Question

Whether the homepage leads directly or very quickly to the most frequently needed information 20/21

Direct access from the homepage

Academic libraries

Address Way to the library Opening times Online catalogue Lending service User card User account ILL/document delivery Reference Databases E-journals Subject access User training News, events 21/21

Direct access from the homepage

Public libraries

Address Way to the library Opening times Online catalogue Lending service User card User account Electronic collection Link collection Reference Fees Services for children Branch libraries Community services News, events 22/21

Access via the homepage

Service direct

information on the homepage, e.g. times open, address, catalogue search

unmistakable term

on the homepage

unmistakable term

on the homepage

unmistakable term

on the homepage

unmistakable term

on the homepage

ambiguous term

on the homepage

ambiguous term

on the homepage

Clicks 0 1 2 3



1 3



1 Points 10 8 6 2 0 4 0

The indicators

B. Use: How are the services accepted?

General Library as place Collections

1. Market penetration 2. User satisfaction 3. Library visits per capita 4. Seat occupancy rate

Information services

5. Number of content units downloaded per capita 6. Collection use (turnover) 7. Percentage of stock not used 8. Loans per capita 9. Percentage of loans to external users 10. Attendances at training lessons per capita

Cultural activities

11. Reference questions per capita 12. Attendances at events per capita

24/21

Library visits per capita

Definition

The total number of library visits per year, either physical or virtual, by members of the population, divided by the number of persons in the population

Method

Physical visits: turnstile Virtual visits: web browser visits IP visits homepage visits 25/21

Library visits per capita: A method for counting homepage visits Counting the deliveries of a Transparent GIF placed on the home page for counting purposes The number of deliveries of the pixel is identical to the number of page deliveries. The advantage of this method is that it easy to use. The disadvantage is that access to other pages is not counted.

German Benchmarking Project BIX http://www.bix-bibliotheksindex.de/ 26/21

The indicators

C. Efficiency: Are the services offered cost-effectively?

General Collection costs Processes - speed Processes - reliability

1. Cost per user 2. Cost per visit 3. Cost per use 4. Ratio of acquisitions costs to staff costs 5. Cost per database session 6. Cost per download 7. Acquisition speed 8. Media processing speed 9. Employee productivity in media processing 10. Lending speed 11. Interlibrary loan speed 12. Correct answer fill rate 13. Shelving accuracy

27/21

Cost per use

Definition

The total operating or recurrent expenditure of the library during the reporting year divided by the number of loans + in-house use + downloads from the electronic collection

Method

loans = lending system in-house use = sampling downloads = suppliers and library servers

Question

The indicator assesses the library costs per case of collection use and therewith the cost-efficiency of library services 28/21

The indicators

D. Potentials and development: Is the library fit for future?

Electronic services

1.Percentage of acquisitions expenditure spent on the electronic collection 2. Percentage of library staff providing end developing electronic services

Staff development Budget

3. Attendances at training lessons per staff member 4. Percentage of library means received by special grants or income generation 5. Percentage of institutional means allocated to the library

29/21

Attendances at training lessons per staff member

Definition

The number of of library staff attendance hours of staff members at formal training lessons during one year divided by the total number

Method

- number of attendants at each training - duration of a training A training of 3 hours with 12 attendants would be 36 attendance hours.

Question

What priority does the library give to staff training and therewith to the library’s ability to cope with development?

30/21

Staff training: What has been counted?

Hours of training per staff member Days of training per staff member (hours calculated in days) Percentage of staff members who received training during the year Number of training lessons per staff member Training hours as percentage of total staff working hours Expenditure for staff training 31/21

Measuring Quality: What we tried to do

new indicators indicators for all services „merged“ indicators

find the ultimate set of measures ??

bibliography for each indicator practical examples for each indicator 32/21

It is difficult to catch a black cat in a dark room especially when

it is not there

Chinese Proverb