Transcript uniwersytety
Library performance indicators does it really make sense to measure them?
Data collection and analysis of library performance a case study for Polish research libraries Błażej Feret The Main Libray of theTechnical University of Łódź, Poland [email protected]
Marzena Marcinek Cracow University of Technology Library, Poland [email protected]
CASLIN Seminar, June 2006
Ackowledgements Special thanks go for Mrs Lidia Derfert-Wolf , a member of the Task Group for Standardisation for Polish Research Libraries, for her kind assistance and advice.
2
1.
1.
2.
3.
4.
Plan of the workshop
Introduction 15 minutes A few general remarks on library quality and performance measurement Quality criteria from different perspectives 10 minutes A Common Project of Polish Research Libraries on Comparable Measures ( i n c l . e x a m p l e s) 40 minutes Czech realities 20 minutes Standardised t e r m i n o l o g y and d e s c r i p t i o n s of library performance indicators based on the ISO 11620 Standard Information and
documentation – Library performance indicators
20 minutes Conclusions 15 minutes 3
A few general remarks on library quality and performance indicators
4
Quality: fitness for purpose fitness for use conformity to requirements absence of defects …… Quality of libraries (ISO 11620):
totality of features and characteristics of a product or services that bear on the library's ability to satisfy stated or implied needs
5
Quality of research libraries from the perspective of:
Users Funding bodies Librarians
Please consider all these perspectives
6
Model of measuring quality of libraries
inputs - the raw data such as finance, collection, equipment, users and staff, space, seats outputs - the work done, i.e. circulation, cataloguing, reference services, preservation, interlibrary lending, facilities usage and e-sources searches outcomes – user satisfaction and the impact of library services on users at the local institution and society quantitative data qualitative data from the perspective of: users funding bodies librarians
group work
7
Library quality assessment - what is required?
set of quality criteria objectives librar set of performance indicators y’s goals and national library statistics system standardisation on local, national and international level but ...
8
“ The level of quality cannot be defined once and for all, since both the criteria and evaluation methods, as well as the assessment of the results achieved, may change. This is caused by various factors, for example technological, political, economic, as well as the ones connected with the community in which and for whom information services work. It is crucial that these varying criteria and methods, as well as the dissimilarity in the level of the quality achieved by a given country, are taken into account when aiming at adopting the international standards on the quality of information work. Countries differ in living standards they achieve, in the level of education, and the way they introduce innovations. Setting the
goal is of fundamental importance, since its accomplishment or failure to reach it may be treated as a
basis for quality assessment”.
W. Pindlowa
9
Activities results
and conducted by the Group for Standardisation for Polish Research Libraries,
plans
for the future
CASLIN Seminar, June 2006
The Group for Standardisation for Polish Research Libraries coordinated by the Adam Mickiewicz University in Poznań Lidia Derfert-Wolf (ATR, Bydgoszcz) Ewa Dobrzyńska-Lankosz (AGH, Krakow) Mirosław Górny ( Adam Mickiewicz University, Poznan) Elżbieta Górska (Warsaw Public Library) Marek Górski (Cracow University of Technology) Artur Jazdon ( Adam Mickiewicz University, Poznan ) Dariusz Pawelec (Silesian University, Katowice) Anna Sokołowska-Gogut (Academy of Economics, Krakow) Teresa Wildhardt (Pedagogical University, Krakow) 11
Objectives
to gather libraries' statistical data for a computer database to select a set of performance indicators and standards for library performance (quantity, quality and effectiveness) to conduct a comparative research to prepare and publish yearly reports to define methods for the assessment of Polish research libraries 12
Project for the Analysis of Polish Research Libraries
13
Some facts as of May 2006 • Libraries registered – 59 54 academic libraries (42 state-owned and 11 non state-owned) 3 public libraries 2 special libraries • Questionnaires for 2004 completed 31 libraries • Questionnaires for 2003 completed 29 libraries • Questionnaires for 2002 completed 17 libraries 14
Software for the collection and analysis of data - general requirements on-line access to the questionnaire (submission, modification) selected performance indicators automatically calculated and presented automatic control and verification of the accuracy of submitted data access to analysing functions for individual libraries possibility to conduct a multi-aspect comparative analysis of selected data and performance indicators 15
Web-based Application for: • General users • Library directors • Analysts • Administrators 16
General user
General information about the Project Instructions • Registration • Rules for filling up questionnaires • Data analyses Questionnaires (patterns in doc. format) Example results Useful links 17
http://ssk2.bu.amu.edu.pl/standaryzacja/index.htm
General information Instructions Questionnaires Results
handouts 1
Useful links
18
19
Library Director
Filling up questionnaires; Analysis of the own library data sorted by years, question categories or indicators; Access to calculated automatically indicators Comparison of the own library data with average results of other libraries in the same category or in the country (data published in the form of tables).
Establishment and modification of access rights for the library staff 20
Login 21
For directors of registered libraries
Users Questionnaires Send e-mail Change password Logout Analyses 22
handouts 2 Questionnaire 48 questions of various types refer to easily accessible or computable data (e.g. size of collection, number of users etc.) closed questions about the services offered (e.g. on-line reservation: Yes/No) 88 performance indicators 19 calculated by librarians 69 calculated automatically 23
Questionnaire • addresses all the elements of a library system, its environment, library processes and services • divided into chapters: •Staff •Collection •Budget •Infrastructure •Circulation •Information services •Didactics •Publications and data bases created by the library •Library cooperation •Organisation of library events •Professional activity of library staff 24
Examples of performance indicators required to complete the questionnaire library ’s total expenditures expenditures for library materials/books ratio of library budget to the budget of its parent university time required for the technical processing of a document collection on the computer system as a % of the whole collection of the library percent of catalogue descriptions acquired from outside resources 25
Examples of performance indicators calculated automatically library expenditure per student/user expenditures for library materials/books per student/user library registered users as a percent of potental users space of the library per user collection on the computer system as a % of the whole collection of the library number of user training hours per one staff member 26
For university libraries data concern
main library university library and information system No of library staff
categories education
Survey form
No of libraries in the university library and information system (incl. the main library)
Staff
commentary
age
27
Collection Acquisition
28
Budget 29
Infrastructure Circulation 30
31
Standard lending period Information services Electronic sources usage Didactics Library’s own publications and databases 32
Library’s own publications and databases Interlibrary cooperation, staff professional activities 33
Selected performance indicators 34
Once the questionnaire is validated the indicators are calcullated automaticly Indicators calculated automatically
Admin can add other indicators based on the data gathered no action by director required
35
Adding and modifying users 36
values statistical data Director can analyse performance of his/her own library for selected years 37
categories for the analysis Tyle kategorii ile pytan i wskaznikow wpisanych i automatycznych
Number of categories = number of data and indicators (entered and calculated automatically)
38
Total library space per student No of loans per registered user No of loans per staff membber 39
No of years analysed: 3 Total library space per student No of loans per one registered user No of loans per staff member 40
Analyst •Access to all the questionnaires •No modification rights Analysis Questionnaires 41
Selection of the library categories and years
Type: academic Other types Source of funding
Analyse the years Sort by (years, types) 42
Staff 180 categories for the analysis found, please select Collection 43
Indicators calculated automaticaly Total library space per one user No of loans per one registered user 44
Research libraries performance in 2003 Total library space per user No of loans per one registered user 45
Research libraries performance in 2004 No of loans per one registered user No of loans per staff member 46
Performance indicators
Cost per user in PLN Acquisition cost per user in PLN Library budget as % of institutional budget Registered users as % of potential users Total books per user Books added per user Loans per registered user Loans per library staff member Users per library staff member Total library space per user Users per seat Open access printed books as % of total printed books User services staff as % of total staff Staff with higher LIS education as % of total staff Time of document acquisition and processing in days
University libraries 2002
191,2 45,6 4,81 73,8 21,8 0,2 7,7 2038 341 0,34 116 8,7
2003
182,1 53,8 4,75 70,6 20 0,26 8,9 2447 364 0,26 109 10,7
2004
188,6 52,1 4,23 70,9 19,9 0,27 8,7 2550 367 0,25 113 11,3
University of technology libraries 2002
157,8
2003
171,3
2004
181,1 56,6 56,8 54,9 2,57 72,2 11,4 0,2 7,9 2905 451 0,18 86 11,1 2,59 65,9 12,4 0,18 7,9 2215 432 0,18 95 9,6 2,55 68,1 12,8 0,19 8,4 2421 424 0,18 92 9,5 48,9 34,8 16 54 43,6 37,6 54,2 44 32 57,9 50 14,4 58,6 51,5 16 58,3 52,9 47 17
Administartor’s Module Libraries Users Questionnaires Forms Outstanding questionnaires Send e-mail logout Change password Database statistics 48
Registration form Send via snail mail to….
49
Add / modify library Sortowanie Sort by Source of funding Type Name 50
Select the library 51
List of Libraries with outstanding questionnaires Questionnaires missing not validated validated by director Validated by administrator 52
The analysis of data – examples
(no complete data for 2005 yet)
53
Performance indicators
Cost per user in PLN Acquisition cost per user in PLN Library budget as % of institutional budget Registered users as % of potential users Total books per user Books added per user Loans per registered user Loans per library staff member Users per library staff member Total library space per user Users per seat Open access printed books as % of total printed books User services staff as % of total staff Staff with higher LIS education as % of total staff Time of document acquisition and processing in days
University libraries 2002
191,2 45,6 4,81 73,8 21,8 0,2 7,7 2038 341 0,34 116 8,7
2003
182,1 53,8 4,75 70,6 20 0,26 8,9 2447 364 0,26 109 10,7
2004
188,6 52,1 4,23 70,9 19,9 0,27 8,7 2550 367 0,25 113 11,3
University of technology libraries 2002
157,8
2003
171,3
2004
181,1 56,6 56,8 54,9 2,57 72,2 11,4 0,2 7,9 2905 451 0,18 86 11,1 2,59 65,9 12,4 0,18 7,9 2215 432 0,18 95 9,6 2,55 68,1 12,8 0,19 8,4 2421 424 0,18 92 9,5 48,9 34,8 16 54 43,6 37,6 54,2 44 32 57,9 50 14,4 58,6 51,5 16 58,3 52,9 54 17
12,09 3,49
University Libraries - 2002
11,65 28,05 Collection Staf f expenditures Automation Premises Others expenditure 44,72 5,38 2,73
Technical Univ. Libraries - 2002
12,02 34,45 Collection Staf f expenditures Automation Premises Others expenditure 45,43
All examined academic libraries - 2002
13,54 30,42 7,72 2,85 45,47 Collection Staf f expenditures Automation Premises Others expenditure 8,14 2,49
University Libraries - 2003
12,4 28,41 Collection Staf f expenditures Automation Premises Others expenditure 48,56 6,35 2,37
Technical Univ. Libraries - 2003
11,09 31,63 48,58 Collection Staf f expenditures Automation Premises Others expenditure
All examined academic libraries - 2003
11,73 6,11 2,33 29,15 Collection Staff expenditures Automation Premises Others expenditure 50,7 55
Structure of budget in state university libraries in the years 2002 and 2003
48,56
50 45 40
44,72
35 30
28,05 28,4
25 20 15
10,3 10 10,32 6,96
10
3,49 2,49 3,44 3,27
5 0 zb io ry collection salaries pła ce po m ie sz cz en ia space ko m pu te ry i op ro gr .
hardware &software ad m in is tra cj a administration in ne other 56
Structure of budget in state technical universities libraries in the years 2002 and 2003 50 45 15 10
45,43
40
34,44
35
31,63
30 25 20 5 0 collection zb io ry pła salaries ce
48,58 4,84 5,68 2,73 2,37 2,44 2,65 10,12 9,09
po m ie sz cz en ia space ko m pu te ry i op ro gr .
hardware &software ad m in is tra cj a administration in ne other 57
Selected indicators for state academic libraries No of books per one user [vols]
Liczba vol. książek na 1 użytkownika 15 10 5 35 30 25 20 17,47 31,45 11,44 12,45 21,84 20,03 0
all academic libraries
wszystkie akademickie politechniki
technical universities
uniwersytety
universities
2003 2002
58
Selected indicators for state academic libraries No of new documents (books) per user [vols.]
Liczba vol. nabytków (książki) na 1 użytkownika 0,4 0,3 0,2 0,1 0 1 0,9 0,8 0,7 0,6 0,5 0,25 0,19 wszystkie akademickie 0,2 0,18 0,2 0,26 politechniki uniwersytety 2003 2002
59
Selected indicators for state academic libraries
195,5 Wydatki na 1 użytkownika (w PLN)
Expenditures per user
180,3 wszystkie akademickie
All academic libraries
157,8 171,3 politechniki
Technical universities
191,2 182,1 uniwersytety
universities
0 2003 2002 200
Expenditures for new collection per user
Wydatki na zakup zbiorów na 1 użytkownika (w PLN) 180 160 200 140 180 120 160 100 80 140 120 60 100 40 20 80 60 56,6 57,4 45,6 40 55,7 56,8 53,8 20 0 wszystkie akademickie politechniki uniwersytety
All academic libraries Technical universities Universities 60
Selected indicators for state academic libraries Library budget as percent of parent university budget Budżet biblioteki jako procent budżetu uczelni -
rok 2002 Uniwersytety Politechniki Wszystkie akademickie 4,81 2,57 3,87 rok 2003 Politechniki Uniwersytety 4,75 2,59 Wszystkie akademickie 4,81
61
Selected indicators for state academic libraries Staff with MA degree in librarianship as a percent of the whole library staff Pracownicy z wyższym wykształceniem bibliotekarskim jako odsetek pracowników działalności podstawowej
50 45 40 15 10 5 0 35 30 25 20 30,87 40,16 48,1 49,43 39,96 43,32 2002 2003 Uniwersytety Politechniki Wszystkie akademickie
62
Feedback loop
63
Problems arisen when receiving data • lack of statistical data required to complete the questionnaire • lack of comparable data on the use of electronic resources (incl. differences in usage statistics generated by various providers) • differences in library structure and budgeting within university • difficulties with validation – mistakes (e.g. wrong ratio) need correction, misunderstanding of data requirements, wrong interpretation of questions 64
product ...and solving them • Verivfication of data required • Verification of indicators • Software tools to control data in the fields • Detailed commentaries testing problems noticed by librarians and analysts improvements 65
Selection of performance indicators – the reasons for measurement What? Why? At what cost?
• Part of reporting mechanism (statistics) • Part of internal assessment • Support for decision making (locally, within parent organisation, at the national level) • External requirements 66
Library performance indicators in the ISO 11620:1998 Standard
67
The purpose of library performance indicators • To function as tools to assess the quality and effectiveness of activities and services provided by a library • To assess the efficiency of resources allocated by the library to its activities and services 68
Required features of performance indicators • Informative content • Reliability • Validity • Appropriateness • Practicality • Comparability 69
Informative content
The indicator has to be informative as a tool for measuring activity, for identyfying achievements and problems in the performance of the library so that action can be taken to remedy this. It should provide information for decision-making, e.g. goalsetting, budget allocation, prioritizing services and activities.
70
Reliability
A performance indicator should be reliable in the sense that it consistently produces the same results when used repeatedly under the same circumstances.
71
Validity
A performance indicator must be valid, that is, it must measure what it is intended to measure.
72
Appropriateness
A performance indicator must be appropriate for the purpose to which it is to be put. That is, the units and scale must be suitable and the operations necessary to implement the process of measurement should be compatible with the library’s procedures, physical layout, etc.
73
Practicality
A performance indicator has to be practical in the sense that it uses data that can be made available by the library with a resonable amount of effort in terms of staff time, staff qualifications, operational costs and user’s time and patience.
74
Comparability
A performance indicator allows comparison between libraries if the same score, making allowance for the accuracy of the score, means the same level of quality of services or the same level of efficiency in the libraries to be compared.
75
Some definitions – group work •Performance •Indicator •Evaluation •Accessability •Effectiveness •Efficiency •Reliability •Validity •Loan •Recurrent expenditure •Resources •Target population Handouts 3 and 4 afterwards 76
Performance indicators – a descriptive framework Name Objective Scope Definition Method Interpretation Factors affecting the indicator Related indicators Handout 5 77
Uses of performance indicators The quality and effectiveness of the services of the library as well as the efficiency of the uses of the resources are evaluated against the mission, goal and objectives of the library itself.
Performance indicators should be linked to systematic library planning and evaluation.
Indicators are useful for comparison over time within the same library.
Comparison between libraries is possible but careful interpretation is required.
78
Comparability of performance indicator data Purposes of using library performance indicators: • self-diagnosis (within the same library) e.g. comparisons of one year’s performance with another; •comparison accross different libraries in full recognition of the limitations of such comparisons and with respect for each library’s: •mission, goals and objectives •resources •user groups •governance / funding structure •procedures 79
Limitations It is impossible to achieve optimum scores simultaneously on all performance indicators (eg. user satisfaction vs. expenditure per user) „The scores on performance indicators must be interpreted in the light of what the library intends to accomplish, not simply in terms of optimizing scores on particular indicators.” Degree of accuracy • Sampling errors • Subjective aspects of the measuring process • Inadequate time or resource for measuring process Users skills vs. library performance perception Linking resources to services, management approaches, staff skills etc.
80
What factors would you consider when making decision for registering your library to the library project for comparable measures?
- group work
81
Consider
• Relevancy to to the services being analysed • Strategic purpose of the measurement activity • Staff involved in the measurement process and their awarness of the fact that measurment process is part of their normal flow of work • Accuracy • Reliability • Consistency over time • Practical aspect 82
Success conditions Simplicity versus complexity (goals!) The effort involved in data collection versus the expected outcomes Possibility to integrate data collection with the already carried work Trends are (usually) more important than figures For some measures (e.g. time taken to handle information queries) setting aside certain periods throughout the year and extrapolating findings to a full year Representative samples Do not collect data for the sake of it 83
Thank you!
84