Transcript Qualitative Assessment in Libraries: Adding Meaning to

Qualitative Methods in
Assessing Libraries, Users
and ICT Applications
Dr. R. Sevukan
Associate Professor & Head
Dept. of Library & Information Science
Pondicherry University
Puducherry, India
[email protected]
NACLIN, 9-11 December, 2014
What is an Assessment?

It is the process of gathering information
from the stakeholders using various
methods to systematically gauge the
effectiveness of an institution.
Why is assessment important?

Are we delivering what we think we are
delivering?

Are customers getting what they are
supposed to be getting?

Are customers satisfied with what they are
provided?

Is there a way to deliver the information
better, thereby promoting satisfaction?
Answers to these questions

Information sources needed for research,
teaching, and learning;

Reasons and frequency of library use;

Campus computer network connectivity;

Use of electronic resources;

Instructional needs and effectiveness;

Satisfaction on the services available; and

Enhance the quality of services rendered.
Methods of Assessment
(Source: Niglas, 1999)
Qualitative Vs. Quantitative

Qualitative Research is primarily exploratory research
used to gain an understanding of underlying reasons,
opinions, and motivations. It provides insights into the
problem or helps to develop ideas or hypotheses for
potential quantitative research.

Quantitative Research is used to quantify the problem by
way of generating numerical data or data that can be
transformed into useable statistics. It is used to quantify
attitudes, opinions, behaviors, and other defined
variables – and generalize results from a larger sample
population
Qualitative Methods:
Characteristics

Naturalistic –

Emergent –

Purposeful –
Investigate social processes as they unfold
naturally rather than reasoning from either the conditions
under which they occur or the outcomes that correlate with
them
Willingness to adapt methods of analysis as
knowledge and understanding deepens
Cases are selected because they are
“information rich”
Source: Patton, Michael Q. Qualitative Research and Evaluation Methods. 3rd ed. (Thousand Oaks, CA: Sage, 2002), pp. 4041.
Data Collection Characteristics of
Qualitative Method

Thick description –

Personal experience and engagement –

Empathic stance –

Attention to process –
The goal is deep understanding
about phenomena and experiences.
The
researcher is the primary instrument of data collection and
analysis.
Sensitivity, respect, awareness,
responsiveness, and openness.
Situational and contextual
changes occur throughout the life of the study.
Analytical Characteristics of
Qualitative Method

Unique case orientation – Every case is special and
unique

Inductive and creative – Research process guided by
analytical principles rather than rules

Holistic perspective – Phenomena under study are
complex and cannot be reduced to a few variables

Context sensitivity – Avoid broad generalizations;
investigation does not rest in isolation from the larger social/
cultural/ economic/political environment

Reflexivity – Critical self-awareness of one’s own
presence in the research process
Selective Applications in
Library Settings

To assess individualized outcomes
 To explore internal dynamics of programs and services
 To evaluate critical cases of program or service utilization
 To acquire details about program implementation
 To obtain information about the nuances of program or service
quality
 To personalize the evaluation process and/or outcomes
assessment
 To discover effects of specific programs or services
 To add depth, detail, and meaning to statistical data
 To provide privileged access to processes, causes, and effects.
Source: Patton, Michael Q. How to Use Qualitative Methods in Evaluation. 1st ed. (Thousand Oaks, CA: Sage, 1987), pp. 41-42.
Some Techniques for Collecting
Assessment Data
In-Depth Interviews – speaking with and listening to
our users
Observation – witness what is going on in a particular
setting and explore the experiences of others
Qualitative Case Studies – holistic understanding of
social phenomena in library settings
In-Depth Interviews
Outcomes for Assessment

What is meaningful or important in the respondent’s
own voice?

Researcher’s knowledge, expertise, and interpersonal
skills can be used to explore interesting or unexpected
concepts or issues raised by respondents

Opportunity for deep probing of issues
Source: Patton, Michael Q. Qualitative Research and Evaluation Methods. 3rd ed. (Thousand Oaks, CA:
Sage, 2002)
Observation: Outcomes for
Assessment

Can facilitate multidimensional and long-term
imbeddedness in the “natural setting”

Melds together aspects of looking and listening
and watching and asking (connect what people are
saying to what they are actually doing)

Unmasks everyday interaction between people
and organizations; reveals the mundane

Cases can be revisited repeatedly
Case Studies Design Considerations

Individuals – comparative sample of individual experiences

Groups – sample of individuals who share a common identity

Communities – sample of individuals with ethnic, cultural,
spiritual or other types of characteristics in common

Programs – comparative sample of projects or services
intended to meet a public need

Policies – comparative sample of plans of action

Organizations – comparative sample of a formal group of
people with one or more shared goals
Qualitative Case Studies
Applications for Assessment

Illustrative: Descriptive and intended to add “realism” and in-depth
examples to other information about a program or policy.

Exploratory: Also descriptive, but aimed at generating hypotheses for later
investigation rather than for purposes of illustrating.

Deviant/Critical Instance: Examines a single instance of unique interest
or serves as a critical test of an assertion about a program, problem, or
strategy.

Program Implementation: Investigates operations, often at several sites,
and with a subjective, value laden, and/or affective emphasis.

Program Effects: This application uses the case study to examine causality
and usually involves multi-site, multi-method assessments.

Cumulative: This brings together findings from many case studies to
answer an evaluation question, whether descriptive, normative, or causeand-effect.
Assessment Tools: A Glance
The Challenge of
Assessment in Libraries

Traditional statistics emphasize inputs, expenditures,
acquisitions, holdings, etc.

Helping funding
investment

No demonstrable relationship between expenditures and
service quality

Lack of metrics describing outcomes: success from the
user’s point of view

Need to redesign library services to better meet
changing patterns of use

Building the climate, tools, and skill set for library
assessment
agencies
understand
success
of
No Single Method!!!

“The difficulty lies in trying to find a single model or set
of simple indicators that can be used by different
institutions, and that will compare something across
large groups that is by definition only locally applicable –
i.e., how well a library meets the needs of its institution.
Librarians have either made do with oversimplified
national data or have undertaken customized local
evaluations of effectiveness, but there has not been
devised an effective way to link the two.”
Sarah Pritchard, Library Trends, 1996
Successful Model
Assessment
Tools
Mixed
Methods
Quantitative
Qualitative
Methods
Methods
“Different is not
always better but
better is always
different”
“Let us replicate but not duplicate research”
--- H.K.Kaul