Transcript Help! My Assessment Results Are Not What I Expected -

Why Bother? The Road To A
Quality Information Literacy
Assessment Study
Lyn LaVigne, M.A., M.L.I.S.
Malka Schyndel, M.S.L.S.
Darlene Ann Parrish, Ph.D.
Florida Atlantic University Libraries
2010
Special thanks to
Tony Scott Parrish, M.S. in M.I.S.,
Systems Administrator at FAU,
for his technical assistance.
Introduction
Reference/instruction librarians from Florida
Atlantic University (FAU) wanted to find out if their
library sessions have a lasting impact on student
research. For the first time, an effort was made to
develop a model that could be used as a method of
continuous assessment and improvement.
In partial preparation for a Southern Association of
Colleges and Schools (SACS) re-accreditation visit, a
committee of reference/instruction librarians
developed a 15-item multiple choice questionnaire to
measure what specific library research skills first
semester freshman students enrolled in a Strategies
for Learning Success (SLS 1503) class have prior to
library instruction, and if there is any improvement
after library instruction.
Introduction (con.)
A pretest-post-test study design was used, and
the pretest was administered prior to library
instruction during a 50-minute on site scheduled
library instructional session. The post-test, which
was identical to the pretest, was administered at the
end of the semester in the students’ regularly
scheduled class location. Pretests and post-tests
were administered to more than 300 students in a
total of 22 SLS classes.
Why is library assessment
necessary?
• Accreditation purposes.
• Fits into other parts of organization:
strategic plan, mission/vision, annual
reports.
• Result of financial constraints and
increased accountability.
ASSESSMENT PROCESS FLOW CHART
Start with a strategic plan
containing goals and objectives.
Develop self-studies to measure
goals and objectives.
Analyze results, implement changes,
and revise strategic plan.
Problem Areas in Developing
Information Literacy Studies
• Identification of what needs to be
assessed (e.g. learning outcomes).
• Determination of population and sample
size.
• Time-consuming nature of information
literacy studies.
Problems in Developing
Information Literacy Studies
• Selection, creation and implementation
of appropriate tools/methodologies.
– Which assessment models are most
appropriate?
– Are there models that already exist?
– Do you need to create an in-house
one?
• Interpretation and application of results.
“Best Practices”
• The information literacy assessment study
should be linked to the library’s strategic
plan with goals and objectives.
• It is important to create a time line for any
information literacy study.
• The purpose of the study should be
clearly defined and agreed upon by all
investigators at the beginning.
“Best Practices”
• The principal investigator needs to keep
the study on schedule.
• If a research methodology is created for a
study, it will increase study time, but can
be used for continuous assessment and
improvement.
• Coordinators of library assessment need
to have some formal training in applied
or basic research.
Phases of the SLS Information
Literacy Study
Study Design Phase
• December 2007 – Committee of reference/
instruction librarians began to meet.
• Spring 2008 – Committee began to create
survey instrument for SLS study.
• September 2008 – Received Institutional
Review Board (IRB) approval for SLS
study.
• Fall 2008 - Pretested survey instrument
using students working for the Library.
Data Gathering Phase
• August-October 2009 – Administered
pretests to 21 SLS classes in Library,
before library instruction.
• November-December 2009 – Administered
post-tests to 22 SLS classes during their
regular class time.
Test Scoring and Data Input
Phase
• March 2010 – Completed the scoring of
individual student pretests and posttests.
• March 2010 – Completed the data input in
an Excel spreadsheet.
Data Analysis Phase
• March 2010 – Compared the mean for
•
correct responses for pretests with posttests to see if there was an improvement.
April 2010 – Will create a report with
recommendations for improvement.
Results
• Mean score was 9.46 for pretests and
10.10 for post-tests out of 15 possible
correct responses, which is about a 6%
improvement.
• Maximum number of correct responses for
pretests and post-tests was 14, so there
was no change.
• Minimum number of correct responses for
pretests was 2 and increased to 3 for posttests.
SURVEY QUESTION TOTALS FOR PRE-TEST AND POST-TEST
400
350
Number of correct answers
300
250
200
150
100
50
0
PRETESTS
POST-TEST
Q1
298
291
Q2
335
317
Q3
106
171
Q4
104
102
Q5
329
316
Q6
170
191
Q7
158
183
Q8
205
196
Q9
246
263
Q10
268
271
Q11
252
246
Q12
10
17
Q13
325
306
Q14
138
134
Q15
281
287
What have we learned?
• Individual course instructors could
have positively impacted survey
results:
– Some instructors provided library
instruction prior to the library session.
– Some instructors required knowledge of
library skills in order to complete
assignments.
What have we learned?
• Results showed that the SLS students
who were surveyed did better on the
pretests than expected.
• Results of study may indicate that “one
time” library sessions, if not tied to an
assignment, are less effective.
What have we learned?
• Results lead to the assumption that
instructors/professors whose students
have a “need to know” in order to
complete assignments benefit more from
library instruction.
• Students are more motivated to learn
library skills when a library instructional
session is tied to a meaningful
assignment.
Where do we go from here?
• This study will be used as a baseline for
future assessment studies in information
literacy.
• Results of this study support the Library’s
current direction of embedding librarians
in course instruction.
• Assist course instructors/professors in
their understanding of the importance of
the relationship between information
literacy and its impact on student
learning.
Where do we go from here?
• Compare results of the SLS study with
a current ENC 1102 (Freshmen English)
study that is using the same design
and survey instrument.
• The comparison of the SLS and ENC
studies may help to determine if project
based library instruction is measurably
more effective.
Where do we go from here?
• Recommendations will be made to
continue supporting the SLS classes with
library instruction.
• The emphasis will be on embedding
librarians in course instruction, which
would include the planning of course
assignments.
For More Information