Transcript The Collegiate Learning Assessment

The Collegiate Learning
Assessment (CLA)
Roger Benjamin
CAE
April 15, 2008
Mission: Improve Teaching and
Learning---Higher Order Skills
• Strategy
– Initial institution-based comparison of student
learning outcomes relative to the development
of higher order skills
– Encourage CLA blue print for continuous
improvement
– Launch Institute for Performance Assessment
to advance role of performance assessment
in education
Rationale
• We do not have strong theories of
teaching and learning
• Comparative method useful to provide
investigator with more cases to generalize
from
– Comparative based approach only makes
sense if there is sufficient variation between
colleges and universities on student learning
outcomes
Collegiate Learning Assessment
(CLA)
• The performance tasks are designed to
measure higher order skills, critical
thinking, analytical reasoning, problem
solving and writing initially at institution
level of analysis.
• CLA only measures certain aspects of
undergraduate education.
Overview
•
•
•
•
•
Principles driving the CLA
CLA’s distinguishing features
CLA’s measures
Reporting results
The CLA Blue Print for
Continuous Improvement
• The Case for an Institute of
Performance Assessment
Principles Driving the CLA
1. Cannot assess everything that is
important
2. Can measure some important
abilities
3. Need benchmarks
4. Use results to support improvement
and curricular reforms
5. Use psychometrically sound
1. One measure is not
sufficient
• Colleges have different missions
• Students have different majors
• Multiple important outcomes require
multiple measures
2. Can measure some
important abilities
• Competencies listed in college
mission statements are applicable
to all students
–
–
–
–
Writing – clear, organized, etc.
Critical thinking
Analytic reasoning
Problem solving
• Integrated rather than separate
abilities
3. Need benchmarks to
assess progress
• How much did our students
improve from the time they
entered college?
• Is that degree of improvement
comparable to that obtained by
equally able students at other
colleges?
5. Use psychometrically
sound tests
• Valid – assess important skills
(and not something else)
• Reliable – consistent (not chance)
• Fair – standardized, calibrated,
and unbiased measures
• Cost effective in terms of testing
time and expense
Some of CLA’s Distinguishing
Features
• Rely on open-ended
holistic tests that
are:
– realistic “work samples”
– engaging
– applicable across academic majors
• College (not student) is the unit of
analysis
• Sample students within schools
• No arbitrary standards, such as
“proficient”
• Paperless administration, scoring, &
CLA Is Not Like NCLB
• Constructed rather than selected
response tests
• Use assessment for improvement (low
stakes) rather than accountability (high
stakes)
• Same tests used nationwide rather than
a patchwork of measures and standards
• Matrix sampling of measures (like
NAEP) rather than require that all
students take all measures
• Test sample of students at each school
CLA’s Measures
• Analytic writing (essay) prompts
– Make-an-argument (45 minutes)
– Break-an-argument (30 minutes)
• Performance tasks (90 minutes)
– Document based
– Contextualized questions
– Split screen/dialogue box format
– Analytic and holistic scoring
• Background questionnaire
Make-An-Argument Prompt
“In our time, specialists of all kinds
are highly overrated. We need
more generalists – people who can
provide broad perspectives.”
Directions: 45 Minutes, agree or
disagree and explain the reasons
for your position. Student selects
one of two prompts to answer.
Break-An-Argument Prompt
Students are asked to discuss how well
reasoned they find an argument to be
(rather than simply agreeing or disagreeing
with it).
A respected professional journal with a
readership that includes school principals
published the results of a two-year study on
childhood obesity. This study sampled 50
children, ages 5-11, from Jefferson School. A
fast food restaurant opened near the school just
before the study began. After two years,
students who remained in the sample were
more likely to be overweight—relative to the
Performance Tasks
• 90-minute real life problems
• General directions and context
• Need to combine information from
different types of sources
• A few open-ended questions, no one
“right” answer—must explain rationale
• Split screen
Right: Document Library
Left: Question and answer block
Swiftaire 235
• You advise the president of
DynaTech
• DynaTech makes airplane
navigation and communication
equipment
• DynaTech’s sales manager
suggests buying a Swiftaire 235 to
visit customers and demo its
products
• Recent accident – wing came off in
flight
Document Library
• Newspaper article about recent
accident
• Transcript of interview about
accidents
• DynaTech email exchanges
regarding reasons to buy a
Swiftaire 235
• Trade magazine article that
compares Swiftaire 235’s
performance and safety
characteristics to similar planes
Scoring Rubric
• Writing skills – clear, organized,
persuasive
• Analysis, problem solving, & reasoning
skills
– Integrates information from different
sources
– Recognize flaws and issues—not swayed
by emotional arguments, faulty logic,
irrelevant information, etc.
– Anticipates consequences and implications
of alternative solutions and strategies
– See pros and cons of competing
explanations, points of view, and
arguments
Two Definitions of ValueAdded
• Do seniors score higher than
freshmen with the same SAT
scores?
• Does a college’s seniors score higher
than what would be expected given
(1) their average SAT score and (2)
the typical relationship between
average SAT and CLA scores at
other colleges?
Fig. 3: Relationship Between Mean
ACT Scores and Mean Total CLA
Scores for Freshmen and Seniors
31
27
CLA
Score 23
19
15
15
19
23
ACT Score
27
31
Status of the CLA, 07-08
• 238 four-year institutions testing (roughly
mirrors the Carnegie classification)
• Community college and independent and
suburban high school pilots underway
• Harvard Medical School-based consortium
• Feasibility study for OECD international
version of CLA underway
The CLA Comparison Strategy:
From the Institution to the
Classroom
• There is up to 3.0 standard deviation of
difference between similarly situated institutions
along the regression plot. These large
differences provide faculty and administrators a
signal about where their institution stands
• This means there is a large canvas for studying
best practices in the institutions that do better
than predicted.
The CLA Blue Print for Continuous
Improvement
• Correlate inputs, processes, and outputs
with CLA results
• Drill down to departments and programs to
discover which are strong or weak
contributors to their CLA results
• Conduct audit of existing assessments
• Examine best practices, e. g., more
analytic based writing appears to help
Get Published Performance Tasks
Into The Classroom
• Assist faculty to develop performance tasks that
are based on the scoring guide of the published
tasks
• Choose case studies and problems for text
material that is congruent with the documents in
the performance tasks
• Adopt a student-centered approach to teaching
that calls for more analytic-based writing
The Continuous Improvement
System
• Do comparative value added institutionbased testing with CLA
• Carry out variety of responses, particularly
those focused on the classroom
• Re-test using CLA value added institutionbased testing
• Repeat the cycle
The Evolution of the CLA (1)
• Attract a sufficient number of institutions
using the CLA protocol to corroborate the
hypothesis there is wide variation between
similar institutions in producing CLA
results
• Develop diagnostic logic necessary to
“unpack” CLA results at institution level
The Evolution of the CLA (2)
• Develop resources useful for faculty and
administrators to employ CLA blue print for
improvement,
– Analyze test results to provide more detailed
information about strengths and weaknesses
of students’ performance
– Create CLAintheclassroom for classroom use
– Launch CLA Academy workshops to teach
faculty how to develop performance tasks
– Design on line version of CLAintheclassroom
The Kalamazoo Story
• The differences between academic
divisions---humanities, social sciences,
natural sciences appears to be a function
of writing
– Students who write well and who have had
more experience making judgments abut the
value of information appear to perform better
on the CLA
Sample Cabrini College Faculty
Response
• If a faculty member is going to use this
kind of assessment, they are going to
rethink the way they teach because this
type of assessment asks for students to
pull together various forms of information
and analyze it at a higher level.
The Case for an Institute of
Performance Assessment
• Knowledge economy requires stronger higher
order skills. This development portends
significant changes in teaching and learning,
– Shift from lecture format to student-centered
approach that emphasizes more analytic writing
– Change in the balance of curricular and textbook
focus from emphasis on content to case and problembased approach that requires students to apply what
they know to new situations
– Change from multiple choice tests to open-ended
essay tests better aligned with the other two goals
Rationale for Institute
• Lesson of human capital school---its time
to focus transforming the content of the
stock of knowledge and skills of our
citizens
• Multiple choice tests no longer sufficient
• The field of performance assessment must
be developed
• Comparative advantage of CLA to launch
the Institute
Frequently Asked Questions
1. Are not selective institutions penalized if value
added growth is the metric?
2. What about absolute levels of performance
compared to value added growth?
3. What is the methodological importance of
establishing the base line characteristics of the
students taking the CLA?
4. What about the motivation issue?