Transcript 360 Degree Academic Performance Assessment Model

Assessing for Critical Thinking
Narrowing the Gulf Conference
March 2009
St. Petersburg College
Assessing for Critical Thinking
Presenters
 Dr. Carol Weideman, Mathematics Professor,
and Assessment Coordinator for the QEP
 Maggie Tymms, Assessment Coordinator for
Academic Programs
 Dr. James Coraggio, Director of Institutional
Research and Effectiveness
March 27, 2009
Narrowing the Gulf Conference
2
St. Petersburg College
Assessing for Critical Thinking
 SPC, established in 1927, is the oldest
2-year college in Florida
 First Community College in Florida to offer 4
year degrees (2002)
 9 Campuses throughout the county
 2007-08 FTE: 16,086 (LD), 1,415 (UD)
 Opening Fall 2008 credited enrollment: 27,076
 Annual 2007-08 headcount: 62,200
March 27, 2009
Narrowing the Gulf Conference
3
SACS
Assessing for Critical Thinking
 SPC was recently reaffirmed in 2008 as part of
the SACS affirmation process. Keys to the
process included…
 Compliance with all SACS requirements and
standards
 Assessment of all academic programs and support
processes
 Credentialing of all faculty and key staff
 Continuous improvement of student learning through
a Quality Enhancement Plan (QEP)
March 27, 2009
Narrowing the Gulf Conference
4
The Quality Enhancement Plan (QEP)
Assessing for Critical Thinking
 Second major requirement by SACS (after
reaffirmation). Requirements include…






Focus: Significant issue related to student learning
Faculty-driven
Broad-based involvement
Institution demonstrates ability to carry out plan
Includes methods of assessment
Report results to SACS after five years
March 27, 2009
Narrowing the Gulf Conference
5
Broad-based involvement
Assessing for Critical Thinking
 Selected ‘improve student learning
through the enhancement of CT skills’
 First step after deciding on a topic was
the development of a standardized
definition of CT
 Involved the entire college including a
web blog review process to gather
information
March 27, 2009
Narrowing the Gulf Conference
6
St. Petersburg College
Assessing for Critical Thinking
SPC’s definition of Critical Thinking:
“Critical thinking is the active and systematic
process of communication, problemsolving, evaluation, analysis, synthesis,
and reflection, both individually and in
community, to foster understanding,
support sound decision-making, and guide
action.”
March 27, 2009
Narrowing the Gulf Conference
7
Measureable Learning Outcomes
Assessing for Critical Thinking
In order to link specific and measureable
student learning outcomes, SPC’s
definition of critical thinking was
operationalized in order to provide a more
concrete and less abstract linkage or
bridge between the original SPC definition
of critical thinking and the student learning
outcomes.
March 27, 2009
Narrowing the Gulf Conference
8
Measureable Learning Outcomes
Assessing for Critical Thinking
March 27, 2009
Narrowing the Gulf Conference
9
Learning Outcomes
Assessing for Critical Thinking
March 27, 2009
Narrowing the Gulf Conference
10
Learning Outcomes
Assessing for Critical Thinking
March 27, 2009
Narrowing the Gulf Conference
11
Linkage between Student Learning
Outcomes and Assessments
Assessing for Critical Thinking
 SPC’s QEP contained a number of direct
measures and indirect measures for use in
assessing student learning in the area of critical
thinking.
 These assessments have been linked to the
specific Core Operational Elements (KSAs) and
student learning outcomes.
 Provides the relationship between the Core
Operational Elements (KSAs), the student
learning outcomes, and the specific assessment
measures.
March 27, 2009
Narrowing the Gulf Conference
12
Linkage between Student Learning
Outcomes and Assessments
Assessing for Critical Thinking
March 27, 2009
Narrowing the Gulf Conference
13
Linkage between Student Learning
Outcomes and Assessments
Assessing for Critical Thinking
 SPC will determine improvement in
students’ critical thinking skills using the
multiple measures.
 These include standardized instruments,
authentic assessments, and indirect
methods.
 Before we can measure Critical Thinking…
we need to understand what it looks like.
March 27, 2009
Narrowing the Gulf Conference
14
Class Without CT
Assessing for Critical Thinking
 Didactic instruction where students are
presented with factual information from a
text book
 Assessment is primarily multiple choice
items where students are expected to
regurgitate factual information
http://www.youtube.com/watch?v=ITmxpe5wjJo
March 27, 2009
Narrowing the Gulf Conference
15
Class With CT
Assessing for Critical Thinking
 Interactive learning environment where students
not only learn facts but the relationship between
the facts and the application of that information
http://www.youtube.com/watch?v=UKnZc-MJbTE
 Authentic assessment where students are able
to model the applications of the discipline
through simulations, projects, etc.
http://www.youtube.com/watch?v=UrK2mg9h7kE
March 27, 2009
Narrowing the Gulf Conference
16
Linkage between Student Learning
Outcomes and Assessments
Assessing for Critical Thinking
 How do you measure Critical Thinking in
your classroom?
March 27, 2009
Narrowing the Gulf Conference
17
Authentic Assessments
Assessing for Critical Thinking
 Authentic assessments serve dual purposes of
 encouraging students to think critically and
 providing assessment data for measuring improved
student learning.
 These assessment techniques fall into three
general categories:
 criterion-referenced rubrics,
 student reports (reflection or self-assessments), and
 student portfolios.
March 27, 2009
Narrowing the Gulf Conference
18
Assessment Rubric for
Critical Thinking
Assessing for Critical Thinking
 SPC developed a multi-measure model to
assess critical thinking skills, which
includes a critical thinking rubric.
 Assessment Rubric for Critical Thinking
(ARC)
 A global rubric template developed to
provide a snapshot view of how student
learning is being affected by the critical
thinking initiative.
March 27, 2009
Narrowing the Gulf Conference
19
Assessment Rubric for
Critical Thinking
Assessing for Critical Thinking
 Designed to be flexible enough to address
a number of student project modalities
including written and oral communications.
 Will evaluate the student’s use of critical
thinking skills in the development of the
paper as opposed to specifically
evaluating the quality of student’s writing
skills.
March 27, 2009
Narrowing the Gulf Conference
20
Assessment Rubric for
Critical Thinking
Assessing for Critical Thinking
ARC was designed by the QEP staff and
the Faculty Champions to…
 Enhance the QEP
 Align with the College’s definition of critical
thinking
 Be flexible for use in multi-disciplines
March 27, 2009
Narrowing the Gulf Conference
21
Assessment Rubric for
Critical Thinking
Performance
Exemplary
Assessing
for Critical
Thinking Proficient
Element
(4)
(3)
Developing
(2)
Emerging
(1)
Not Present
(0)
I. Communication
Define problem in
your own words.
Identifies the main idea or
problem with numerous
supporting details and
examples which are
organized logically and
coherently.
Identifies the main
idea or problem with
some supporting
details and examples
in an organized
manner.
Identifies the main
idea or problem with
few details or
examples in a
somewhat organized
manner.
Identifies the main idea
or problem poorly with
few or no details or
states the main idea or
problem verbatim from
the text.
Does not
identify the
main idea or
problem.
II. Analysis
Compare & contrast
the available
solutions.
Uses specific inductive or
deductive reasoning to
make inferences
regarding premises;
addresses implications
and consequences;
identifies facts and
relevant information
correctly.
Uses logical
reasoning to make
inferences regarding
solutions; addresses
implications and
consequences;
Identifies facts and
relevant information
correctly.
Uses superficial
reasoning to make
inferences regarding
solutions; Shows
some confusion
regarding facts,
opinions, and
relevant, evidence,
data, or information.
Makes unexplained,
unsupported, or
unreasonable inferences
regarding solutions;
makes multiple errors in
distinguishing fact from
fiction or in selecting
relevant evidence.
Does not
analyze
multiple
solutions.
III. Problem Solving
Select & defend your
final solution.
Thoroughly identifies and
addresses key aspects of
the problem and
insightfully uses facts and
relevant evidence from
analysis to support and
defend potentially valid
solutions.
Identifies and
addresses key aspects
of the problem and
uses facts and
relevant evidence
from analysis to
develop potentially
valid conclusions or
solutions.
Identifies and
addresses some
aspects of the
problem; develops
possible conclusions
or solutions using
some inappropriate
opinions and
irrelevant information
Gulf
Conference
from analysis.
Identifies and addresses
only one aspect of the
problem but develops
untestable hypothesis;
or develops invalid
conclusions or solutions
based on opinion or
irrelevant information.
Does not select
and defend a
solution.
March 27, 2009
Narrowing the
22
Assessment Rubric for
Critical Thinking
Performance
Exemplary
Assessing
for Critical
Thinking
Element
(4)
Proficient
(3)
Developing
(2)
Emerging
(1)
Not Present
(0)
IV. Evaluation
Identify weaknesses in
your final solution.
Insightfully interprets
data or information;
identifies obvious as
well as hidden
assumptions, establishes
credibility of sources on
points other than
authority alone, avoids
fallacies in reasoning;
distinguishes appropriate
arguments from
extraneous elements;
provides sufficient
logical support.
Accurately interprets
data or information;
identifies obvious
assumptions,
establishes
credibility of sources
on points other than
authority alone,
avoids fallacies in
reasoning;
distinguishes
appropriate
arguments from
extraneous elements;
provides sufficient
logical support.
Makes some errors
in data or
information
interpretation; makes
arguments using
weak evidence;
provides superficial
support for
conclusions or
solutions.
Interprets data or
information
incorrectly;
Supports conclusions
or solutions without
evidence or logic;
uses data,
information, or
evidence skewed by
invalid assumptions;
uses poor sources of
information; uses
fallacious
arguments.
Does not evaluate
data, information, or
evidence related to
final solution.
V. Synthesis
Suggest ways to
improve/strengthen your
final solution.
Insightfully relates
concepts and ideas from
multiple sources; uses
new information to
enhance final solution;
recognizes missing
information; correctly
identifies potential
effects of new
information.
Accurately relates
concepts and ideas
from multiple
sources; uses new
information to
enhance final
solution; correctly
identifies potential
effects of new
information.
Narrowing the
Inaccurately or
incompletely relates
concepts and ideas
from multiple
sources; shallow
determination of
effect of new
information on final
solution.
Poorly integrates
information from
more than one
source to support
final solution;
Incorrectly predicts
the effect of new
information on final
solution.
Does not identify
new information for
final solution.
March 27, 2009
Gulf Conference
23
Assessment Rubric for
Critical Thinking
Assessing for Critical Thinking
Performance
Element
Exemplary
(4)
Proficient
(3)
Developing
(2)
Emerging
(1)
Not Present
(0)
VI. Reflection
Reflect on your own
thought process.
“What did you learn
from this process?”
“What would you do
differently next time
to improve?”
Identifies strengths and
weaknesses in own
thinking: recognizes
personal assumptions,
values and perspectives,
compares to others’, and
evaluates them in the
context of alternate
points of view.
Identifies strengths
and weaknesses in
own thinking:
recognizes personal
assumptions, values
and perspectives,
compares to others’,
with some
comparisons of
alternate points of
view.
Identifies some
personal
assumptions, values,
and perspectives;
recognizes some
assumptions, values
and perspectives of
others; shallow
comparisons of
alternate points of
view.
Identifies some
personal
assumptions, values,
and perspectives;
does not consider
alternate points of
view.
Does not reflect
on own thinking.
March 27, 2009
Narrowing the Gulf Conference
24
Sample Scenario (Deer)
Assessing for Critical Thinking
Three teenagers were seriously injured in a car
accident when swerving to avoid a deer on a
two-lane road near a small, rural town in Florida.
The residents of the town have seen more and
more deer enter the town’s populated areas over
recent years. Local law enforcement has been
called numerous times this year to remove the
animals from backyards and neighborhood
streets, and one deer even caused considerable
damage as it entered a restaurant in town. The
mayor has been charged by the city leaders to
keep the town residents safe.
March 27, 2009
Narrowing the Gulf Conference
25
Sample Scenario (Deer)
Assessing for Critical Thinking
Local crops have even been damaged by the
animals. Some long time residents have
requested that the hunting season and catch
limits be extended in order to reduce the deer
population. One city leader even proposed that
the city purchase electronic devices to deter the
deer from entering populated areas. Health
concerns have recently been elevated as three
deer carcasses were found at the edge of town
and local law enforcement suspect that the
animals had been poisoned.
March 27, 2009
Narrowing the Gulf Conference
26
Group Project
Assessing for Critical Thinking
Task 1: Communication
 Define the main idea or problem in your
own words.
March 27, 2009
Narrowing the Gulf Conference
27
Small Group Project
Assessing for Critical Thinking
Task 2: Analysis
 Identify the Three Available Solutions
 Compare and Contrast Available Solutions
to the Problem.
March 27, 2009
Narrowing the Gulf Conference
28
Sample Scenario (Deer)
Assessing for Critical Thinking
Possible Solutions:
1. Some long time residents have requested that the
hunting season and catch limits be extended in
order to reduce the deer population.
2. One city leader even proposed that the city
purchase electronic devices to deter the deer from
entering populated areas.
3. Health concerns have recently been elevated as
three deer carcasses were found at the edge of
town and local law enforcement suspect that the
animals had been poisoned.
March 27, 2009
Narrowing the Gulf Conference
29
Task 3: Problem-Solving
Select the best solution from the possible
solutions analyzed.
Assessing for Critical Thinking
March 27, 2009
Narrowing the Gulf Conference
al
s
0%
A
ni
m
to
de
vi
ce
s
El
ec
tr
on
ic
H
un
ti n
g
se
as
on
an
d
0 of 30
0%
...
c.
..
0%
Po
is
on
1. Hunting season and
catch limits
2. Electronic devices to
deter the deer
3. Poison Animals
30
Group Project
Assessing for Critical Thinking
Task 4: Evaluation
 Critically evaluate the weaknesses to the
solution you chose.
March 27, 2009
Narrowing the Gulf Conference
31
Small Group Project
Assessing for Critical Thinking
Task 5: Synthesis
 Explain ways to improve and strengthen
the solution you chose, and improve any
weaknesses you identified. (You may
introduce new information)
March 27, 2009
Narrowing the Gulf Conference
32
Group Project
Assessing for Critical Thinking
Task 6: Reflection
 What did you learn from this process?
 What were the strengths and weaknesses in
your thinking process?
 What personal assumptions influenced your
thinking during the process?
 What could you do differently to improve your
thinking process?
March 27, 2009
Narrowing the Gulf Conference
33
The Deer Scenario was a Helpful
Critical Thinking Exercise.
Assessing for Critical Thinking
Strongly Agree
Agree
Disagree
Strongly Disagree
March 27, 2009
Narrowing the Gulf Conference
0%
is
ag
re
e
0%
St
ro
ng
ly
D
0%
D
is
ag
re
e
A
St
ro
ng
ly
0 of 30
gr
ee
0%
A
gr
ee
1.
2.
3.
4.
34
Lessons Learned
Assessing for Critical Thinking





Provide rubric to students prior to
assessment
Anonymous papers
Anchor papers defining levels of
proficiency for reference
Use of multiple scorers
Interrater reliability statistics during
training and grading
March 27, 2009
Narrowing the Gulf Conference
35
Other Critical Thinking
Assessments
Assessing for Critical Thinking
 The CAT - Critical Thinking Assessment Test is designed to
assess and promote the improvement of critical thinking and realworld problem solving skills. The instrument is the product of
extensive development, testing, and refinement with a broad range
of institutions, faculty, and students across the country.
 The iSkills™ assessment (former ICT Literacy Assessment),
externally developed by ETS, Inc., is a comprehensive test of
Information and Communication Technology proficiency that uses
scenario-based critical thinking tasks to measure both cognitive and
technical skills.
 The Measure of Academic Proficiency and Progress (MAPP)
test, externally developed by Educational Testing Services, Inc.
(ETS), Inc., is a measure of college-level reading, mathematics,
writing, and critical thinking in the context of the humanities, social
sciences, and natural sciences.
March 27, 2009
Narrowing the Gulf Conference
36
Critical Thinking Assessment Test
Assessing for Critical Thinking
“The CAT instrument is a unique tool designed to
assess and promote the improvement of critical
thinking and real-world problem solving…
The test was designed to be interesting and
engaging for students. Most of the questions
require short answer essay responses and a
detailed scoring guide helps insure good scoring
reliability”
(Tennessee Tech University, Critical Thinking
Assessment Test Overview).
March 27, 2009
Narrowing the Gulf Conference
37
Critical Thinking Assessment Test
Assessing for Critical Thinking
March 27, 2009
Narrowing the Gulf Conference
38
Critical Thinking Assessment Test
Assessing for Critical Thinking
March 27, 2009
Narrowing the Gulf Conference
39
Critical Thinking Assessment Test
Assessing for Critical Thinking
 In collaboration with Tennessee
Technological University and with support
from the National Science Foundation, St.
Petersburg College received a grant to
administer the Critical Thinking
Assessment Test (CAT) instrument to a
representative sample of approximately
100 students enrolled in the College
during 2008.
March 27, 2009
Narrowing the Gulf Conference
40
Critical Thinking Assessment Test
Assessing for Critical Thinking
 SPC administrators attended regional
training workshops at Tennessee
Technological University in 2008 and
2009.
 Eighty-seven CAT assessments were
administered during 2008 to SPC students
enrolled in five courses.
 A CAT Scoring Session was held on July
11th, 2008 to score these assessments.
March 27, 2009
Narrowing the Gulf Conference
41
Critical Thinking Assessment Test
Assessing for Critical Thinking
:
March 27, 2009
Narrowing the Gulf Conference
42
I Will Be Able To Apply The Information I
Learned In This Presentation.
Assessing for Critical Thinking
1.
2.
3.
4.
Strongly Agree
Agree
Disagree
Strongly Disagree
0
of
30
0%
y
gl
n
ro
St
March 27, 2009
e
re
g
A
Narrowing the Gulf Conference
0%
e
re
g
A
0%
Di
e
re
g
sa
y
gl
n
ro
St
0%
D
e
re
g
a
is
43
I Enjoyed This Presentation!
Assessing for Critical Thinking
Strongly Agree
Agree
Disagree
Strongly Disagree
0
of
30
March 27, 2009
Narrowing the Gulf Conference
0%
is
ag
re
e
0%
St
ro
ng
ly
D
ee
0%
D
is
ag
r
St
ro
ng
ly
A
gr
ee
0%
A
gr
ee
1.
2.
3.
4.
44
Assessing for Critical Thinking
Narrowing the Gulf Conference
March 2009