Transcript Demystifying Assessment: Engaging Faculty with Fundamental Assessment Practices for Quality Enhancement SACS Annual Meeting December 4, 2011 Patricia R.

Demystifying Assessment:
Engaging Faculty with Fundamental Assessment Practices for
Quality Enhancement
SACS Annual Meeting
December 4, 2011
Patricia R. Payette, Ph.D.
Executive Director, Ideas to Action
Associate Director, Delphi Center for Teaching and learning
[email protected]
1
Goals of presentation
 Examine the University of Louisville as a case
study regarding QEP assessment practices &
principles (focus: critical thinking)
 Share challenges related to assessment
 Share effective practices and best practices
 Apply relevant concepts from the presentation to
participants’ own contexts
What else you need to know about this session...!
2
“Logic of” Assessment




One word…
Purpose
Key Question or Problem
Point of View
3
Logic of Assessment at
UofL

One word: meaningless

Purpose: to foster accountability &
improvement

Key Question or Problem: how do
we contend with our uneven,
decentralized assessment practices to
report systematically?

Points of View: students, faculty,
staff, SACS, disciplinary norms, best
practices in assessment
4
Words of Wisdom from
UofL
Check for, and reinforce, a shared
conceptual understanding of
• Goals
• Rationale
• Practices
• Protocols
related to assessment.
5
Example of demystifying
assessment: NSSE
http://louisville.edu/nsse/nsseatuofl
http://louisville.edu/institutionalresearch/files/assessments/EveryCardCountsMarketing.pdf/at_download/file
6
University of Louisville
Mission: Kentucky’s premier,
nationally recognized metropolitan
research university
•
•
•
•
•
•
•
•
•
•
Established: 1798 in Louisville, KY
Total Student Headcount: 22,031 (Fall 2009)
Faculty: 2,125 and Staff: 3,961
Operating Budget (2007-2008): $946 million
• Academic Programs (Degrees offered):
– Undergraduate degrees, 78 programs (includes certificates, associate degrees,
baccalaureate degrees, and post-baccalaureate certificates)
– Graduate degrees, 106 programs (includes master’s degree, post-master’s
certificates, doctoral degrees)
– Professional degrees, 3 programs
Source: University of Louisville’s
“Just the Facts,” 2009-2010 (http://www.louisville.edu)
Checking my assumptions
Are you engaged in your developing,
implementing or assessing your QEP ?
Fist to Five:
Zero: working on our topic
One: have our topic, working on proposal
Two: SACS visit is coming up, polishing proposal
Three: Topic approved, early launching stage
Four: Implementing right now
Five: Preparing QEP report or just completed it
8
Conceptualizing the QEP
… is designed to enhance student learning and/or the learning environment;
… is driven by institution’s mission and what is achievable and affordable;
… is determined through broad-based involvement of stakeholders;
… is to be assessed by SACS through a five-year report that includes
achievements, adjustments, and impact data (expected and unexpected)
What Makes a Success QEP in relation to Student Learning Outcomes by Dr. Rudolf Jackson:
www.sacscoc.org/staff/rjackson/Jackson%20Excerpt.Governance%20Seminar.June%202010.pdf
9
SACS acknowledges this reality:
Your agenda will change the institution,
and your institution will change your
agenda.
10
Our QEP: Ideas to Action
Ideas to Action (i2a): Using Critical
Thinking to Foster Student Learning
and Community Engagement
“Our extensive consultation with all university constituencies
yielded a surprisingly strong and clear call for education focused on
the skills and knowledge needed to deal with real-world issues
and problems.” (i2a SACS proposal, p.22)
Proposal: louisville.edu/ideastoaction/resources/archives
11
Focus of this presentation:
critical thinking
Sharpen our
existing focus on
building critical
thinking skills in
the general
education
program…
…..continuing
through
undergraduate
major courses
with an emphasis
on applying and
refining those
skills…
…resulting in a
culminating
undergraduate
experience
(CUE), such as a
senior thesis,
research, service
learning project,
internship, or
capstone project
that reflects
authentic
engagement
12
How are we measuring
critical thinking?
Qualitative
Impact Data
aka “testimonials”
Assessments
CAAP
General Education
Critical thinking rubric
FSSE and NSSE
Note about standardized tests
13
Barbara Walvoord
Purpose: Informed decision making (continuous
improvement)
Key Questions:
• Are students learning what we want them to?
• How can we better help them learn?
Key Concept:
Assessment includes 3 steps:
goals, information, and action
14
Who engages in this cycle? How to
pull people together?
Pull together a powerful assessment
team that reflects assessment interests and expertise
of key stakeholders:
 At the administrative/executive level
 At the QEP/SACS liaison leadership level
 At the faculty governance level
 At the department/unit level
 At the individual faculty/staff/student level
We created an 2a assessment subcommittee with those who can speak across academic15
cultures
i2a Assessment Vision: Systematic, ongoing
process to assess the evidence of undergraduate
students’ ability to think critically and connect
student learning to community for the purpose of
enhancing the quality of the undergraduate
educational experience and documenting
accountability to accreditation agencies.
https://louisville.edu/ideastoaction/what/assessment
16
Goals drive ongoing
decisions
Our QEP Assessment Goals:
 Triangulation of direct & indirect assessments
 Consistent with one critical thinking
framework
 Assessment of outcomes and changes/revisions
 Assessments that are “valued-added” to
existing assessment processes
 Faculty input and participation
17
Words of Wisdom
• Get comfortable with evolutionary change
agenda regarding assessment
• Expect QEP will grow beyond original boundaries and conceptual
frame
• Keep SACS expectations and Five-Year Interim Report frame in
mind
• Don’t wait for all change conditions to be “perfect” to proceed
with assessment
18
Reflect/Apply
Considering your assessment focus right
now, try to describe your assessment
vision (what, why and how) in one
sentence:
19
First step: operationalizing our
QEP concepts
What do we mean when we talk about critical
thinking?
How will will teach for these skills?
How do we measure it across our campus?
20
Richard Paul-Linda Elder Framework
http://www.criticalthinking.org
21
Miniature Guide, 2008, p. 3-6
University-level
assessment buy-in
 We mapped Paul-Elder framework to existing
assessment of critical thinking through our General
Education Curriculum Committee (see handout)
 Considered the fit of standardized tests to our campus
learning priorities
 MAPP, CLA or CAAP critical thinking test? CAAP
 NSSE items related to critical thinking and our QEP
Resource on mapping you can Google:
Stassen, Martha L.A., Anne Herrington, and Laura Henderson. “Defining Critical Thinking in Higher Education:
22
Determining Assessment Fit.” To Improve the Academy. Vol 30. 2011
Getting on the same page: shared
concept of assessment
What are our goals?
What did we learn
related to our goals?
What inferences,
conclusions can we
draw in order to
make informed
decisions
for our next steps?
What specific
measures/instruments
are complementary
to our goals?
Gather our
data using our
instruments.
Analyze my data
23
University-level
Assessment at UofL
Our room
for growth
Its not assessment itself that leads to
improvement, it’s the action taken as a result of
the assessment. - B. Walvoord
24
University-level
Assessment: what we did
• Tie QEP reporting to existing annual reporting
processes for units.
• Work with Office of Academic Planning and
Accountability to decide on software needs.
• Worked with GECC to align our goals
• Transparent/clear in our assessment goals and
resources.
• Created together tutorials on student learning
outcomes (SLOs) to help with uneven skill level:
louisville.edu/institutionalresearch/institutionaleffectiveness/student-learning-outcome.html
25
Student Learning Outcomes Online
Tutorials
26
Words of Wisdom
 Define terms and concepts
 Create outcomes collaboratively
 Create protocol or plan to explain it all
https://louisville.edu/ideastoaction/what/assessment
 Tie into existing reporting methods
 Don’t try to hide your campus reality
 Don’t wait for conditions to be perfect:
Well, what can we realistically find out now from where we
sit? What can we do with that information?
27
Reflect/Apply
• Where do you see the strengths of
your institution’s or colleagues’
engagement with the assessment
cycle?
• Where are the opportunities for
improvement? How might you
leverage those?
28
Assessment at Unit Level
• Use disciplinary norms and
accreditation expectations to align with
QEP goals and outcomes
• Customized workshops and training
• Grant funds to bridge the gap
29
Critical thinking…
30
J.B. Speed School at UofL
and critical thinking
ENGR 100: Intro to Engineering
• Critical thinking is using logic
logic to
todecide
decidewhat
whatto
to
believe
believe
based
based
onon
accurate
accurate
and and
objective
objective
evidence.
evidence.
• Critical thinking is the ability to think clearly
clearly
and rationally.
rationally.
• Critical thinking is the process of
conceptualizing,
conceptualizing, applying,
applying,
analyzing,
analyzing,
synthesizing, and/or evaluating information
information as
as
a guide
a guide
to belief
to belief
andand
action.
action.
Intellectual Standards = blue
Pat
Elements of Thought = red
Workshops/Training
Example
“The goal of the workshops, in addition to
exposing these faculty members to the
Paul Elder critical thinking model, is to
develop a grid showing where each of the
Elements of Thought and Intellectual
Standards will be emphasized within the
College of Business core curriculum.”
(see curriculum mapping handout).
32
Supporting Undergraduate
iNnovation (SUN) Grants
This internal funding program is designed to give
faculty, staff, or teams of faculty and staff grants of up
to $12,500 to develop, implement, and assess projects
that will directly and significantly support the
sustained incorporation of selected i2a
outcomes into undergraduate academic, cocurricular and student support projects, programs,
courses and curricula.
http://louisville.edu/ideastoaction/grants
33
Sample SUN Grant assessment projects
• Individual course project:
“Incorporating Critical Thinking into the Required Business Writing Course
(ENGL306),” Joanna Wolfe and Steve Smith,
Department of English
• Program-wide project:
“Bachelor of Social Work Program Curriculum Development,”
Noell Rowan and Lynetta Mathis,
Kent School of Social Work
• Academic Affairs project:
“Academic Improvement Model (AIM)”: Janet Spence and Nora Scobie;
Undergraduate Advising
34
Words of Wisdom
• Respect disciplinary norms and readiness
• Balance consistency of curriculum
with autonomy for the units
• Sometimes you must focus on quality vs.
quantity: who is “on the bus” is more
important than how many (Jim Collins, “Good to Great”)
Bottom line: one size does not fit all
35
Reflect/Apply
• Who are/could be your supporters and
early adopters? (or: who is your next important
group?)
• What resources do you need to get them
on board and moving forward?
36
Assessment and individual faculty:
meet them where they are.
“I’m already doing critical thinking!”
37
Critical Thinking & Faculty
• Paul, 1996
• 140 randomly sampled California college
faculty
• 89% indicate critical thinking is a primary
objective of their instruction
– 19% could give a clear explanation of
critical thinking
– 77% had difficulty describing how to
balance content coverage with fostering
critical thinking
– 8-9% could articulate how to assess critical
thinking
Focus on: your students,
not SACS
Think of a specific course that you teach, or a specific
learning context in which you teach and/or mentor
students to think critically.
Describe in a short list the changes in students’ mindset (or
“mental models”) you want to see in them at the end of your
time with them in the classroom, lab, etc. (e.g. ask relevant
questions).
39
Example from UofL faculty
Public health faculty want students to be
able to….
 Monitor the quality of their own thinking (e.g. “am I fooling






myself?”)
Discern dominant discourse from alternative discourse
Translate ideas/thoughts into a testable hypotheses
Believe there are no stupid questions, just stupid answers
Critically evaluate a situation from multiple points of view
Postulate an argument and rationally support it
Develop an independent voice (despite grades)
40
Making critical thinking visible:
A Well-Cultivated Critical Thinker
 Raises vital questions and problems, formulating them clearly
and precisely
 Gathers and assesses relevant information, using abstract ideas
to interpret it effectively
 Comes to well-reasoned conclusions and solutions, testing them
against relevant criteria and standards
 Thinks open mindedly within alternative systems of thought,
recognizing and assessing, as needs be, their assumptions,
implications, and practical consequences
 Communicates effectively with others in figuring out solutions
to complex problems
The Miniature Guide to Critical Thinking, 2008, page 2
41
Formative Assessment: Making
critical thinking process visible
The SEE-I prompt
S: State it
E: Elaborate
E: Exemplify
I: Illustrate
42
Gerald Nosich on the SEE-I
“If you can accurately S,E,E,
then I a concept or principle in a
course, it means you almost
certainly have a good grasp of
it, that you understand it to a
much greater degree than if you
are merely able to state it.”
(ASSESSING CLARITY AND ACCURACY)
Nosich, G. “Learning to Think Things Through: A Guide to Critical
Thinking Across the Curriculum.” (2009). p. 35.
43
SEE-I for “civil
disobedience”
• State – A definition is the “active refusal to
obey certain laws & a primary method of
nonviolent resistance.”
• Elaborate – In other words, a means by which
citizens can actively rebel against unfair laws
and the demands of the state
• Example –An example would be like Rosa
Parks refused to give up her seat on the bus
• Illustrate – It’s like being a cliff at the edge of
the ocean, waves crash against it, but the cliff
remains there.
44
When to use a SEE-I
• Use it as a starting point for discussion
• “I’ll make a statement, and I want to someone to
elaborate”
• Use it between students or groups during a
discussion.
• During silence to help understanding
• SEEI can also be used as a catalyst.
• Provide an elaboration of a concept and ask students to
provide examples. Connect it to other concepts and have
them start the process over again.
• “Who has an example of…?”
More info: http://en.wikipedia.org/wiki/SEE-I
45
Words of Wisdom
• Give theory/research and
practical tools
• Help faculty reflect on
their learning goals,
don’t lead with a “deficit • Enlist early adopters who
model”
can provide examples,
testimony across the
curriculum
• Guide faculty to make
an immediate
connection to their
teaching
http://louisville.edu/ideastoaction/resources/faculty-exemplars46
Faculty perspective
“I think that for decades I have given my students many
opportunities to engage in critical thinking, and I have
modeled critical thinking in class discussions. But I
don’t think I can claim ever to have taught critical
thinking in a systematic way. The framework gives me a
way to share a critical thinking vocabulary with students
and to chart their progress. I know and can tell my
students exactly what I am looking for.”
Faculty Learning Community participant,
Department of English
47
Reflect/Apply
What are the insights or ideas
generated by this session you are taking
back to your campus?
What are your next steps?
48
Unanticipated outcomes of QEP
• Partnering to produce i2a research/scholarship
• Supporting projects launched by staff and
students
• Gathering powerful stories of transformation
• Learning from challenges of campus culture
• Serving as regional/national leaders
• Developing unique community
and campus partnerships
49
Focus for QEP Committee:
 Organizing and interpreting qualitative and
quantitative data
 Synthesizing lessons learned
 Providing robust implementation evidence in 10 pages
 Deciding on pointed examples
 Balancing expected “achievements” with unanticipated
“adjustments”
50
Resources on i2a
Ideas to Action:
http://louisville.edu/ideastoaction
Annual Reports:
http://louisville.edu/ideastoaction/grants
QEP Milestones:
Phase VI and beyond:
Assessing, Engaging and
http://louisville.edu/ideastoaction/what/implementation
Reporting
SACS Speak! Videos:
Faculty, Students
and toStaff
http://louisville.edu/ideastoaction/resources/media
Phase VI and beyond:
Assessing, Engaging
and
Evaluation/Assessment
Protocol:
Reporting to SACS
https://louisville.edu/ideastoaction/what/assessment
Learning Communities:
http://louisville.edu/ideastoaction/aboutlc
Phase VI and beyond:
Info: Dr. Cathy Bays, i2a specialist of assessment:
cathy.bays@ louisville.edu
51