Transcript ePortfolio-Based Outcomes Assessment Bellwether

ePortfolio-Based
Outcomes Assessment
Bellwether Presentation:
2012 Community College
Futures Assembly
Paul Arcario, Bret Eynon,
Marisa A. Klages
LaGuardia Community
College/CUNY
Key Questions for Outcomes Assessment
1. What competencies/knowledge
do we want our students to
graduate with?
2. How do we know our students
are graduating with those
competencies?
3. How do we use assessment
data to improve our programs?
Outcomes Assessment
at LaGuardia
1. What competencies/
knowledge do we want
our students to graduate
with?
2. How do we know our
students are graduating
w/ those competencies?
3. How do we use
assessment data to
improve our programs?
Assess student growth over time; use
data to improve pedagogy & programs
Assess student achievement of
Programmatic & General Education
Core Competencies.
Assessment of student work, gathered
with ePortfolio, complements
standardized test scores and IR data
Supports Middle States & Program
accreditation processes
What Should Students Know & Be Able to Do?
• Core Competencies: Gen Ed
– Critical Literacy (Reading,
Writing, Critical Thinking)
– Quantitative Reasoning
– Oral Communication
– Information & Research
Literacy
– Technological Literacy
• Programmatic Competencies
– Defined by programs & majors
– As appropriate, defined by
outside accrediting bodies
An Authentic Assessment Process:
How do we know our students are graduating with
these competencies?
• Faculty review samples of student
coursework, via the ePortfolio
Assessment Database.
• Student work is assessed by faculty,
using faculty-developed rubrics for
each core competency.
• Longitudinal outcomes: faculty
compare freshman & senior work.
• Analysis of outcomes leads to
recommendations for classroom
change & program improvement.
X
X
X
X
Programs Integrate
Core Competencies
into their Curricula
Competency–Focused Assignments
• Critical Literacy
▫ Popular Legal Topics evaluation
(Paralegal)
▫ Fundamental Skill Paper (Occupational
Therapy Assistant)
• Quantitative Literacy
▫ Analyzing an Ecological Footprint
(Philosophy)
▫ Physical Disability Case Study
(Occupational Therapy Assistant)
• Oral Communication
▫ Class critique of photographic
assignments (Photography)
▫ Oral Presentation of Business Report
(Business)
• Research and Information Literacy
▫ Total Quality Management Theories
(Foodservice Administration)
▫ Capstone Project: An Evidence-Based
Intervention (Physical Therapist Assistant)
Using the ePortfolio for Assessment
• Vehicle for collecting
student work: 35,000+
artifacts catalogued
• Delivers samples for Program
Assessment & Benchmark
Assessment Readings
• Connects assessment to the
classroom.
• Provides flexible data: by course,
by program, by competency—
can aggregate in many
configurations.
• Involves students in
reflection & self-assessment.
25,000
A Growing Body of Evidence
21,226
20,000
Number of Artifacts Deposited for Assessment, by Year
15,000
10,000
7,622
5,648
5,000
3,465
0
2007-8
2008-9
2009-10
2010-11
Faculty can track student
submissions, ensuring that
students submit work for
required competencies.
Assessment of Student Work:
PPR and Benchmark Readings
• Periodic Program Reviews (PPR): On a 5-year
cycle, faculty review work from students in
their majors, drawn from the ePortfolio Data
Base. Other artifacts/measures assessed, as
determined by a program.
• Benchmark Assessment Readings: Yearly
reviews provide college-wide measure of how
students are performing on the Gen Ed Core
Competencies.
Faculty form reading
teams around Core
Competencies.
Reading and scoring is
done on-line through an
embedded rubric
The PPR Calendar
• All Periodic Program Reviews will happen on a
3 year calendar within a 7 year cycle.
– Year 1: Preparation for PPR
– Year 2: PPR
– Year 3: Implement recommendations
• Assessment becomes an ongoing cyclical activity. In off
years, programs are collecting student artifacts and
assessing them informally.
PPR Assessment:
Broadening the Base with ePortfolio Data
• 2007-08 Accounting PPR
evaluated 32 student
artifacts, assessing Critical
Literacy.
• 2010-11 Liberal Arts PPR
evaluated 418 artifacts,
assessing Critical Literacy,
Quantitative Literacy, Oral
Communication, Research
& Information Literacy.
2007-8
2008-9
2009-10
2010-11
Collecting & Analyzing Data
2008 Nursing Program PPR
• Critical Literacy core competency
was assessed through student
ePortfolio work.
• A total of 101 student samples
assessed by faculty using the
critical literacy rubric.
• Student written work from the
first course in the nursing clinical
sequence compared to work in
the final course in the sequence
Nursing PPR
Analyzing Critical Literacy Outcome
5
4.9
Writing Samples Scored Against CL Rubric (1-6)
n=101
4.8
4.7
4.6
4.5
4.34
4.4
4.3
4.28
4.2
4.1
4
Baseline Nursing Students Capstone Nursing Students
Closing the Loop
Nursing PPR: Program Improvement
Faculty analysis of authentic student work
spotlighted problems, leading to recommendations
& change in faculty practice:
• Created & implemented syllabus guidelines
requiring students to provide full reference
information and to avoid using overly long
quotations from a text
• Developed, tested ,& refined carefully scaffolded
writing assignments for “SCR 150: Perspectives of
Nursing” & “SCR260: Trends in Nursing”
Assessing Overall Student Progress
Benchmark Assessment Readings:
• Examine artifacts from students w/ 25 credits
or less vs. students w/ 45 credits or more
(Beginning vs. End of student career)
• 1072 samples of student work scored.
• 29 faculty from 13 different areas participated
as evaluators.
• Two readers scored work on a scale of 1-6
yielding a combined score for each student
ranging from 2-12.
Making Progress: Avg. Gain Across Competencies = .87
Where We Want to Be: Outcomes = 10+
12
11
Score Against Rubric
25 or Under Credits vs. 45 or Over
10
9
8
7.44
7
6
7.80
7.58
6.83
6.52
5.68
5.64
5
4.19
4
3
Critical Literacy
Rsch & Info Lit
Oral Com'n
Quant. Literacy
Effect Size
0.7
Gains Small Yet Meaningful
Effect Size (Cohen’s d) by Core Competency
0.6
0.5
Critical Literacy
0.4
Oral Communication
0.3
Quantitative Literacy
0.2
Research & Information
Literacy
0.1
0
Effect Size
Closing the Loop
CTL Mini-Grant Program
Examples of Action Based-on PPRs:
Accounting: used grant to develop new writing
assignments, given that scores on critical literacy
rubric needed improvement.
Computer Science: used grant to revise 7 courses
to address Gen Ed competencies (e.g. revised
“Data Structures” to improve Research & Info Lit
competency, etc.)
Business: used grant to develop oral presentation
assignments and create faculty “intervention” on
how to do more effective presentations.
The LaGuardia Center for Teaching and Learning
Supporting Change
• Mini-Grant program allows programs to
follow up on recommendations in the PPR.
• Core competencies and development of
assignments threaded into CTL seminars.
• Faculty discuss, create & exchange
ePortfolio-specific assignments.
• Capstone Seminar helps faculty create
discipline-based ePortfolio assignments
that showcase integration, culmination &
key learning in Core Competencies.
Lessons Learned
• Make Assessment Authentic
– Build assessment around the day-to-day work of students & faculty
– Basing assessment on classroom work facilitates more effective
recommendations and “closing the loop”
• Allow Faculty Ownership
– Assessment director is a faculty member, additional faculty on the
Assessment Leadership Team
– Frame assessment as an intellectual enterprise- ongoing work of faculty
rather than an external enforcement.
• Commit Time and Resources
– Provide support to make change based on assessment (mini-grants,
seminars)
– Assessment is a learning curve. Give people time to understand assessment.
– Evolution and revision are constant.
• Go Public
– Sharing assessment data is key to creating community around assessment
activities
Questions?