Transcript Document

How to Conduct
Effective
Assessment
When Nobody
Wants to
They Can’t
Agree What to
But We Have to
The history of a large community
college English Department’s
struggle to implement a district
wide assessment program.
Presented by: Beth Baker-Brodersen, Bret Ross, and Chelli Gentry
DMACC
DMACC
Founded March
1966
Student
Enrollment
25,000
Offer 168 career programs,
certifications, and technical
programs as well as liberal
arts transfer degrees
6 campuses
Ankeny
Boone
Carroll
Newton
Urban
West
Student
Demographics
55% Female
45% Male
Average age is 23
40% Full-time
60% Part-time
Purpose
• Accountability
– HLC
– AQIP
• Improve
– Student learning
– Teaching
Assessment at DMACC
• Shared process of purposeful, systematic measurement
used to document, reflect upon and improve student
learning.
• Engaging in assessment activities involves four
processes that result in improved student learning.
– Collaboration with peers to identify clear, valid and
appropriate course and program competencies.
– Systematic collection of evidence that the identified
competencies are addressed.
– Thoughtful dialog to arrive at a collective interpretation of
the data.
– Agreement to use data to improve both teaching and
learning.
Assessment Process
Curriculum
• Develop
• Modify
Refine
Competencies
• Based on
data
• Learning
Outcomes
Collect
• Discuss
• Analyze data
Measure
• Create
Course Level Assessment
• Represents the largest part of DMACC’s
comprehensive assessment program
• Faculty work collaboratively to develop an
assessment tool to assess the course
competencies for a target course
• The assessment enables faculty to identify
trends in student learning to analyze, monitor
and enhance the course curriculum
Continued
• Course assessment works well for liberal arts
courses for transfer degrees
• Appropriate methods of course level
assessment include: testing, portfolios,
projects evaluated by rubrics, or any other
systematic method of collecting data
• Use same method of assessment across the
district
DMACC ENG Assessment:
A Brief History, 2002-2009
Beth Baker-Brodersen
English professor
District Chair of Communications
DMACC West Campus
A Delicate Endeavor
“Comp. I Assessment Analysis: Fall 2006 through Spring 2009”
Many of our composition faculty feel alienated
or are just plain disinterested in the subject of
writing assessment. They especially feel
disconnected by an assessment that happens
outside of the classroom for purposes of
program evaluation. This distrust is
understandable considering the teaching load
most composition faculty are faced with every
semester and the relative isolation created by a
multi-campus institution.
Guiding Principles of Assessment
Conference on College Composition and Communication,
“Writing Assessment: A Position Statement”
• Designed/evaluated by well-informed faculty
for clearly understood purposes
• Elicit a variety of pieces, preferably over time
• Encourage and reinforce good teaching
practices
• Grounded in current and relevant research on
language learning and best practices
2002-2009
• 2002/2003: developed and implemented a
timed-writing based on reading prompt, with
opportunity for revision later in the semester
• 2004: Ed White, recognized expert in writing
assessment, visits DMACC and suggests
portfolio approach
• 2005-2009: Portfolio assessment developed,
refined, and implemented; Ed White visits
again
DMACC’s English 105 Assessment Project:
The Nuts and Bolts (Mostly Nuts)
Presenter: Bret Ross
Chair, ENG 105 Assessment
Committee
My Qualifications
• DMACC English faculty member
• Member of the first project’s assessment
committee
• Baseball scorer
Section 2 of ENG 105 (Comp 1)
Competencies
• 2. Practice reading as an active part of the writing
process.
– 2.1 Use effective reading techniques such a rereading,
annotating, and summarizing.
– 2.2 Differentiate between main and supporting ideas.
– 2.3 Distinguish between objective and subjective material.
– 2.4 Understand connotation and denotation.
– 2.5 Demonstrate sensitivity to discriminatory language.
– 2.6 Analyze the content, expression, and context of the
writing.
– 2.7 Adapt material for specific writing purposes.
The Former ENG 105 Assessment
• Borrowed ideas from the old ENG 105
Portfolio Assessment
–Low stakes
–Rubric design
–Universality
–Holistically scored writing
–Data Collection
The Resurrection—Combining Old and
New
• New ideas
– Assess a single competency
• 2.6 Analyze the content, expression, and context of
verbal or visual text. (revised)
– Key Idea: No single essay prompt
• Each ENG 105 Instructor should choose a text for student analysis
• Assessment as Experiment—Hypothesis:
‘Tradeoff sample stability for increased
instructor involvement’
Results
• Two pilot scoring sessions conducted by Boone
DMACC ENG faculty
• Boone project given go ahead in February, 2013
• ENG 105 Assessment Committee formed in
Spring 2013
– Developed student essay constraints
– Developed phased-in faculty involvement schedule
– Revised scoring rubric
Results: Scoring Rubric
0
1
2
3
Inappropriate thesis;
paper’s focus is not on
critical reading; paper
reads as a character
description, narrative,
short story, etc.
No clear analytical
thesis statement
May have an analytical
thesis statement, but
material does not
support it; the paper’s
focus is unclear
Has an analytical thesis Has an analytical thesis Has an insightful
statement
statement; may also
analytical thesis
forecast techniques to statement
be explored in paper
No use of analytical
terminology to
describe or interpret
the text
May include some
analytical terms, but
terms are misused to
describe text
May understand
analytical terms, but
does not cite textual
evidence
of their use
Understands analytical
terms, but description
of text is haphazard
or weak
Understands analytical
terms and describes
text generally well;
adequate use of textual
evidence
Has command of
analytical language to
describe text; uses
interesting textual
evidence
No interpretation and
does not include any
reference to a text at
all
Little or no
interpretation:
Material is dominated
by summary of the text
or personal taste asides
Unclear interpretation: Interpretation of text is
may try to analyze, but apparent but not
ends up with broad
developed adequately
summary of points or
personal tastes
Interpretation of text is
generally thorough, but
may have isolated
logical gaps in
development
Interpretation of text is
thorough and
consistent; may exceed
assignment
expectations
Exhibits no written
Has little consideration
consideration of text’s of the text’s audience,
audience, purpose, or purpose, or context
context
Has little consideration
of the text’s audience,
purpose, or context
Exhibits some
consideration of the
text’s audience,
purpose, or context
Adequately considers
the text’s audience,
purpose, or context
Weaves audience and
contextual
considerations into
analysis seamlessly
Material lacks
sufficient
development: paper
appears to be short
answer exercises, or
may be only a
paragraph or two in
length
Material may have
initial analytical focus,
but deviates; lacks
clear organization and
expression
Material is focused on
analysis, but occasional
weak expression and
organizational
problems occur
Analysis is well
focused, organized,
and expressed, but
isolated problems exist
Superficial errors may
exist, but analysis
shows strong
composition skills
overall
Material lacks
analytical focus,
organization, and clear
expression
4
5
Results: Essay Scoring Excerpt (from Fall 2014)
Folder
12 *(out of 15)
Reader
I
Reader
II
ADJ/AV
Score
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
1
1
0
2
0
2
1
2
3
1
3
3
2
0
1
1
0
1
1
1
0
3
0
3
1
5
4
3
5
3
4
4
3
3
1
2
1
4
2
3
1
1
2
2.5
DQ
5
3
2.5
3
1
3.5
3.5
2.5
1
1
1.5
1
2
1.5
1
Key:
Regular=Averaged
Bold=Adjusted
Green=Adjudicate
DQ=Disqualified
B.E.G.I.N.= Coded
Essays (Concurrent
Student Essays)
20 Essays/15 Folders=
300 Student Essays
Results: Fall 2013 (FT only) & Spring 2014 (FT +ADJ)
2-Reader Scoring Differentials
ENG 105 Assessment, Fall 2013
2-Reader Scoring Differentials
ENG 105 Assessment, Spring 2014
3+ pt.
12%
3+ pt.
7%
2 pt.
20%
Identical
29%
1 pt.
44%
Identical
35%
2 pt.
19%
1 pt.
34%
Results: Fall 2013 & Spring 2014
Single Reader Scores, Spring 2014
(Out of 540 total reads)
143
178
* 22 DQ Essays
* 68 DQ Essays
117
102
133
129
85
95
63
55
30
28
Score 5
Score 4
Avg. Score: 2.24
Score 3
Score 2
Score 1
Score 0
Passing Scores (>=3) 77 (27.6%)
Score 5
Score 4
Score 3
Score 2
Score 1
Score 0
Avg. Score: 2.27 Passing Scores (>=3) 64 (31.8%)
*Results—Fall 2014 (All ENG 105 Faculty)
Single Reader Scores (600 reads)
Two-Reader Scoring Differentials
2/20/15
162
3
7%
134
Same
29%
2
20%
123
102
52
26
1
44%
*Essays screened for DQs before scoring
5
4
AVG. Score: 2.45
3
2
Scores 0-5
1
0
Passing Essays 110 (37%)
Deliverables
• Process Benefits
–
–
–
–
Steady improvement in faculty buy-in
Increased communication between faculty segments
Spurred development of other assessment projects
Created “Assessment Community” projects
• Product Benefits
– Fulfillment of AQUIP requirements
– Results of Assessment Communities (pending)
– Improvement of student reading skills (?)
Conclusions and Outlook
• At this point process benefits trump product
benefits
• In order to improve the product we need to
expand the process
– We need a forum to present and discuss ENG 105
assessment results
– From this discussion, we need to focus on best
practices toward developing students’ reading skills
We need a specific and recurring “ENG Faculty
Development Day”
Wrap Up
• Faculty consensus is not easy
– Do something
– Get as much buy-in as possible
– Find champions
• Cooperation collaboration are a necessary
part of the process
• Produce results
• Remain patient
• Create change
Questions
Appendix
• Works Consulted for the ENG 105 Assessment
Project 2011-Present
• Email copy sent to all DMACC ENG faculty, Fall
2014.
• Example of a ‘Default Essay Assignment’ for
the ENG 105 Assessment