Montana University System Writing Assessment
Download
Report
Transcript Montana University System Writing Assessment
Where have we been?
Where do we go from here?
Spring 2012
http://www.mus.edu/writingproficiency/index.asp
Why MUSWA?
College Readiness: A National Concern
The laments about America’s higher education system are long and
loud………….. And remedial education -- the ‘catch-up’ work now
required for the nearly 40 percent of students who come to
college lacking basic skills needed to succeed -- is a prime candidate
for elimination on almost everybody’s list.
Because colleges have not clearly articulated the skills that students
must possess to be college-ready, students are blindsided when they
are placed into remedial courses, and high schools don’t have a clear
benchmark for preparing students for success.
Jane Wellman and Bruce Vandal, Inside Higher Education, 2011
Writing Proficiency Policy
A. Any student seeking full admission to a four-year degree program… must earn a minimum score of:
7 on the Writing Subscore or 18 on the Combined English/Writing section of the Optional Writing
Test of the ACT; or
7 on the Essay or 440 on the Writing Section of the SAT; or
3.5 on the Montana University System Writing Assessment; or
3 on the AP English Language or English Literature Examination.
D. A student who has not yet demonstrated the ability to meet these standards may be admitted
(without condition) to a two-year degree program or admitted provisionally to a four-year degree
program on any campus of the Montana University System.
E. Before gaining full admission status to a four-year program, the student may prove that he/she has the
appropriate proficiency in the following ways:
1) retake one or more of the listed writing assessments to earn the required score; or
2) within 3 semesters, earn a grade of C- or better in the composition course that is the prerequisite to
the composition course that satisfied the general education program requirements described in
Board Policy 301.10.
MUSWA Turned Policy into Practice
Defined College Readiness in Writing
Provided high school students with information about
placement into college-level or developmental courses
Provided feedback to HS staff on curriculum and
instruction for the transition to college
Established a forum for K-12/Higher Ed collaboration
Provided ongoing professional development in writing
Gave students & high schools an avenue for celebrating
student success in writing
Evolving
The Result? Improved Achievement
80
70
60
51.1
50
40
37.8
54.7
52.4
57
57
65.6
71
73.5
75
42.7
30
20
10
0
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
This graph shows percent at or above Proficient level and includes
data on ALL students, not just those with college aspirations.
N = 3,365 in 2001; 7,685 in 2011
2011
The Outcome? Reduced Remediation in
College Composition
Remediation Rates of High School Graduates
as Freshmen in the MUS
16
14
14.6
12
13.6
12.4
10
11.6
10.7
8
10.3
6
8.4
4
2
0
Fall 05
Fall 06
Fall 07
Fall 08
Fall 09
Actual Placements
Fall 10
Fall 11
MUSWA: Sustained Over Time
2001 First Grader
2011 Junior
Holden Pepprock, Shelby High School, Earned a “6” on 2011 MUSWA
The MUSWA Evolution
2001
2011
3,365 students tested
• 7,685 students tested
73 high schools
• 138 high schools
96 scorers
• 320 scorers
3 regional sites
• 8 regional sites
3.0 average score
• 3.9 average score
0.6% earned “6”
• 1.7% earned “6”
37.8% scored college-ready
• 75% scored college-ready
100% of tests handwritten
• 77% of tests word-
processed and submitted
online (2% handwritten)
MUSWA in Three Modes
MUSWA Adapts to Change
From ACT to MUSWA
Expansion & Refinement
2001-2003
Training by ACT
Qualifying Set
2004
Training of Trainers
Calibration Set
Strengths and Weaknesses
College Credits
2005-2006
Online testing
Mixed prompt packets
2007
Eight scoring sites
Over 300 scorers
2008
Consensus Set
Learning > Scoring
2009
The AHA! Essay
2011
Common Core
Hundreds Participate
300 to 370 workshop participants give two days to
MUSWA each year. (Renewal Units, College Credit)
40-50 Trainers give two days to preparation and
two days to workshops each year.
Benefits:
Improved teaching skills and confidence
Useful tools—prompts, rubric, training process
Important role of MUSWA for students, teachers, state
Collaboration among teachers
Personal regeneration from stimulating conversations
How do we score?
Data in Useful Formats
WEBSITE for score retrieval: www.muswa.com with
password:
Distribution tables for state, school, classroom
Student scores (with S & W) by teacher and class
Spreadsheet, by student, with all data
Individual memos, by student, to parents explaining
score
MAILINGS with:
Awards
Transcript labels
Newsletters
Scores honored in
UT, WA, ID, ND & SD
MUSWA Recognizes Excellence
Awards of Merit for Schools in Top
Quartile
Letters of Recognition for students
with scores of 6 and 5.5
Poplar High School Won Awards of
Merit in 2009 and 2010
In 2011, nine American Indian
students received Letters of
Recognition for earning scores of 5.5
or 6.
Ensure accuracy, reliability, validity
Ensure accurate scoring
Does EVERY score match the rubric?
Is EVERY writer scored fairly?
Ensure reliable scoring
Would you give the same score later?
Is everyone giving about the same score?
Ensure that the samples are valid
Were they written in class by the student?
Did the student use a suitable mode?
Question Features of Writing
Does this essay address the prompt?
What is the organizational pattern and is it logical,
coherent, appropriately sequenced?
Is this “a little” elaboration or “some” elaboration?
Is this “precise” word choice, or is it even
appropriate?
How does an essay generated in a testing
environment differ from one produced as an
assignment for a particular classroom teacher?
Scoring Protocol
Refer to the rubric often
Compare to anchor papers
Discuss questionable scores
Weigh strengths and weaknesses
Print and bubble scores carefully and accurately
For online tests, print and bubble test number
Write score at bottom of essay itself
Strength and Weakness Data
Purpose:
Provide feedback to schools and students
Procedure:
Mark a strength or weakness that impacts the score (that
feature keeps the score at “x” or drops/raises the score to
“x”
Mark a strength or weakness to “set aside” a feature that
prevents you from scoring other features fairly
Both scorers may bubble in, but they must not be
contradictory
Take care with score sheets
If student didn’t do it, bubble in prompt number
If printed prompt is wrong, bubble in right number
Bubble in Solution: 1, 2, or 3 (other)
All score sheets must have Reader 1 & 2
Reader 3 (resolver) bubbles in 2 scores
All readers share Comments: Strength and Weakness
Online score sheets MUST show two matching test #’s
After scoring, scorers may check for agreement
Table leader should check for accuracy and
discrepancies, then hold discussions outside—
particularly early in the scoring process
Keep Materials Neat
Keep score sheets with tests until information and
scores are final (use paper clips)
Ensure score sheets are scanner-ready
Completed neatly and correctly
Stacked with same orientation
Stack tests by score for easy research
Collect complete, ordered training materials: ready for
next scoring site
Remember: Each essay represents a student
What strengths does this writer
demonstrate?
How does this essay reflect the rubric at
score X? At score Y?
How would you help this writer improve?
Does this writer demonstrate the capacity to
succeed in a college-level composition
course?
MUSWA Faces the Future
In 2013, all of Montana’s juniors will have the
opportunity to take the ACT Plus Writing, paid for
through new GEAR UP funding.
In 2014, all of Montana’s juniors will be required to
take Smarter Balanced Assessment Consortium
tests.
Can a professional development program modeled
after MUSWA be designed and funded?
Comparing Writing Tests
MUSWA
ACT
Choice of 2 prompts
One prompt
Choice of handwritten,
Handwritten only
word-processed, online
40 minutes
One class period, chosen
by teachers
Scored collaboratively by
Montana teachers
30 minutes
After 4-hour MC test on
a state test date
Scored individually by
paid scorers on
computers
Put the Writing Summit on Your School Calendar!
September 23-25, 2012
Red Lion Colonial Inn, Helena
Co-sponsored by MATELA & Title II
Featuring social event, luncheons, banquet, speakers
such as Carol Jago (CA), Kathleen Blake Yancey, Donna
Miller, Beverly Ann Chin, and others
Presentations from MUSWA trainers and teachers on
writing instruction, writing research, writing
assessment, college readiness, and Common Core.
Participate in a New Writing Program
The MUS Writing Alternative
A limited number of students take the MUSWA, providing student
samples for Writing Assessment Workshops across the state.
Integrating Literacy in Science and Technical Subjects
Schools form teams of English, science, and library media teachers
to learn, through an online class, readings, and/or workshops to
teach students to read and write arguments developed with relevant
data and sources. Teams would convene to score these papers,
much like a science fair, but based entirely on student writing.
Arguing with Statistics
Teachers would take a course to help students select appropriate
statistics to analyze and solve an economic, policy, or social
problem. Their arguments would be scored by groups of teachers.
Can you help us plan?
Will you participate in the Writing Summit?
Would your school participate in another
kind of Writing Assessment/Professional
Development effort?
Would you school take the MUSWA, in
addition to or as an alternative to Plus
Writing?
Contact
Jan Clinard, Ed.D.
The University of Montana Helena
1115 North Roberts, Helena, MT 59601
406-444-0652
[email protected]
http://www.mus.edu/writingproficiency/index.asp
On Facebook: MUS Writing Assessment.com