Transcript Correlations between Longer Student Papers and Higher

Hello from Ohio!
Tom Beery [email protected]
and John Fallon [email protected]
English Faculty
Rhodes State College, Lima, OH USA

Read the prompt and take two minutes to grade the
paper

Holistic scoring: Give the paper one score on a scale
1 (very low) – 8 (very high); no half scores.

When scoring, you may generally consider such traits as:
▪
▪
▪
▪
▪
Organization
Development
Critical thinking
Style
Mechanics
3
In 2005, SAT and ACT add a writing
sample to their tests.
Writing experts say this is no way to test
writing ability.
4
• Are not representative of the writing
“process”: brainstorming, drafting,
revising, editing, and proofreading
• Are not supported by the Conference on
College Composition and Communication
5
“Any individual's writing ability is a sum of a
variety of skills employed in a diversity of
contexts….”
“One piece of writing—even if it is generated
under the most desirable conditions—can never
serve as an indicator of overall writing ability,
particularly for high-stakes decisions.”
6

Adds time to the test

Fosters changes in classroom pedagogy

Doesn’t equate to writing ability (CCCC)

Leads to inaccurate evaluations (e.g., test
procedures)
7
Dr. Les Perelman found a 90% correlation
between paper length and score:
“I have never found a quantifiable
predictor in 25 years of grading that was
anywhere as near as strong as this one is.”
“Write long, badly, and prosper!”
8
1.
High stakes testing is too important to leave
to private testing agencies
2.
Have students write two essays over the
course of a day
3.
Gather graders together for a weekend to
grade (e.g., calibrate graders on the rubric)
9
Is paper length correlated to
paper score?
10
•We needed someone to take the test
Regular and CP high school juniors
from Ohio Hi Point Career Center in
Bellefontaine, OH
•We needed someone to grade the test
20 teachers: 8 college and 12 high
school
11
•Give 12 teachers two hours of rubric training
•Divide into 6 teams – 3 high school and 3
college
•Assign each team one rubric trait
• Discuss proposed revisions
• Rewrite rubric to reflect consensus changes
12
Our Rubric
SAT Rubric
ACT Rubric
Critical thinking
Critical thinking
Organization
Organization
Express judgments by taking a
position on the issue in the
writing prompt
Organize ideas in a logical way
Maintain a focus on the topic
throughout the essay
Development
Development
Develop a position by using
logical reasoning and by
supporting their ideas
Style
Use of language and vocabulary
Mechanics
Sentence structure
Use language clearly and
effectively according to the rules
of standard written English
Scored 1-8; two readers; trait
scores are averaged and totaled
Grammar and usage
Scored holistically 1-6; two
readers; scores are averaged
Scored holistically 1-6; two
readers; scores are combined
13
All 10 teams scored all 42 papers.
Did longer papers receive higher scores?
Yes, longer papers received higher scores.
14
51-159 Words
Average score 15
160-267 Words
Average score 21
268-375 Words
Average score 24
376-477 Words
Average score 26
15
The more the points cluster tightly about the line, the higher the magnitude of
the correlation (between word count and score). We have a tight cluster!
16
Room side A
Room side B
________
________
17
1. If a paper even looks longer, it will receive a
higher grade.
2. Trained readers (6 teams) scored this paper 17.3
3. Untrained readers (4 teams) scored this paper
23.25
4. Rubric-trained teams and rubric untrained
teams scored very differently.
18
Mean
26.4048
24.0238
19.9881
19.3690
Team
High School Untrained
High School Untrained
College Untrained
College Untrained
18.7143
17.4881
16.3452
16.0000
15.2381
15.2262
High School Trained
College Trained
College Trained
High School Trained
High School Trained
College Trained
19
Levels of
Magnitude
.1 small
.3 medium
.5 large
This chart looks at the strength of the relationship
between word count and paper score. A correlation of + 1
is a perfect positive correlation.
20
Rubrics may be used for two entirely separate activities:
evaluation and assessment.
1.Evaluation: Use of a trait-scoring rubric would help eliminate
biases in teacher evaluation (risking one trait being too
prominent in how a reader scores the whole paper, e.g., the
students can’t spell; therefore, they can’t write).
2.Assessment: Using a trait rubric for assessment in the
classroom would promote discussion for what specifically makes
“good” writing before the students actually write.
21
1. Grading with a rubric does not ensure standards.
2. Even just a couple hours of rubric training is
enough to norm evaluators to a standard.
3. Moreover, rubrics can be used for two separate
purposes: evaluation and assessment.
4. Perelman was right. Word count does influence
paper score; however, so does rubric training.
22
Questions?