Transcript WEB-BASED WRITING ASSESSMENT

Setting Accuplacer Cut Scores
for WritePlacer Plus and
College Math
Becky J. Mussat-Whitlow, M.A., Ph.D.
Director of Institutional Assessment
Winston-Salem State University
Robert Ussery, M.S.
Assistant Vice Chancellor for Academic Affairs
North Carolina A&T State University
18th Annual Accuplacer National Conference
Fort Lauderdale, Florida
June 26, 2008
Introduction
• Historical context


Local bubble sheet test for math placement
ETS’s online Criterion for English placement
• Executive decision to use Accuplacer


For math and English placement
Single vendor
Project Timeline
• November, 2007


Assign project manager
Complete project plan
• December, 2007 – Secure funding
• January, 2008


Provost appoint Placement Committee
Recruit cut score study consultant
• February, 2008


Appoint cut score study panels and chairs
Conduct Round 1 of student testing
• March, 2008



Conduct Cut Score study during Spring Break (March 3 – 7)
Round 2 of student testing
Cut score recommendations to Placement Committee
Creating Defensible Cut Scores
The standard setting process should pay
careful attention to:
1. Selection of panelists
2. Training
3. Aggregation of data into a final set of standards
4. Validation of performance standards
5. Careful documentation of the process.
Reference: Hansche, L.N. (1998). Handbook for the development of
performance standards. Bethesda, MD: US Department of Education,
Council of Chief State School Officers.
Critical Question
At what point along the scale should the
passing mark be set?
Classification Error
Cut scores split a continuous distribution of
knowledge, skills, and abilities into separate
regions.
Need to determine the preference for
classification error.

Do you prefer to pass students who deserved to fail?
OR

Do you prefer to fail students who deserved to pass?
History of Standard Setting
• 2000 BC – Chinese Military Selection
• 1800s - Integration of Psychology and Statistics
• 1970s – Mandated pupil proficiency testing
• 1978 Journal of Educational Measurement
• 1982 ETS Passing Scores publication
“People have been setting cut scores
for thousands of years, but it is only
since the middle of the 20th century that
measurement professionals began to
pay much attention.”
~ From: Cizek, G.J. (2001). Setting
Performance Standards. Hillsdale, NJ:
Lawrence Erlbaum Associates. (page 20)
Overview of Standard Setting

All standard setting methods involve
judgment

Performance standards (cut scores)
may be set too high or low

Need a reasonable process to arrive at
decision
Two Categories of Setting
Performance Standards

Test-based methods – methods in
which panelists are focused on a
review of test content

Student based methods – methods
that focus on students
Guiding Principles to Ensure
Fairness in Establishing Cut Scores

Those who set the standards should be thoroughly
knowledgeable about the content domain that is to be
assessed, the population of examinees who are to
take the assessment, and the uses to which the
results are to be put. (p. 316)

Proficiency classifications should have the same
meaning for all sub-groups. (p. 317)
~ From: Bond, L. (1995). Ensuring fairness in the setting of
performance standards. In Proceedings of Joint Conference
on Standard Setting for Large-Scale Assessments (pp. 311324). Washington, DC: National Assessment Governing
Board and National Center for Education Statistics.
Guiding Principles to Ensure
Fairness in Establishing Cut Scores

If the assessment is to be used as a screen for future
educational opportunities, the content of the
assessment and the level of proficiency required
should be demonstrably related to future success. (p.
317)

Attention must be paid to the consequences of
particular uses of an assessment. (p. 318)
~ From: Bond, L. (1995). Ensuring fairness in the setting of
performance standards. In Proceedings of Joint Conference
on Standard Setting for Large-Scale Assessments (pp. 311324). Washington, DC: National Assessment Governing
Board and National Center for Education Statistics.
Guiding Principles to Ensure Equity in
the Process of Setting Cut Scores





Adequate notice of proposed actions.
Ample provision of opportunities for participation
Adequate records of all discussions and decisions by
the participants.
Timely distribution of minutes and ballot results
Careful attention to minority opinions.
~ From: Collins,B. L. (1995). The consensus process in
standards development. In Proceedings of Joint Conference
on Standard Setting for Large-Scale Assessments (pp. 203220). Washington, DC: National Assessment Governing
Board and National Center for Education Statistics.
Common Problems
Use of ambiguous descriptions of
performance standards
 Failure to involve key stakeholders in
standard setting process
 Failure to devote sufficient time to
establish cut scores
 Failure to document the process
 Failure to validate the process

Questions to Consider
How many cut scores are needed?
Two (1 to differentiate between remedial and regular, 1 to differentiate
between regular and advanced)
VERSUS
One (to differentiate between remedial and regular course
placement)
Desirable Characteristics of
Cut Scores

Should be understandable and useful for all
stakeholders

Clearly differentiate among levels

Grounded in student work

Built by consensus

Focus on learning
Contrasting Group Method

A group of panelists qualified to assess the
content domain and students being assessed
are asked to classify students into two groups
(master vs. non-masters)

Panelists initial make judgments regarding
grouping of students. Then, the performance
of examinees empirically determined.
Contrasting Group Method

Two distributions created to represent the
students’ actual (obtained) scores on the
assessment separately; one for those
students judged to have acceptable skills by
the standard setters and another for those
students whose performances were judged to
be unacceptable.

The point at which the two distributions
intersect may be chosen as the cut score
location.
Hypothetical Illustration of
Contrasting Groups Distributions
Masters
Distribution
ƒ
Nonmasters
Distribution
Cχ
Score Scale
Modified Contrasting
Group Method



Students were classified into three broad
performance categories of average, below
average, or above average and administered
the placement tests.
Score distributions were plotted to represent
the students’ actual (obtained) scores on the
assessment separately
The distribution plot was visually analyzed to
identify an appropriate cut score.
Modified Contrasting
Group Method



Panelists completed the placement test role
playing as a student with average and above
average ability
Score distributions were plotted to represent
the panelists’ actual (obtained) scores
The distribution plot was visually analyzed to
help guide the establishment of an
appropriate cut score.
Multi-Step Approach
to Establish Cut Scores



Panelists created performance level descriptions
(PLDs) and mapped these PLDs to College Board
proficiency level statements
Panelists reviewed student score distributions
Panelists completed placement test and reviewed
their score distributions.
Additionally, for English, panelists retrospectively categorized
papers into 2 groups (remediation required vs. no remediation
required) and used this information to guide cut score
establishment.
Three Day Panel Sessions

Panels met for 3 hour sessions on 3
consecutive days.

Math Panel: 12 faculty

English Panel: 12 faculty
Panel Session I

Panel Session 1:
 Identified the number of cut scores
needed and the different courses into
which students would be placed
 Developed clear and concise
performance level descriptions for
each placement level
Characteristics of Performance
Level Descriptions

Written in positive terms

Parallel in organization, language,
and style

Written in clear and concise
language without using
unmeasurable qualifiers (e.g., often,
seldom, etc.)
Panel Session II

Panel Session 2:
 PLDs Approved
 Initial round of cut score setting
based upon the results of student
testing and faculty role play
performance
 For English, student work categorized
by panelists into 2 groups.
Illustration of Score Distributions
Illustration of Score Distributions
1st Cut Score = 30
Students scoring
less than 30 –
Remedial
Placement
2nd Cut Score = 42
Students scoring
from 30 to 41 – Math
101
3rd Cut Score = 65
Students scoring from
42 to 64 – Math 110 or
111
Students scoring 65 or
higher – Math 131
English Panelist Rating Information
Remedial Indicated by Two
or Fewer Panelists
Remedial Indicated by Three
or More Panelists
Mean
SD
7.85
1.37
6.90
1.07
English Panelist Rating Information
Remedial Indicated by None
or Some Panelists
Remedial Indicated by All
Panelists
Mean
SD
7.61
1.28
5.0
1.01
Panel Session III

Panel Session 3:

Cut Scores were aligned across
various methods used.
PLDs Mapped To
Proficiency Statements

MATH 101
 PLD: Perform addition, subtraction,
multiplication and division of polynomials;
identify factors, terms, constants, variables,
exponents, and coefficients; and recognize
graphs of polynomial functions.
 Lower than ACCUPLACER 40
 Cut Score 30
Alignment of Recommendations



Performance level descriptions written by
NC A & T faulty were mapped to College
Board proficiency level statements
Review score distributions for student
performance groups to determine cut score
Cut scores indicated on the basis of the
student data were considered in conjunction
with performance level descriptions.
Example of Alignment Process
Based on mapping
to Accuplacer
Proficiency
Statements. Cut
score should fall
between 40 & 63.
Math 110 or 111
Follow-Up Testing and Cut-Score
Refinement

Additional Students Tested

Cut Scores Revised

Recommendations Made By Panelists
Project Timeline after Study





March, 2008
 Committee review and recommendations
 Executive review and policy
April, 2008
 Develop student score validation and upload procedure
May, 2008
 Train proctors and faculty advisors
 Final system test
June
 Go live with Accuplacer placement testing
 500 students per week
July, 2008
 Process evaluation and recommendations
Campus Collaboration






Admissions – New students
Student Affairs – New student orientation
Registrar – Validate student data
IT – Upload student data to campus system
Academic Affairs - Proctors and advisors
Institutional Research – Management and
support
Accuplacer Support

Helpline

Online procedure guides
Early Results
Five (5) orientation sessions in June
 Overall, the system works
 Some Accuplacer interface weaknesses

WHERE TO FROM HERE

Develop alternative placement
measures

Implement off-site testing
Thank you for attending!
Questions?
NCA&T Support Site
http://qed.ncat.edu/ir&p/availabl.htm