Transcript No Slide Title

Standard Setting
Different names for the same thing
Standard
Passing Score
Cut Score
Cutoff Score
Mastery Level
Bench Mark
Standards exist for as long as there are tests
Canada: 50
China: 60
America: 70
France: 2/3
Examinee-Centered Methods:
Borderline Group Method
Contrasting Groups Method
Problem: How to identify the
examinees.
Test-Centered Methods:
Nedelsky Method (1954)
Angoff Method (1971)
IDEA Method (2004)
Problem: Judgmental errors.
Test-Centered Methods:
Conceptualization of examinee
competency
Judgmental item analysis
Aggregating item difficulty estimates
Training of Judges:
Conceptualization of minimum competency
The use of a standard-setting method
Intrajudge inconsistency
Iterative process
Documentation
What do experts say?
"We have come to realize that there is no
objectively correct way to set standards. But we have
also come to realize that there is nothing wrong with
using judgments appropriately." (Zieky, 1995, p.5)
"Determination of a minimum acceptable
performance always involves some rather arbitrary and
not wholly satisfactory decisions." (Ebel, 1972, p.492)
"Researchers agree with Glass (1978) that
standards are all arbitrary. But they reject the
notion that being arbitrary, in the sense of being
judgmental, is a fatal flaw." (Zieky, 1994, p28)
"If competence is a continuous variable, there
is clearly no point on the continuum that would
separate students into the competent and the
incompetent. A right answer does not exist, except
perhaps in the minds of those providing the
judgments." (Jaeger, 1989, p.492)
1. All standard-setting is judgmental
2. Standard-setting leads to errors of classification
3. Standard-setting is and will remain controversial
4. There is no purely absolute standard.
5. There is no one right method
6. Choosing judges is more important than
choosing methods
7. Standard-setting is a process
Failed Standard-Setting Exercises
Due to Legal Matters
Tenured teachers can not be decertified.
Contracted teachers can not be decertified.
Candidates for becoming teachers can be decertified
Due to Psychometric Matters
Practice analysis failed to support job-relatedness of the test.
Teachers’ concern about the objective of the test was not
addressed.
Items did not reflect judgment of content committee.
Teachers were excluded from the standard-setting processes.
Changing cutscore without justifications.
Defensible Standard Setting Steps
Subject matter experts are asked to review test items.
Sensitivity review to check biases against certain groups.
Documentation of the standard setting processes
Description of subject matter experts
Selection criteria and procedures
Standard-setting methods and justifications
Training procedures
Independent evidence that the cutscore is “reasonable.”
Any indices of reliability, item analysis information,
distractor analysis
Intrajudge and interjudge consistency evidence, e.g., split half
Steps and Procedures in Developing an Achievement Test
1. Define domain content
Intelligence tests and theories,
20%
Personality tests,
20%
Item characteristics,
10%
Reliability,
20%
Validity,
15%
Test development,
15%
Table of specifications
High
Low
Total
Intelligence tests and theories
5%
15%
20%
Personality tests,
5%
15%
20%
Item characteristics,
5%
5%
10%
Reliability,
15%
5%
20%
Validity,
10%
5%
15%
Test development,
5%
10%
15%
2. Write Items
Normally done by subject teachers.
3. Item Analysis
• Item variance
• Item difficulty
• Item discrimination
4. Setting Standards
5. Test reliability
• Test-retest
• Parallel form
• Split half-internal consistency
6. Test Validity
• various validity issues
"Anyone who expects to discover the "real"
passing score by any of these approaches, or any
other approach, is doomed to disappointment, for a
"real" passing score does not exist to be discovered.
All any examining authority that must set passing
scores can hope for, and all any of their examinees
can ask, is that the basis for defining the passing
score be defined clearly, and that the definition be
as rational as possible." (Ebel, 1972, p.496)
"At a minimum, standard-setting procedures
should include a balancing of absolute judgments
and direct attention to passing rates." (Shepard,
1980, p.463)
Uses of Standards
Exhortation
To exhort, encourage or urge the students,
schools, and public to exert more or different kinds
of effort to achieve established standards of
performance.
Exemplification of Goals
To provide clear specifications of the
achievement levels that students are expected to
attain.
Accountability
In the U.S., schools not students can obtain
rewards or sanctions depending on the progress
achieved in meeting performance standards.
"The thing that hath been, it is
that which shall be; and that which is
done is that which shall be done: and
there is no new thing under the sun."
Ecclesiastes (1:9).