Transcript Inside the black box: Raising standards through classroom
Not invented here: the baffling insularity of assessment practices in higher education Dylan Wiliam www.dylanwiliam.net
Keynote presentation at the University of London External System’s 150th anniversary Assessment Symposium
Overview: some assessment tensions
Function
Formative versus summative
Quality
Validity versus reliability
Format
Multiple-choice versus constructed response
Scope
Continuous versus one-off
Function Quality Format Scope
A statement of the blindlingly obvious
You can’t work out how good something is until you know what it’s intended to do… Function, then quality
Formative and summative
Descriptions of
Instruments Purposes Functions An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of that evidence.
Gresham’s law and assessment
Usually (incorrectly) stated as “Bad money drives out good” “The essential condition for Gresham's Law to operate is that there must be two (or more) kinds of money which are of equivalent value for some purposes and of different value for others” (Mundell, 1998) The parallel for assessment: Summative drives out formative The most that summative assessment (more properly, assessment designed to serve a summative function) can do is keep out of the way
Function Quality Format Scope
Validity
Traditional definition: a property of assessments
A test is valid to the extent that it assesses what it purports to assess Key properties (content validity) Relevance Representativeness
“Trinitarian” doctrines of validity
Content validity Criterion-related validity Concurrent validity Predictive validity Construct validity
Validity
Validity is a property of inferences, not of assessments “
One validates, not a test, but an
interpretation of data arising from a specified procedure” (Cronbach, 1971; emphasis in original)
The phrase “A valid test” is therefore a category error (like “A happy rock”)
No such thing as a valid (or indeed invalid) assessment No such thing as a biased assessment
Reliability is a pre-requisite for validity
Talking about “reliability and validity” is like talking about “swallows and birds” Validity
includes
reliability
Modern conceptions of validity
“Validity is an integrative evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment ” (Messick, 1989 p. 13)
Validity subsumes all aspects of assessment quality
Reliability Representativeness (content coverage) Relevance Predictiveness
Meanings and consequences
Result interpretation Evidential basis Content validity Consequential basis Value implications Result use Construct validity+ utility Social consequences
Adverse social consequences … are not in themselves indicative of invalidity (Messick, 1989, p. 89) Right concern, wrong concept (Popham, 1997)
Threats to validity
Inadequate reliability Construct-irrelevant variance
Differences in scores are caused, in part, by differences not relevant to the construct of interest The assessment assesses things it shouldn’t The assessment is “too big”
Construct under-representation
Differences in the construct are not reflected in scores The assessment doesn’t assess things it should The assessment is “too small”
With clear construct definition all of these are technical —not value—issues But they interact strongly…
Function Quality Format Scope
Item formats
“No assessment technique has been rubbished quite like multiple choice, unless it be graphology” Wood, 1991, p. 32) Myths about multiple-choice items
They are biased against females They assess only candidates’ ability to spot or guess They test only lower-order skills
Mathematics 2
What can you say about the means of the following two data sets?
Set 1: 10 Set 2: 10 12 12 13 13 15 15 0
A.
B.
C.
The two sets have the same mean.
The two sets have different means.
It depends on whether you choose to count the zero.
Mathematics 3
Which of the shapes below contains a dotted line that is also a diagonal?
Science
The ball sitting on the table is not moving. It is not moving because: A.
B.
C.
D.
E.
no forces are pushing or pulling on the ball. gravity is pulling down, but the table is in the way.
the table pushes up with the same force that gravity pulls down gravity is holding it onto the table. there is a force inside the ball keeping it from rolling off the table
Wilson & Draney, 2004
OU S354: Understanding space & time
Below are five statements about the cosmic background radiation of our Universe. Select two options that are correct, according to the standard model of the Universe.
A.
The microwave radiation collected on Earth is dominated by signals of cosmic origin B.
The total energy of the cosmic background radiation is currently much greater than that of matter C.
In a closed universe, the cosmic background radiation would eventually appear as visible light D.
The temperature of the cosmic background radiation was equal to that of the matter in the Universe until the appearance of galaxies E.
The number of photons in the cosmic background radiation has remained approximately constant since the era of decoupling
English
Where would be the best place to begin a new paragraph?
No rules are carved in stone dictating how long a paragraph should be. However, for argumentative essays, a good rule of thumb is that, if your paragraph is shorter than five or six good, substantial sentences, then you should reexamine it to make sure that you've developed the ideas fully. A Do not look at that rule of thumb, however, as hard and fast. It is simply a general guideline that may not fit some paragraphs. B A paragraph should be long enough to do justice to the main idea of the paragraph. Sometimes a paragraph may be short; sometimes it will be long.
C On the other hand, if your paragraph runs on to a page or longer, you should probably reexamine its coherence to make sure that you are sticking to only one main topic. Perhaps you can find subtopics that merit their own paragraphs. D Think more about the unity, coherence, and development of a paragraph than the basic length.
E If you are worried that a paragraph is too short, then it probably lacks sufficient development. If you are worried that a paragraph is too long, then you may have rambled on to topics other than the one stated in your topic sentence.
English 2
In a piece of persuasive writing, which of these would be the best thesis statement?
A.
B.
C.
D.
E.
F.
G.
H.
The typical TV show has 9 violent incidents There is a lot of violence on TV The amount of violence on TV should be reduced Some programs are more violent than others Violence is included in programs to boost ratings Violence on TV is interesting I don’t like the violence on TV The essay I am going to write is about violence on TV
History
Why are historians concerned with bias when analyzing sources?
A.
B.
C.
D.
E.
F.
People can never be trusted to tell the truth People deliberately leave out important details People are only able to provide meaningful information if they experienced an event firsthand People interpret the same event in different ways, according to their experience People are unaware of the motivations for their actions People get confused about sequences of events
Automated scoring technologies
High-order simulations e-rater m-rater Multiple choice items Low-order structured c-rater evidence structure unstructured
Function Quality Format Scope
Continuous vs. one-off assessment
Continuous assessment
Pros High validity (including reliability) Reduced stress (for some students) Cons Comparability of work done at different times Questions about the accumulation of learning over the programme
One-off assessment
Pros Synoptic Comparability issues minimized Cons Limited validity (especially reliability) Stressful for some students (construct-irrelevant variance)
Reflections
The challenge
To design an assessment system that is:
Distributed So that evidence collection is not undertaken entirely at the end Synoptic So that learning has to accumulate Extensive So that all important aspects are covered (breadth and depth) Manageable So that costs are proportionate to benefits Trusted So that stakeholders have faith in the outcomes
The minimal take aways…
No such thing as a summative assessment No such thing as a reliable test No such thing as a valid test No such thing as a biased test “Validity including reliability”