Transcript Two Categories of Test Accommodations for English Language

C RE SS T/U C LA
Model-Based Assessment:
Why, What, How, How Good,
and What Next?
Eva L. Baker
UCLA Graduate School of Education & Information Studies
Center for the Study of Evaluation
National Center for Research on Evaluation, Standards, and Student Testing
National Research Council, Board on Testing and Assessment
Bridging the Gap Between Classroom and Large-Scale Assessment Workshop
January 23-24, 2003
Washington, DC
C RE SS T/U C LA
Why? Assessment
Knowledge: Usable and Useful
Usable Knowledge
 In a form that can be understood
 In a form that can be applied
 Timed appropriately
 May cause rethinking of the problem
Useful Knowledge
 Rethinking indicates a new solution path
 Adapted to situation
 Sufficient to guide solution
 Improved outcomes occur as a result
C RE SS T/U C LA
Why Are Some Schools
Successful in Using
Assessment Knowledge?
 Focus on learning (students and adults)
 Constant use of appropriate information
(formal and informal)
 Focus on feedback and change
 Public display and exchange
 Community pride in outcomes of students
and place
C RE SS T/U C LA
Goals for CRESST Model-Based
Assessment (MBA)
 Assessment components share a common
framework. MBA starts with thinking skills
and applies them to content domains to
support
 Coherent, sustained learning
 Spiral teaching-common language
 Transfer (application to new situations)
 Multipurpose
 Learning organization
C RE SS T/U C LA
CRESST Model-Based
Assessments (MBA)
Features
 Research based
 Focus on cognition and learning
 Abstracted in models based on key learning
elements—principles guiding test design and
instruction
 Operationalized in templates
 Reusable and cost-sensitive design/training/scoring
C RE SS T/U C LA
Model-Based Assessment
Cognitive Families
Content
Understanding
Teamwork and
Collaboration
Communication
Learning
Problem
Solving
Metacognition
C RE SS T/U C LA
Model-Based Assessment
Design
 Models to templates (to specification) to tests
 Template contains domain-independent
(transfer) and domain-specific (strategy and
knowledge) components
 Templates that allow common domain-specific
design approaches to be used, e.g., primary
sources in history
 Scoring requirements
C RE SS T/U C LA
Expert Model—Deep
Understanding of Content
(Domain Independent)
 Principles or themes (big ideas)
 Use of prior knowledge
 Explicit relationships
 Avoiding misconceptions
 Expert performance-based scoring
C RE SS T/U C LA
Specifications for
Large-Scale Use
 Standards reference
 Place in sequence
 Content domain (what’s in and out)
 Proportion of effort
 Format options
 Interpretation rules
 Time
C RE SS T/U C LA
Template
 Task(s)
 Format(s)
 Prompt(s) and requirements
 Scoring
 Directions
 Sample
C RE SS T/U C LA
Three Templates for the
Model of Deep Content
Understanding
 Explanation
 Explanation with explicit knowledge
 Graphical representation of
relationships
C RE SS T/U C LA
Deep Content
Understanding
 Primary source materials in each
domain
 Student required to integrate prior
knowledge and principles to succeed
 Scored by using expert model in
subject matter
C RE SS T/U C LA
Content Understanding
Template #1
Explanation
 An array of primary source materials
 A prompt that asks for an explanation
in context
 Constructed (written) answer
 Evaluated by means of a scoring rubric
that operationalizes learning model
C RE SS T/U C LA
Hawaiian History Writing
Assignment: Bayonet
Constitution
Imagine you are in a class that has been studying Hawaiian history. One of
your friends, who is a new student in the class, has missed all the classes.
Recently, your class began studying the Bayonet Constitution. Your friend is
very interested in this topic and asks you to explain everything that you have
learned about it.
Write an essay explaining the most important ideas you want your friend to
understand. Include what you have already learned in class about Hawaiian
history, and what you have learned from the texts you have just read. While
you write, think about what Thurston and Liliuokalani said about the Bayonet
Constitution, and what is shown in the other materials.
Your essay should be based on two major sources:
1. The general concepts and specific facts you know about Hawaiian history,
and especially what you know about the period of the Bayonet Constitution.
2. What you have learned from the readings yesterday.
Be sure to show the relationships among your ideas and facts.
C RE SS T/U C LA
EXCERPTS from HAWAIIAN HISTORY
PRIMARY SOURCE DOCUMENTS
LILIUOKALANI
For many years our sovereigns had welcomed the advice of American
residents who had established industries on the Islands. As they became
wealthy, their greed and their love of power increased. Although settled
among us, and drawing their wealth from resources, they were alien to us
in their customs and ideas, and desired above all things to secure their own
personal benefit.
Kalakaua valued the commercial and industrial prosperity of his kingdom
highly. He sought honestly to secure it for every class of people, alien or
native. Kalakaua’s highest desire was to be a true sovereign, the chief
servant of a happy, prosperous, and progressive people.
And now, without any provocation on the part of the king, having matured
their plans in secret, the men of foreign birth rose one day en masse, called
a public meeting, and forced the king to sign a constitution of their own
preparation, a document which deprived [him] of all power and practically
took away the franchise from the Hawaiian race.
C RE SS T/U C LA
Explanation
Scoring Rubric
 General impression of content quality
 Principles or concepts
 Prior knowledge
 Examples
 Misconceptions
 Argumentation
C RE SS T/U C LA
Template #2
Prior Knowledge and
Explanation
 Explicit measurement of knowledge domain
before explanation
 Uses short answer or selected response
 Helps interprets explanation performance
C RE SS T/U C LA
Template #3
Knowledge Representation
 Same prompts
 Key aspects of ideas, supporting facts
and views and their relationships
 Relationship is explicit
 Organizational options
 Core and peripheral
 Hierarchical
 Cause-and-effect
 Chronological
 Expert scoring
C RE SS T/U C LA
History
C RE SS T/U C LA
Genetics
C RE SS T/U C LA
Bicycle Pump
C RE SS T/U C LA
Brief History of MBA in LAUSD
 Content understanding and problem-
solving models
 Explanation templates
 4 subjects, 3 grade levels, 2 languages
 Purposes: (1) to clarify expectations;
(2) to provided instructionally
embedded assessment; (3) to get a
measure of school performance
 CRESST-managed teacher involvement
C RE SS T/U C LA
LAUSD Process
 Teacher design teams
 LAUSD standards first
 Adapted to success standards
 Training cadre of scorers
 Training trainers
 Supervising scoring
C RE SS T/U C LA
LAUSD Process (cont’d)
 Shift in four-topic focus (capacity based)
to two and then to one, now back to two
 Continual assaults
 Curriculum mandates
 Accountability pressure (API)
 Long-term embedded approach
resurfacing
C RE SS T/U C LA
Present LA Situation
 Administered in 2, 3, 4, 5, 6, 7, 8, 9
 Purpose added regarding promotion
 Teacher scored with an audit reported to
school
 Local sub-districts managing activity
 Ongoing validity studies
 District review of alternative assessments
C RE SS T/U C LA
C RE SS T/U C LA
CRESST Validation Studies
 Score reliability
 Task and rater generalizability
 Stability of student performance over time
 Relationships among measures
 Instructional sensitivity
 Opportunity to Learn (OTL)
 Effect of school composition on performance
 Cut-score modeling
C RE SS T/U C LA
Evidence of Predictive Validity
% of Students in Different Categories
of Performance in CA Standards Test
LAUSD Grade 7 Students’ Achievement Levels: Comparison
of 2002 California Standards Test and Performance
Assignment Scores
100.0%
Below Basic
Basic
80.0%
Above Basic
73.7%
59.0%
60.0%
49.1%
41.2%
36.3%
40.0%
33.8%
31.7%
25.1%
21.4%
20.0%
14.5%
9.3%
4.9%
0.0%
Not Proficient
Partially
Proficient
Proficient
2001 Performance Assignment Scores
Advanced
C RE SS T/U C LA
LA Scale-Up
 Cost and time driven
 Maintained by board and union support
 Transfer of responsibility
 Reduction in technical quality
 Reduction in range of measures
 Positive evaluation from independent
group focusing on changing teaching
practices
C RE SS T/U C LA
Continuing R&D Areas
 New contexts
 Trade-offs (limited number of templates
vs. wide range of formats)
 Performance over time
 Scalability in the long run
 Authoring systems to support teacher-
developed assessments linked to largescale assessment
C RE SS T/U C LA
Summary of Assessment
Knowledge Requirements
 Knowing why
 Knowing what to assess: content plus
cognitive demands (problem solving,
communication, learning to learn, teamwork,
content knowledge)
 Knowing how: transfer (application to other
topics and situations)
 Reflecting: applying MBA to teaching
C RE SS T/U C LA
Model-Based Assessment
Cognitive Families
Content
Understanding
Teamwork and
Collaboration
Communication
Learning
Problem
Solving
Metacognition
C RE SS T/U C LA
Context for Success of
Knowledge-Based Reform
 Local ownership of knowledge
 Infrastructure and stability
 Capacity to investigate
 Learning
 Congruence or peace with
external mandates
C RE SS T/U C LA
Usable Knowledge and
Support May Get to Useful
Knowledge
 For assessment knowledge to be
useful, it depends upon the context,
capacity, and communication of the
teaching system
 For assessment knowledge to be useful
to students, it must go to the heart of
why, what, and how they learn