Transcript Document

Independent Study of
State End-of-Course Assessments
Presentation of Findings
House Education Committee
February 1, 2008
Introduction
• The Washington State Board of Education (SBE)
contracted with Education First Consulting to conduct an
independent study of state end-of-course assessments.
• Our research team includes:
– Jennifer Vranek, President of Education First Consulting
– Jessica de Barros, Education Policy Consultant
– Richard Brown, Psychometrician and Assistant Professor in the
USC Rossier School of Education
– Betheny Gross, Senior Research Analyst in the University of
Washington’s Center on Reinventing Public Education
– Christopher Mazzeo, Education Policy Consultant, Consortium
for Chicago School Research
Slide 2
Our Approach
Our team researched the following:
• What lessons can Washington State learn from the
literature on end of course tests (EOCs),
comprehensive assessments and exit exams?
• What have been the experiences of other states
implementing EOCs?
• What alternatives measure Washington’s standards?
• What are the policy implications for Washington’s
high school assessment system based on lessons
learned from other states with EOCs?
Slide 3
Methodology
• First, the team conducted a thorough review of the primary and
secondary literature on EOCs and comprehensive tests, as well as
the use of these formats for exit exams and school accountability.
• Next, we researched and summarized nine states’ EOC programs.
– CA, IN, MD, NJ, NY, SC, TN, TX and VA
• We simultaneously conducted 30 interviews of education,
government and business leaders in seven states.
– CA, IN, NJ, SC, TN, TX and VA
• Finally, we used our research findings to consider policy
implications for Washington State’s high school assessment
system.
Slide 4
What are EOCs?
What is the WASL?
End-of-course assessments are tests that:
– Are given at the end of a particular course and measure the
content found only in the standards for that course.
– Are taken only by students taking a particular course, so not
all students are assessed.
The 10th grade Washington Assessment of Student
Learning in reading, writing, mathematics and science is a
comprehensive assessment because:
– It is given at a common point in time (to all 10th graders) and
– It measures content and skills from standards that are taught
over several years, rather than in a particular course.
Slide 5
Purposes
Of Assessment
There are many ways to describe the purposes of
statewide high school assessment. We summarize
these as four main purposes:
– Supporting student learning;
– Holding students and/or schools accountable;
– Determining readiness for postsecondary education
and training; and,
– Ensuring high-quality and efficient operations.
Slide 6
Assessments:
Means to an End
• On balance, both comprehensive and EOC
assessments can meet these four purposes
equally well.
• However, each approach has different strengths
and meets the purposes in different ways.
• The advantages and disadvantages of either test
format depend on the state’s purposes for
assessment.
Slide 7
Comparing Use
Of Assessments
• Comprehensive assessments are more
common today than EOCs…
• But more states are shifting to EOCs.
– 16 states currently use EOCs
– 11 additional states are planning to use EOCs
Slide 8
States with EOCs
(In Place or Planned)
16
Have EOCs in place (or field-testing) in 2007-2008 school year:
AR, CA, GA, IN, LA, MD, MA, MS, NJ,
NY, NC, OK, SC, TN, UT, VA
11
Report plans to have EOCs in at least one subject area:
AZ, FL, HI, KY, MI, NM, OH, PA, RI, TX, WV
7
Planning to keep both EOCs and comprehensive assessments:
AR, CA, GA, MA, MI, NJ, SC
Slide 9
States with EOCs as Exit Exams
or for School Accountability
(In Place or Planned)
All states currently or planning to have exit exams for students:
26
AL, AK, AR, AZ, CA, FL, GA, ID, IN, LA, MD, MA, MN, MS,
NV, NJ, NM, NY, NC, OH, OK, SC, TN, TX, VA, WA
10
3
12
States currently or planning to use EOCs to hold students accountable:
AR (2010), IN (2000), MD (2009), MS (2006), NY (2000),
NC (2010), OK (2012), TN (2005), TX (2012), VA (2004)
States that will use comprehensives in English/math and EOCs in
other subjects:
MA (English/math 2003, science EOCs 2010, U.S. History EOC 2012)
NJ (English/math 2003, Biology EOC 2010)
SC (English/math 2006, Biology and U.S. History EOCs 2010)
Use or plan to use some or all of their EOCs for school accountability
under NCLB:
AR, MA, MD, MS, NJ, NY, NC, OK, SC, TN, UT, VA
Slide 10
Key Findings:
Comprehensive Assessments
Overall, we found that comprehensive testing systems:
• Usually focus on 10th grade or lower standards;
• Assess a slice of high school standards, rather than deep
knowledge of particular subjects;
• Can potentially narrow the delivered curriculum to what is
tested;
• Provide a “snapshot” of system performance;
• Often take up less testing time overall and cost less;
• Take a more straightforward approach to exit exams and
school accountability; and
• Rarely provide information on students’ readiness for
postsecondary education coursework and training.
Slide 11
Key Findings:
EOC Assessments
Overall, we found that EOC testing systems:
• Vary widely with respect to number and kinds of courses assessed;
• Will measure a broader and deeper range of standards, but only if there are a
sufficient number of EOCs in each subject;
• Do not assess all students against common standards, unless states require all
students to take certain courses and/or require all students to take certain EOCs;
• Can promote more consistency of teaching and provide more timely
information on learning and course quality;
• Motivate students to learn through exit exams as well as other forms of lesser
student stakes (e.g. counting results in course grades;
• Make it more complicated to hold schools accountable, yet offer the potential
to produce more validity and reliability; and
• Can be better-suited for placing students in postsecondary education
courses than comprehensive tests given by states in the 10th grade.
Slide 12
Policy Implications
for Washington
• Our research revealed that both comprehensive and
EOC assessments have strengths and limitations,
depending on the state’s purpose of assessment.
• We believe Washington policymakers must first
determine what their policy priorities are for student
assessment in order to choose the most appropriate
testing format.
Slide 13
A Decision Framework
For Policy Priorities
• To help policymakers compare how each test format
meets various policy priorities, we developed a
“decision framework” analyzing each type of
assessment according to the four purposes of
assessment:
– Supporting Student Learning;
– Holding Students and Schools Accountable;
– Determining readiness for postsecondary education and
training; and
– Ensuring High-Quality and Efficient Operations.
• Within each issue area, we analyzed the relative
advantages and disadvantages of comprehensive
assessments and EOCs according to multiple criteria.
Slide 14
Supporting
Student Learning
Issue Area
Supporting Student Learning
Measuring the breadth and depth of standards
Assessing students near the point of curriculum
delivery
Assessing students with the same test
Choice and quality of test question types
Slide 15
Advantage to…
EOCs (slight)
EOCs (strong)
Comprehensive
No clear
advantage
Holding Students &
Schools Accountable
Issue Area
Holding Students and Schools Accountable
Validity and reliability of assessments
Holding students accountable
Reporting results at the classroom or course
level
Holding schools accountable
Slide 16
Advantage to…
EOCs (slight)
No clear
advantage
EOCs
No clear
advantage
Determining Readiness
for Postsecondary Education
Issue Area
Advantage to…
Determining Readiness for Postsecondary Education &
Training
Measuring readiness for postsecondary
EOCs (strong)
education
Providing access to rigorous courses while
EOCs (slight)
preserving flexibility
Slide 17
Ensuring Quality &
Efficient Operations
Issue Area
Advantage to…
Ensuring Quality and Efficient Operations
Testing window and turnaround time for
No clear
results
advantage
Costs and time spent on testing
Comprehensive
Impact of administration on schools
No clear
advantage
Test security
No clear
advantage
Slide 18
Sequencing
Policy Decisions
• Assessments are a means to the end of understanding how well
students, schools and districts are meeting state standards.
• It is important to consider the assessment policy within the larger
context of the state’s K-12 education system and efforts to improve
learning and teaching.
• Over the next several months, the statewide system of standards,
curriculum, instruction, assessment and accountability is being
reshaped.
• State policymakers should first address key questions about high
school education policy and then determine which format for the
high school assessments is most relevant.
Slide 19
Recommended
Policy Questions
• What skills and knowledge do students need to be successful after
exiting Washington’s K-12 public education system? How important
is measuring readiness for postsecondary education?
• Is it important for the state to assess academic standards close to the
time when students learn content or at a common point in time?
• When the new graduation requirements are adopted, what course
credits will all students be expected to earn in English, math,
science and social studies?
• Should the main high school assessments also be used to measure
readiness for postsecondary education?
• Does the state want additional ways to hold students accountable,
such as tying test results into course grades?
• Are more statewide measures needed for schools or are the four
current measures sufficient for the state accountability system?
Slide 20
EOC-Specific
Recommendations
If, after addressing these questions, Washington policymakers
consider transitioning to an EOC-based system, we recommend
that policymakers:
– Minimize cost and development time by working in
collaboration with other states to implement standards-based,
criterion-referenced assessments;
– Require all students to earn a common set of course credits,
and require all students to take the corresponding EOCs in these
subjects, to ensure equity.
– Maintain the 10th grade WASL in reading and writing, rather
than creating EOCs in these subjects.
Slide 21
Discussion
Slide 22
Back-Up Slides
Slide 23
Detailed Findings
1.
Well-designed comprehensive test systems and EOC systems
share many characteristics of high-quality testing
systems, such as
–
–
–
–
Slide 24
Measuring state standards with reliability and validity.
Assessing standards with a variety of formats (multiplechoice, constructed-response).
Turning around test results in a timely manner.
Scoring costs may increase with EOCs, but this depends
on number of tests/test forms, types of item formats and
scoring turnaround needs.
Detailed Findings
2.
State EOC systems vary widely on the number and kinds of
courses that are assessed.
–
–
–
–
–
EOC systems can assess a broader and deeper range of content and
skills—if there are multiple assessments in a subject area, assessments
for advanced-level courses and/or multiple subject areas.
Comprehensive tests typically focus on 10th grade standards and assess
all students at a common point in time.
Most EOC systems still measure English with a comprehensive test.
Unless states require all students to pass a certain series of courses
and/or require all students to take certain EOCs—different groups of
students will be tested on different content.
E.g.: 700,000 California students took the Algebra I EOC (a course
required for graduation) in 2006, but only 300,000 took the Geometry
EOC.
Slide 25
Detailed Findings
3.
While both types measure standards, EOCs are typically
chosen by states to promote more consistency of teaching.
–
–
–
Slide 26
EOC exams are designed to measure the impact of a specific course
on student learning.
Comprehensive exams are designed to measure the impact of
instruction over several years on student learning.
CA, IN, TN and VA education leaders told us they administer EOCs
because the standards, content, instruction, curriculum and
assessment are more closely aligned than in comprehensive systems.
Detailed Findings
4.
Changing test formats will not necessarily improve student
learning of state standards or increase student performance.
–
Performance on EOCs and comprehensive tests varies widely.
•
–
It not possible to conclude that test format type has any influence on
the number or proportion of students meeting standards. Factors that
impact test scores include:
•
•
•
•
•
–
Slide 27
Student results on the high-stakes California and South Carolina 10th
grade exit exams are much higher than on those states’ EOCs.
Content measured;
Rigor and complexity of test questions;
Performance standards (“cut scores”);
Alignment of standards and local curriculum with the assessment; and
Students’ opportunity to learn the material tested.
Most exit exam states—regardless of test format—have high levels of
student performance overall (and many have closed performance gaps),
especially when results from retakes are added.
Detailed Findings
5.
Both EOCs and comprehensive assessments can be used as
exit exams. In addition, EOCs can be used as part of student
course grades.
–
–
–
Slide 28
A handful of states—including CA, IN, LA and UT—currently attach
no stakes for students to their EOCs.
Several states—including GA, NC, SC and soon TN and TX—require
districts to count EOC results as a portion of final course grades
(usually 15-25%).
13 states currently or will require students to pass at least one EOC in
order to graduate. 10 of these states use only EOCs, while MA, NJ and
SC use comprehensive and EOCs. This includes IN and TX, which
plan to transition away entirely from their comprehensive tests.
Detailed Findings
6.
It can be more complicated to hold schools accountable
with EOC tests.
–
It is more straightforward to use comprehensive tests for school
accountability.
EOCs can paint a more complex—and more reliable—picture of high
school performance. More EOC tests will yield more data and have the
potential for greater validity and reliability.
–
7.
States can use EOCs to meet No Child Left Behind
requirements in reading, math and science.
–
–
Slide 29
10 states currently use EOCs to determine AYP.
Two additional states are planning to use EOCs for AYP.
Detailed Findings
8. EOCs are better-suited to determine student readiness for
postsecondary education and training than comprehensive
tests given in 10th grade.
–
–
–
Most states planning to use high school tests for readiness and placement
purposes are moving towards EOCs, in large part because EOCs given
in advanced subjects can be used more easily to assess readiness than
comprehensive tests given in the 10th grade.
E.g. CA, LA, NJ, NY, TX and TN do or plan to use EOCs to signal
readiness for postsecondary education.
States also are adapting the SAT, ACT, Compass/Accuplacer to
administer to all 11th-graders.
Slide 30