Transcript text
Revisiting Performance Assessment 2.0 st in 21 Century Accountability Eva L. Baker National Center for Research on Evaluation, Standards, & Student Testing Graduate School of Education & Information Studies University of California, Los Angeles Korea Institute for Curriculum and Evaluation Seoul, Korea 27 October 2010 KICE 1 © Regents of the University of California Assessment Revisited: Update on U.S. Educational Policy 2.0 • What have been the major issues in U.S. plans for reform, assessments, and accountability? • What is Race to the Top? • Where are we now? Performance assessment? • What practical changes will need to occur? • Comments and concerns KICE 2 © Regents of the University of California Educational Reform Update • Continuation and extension of President Bush’s policies • Educational performance (NAEP) is flat with some small gap closing; international standing poor • Belief is that large bureaucracies (districts) and teacher unions cause low growth • Adopting of policies to centralize some aspects of education and decentralize others • Competition among States for resources KICE 3 © Regents of the University of California Race to the Top Provisions • Adopting standards and assessments that prepare students to succeed in college and the workplace (Common Core of Standards 40+ states) • Building data systems that measure student growth and success, and inform teachers and principals how to improve instruction • Recruiting, developing, rewarding, and retaining effective teachers and principals, especially where they are needed most • Value Added Modeling (VAM) • Turning around their lowest-performing schools KICE 4 © Regents of the University of California Where Are We Now? Race to the Top • Competition among states for discretionary education money • Competition heated because of poor economy and layoffs from schools (40 states have applied) • Need to get State legislative approval to conform to federal requirements—charter schools; individual and teacher databases; different union relationships • Note: “good” schools and districts have high test scores, despite evidence of direct practice, cheating, and reduction in curriculum offerings • Teachers are mostly blamed for failure, and unions are demonized, charter schools are the answer KICE 5 © Regents of the University of California Additional Efforts • Standards focus on college- and careerreadiness in reading and math (science and history) • Create pipelines and incentives to put the most effective teachers in high-need schools • Alternative pathways to teacher and principal certification • Charter school legislation (State-lifting caps) • STEM KICE 6 © Regents of the University of California Race to the Top—Round 1 • 10 billion available • Tennessee and Delaware won round one 1. Tennessee $500,000,000 2. Delaware $100,000,000 KICE 7 © Regents of the University of California Race to the Top Awards— Round 2 1. Massachusetts $250,000,000 2. New York $700,000,000 3. Hawaii $75,000,000 4. Florida $700,000,000 5. Rhode Island $75,000,000 6. District of Columbia $75,000,000 7. Maryland $250,000,000 8. Georgia $400,000,000 9. North Carolina $400,000,000 10. Ohio $400,000,000 KICE 8 © Regents of the University of California Common Core State Standards (CCSS) • Mathematics and Literacy, for every grade • Literacy includes “informative reading and writing,” topics include biological sciences and history • Standards developed at the initiative of the State education officer and governor • 35+ states participate, may have additional standards KICE 9 © Regents of the University of California Quality of Common Core State Standards • Generally regarded as excellent • Fewer, Clearer, Higher = Many, Clearer, Higher • Good illustrative material • Sequences unverified • Use of ontologies by CRESST to clarify meaning and internal relationships funded by the Gates Foundation • Developing formative and curriculum-based assessments and evaluating existing options KICE 10 © Regents of the University of California Common Core Standards Progression KICE 11 © Regents of the University of California Formal Representation 1X5 KICE 12 © Regents of the University of California Assessment Funding for RTT • To avoid redundant procurement, States banded together in three different consortia to bid for resources to design and specify assessments • Assessments to include formative, for classroom use, interim for prediction within grade, accountability • Reporting by growth and value-added modeling • Enables performance assessment as part of the package KICE 13 © Regents of the University of California Two Consortia Won • Smarter Balance, with a bottom-up teacher approach that promises the use of computer adaptive testing (45) states ($160M) dollars • PARCC supported by Achieve, Inc., a Washington think-tank (26 states; $185.87M) • Period of performance – 4 years, collaboration with universities, commercial groups • Both groups at this point seem relatively conservative in their views KICE 14 © Regents of the University of California Financing Assessment • Total awards for assessment in the $350M area • Testing companies are anxious, needing to win the RFPs from the States even if they don’t agree with their plans • Maintenance would go to publishers with States providing oversight • Who owns standards? Assessments? KICE 15 © Regents of the University of California Smarter Balanced Consortia • New generation assessment system—deep disciplinary understanding and higher-order thinking skills demanded by knowledge-based economies • Relevant to ongoing improvements in instruction and learning • Useful for all members of the educational enterprise: students, parents, teachers, school administrators, members of the public, and policymakers • Formative and summative assessments, organized around Common Core Standards, that support high-quality learning and the demands of accountability • Feasible, innovative assessment and fiscally sustainable • RFPs to private companies KICE 16 © Regents of the University of California Smarter Balance • Assessments are integrated into a coherent learning system of standards, curriculum, assessment, instruction, and teacher development • Teachers and other instructional experts develop formative and summative assessments of standards—supports are provided to enable thoughtful teaching • Teachers are integrally involved in the design, development, and scoring of assessment items and tasks • Teachers’ roles include the construction and review of items/tasks, the definition of scoring guides, selection of student work exemplars, and scoring KICE 17 © Regents of the University of California 21st Century Skills • Assessments include evidence of actual student performance on challenging tasks that evaluate standards of 21st Century learning • They emphasize deep knowledge of core concepts within and across the disciplines, problem solving, analysis, synthesis, and critical thinking • Technology is designed to support assessment and learning systems • by delivering the assessments; enabling adaptive technologies to better measure student abilities across the full spectrum of student performance • evaluate growth in learning • supporting online simulation tasks that test higher-order abilities KICE 18 © Regents of the University of California Performance Assessment • Open ended • Multiple steps • Synthesis of standards • More than one content area? • Team or individual • Judged by raters or markers KICE 19 © Regents of the University of California English Language Arts KICE 20 © Regents of the University of California Technology-Driven • Semi-automated design with templates • Just-in-time steps • Help possible • Simulations of processes • Complex problem solving and decision making • Written products, demos, research KICE 21 © Regents of the University of California Technical Issues—Again • Few “items”—old psychometrics • Comparability of tasks, quantitative and qualitative features • Confounded “difficulty” • Vertical scaling • Reporting for accountability, instructional feedback • Cost • Renewal KICE 22 © Regents of the University of California Technical Needs • Robust and rapid online scoring • Diagnostic and accountability accuracy • Instructional sensitivity • Extensive or inappropriate help • Memorable tasks need to be replenished • Cost (back to reusable designs) • Game frameworks KICE 23 © Regents of the University of California Additional Concerns • Economics suggest that total amount available from Federal government will be reduced • Likely outcome is reduced innovation • What kinds of validity studies should be conducted? • CRESST scholars have proposed studies of accuracy of classification, utility of findings for improvement, instructional sensitivity, accommodations, and other fairness approaches—a struggle since there will be few revision resources and no full trial of the system KICE 24 © Regents of the University of California Research and Development Needs • Validity approaches • Modeling • Sampling • Verification of sequences of CCS and assessments • Utility to teachers of different levels of training and experience • Openness of feedback • Continuous modification and longitudinal data • Value-added issues KICE 25 © Regents of the University of California Speaking Truth to Power • Not easy unless you are friends • Policymakers in U.S. have odd mix of research-based ideas and preferences—now almost an overwhelming wave in one direction • Desire conflicting goals without recognizing technical conflicts • What should researchers do? KICE 26 © Regents of the University of California http://www.cse.ucla.edu Eva L. Baker voice fax email KICE 27 310.206.1530 310.267.0152 [email protected] © Regents of the University of California