Transcript Setting Standards for the MCAS HS STE

Standard Setting:
MCAS High School Science &
Technology/Engineering (STE) Tests
Sheraton Four Points Hotel
Norwood, MA
August 14-16, 2007
Tuesday, August 14
Overview of Plenary Session






Welcome/Introductions
Overview of MCAS Program and the High
School STE Tests
Purpose of 2007 Standard Setting
Body of Work Method and Procedures
Ground Rules for Standard Setting
Agenda (Tuesday-Thursday)
Department of Education











Sal Beatini, MCAS Science Test Developer
Bob Bickerton, Associate Commissioner
Joyce Bowen, Science Specialist
Katie Bowler, MCAS Science Test Development Lead
Per Christiansen, MCAS Science Test Developer
Kevin Dwyer, MCAS History Test Development Lead
Jake Foster, State Science Coordinator
Mark Johnson, Director of MCAS Test Development
Bob Lee, MCAS Chief Analyst
Matt O’Connor, Administrator for Administration, Analysis and
Reporting
Kit Viator, Director of Student Assessment
Measured Progress













Ann Adjutant, Physics Developer
Liz Burton, Psychometrician
Lee Butler, Administrative Assistant
Lisa Ehrlich, Assistant Vice President
Kevin Haley, Manager of Data Analysis
Renee Jordan, Service Center Representative
Mark Peters, Program Assistant
Miechelle Poulin, Program Assistant
Michael J. Richards, Program Manager
Kevin Sweeney, Assistant Vice President, Research & Analysis
David Tong, Assistant Director, MCAS Program Management
Karen Whisler, Lead Science Developer
Eric Wigode, Director of MCAS Test Development
Biology Panelists
Erin Bengiovanni
Carol Bernon
Sheila Blair
Ronit Carter
Andrew Creamer
Amy Deacon
Jennifer Dube
Eileen Dufour
Mary Dulko
Richard Fardy
Martha Hogan
Elizabeth Hufnagel
Julie Jonelis
Sarah Juhlin
Kelly Kallin
Aaron Mathieu
Jim Perry
Beth Raccula
Dianne Rees
Julie Robertson
Carla Romney
Deborah Sakelakos
Ernestine Struzziero
Biology Teacher
Science Department Head
Science Dept. Head/Biology Teacher
Educational Consultant
Biology Teacher
Biology Teacher
Science Instructor
Biology II Instructor
Biology Teacher
Biology Teacher
Academic Chair, Science
Science Teacher
Biology Teacher
Science Instructor
Biology Teacher
High School Biology Teacher
Science Chair
Science Teacher
Director of Science
Upper School Biology Teacher
Faculty Member
MCAS Program Administrator
Science Teacher
Joseph P. Keefe Technical
Barnstable High School
Hull High School
In Harmony with Education
Classical High School
Pentucket Regional High
Greater Lawrence Technical High
Greater Lowell Technical High
Sharon High School
Wilmington High School
Salem High School
Brookline High School
Sabis International Charter
Blackstone Valley Regional
Sharon High School
Acton-Boxborough Regional
Mansfield High School
Walpole High School
Braintree High School
Foxborough Regional Charter
Boston University
Bedford High School
Lynnfield High School
Chemistry Panelists
Steven Abdow
Thomas Bartosek
Catherine Botsford Milne
Patricia Brandl
Barbara Brigante
Stacia Brown
Lisa Buron
Tony DiLuna
Mary Duane
Anthony Fimognari
Catherine Francis
Boriana Georgieva
Joyce Gleason
Esther Hines
Glenn Horner
Shawn Kenner
Ronee Krashes
Peter Nassiff
Diane Perito
Nitzan Resnick
Michael Thompson
Linda Weber
Tara Yohan
Chemistry Instructor
Biology/Chemistry Teacher
Chemistry Teacher
Science Teacher
Teacher
Science Curriculum Coord.
Science Teacher
Science Teacher
Science Teacher
Chemistry Teacher
Chemistry Teacher
Chemistry Teacher
Educational Consultant
Chemistry Teacher
Executive/Scientist
Chemistry Teacher
Teacher
Head of Science Department
Chemistry Teacher
Director
Chemistry Teacher
Chemistry Teacher
Biology/Chemistry Teacher
Durfee High School
Canton High School
Attleboro High School
Medford High School
Northeast Metro. Regional
Oakmont Regional HS
Mansfield High School
Woburn Memorial High School
Tantasqua Regional High School
East Longmeadow High School
North Reading High School
Swampscott High School
Billerica Memorial High School
DCE Aprilis
Sharon High School
Framingham High School
Burlington High School
Malden High School
South Area Solomon Schecter Day School
Amherst Regional High School
Natick High School
Amherst Regional High School
Introductory Physics Panelists
Anne Marie Brooks
Darren Burris
Norma Chico
Steven Cogger
Marilyn Decker
Michael Doherty
Brian Dukeman
Erin Marie Flanagan
Cynthia Givens
Naila Jirmanus
Christine John
Mon Luke
Kristin Newton
Julie Shea
Mary Ellen Stephen
John Sullivan
Sue Vincent
Amy Winston
Science Teacher
Math Instructor
High School Physics & Chemistry
BMW Technician
Senior Program Director, Science
Teacher
Science Teacher
Teacher
Teacher
Science Program Coordinator
Physics Teacher
Teacher
Physics Teacher
High School Science Teacher
Physical Science & Chemistry Teacher
Retired Physics Teacher
Teacher
Science Department Chair
Wareham High School
Boston Collegiate Charter
Doherty Memorial
BMW Peabody
Boston Public Schools
Andover High School
Whitman Hanson Regional
Wachusett Regional High School
Andover High School
Sabis International Charter
Belmont High School
Bedford High School
Cambridge Rindge & Latin
Revere High School
Plymouth South High School
Quincy Public Schools
Turners Fall High School
Newton North High School
Technology/Engineering Panelists
Joseph Clement
Kevin Connolly
Leigh Estabrooks
Thomas Fischer
Pamela Glass
Ramiro Gonzalez
William Hankey
Rebecca Lothrop
Charles Pouliot
Lee Pulis
Bruce Rawley
Anthony Ruscito
Lynn Russell
Mike Stevens
Victor Valenti
Catherine Wolf
Department Head
Technology Teacher
Inventeams Associate
Technology Eng. Teacher
Math/Science Curriculum Coordinator
Engineering & Technology Teacher
Engineering Teacher
Teacher
Teacher
Curriculum Developer
Technology/Engineering Teacher
Technology Teacher
Biology Teacher
Technology/Engineering Teacher
Retired Engineer
High School Science Teacher
Beverly High School
Bedford High School
MIT
Hopedale Jr. Sr. High School
Westport Community School
Boston Arts Academy
Doherty Memorial High School
Clearway School
Lawrence Public Schools
Museum of Science
Millbury High School
Bedford Public Schools
New Leadership Charter
Maynard Public Schools
Compaq
Brookline High School
Standard Setting
Facilitators

Biology – Lisa Ehrlich

Chemistry – Margaret Hill

Introductory Physics – Dona Carling

Technology/Engineering – Tim Crockett
Purpose of MCAS Program

Inform/improve curriculum and instruction

Evaluate student, school, and district performance
according to Curriculum Framework content standards
and MCAS performance standards

Determine eligibility for high school diploma
(Competency Determination)
Massachusetts High
School Competency
Determination (CD)
Requirements
To earn diploma, a student must:
 Meet local graduation requirements
 Attain passing standard (scaled score of 220)
or higher on high school MCAS tests or
equivalent on MCAS Alternate Assessment
Selected Features of MCAS

Custom developed based on Massachusetts Curriculum
Framework content standards and MCAS performance
standards

100% of questions used to determine student scores
released annually

Measures performance of ALL students educated with
public funds

Results reported according to scaled scores and
performance levels
Historical Background of the MCAS
Tests
Massachusetts
Education
Reform Law
passed
Grade 3 Reading,
grade 6 Math, and
grade 7 ELA tests
introduced
Newly required
NCLB tests
(grades 3-8)
introduced
2001
1993
2006
1998
2003
First MCAS
operational tests
introduced (ELA,
Math, and Science
& Technology,
grades 4, 8, and 10)
Class of 2003 first
graduating class
required to earn a CD
(ELA and Math)
STE tests in grades 5
and 8 introduced
2010
Class of 2010 first
graduating class
required to earn
a CD in STE
Development of Massachusetts STE
Curriculum Framework (CF)
Standards and Assessments
1996
First STE Curriculum Framework introduced
1998-2000
Integrated HS STE test administered
2001-2002
Revised STE Curriculum Framework (major revisions); no STE
testing
2003
Question tryout for tests in Biology, Chemistry, Introductory
Physics, and Technology/Engineering; no results reported
2004-2005
Pilot testing of STE tests; item-level results only
BOE decides class of 2010 to pass one of STE tests to earn CD
2006
2007
High School STE CF revised (minor revisions)
Operational test
9th graders may earn CD by passing one of four STE tests (ELA
and Math requirements also apply)
MCAS STE Testing Program

End-of-course tests offered in grades 9 and 10:
– Biology
– Chemistry
– Introductory Physics
– Technology/Engineering

Students required to pass one of four tests in order to
earn CD
High School STE
MCAS Test Design
Each of the four tests includes:
40 multiple-choice items 1 point each
60 total raw
= score points
5 open-response items 4 points each
MCAS Scaled Scores/Performance Levels
Failing
200
Needs
Improvement
220
240
Proficient
Advanced
260
280
High School Science and Technology/
Engineering Test Reporting Categories
Biology
Chemistry
Rep. Category
% of items
(+/- 5%)
Rep. Category
% of items
(+/- 5%)
Biochemistry and Cell
Biology
25
Properties of Matter
and Thermochemistry
25
Genetics
20
25
Anatomy &
Physiology
15
Atomic Structure &
Periodicity
Bonding & Reactions
30
Ecology
20
20
Evolution &
Biodiversity
20
Solutions,
Equilibrium, & AcidBase Theory
Science and Technology/Engineering
Test Reporting Categories
Introductory Physics
Rep. Category
% of items
(+/- 5%)
Technology/Engineering
Rep. Category
% of items
(+/- 5%)
Motion & Forces
40
Engineering Design
20
Heat & Heat Transfer
15
20
Waves & Radiation
25
Construction &
Manufacturing
Electromagnetism
20
Fluid & Thermal
Systems
30
Electrical &
Communications
Systems
30
Rationale:
End-of-Course STE Test Menu



Promote deeper assessment of content than
integrated test
Support diverse course offerings and
scheduling flexibility for high schools
Provide students choice of assessment for
CD requirement
Efforts to Establish “Equivalence”
of STE Tests
The STE tests:



are based on parallel test design and
development processes
are based on performance level descriptors
that are comparable in scope and rigor
have been designed to have similar
psychometric properties
External Validation of Efforts to
Establish “Equivalence” of STE Tests
 Performance
level descriptors used in standard
setting have been externally validated by
Massachusetts teachers
 Technical/psychometric properties analyzed by
Dr. Ronald Hambleton, University of
Massachusetts, Amherst
 Test design and plans for standard setting
endorsed by the MCAS National Technical
Advisory Committee (TAC)
Content Standards vs.
Performance Standards

Content standards = “What”
Describe the knowledge and skills students
should acquire in a particular content and grade

Performance Standards = “How well”
Describe student work on MCAS tests at the
Needs Improvement, Proficient, and Advanced levels
General MCAS
Performance Level Descriptors
Needs Improvement
Students at this level demonstrate partial understanding
of subject matter and solve simple problems.
Proficient
Students at this level demonstrate a solid understanding of
challenging subject matter and solve a wide variety of
problems.
Advanced
Students at this level demonstrate a comprehensive and
in-depth understanding of rigorous subject matter, and
provide sophisticated solutions to complex problems.
General STE Performance Level Descriptors
Needs Improvement
Proficient
Advanced
On MCAS, a student at this level
On MCAS, a student at this level
On MCAS, a student at this level
Conceptual
Understanding
and Factual
Knowledge
 Demonstrates a partial
understanding of some
facts, concepts, principles,
and theories
 Uses basic scientific terms
 Demonstrates a solid
understanding of many
facts, concepts, principles,
and theories
 Uses appropriate scientific
terms
 Demonstrates a
comprehensive, in-depth
understanding of many facts,
concepts, principles, and
theories
 Applies scientific terms in an
appropriate context
Scientific
Process and
Skills
 Interprets simple data and
creates generalized
questions without
necessarily using the
scientific method
 Identifies a problem to be
solved
 Uses most of the steps of
the scientific method to
design and interpret
experiments
 Finds solutions to a variety
of problems
 Designs and evaluates
scientific experiments and
generates full interpretations
of data
 Finds solutions to complex
problems
Application/
Synthesis
 Makes simple predictions
about a specific topic
 Makes predictions based on
information given
 Justifies predictions in a
general sense
 Makes sophisticated
predictions
 Synthesizes a wide array of
information from multiple
sources
 Applies knowledge to
abstract or novel situations
Purpose
High School Science & Technology/ Engineering:
To establish cut scores for MCAS performance levels
Cut score
needed
Failing
Cut score
needed
Needs
Improvement
Cut score
needed
Proficient
Advanced
Linking Performance Standards
with Student Work

What is standard setting?
Establishment of cut scores to distinguish
between performance levels

What is your job?
Use the PLDs to evaluate student work and
make recommendations for where cut scores
should be set
Purpose of Standard Setting

Determine cut scores for reporting
assessment results

Answer the question:
– How much is enough?
General Phases of Standard Setting

Data-collection phase

Policy-making/decision-making phase
Selected Standard-Setting Methods



Angoff
Bookmark
Body of Work
Choosing a Standard-Setting Method
Prior usage/history
 Recommendation/requirement
by policy-making authority
 Type of assessment

Body of Work method chosen for MCAS tests in
High School Science
& Technology/Engineering
What is the Body of Work
Procedure?
Panelists examine student work (actual responses to
test questions) and make a judgment regarding the
performance level to which the student work most
closely corresponds.
HS STE Standard Setting:
Panelists examine student work
that has not been previously classified
and determine how that work
should be classified.
How to Classify Student Work
Materials you will need:

Performance Level Descriptors
• General
• Content specific

Bodies of Student Work
• Responses to constructed-response questions
• Multiple-choice summary sheet

Rating Forms
How to Classify Student Work

Examine the student’s responses to multiple-choice
questions

Examine the student’s responses to open-response
questions

Judge the student’s knowledge and skills
demonstrated relative to the PLDs

Panelists do not need to reach consensus on the
classifications
How to Classify Student Work
To help prepare you to do these ratings, you will
spend time becoming familiar with the following:



Test
Performance level descriptors
• It is important that all panelists have a clear,
common understanding of the PLDs.
Bodies of student work
• Multiple-choice items
• Constructed-response items
How to Classify Student Work

You will have the opportunity to discuss your
classifications and change them if desired.

Don’t worry! We have procedures, materials, and
staff to assist you in this process.
What Next?
Break into content-area groups:










Take the assessment
Discuss the Performance Level Descriptors
Complete the Item Map
Complete training round
Complete individual ratings
Receive feedback from first round of ratings
Discuss ratings and revise
Receive feedback from revised ratings
Discuss feedback and provide final ratings
Complete an evaluation form
Top 8 Most
Misunderstood Things
about Standard Setting
8. Standard setting is a great opportunity to rewrite
Curriculum Framework standards.
7. The process is rigged.
6. This is a good time to vent about all the things
you hate about MCAS.
5. We should use this time to rework HS Science
and Tech/Eng. performance level descriptors.
Top 8 Most
Misunderstood Things
about Standard Setting
4. Standard setting is scoring.
3. Only HS Science and Tech/Eng. scholars should
be doing this work.
2. Only teachers should be doing this work.
1. Disagreement is bad.
Ground Rules







Role of facilitator is to “facilitate” and keep process on
track
Process solely focused on recommending performance
standards (cut scores) for MCAS
MCAS performance level descriptors are integral to
process but are not up for debate
Panelists’ recommendations are vital; however, final cut
scores determined by the MDOE
Each panelist must be in attendance for the duration of
the process for his/her judgments to be considered
Each panelist must complete evaluation form at the end
of the event
Cell phones off, please!
Agenda
Tuesday, August 14
Plenary
9:00 am – 10:30 am
Break
10:30 am – 10:45 am
Work session10:45 am –12:00 pm
Lunch
12:00 pm – 1:00 pm
Work session
1:00 pm – 4:00 pm
Wednesday, August 15
Breakfast
7:30 am – 8:30 am
Work session
8:30 am – 12:00 pm
Lunch
12:00 pm – 12:45 pm
Work session
12:45 pm – 4:00 pm
Agenda
Thursday, August 16
Breakfast
7:30 am – 8:30 am
Work session8:30 am – 12:00 pm
Lunch
12:00 pm – 12:45 pm
Work session
12:45 pm – Until completion
Room
Assignments
Physics:
Chemistry:
Biology:
Tech/Eng:
101
102
103
104
Questions?