Transcript Scenario Writing Workshop

The State of Material on
2011 Science MSP
OSPI Science Assessment Update
WERA, December 2010
Science Assessment Office
 Elaine Wood—Secondary Specialist
 [email protected]
 Kara Monroe—Elementary Specialist
 [email protected]
 Troi Williams—Administrative Assistant
 [email protected]
Agenda
 2011 Measurements of Student Progress
 Standard Setting
 Resources available
 Lessons learned from scoring
2011
Measurements of Student Progress
•Grades 5 and 8
•First assessment of the K-12 Science Learning
Standards (adopted June 2009)
Big Ideas in 2009 Science Standards
EALRs 1-3
Crosscutting Concepts and Abilities
EALR 4
Domains of Science
EALR 1 Systems
Physical Science
…is a way of thinking that makes it possible
to analyze and understand complex
phenomena.
EALR 2 Inquiry
Force and Motion
Matter: Properties and Change
Energy: Transfer, Transformation and
Conservation
Earth and Space Science
…is a process of asking and answering
Earth and Space
questions about the natural world that forms Earth Systems, Structures and Processes
the bedrock of science.
Earth History
EALR 3 Application
Life Science
…is about the interaction between science Structures & Functions of Living Systems
and technology, and how both can help solve Ecosystems
real-world problems.
Biological Evolution
5
Science Assessment Development Cycle
Develop
Test and Item
Specifications
Scenario & Item
Writing Committee
Scenario & Items
reviewed for
content by
OSPI/contractor
Content Review &
Bias/Sensitivity
Committees
Pilot
Range Finding
Committee
Scenario & Item
Pilot Testing
Items available for
Operational Testing
Operational
Range Finding
Committee
Data Review
Committee
Pilot Scoring
Operational Scoring
Scenarios and Items
 Scenarios provide context for sets of items.
 Systems, Inquiry, and Application
 Physical, Earth/Space, and Life Science
 The items associated with a scenario cover a range of
standards & range of cognitive complexity.
 Items not connected to a scenario are called “Stand
Alone” items, which is clearly marked for students.
From WASL to
Measurements of Student Progress
2009
2010
2011
WASL
MSP
MSP
Multiple Choice
21
26
22-25
Completion
NA
pilot only
1-4
Short Answer
8
4
4
Extended Response
0
0
0
57%
76%
76%
Grade 5
% Points from MC/CP
Embedded pilot items makes the total number of items 35.
From WASL to
Measurements of Student Progress
2009
2010
2011
WASL
MSP
MSP
Multiple Choice
28
30
26-29
Completion
NA
pilot only
1-4
Short Answer
7
5
5
Extended Response
2
0
0
56%
75%
75%
Grade 8
% Points from MC/CP
Embedded pilot items makes the total number of items 40.
Test Specifications Grade 5 & 8
The test map designates the following proportions:
 At least 20% Systems (with connected EALR 4 content)
 30% Inquiry
 20% Application (which will apply EALR 4 content)
 At most 30% EALR 4 domain knowledge that is not
associated with a cross-cutting concepts or abilities*
*EALR 4 content may be assessed in questions that only assess
EALR 4 or in questions that cut across both EALR 4 and a
cross-cutting concept or ability (Systems, Inquiry, Application).
More on Science MSP
Test Length
Grade 5: Single session 75-90 minutes*
Grade 8: Single session 90-110 minutes*
New Item Type
Completion: worth 1 score point
Test Window
May 2 to May 19 for paper/pencil tests
May 2 to June 3 for online tests
Online Testing
Voluntary in grades 5 and 8 in 2011
Goal: 25%
*additional time is needed for passing out materials
and reading directions
Standard Setting: Summer 2011
Following the process used by Mathematics in
grades 3-8 last year.
Science Standard Setting Plan
Date Event
June 2009 New Science standards approved
May 2010 Spring 2010 MSP; New items piloted
Aug 2010 Test Map meeting
Sept 2010 Performance Level Descriptors (PLDs) developed
Oct 2010 Spring 2011 tests built
Science Standard Setting Plan
Date Event
June 2009 New Science standards approved
May 2010 Spring 2010 MSP; New items piloted
Aug 2010 Test Map meeting
Sept 2010 Performance Level Descriptors (PLDs) developed
Oct 2010 Spring 2011 tests built
Feb/Mar 2011 Teachers from across state trained on how to use PLDs to participate in
Contrasting Groups Study (CGS)
Mar/Apr 2011 NTAC and SBE formally approve standard setting plan
April 2011 Teachers participate in CGS by providing ratings of students
May 2011 Spring 2011 MSP administered
Science Standard Setting Plan
Date Event
June 2009 New Science standards approved
May 2010 Spring 2010 MSP; New items piloted
Aug 2010 Test Map meeting
Sept 2010 Performance Level Descriptors (PLDs) developed
Oct 2010 Spring 2011 tests built
Feb/Mar 2011 Teachers from across state trained on how to use PLDs to participate in
Contrasting Groups Study (CGS)
Mar/Apr 2011 NTAC and SBE formally approve standard setting plan
April 2011 Teachers participate in CGS by providing ratings of students
May 2011 Spring 2011 MSP administered
Aug 2011 Standard Setting events:
--Grade level panels (use “Bookmark process”, start with teacher ratings)
--“Policy panel” recommendations
--NTAC certifies the planned process was followed
Aug 2011 SBE Sets Achievement Standards (Reviews all recommendations)
“Contrasting Groups” method affords
broader involvement in standard setting
Everyone in the state who teaches science in grades 5 and 8 is
invited to help set the performance standards.
 Performance Level Descriptors (PLDs) are developed for “Basic”
and “Proficient”
 Web-based training on PLDs is provided to participating teachers
 In April, participating teachers rate each of their students as being
“Basic or below” or “At or above Proficient”
 Two distributions of scores are produced: Scores obtained by
students rated as “Basic”, and scores for “Proficient”
 Where the two distributions merge represents the separation point
between “Basic” and “Proficient.”
Students rated as “Basic or below” using
criteria in PLD for Basic
Students judged to be at or below "Basic"
Number
of
students
0
2
4
6
8
10
12
14
16
18
20
22 23 24
26
28
Raw
score points
Points on MSP (simulated)
30
32
34
36
38
40
Students rated as “Proficient or above”
using criteria in PLD for Proficient
Students judged to be "Proficient" or above
Number
of
students
0
2
4
6
8
10
12
14
16
18
20
22 23 24
26
28
Raw
score points
Points on MSP (simulated)
30
32
34
36
38
40
Intersection indicates a region for where
“Basic” separates from “Proficient”
Students judged to be at or below "Basic"
Students judged to be "Proficient" or above
Number
of
students
0
2
4
6
8
10
12
14
16
18
20
22 23 24
26
28
Raw
score points
Points on MSP (simulated)
30
32
34
36
38
40
PLD training
 Available via Moodle, February 1, 2011
 Learn to use evidence to evaluate 5th or 8th grade students’
proficiency (other grade levels can participate)
 Using Moodle website
 Short videos of presentations
 All documents available for download
 Can accomplish all at once, or in multiple sittings
 Free clock hours (3)
 Encourage groups of teachers to participate together
 To sign up, go to link and create an account
 http://moodle.ospi.k12.wa.us/course/view.php?id=19
Contrasting Groups Study
 Available via EDS, April 11-29, 2011
 After completing the PLD Training, 5th or 8th grade
teachers can predict their students’ performance on the
Science MSP
 5th and 8th grade science teachers can gain access to the
EDS application by contacting their District Assessment
Coordinator
Resources Available
On the OSPI website:
www.k12.wa.us/Science/
OSPI Science Assessment Webpage
www.k12.wa.us
www.k12.wa.us/science
www.k12.wa.us/Science/Assessments.aspx
Select
Science
Assessment
from
Assessment
Drop-down
Select
Assessments
Science Assessment Resources
 Test and Item Specifications—more information about:
 Types of scenarios
 Test map
 Item specifications
 Vocabulary lists
 Cognitive complexity
Test and Item Specs
www.k12.wa.us/Science/Assessments.aspx
www.k12.wa.us/Science/TestItemSpec.aspx
Select
Test and Item
Specifications
Select a
Grade
Test and Item Specs
www.k12.wa.us/Science/TestItemSpec.aspx
Pages 5-14
Page 22
Science Assessment Resources
 Updates for 2011—each grade level contains:
 General information on the 2011 test
 Sample items for classroom use
 Scoring information, including rubrics and sample student
responses
Updates for 2011
www.k12.wa.us/Science/Assessments.aspx
http://www.k12.wa.us/Science/EducatorResources.aspx
Select
Educator
Resources
Select a
Grade
Science Assessment Resources
 Online Testing Website: http://wa-online.caltesting.org/
 Demo
 Flash player
 Tutorial
 CAL testing program must be installed
 Practice items, same as Update for 2011 documents
 Both tools help students become familiar with testing
platform, and tests the computer system
CAL online testing
http://wa-online.caltesting.org
Stand-Alone item from 8th grade tutorial
Select
Downloads or
Demos
Short answer item from
5th
CAL
online
testing
grade tutorial
Short answer item from
5th
CAL
online
testing
grade tutorial
Same item scrolled down to show
symbols button and symbols tool pop-up.
Science Assessment Resources
 Teacher Resource tool—tests based on previous standards:
 Types of scenarios
 Generic questions
 Item specifications (previous standards)
 Cognitive complexity
 State-wide statistics
Teacher Resource Tool
www.k12.wa.us/Science/Assessments.aspx
www.k12.wa.us/TeacherResourceTool2009-10/default.aspx
Select
Teacher
Resource Tool
Select
Science
Select a Grade
and a Year
Teacher Resource Tool
Type of
scenario
Generic question,
rolls over to item
specification
Percent correct
or mean score
Science Assessment Resources
 Powerful Classroom Assessments (PCAs)—currently
linked to WASL released scenarios and items:
 Sample items
 Scoring information, including rubrics
 New PCAs will be developed from future released items
Powerful Classroom Assessments
www.k12.wa.us/Science/Assessments.aspx
http://www.k12.wa.us/Science/EducatorResources.aspx
Select
Educator
Resources
http://www.k12.wa.us/Science/PCAs/default.aspx
Select
PCA
Select
grade level
Lessons Learned from Scoring
Transition to new standards
Grade 5
Grade 8
 Terms :
 “system” and “subsystem”
 “inputs” and “outputs”
 Terms
 “closed” and “open” systems
 “balanced” and “unbalanced”
forces
 Human needs leading to
inventions
 Weight as a measurement
of the force of gravity
 Roles of engineers and
scientists in technological
advances
Common misconceptions (gr. 5)
 When the results don’t match the prediction, students
struggle with how to report honestly
 Students think the mass of an substance changes when
the substance breaks or melts
 Students think that plants receive food from soil
 Students have misconceptions about what makes
something a producer in an ecosystem.
 Students think genetic characteristics are acquired from
the environment
Common misconceptions (gr. 8)
 Students have trouble comparing a model to the
real-world system
 Students think the mass of an substance changes when
the substance changes state or temperature
 Students have trouble calculating speed
 Students don’t understand photosynthesis
 Students struggle to see the how natural selection leads
to changes in a population over time
Going well (gr. 5)
 Variables questions
 65% to 90% of the time
 Students are including data in conclusion items
 28-37% get full points
 Between 30% and 40% give each supporting data point
 Students are attempting energy transfer and new
procedure items
 3.1% to 5.0% blanks
Going well (gr. 8)
 Variables questions
 66% to 84% of the time
 Students are including data in conclusion items
 54% get full points
 Between 55% and 58% give each supporting data point
 Students are attempting new procedure items
 7.9% - 10% blanks
Constructed responses (both)
 Students confuse different forces (gravity, push, pull,
magnetic) with forms of energy (5th: light, heat, sound,
motion, electricity; 8th: light, thermal (heat), chemical,
sound, kinetic, electrical)
 Students need more help writing their own procedures:
 Changing the manipulated variable indicated in the question
 Measuring the responding variable indicated in the question
 Students need to be more detailed when answering
application items; more specific than “it will work” or “it
must be broken, so fix it.”
Upcoming Events
 Performance Level Descriptors training
 Via Moodle website (February-March 2011)
 Scenario Writing Workshops
 5th and 8th grade (February 2011)
 High School (February 2011)
 Contrasting Groups Study (April 2011)
Upcoming Events
 Operational Rangefinding Committees
 High School (Olympia, April 2011)
 5th and 8th grade (Cincinnati, Ohio, May 2011)
 Pilot Rangefinding Committees (July 2011)
 Standard Setting (August 2011)
Opportunities for Involvement
 PEPPER email list
 PreSALTers Enthusiastically Promoting Powerful Educational
Resources
 Get updates about science happenings
 Sign up today with your e-mail address, or e-mail us and ask to
be a PEPPER
 SALT list
 Science Assessment Leadership Team
 Anyone can apply
 Respond to emails to sit on Item Development committees
 Committees are chosen to balance state representation and
content expertise
Science Assessment Office
 Elaine Wood—Secondary Specialist
 [email protected]
 Kara Monroe—Elementary Specialist
 [email protected]
 Troi Williams—Administrative Assistant
 [email protected]