Transcript Smarter Balanced Assessment - WVES | Waterville Valley

NHSAA MEETING
SMARTER BALANCED ASSESSMENT
Scott J Mantie, Ph.D.
Bureau of Accountability & Assessment
New Hampshire Department of Education
September 26, 2014
The Context and Perspective
Why assess?
 For many reasons, but the most important are:
1. Improve teaching and learning
2. Evaluate and improve programs
3. Support accountability uses
 In
serving all three of these purposes, assessments
should reflect intended learning targets, but can
also serve as an important signal to what should be
learned
Learning and Assessment


Meaningful assessment scores depend on tight
linkages among learning targets and assessment
design (ELA/Math Blueprints).
In other words, we want students to develop deep
understanding of important knowledge and skills and
we want to know if/how students are acquiring those
skills/knowledge.
Assessing for Deep Understanding



Students cannot develop deep understanding unless they
are provided opportunities on both learning and
assessment tasks.
A major component of Next Generation Learning is the
ability to solve novel problems—this requires deep
understanding!
Assessment conveys what’s important to learn (a signal)
as well as providing an opportunity to check on students’
understanding and evaluate achievement
How we got here…
The Assumption:
“Assessment must support ongoing
improvements in instruction and promote
meaningful learning experiences for
students that lead to outcomes valued by
all stakeholders.”
Creating our own…
Assessment Test Construction Assumptions:
“Apples to Apples” assessment (i.e. accommodations, glossaries,
adaptive, number of items, etc.)
 300 items/grade/content area for summative (4200 total)
 150 items/grade/content area for interim (2100 total)
 Estimated cost for fully fleshed out item: $2,000
 Summative estimated cost: $8,400,00
 Interim estimated cost: $4,200,00
 Ongoing administrative yearly costs: up to $50/student

How we got here…

SBAC Underlying Principles:
SBAC learning and assessment system grounded in Theory of Action
Research-supported instructional practice
 Technology-supported tools
 Innovative assessments
 State-of-the-art classroom support mechanisms
 Have the broadest access for the greatest number of students


Also, it is essential to have comprehensive and detailed evidence
to support the validity and reliability of SBAC assessment.
How We Got Here…

Based on Research:

Demonstrating evidence – an ongoing process

Materials:


Standards for Educational and Psychological Testing (AERA, 1999)

U.S. Department of Education’s Standards and Assessment Peer Review Guidance (2009b)

Program Evaluation Standards (Yarbrough, Shulha, Hopson & Caruthers, 2011)

Guiding Principles for Evaluators (American Evaluation Association, 2004)
Evidence-based to determine what can be done to improve learning
– this evolves operationally
How We Got Here…

SBAC: Evidence-Centered Design
Domain Analysis – content
 Domain Modeling – structure of components
 Conceptual Assessment Framework – constructs, evidence
needed and items needed to collect the evidence
 Implementation – development of the assessment items or
tasks using specifications
 Delivery – delivery of the assessment

How We Got Here…

Reliability & Validity



The test development process of Smarter Balanced seeks to
make explicit the claims that test users can make when
interpreting student scores.
Detailed & Rigorous Framework
 Planning, assessing construction, delivery, accessibility and
outcomes
 Constant process of evaluating and modifying
Constant Test Development & Test Evaluation processes
Assessments in New Hampshire
 Assessment in New Hampshire




SBAC: THE statewide assessment for mathematics and English language arts
NECAP Science: THE statewide assessment for science
NH ALPs Science: THE statewide alternate assessment for science
Dynamic Learning Maps (DLM): THE statewide alternate assessment for mathematics
and English language arts – pending G&C approval
 Data Privacy:
 NO student identifiable information collected!
 NO non-academic questions are asked!
 On-line, Adaptive Assessment vs. Paper/Pencil
 We want to take advantage of the engaging aspects that adaptive, on-line tests can
have for students.
* Assessment results: 4-6 weeks
Emphasis: Removing the test as being the focus of the assessment
NH Assessment: Important Points
 SBAC Operational Vendor Selected
 In conjunction with NEAC States (CT & VT)
 Approval through G&C in October
 Training, documentation and portion of reporting
 Interim Assessments (K12) & Digital Library ARE included
 NHDOE Assessment Communication Plan
 ALT information has begun to be disseminated
 SBAC materials are being developed (more in a moment)
 Practice & Training Tests, Scoring Guides, Documentation
See: http://sbac.portal.airast.org/
Emphasis: Removing the test as being the focus of the assessment
Student Data Privacy
 NHDOE worked closely with legislators (esp. Rep. Neal Kurk) since 2005 to
ensure that laws protect student data
 Data is protected by state and federal laws:
 Federal law prohibits disclosure of student identifiable data, even to the US ED
 State law guides NHDOE efforts:




Name and student data must be kept separate
Student ID is a completely random number (SASID)
Few at NH DOE can access student identifiable data to even assist schools
Violation of student data laws: $25,000/violation, Class b Felony and involuntary
termination of employment
 New law (189:68): prevents collection of parents name, criminal records, etc.
 Increased transparency
Student Data Privacy
 Data stored onsite at the NH DOE on secure servers
 Smarter Balanced Consortium: States own the/their data. No data
can be released without state permission
 Smarter Balanced Assessment: Online assessment has even greater
security (unique SBAC-ID – no name and no SASID)
 Primary purpose of collecting data: aid schools in improving
instruction
Online vs. Paper/Pencil
• Adaptive (online) adjusts to student vs. non-adaptive (p/p) fixed
• Online – greater security (p/p lost/misplaced paper, changes numerous
hands, etc.)
• Online – faster results (p/p must wait until all test results are in)
• Online – Performance Tasks (p/p no performance tasks)
• Online – more usability, accessibility and accommodations (p/p fewer)
• Language supports (glossaries in 10 languages & dialects, full
Spanish translations of math assessment)
• Deaf or hard of hearing receive tests in American Sign Language,
signed by recorded human interpreters
• Refreshable Braille keyboards and real-time embossers allow blind
students to receive online tests in Braille
• Accommodations uploaded (IEP & 504)
Online vs. Paper/Pencil
• Online – will assess a broader range of students because of
the many adaptive and accessibility features
• Online resources
• P/P is only temporary and shortly must move to online
• Most other assessments are going online
• Students are accustom to electronic interactions – it is part of
their current experience
Smarter Balanced: A Next
Generation Assessment

Rigorous assessments of progress toward “college and career readiness”

Summative, Interim, Performance assessments & Digital Library

Summative & Performance Assessments: 3-8, 11, Interim & Digital Library:
K-12

Common cut scores across all Consortium states

Information about grade-level achievement and growth

Valid, reliable, fair and comparable for all students (students with “significant
cognitive disabilities" will be assessed using another valid, reliable and fair
assessment)

Administered online, using multiple measures

Economy of scale: 19 states
Smarter Balanced: Interim

Interim to Provide valid, reliable and fair information about:

Student progress toward mastery of skills measured by summative assessment

Performance at content cluster level so teachers and administrators can track student
progress throughout the year and adjust instruction accordingly

Individual/group (e.g., school/district) performance at the claim level to determine
whether teaching and learning are on target

Teacher-moderated scoring of performance events as a professional development
vehicle to enhance teacher capacity to evaluate student work aligned to standards

Student progress toward the mastery of skills across all students and subgroups of
students
Smarter Balanced: Digital Library

Digital Library (AKA: Formative Assessment Resources) to Provide measurement tools
and resources to:

Improving teaching and learning

Monitor student progress throughout the school year

Help teachers and other educators align instruction, curricula and assessment

Help teachers and other educators use the Summative, Performance and Interim
assessments to improve instruction at the individual and classroom levels

Illustrate how teachers and other educators can use assessment data to engage students
in monitoring their own learning
This will be made available to NH teachers/educators soon
Assessment Details and Reactions
 Performance tasks and summative items delivered by adaptive engine,
yielding more precise, valid, and timely measures of achievement and growth
for all students to support improved decision making and a fair accountability
system.
 Pilot & Field Test Pro/Con: Generally, positive and productive assessment
experience for all students, including those from under-represented groups,
through the use of universal design, embedded digital accessibility tools and
external accommodations supported by research and practice.
 Some student anxious test takers
 Positive reaction to assessment interface
 Age appropriate items and questions?
Adaptive Assessment
 Adjusts to student’s ability by basing difficulty of future questions on previous
answers
 More accurate measurement of student achievement – particularly for high
and low performing students
 Creates an individually tailored set of questions
 Tested (mathematically) in the Field-Test

CAT implemented 2001, Supporting Research:

A Framework for the Development of Computerized Adaptive Tests, Nathan A.
Thompson, Assessment Systems Corporation, and David J. Weiss, University
of Minnesota

The Road Ahead for State Assessments, Rennie Center for Education
Research & Policy, Policy Analysis for California Education (PACE)
Here is how the
Smarter Balanced system
fits into the school year
Summative Assessments
Performance
Tasks
Optional
interim
assessments
Optional
interim
assessments
Scope, sequence and
timing of interim
assessments locally
determined
Computer-adaptive
accurate, aligned to
Common Core and built to
promote great instruction
Computer
Adaptive
Assessment
Student results
Fast, detailed
and digital with
year-on-year
measurement.
Teacher Development / NH Networks (Formative)
Instruction
School year
22
SBAC Issue Checklist:
Asking non-academic questions…
 Student privacy…
 Validity & reliability of assessment…
 Accessibility to more students…
 Timeliness of scores…

SBAC Issue Checklist:
Asking non-academic questions…
 Student privacy…
 Validity & reliability of assessment…
 Accessibility to more students…
 Timeliness of scores…

DONE!
The Assumption & Foundation
“Assessment must support ongoing
improvements in instruction and promote
meaningful learning experiences for
students that lead to outcomes valued by
all stakeholders.”
Testing Window
 Window can begin:
 66% instructional days: Grades 3-8
 80% instructional days: Grade 11
 Where States are at: ..\..\..\..\Documents\A_SBAC Materials\SBAC
State Test
Windows\2015_Test_Admin_Windows_20140814_Final.xls
 Idea/Suggestion for NH (not final):
 Grades 3-4, weeks 1-3
 Grades 5-6, weeks 4-6
 Grades 7-8 & 11, weeks 7-9
 Makeup, weeks 10-12
 Your thoughts? (Accountability Task Force)
The Test: What We’ve Learned
Are You Ready? Questions?

Smarter Balanced Information and FAQs

SBAC Network: www.education.nh.gov/networks/

Smarter Balanced pages on NHEON.org
www.nheon.org/oet/readiness/

Smarter Balanced Website: You will find the most current
information here (www.smarterblanced.org)
Are You Ready? Questions?
Your school readiness questions:
Technology Information and communication technologies (ICT)
program standards are equal to or beyond SBAC technology
requirements (i.e. Technology Plans).
 Planning room(s), schedule, installing browser(s), etc.
 Practice/Experience test teachers & students

Visit: SmarterBalanced.org
for current information
30
For more information…
603-271-3844
Please contact Dr. Scott Mantie if you have any questions!
[email protected]
603-271-3844
Information & Practice/Training Test
Resources:
http://sbac.portal.airast.org/
Communication Planning –
Assessment Transition


Stakeholder Advisory Group
 Assisting with the development of a communications plan around
standards, competencies and assessments
Preparing communication resource documents



FAQs; talking points; parent resources
Organizing and redesigning public information on the NH DOE
website
Collecting and redesigning educator resources on the NH Network