Transcript BEST Plus and BEST Literacy - RAETAC

BEST Plus and BEST Literacy
Directors’ Training
Feb. 14, 2013
Columbia, SC
What are BEST
Plus and BEST
Literacy?
BEST Plus and BEST Literacy Overview
Who can administer BEST
Plus?
• BEST Plus Test Administrators
must complete and pass a six-hour
training, led by a certified BEST
trainer. They should then
administer 10 practice tests and
observe one experienced Test
Administrator before giving a “real”
test.
• Anyone who has not been
approved as a test administrator by
CAL (Center for Applied
Linguistics) should not assess
students using BEST Plus.
After BP initial training, the Director
and the prospective Test Administrator
will receive:
Who can administer BEST
Plus?
• Newly approved BEST Plus Test
Administrators must complete a
Scoring Refresher training six weeks
to three months after their initial
training and show competency in
scoring all three areas of the test.
• Thereafter, Test Administrators must
attend a Scoring Refresher training
annually and demonstrate scoring
competency.
After each Refresher training, the Director
and the approved Test Administrator will
receive:
Who can administer BEST
Literacy?
• BEST Literacy may be administered
with no formal training after the Test
Administrator has reviewed and
studied the BEST Literacy Test
manual.
CAL recommends that all Test
Administrators (BEST Plus and
BEST Literacy) have a strong
command of the English language
and be familiar with working with
second language learners.
Now to the
“Nuts and
Bolts” of
BEST Plus
Overview
1.
What is BEST Plus?
2.
What does BEST Plus look like?
3.
What is the content of BEST Plus?
4.
How is BEST Plus scored?
5.
What do we know about the reliability and
validity of BEST Plus?
6.
How is BEST Plus used?
What is BEST Plus?
• BEST Plus is . . .
– a performance-based
assessment (an individually
administered face-to-face oral
interview) . . .
– that assesses functional oral
language skills (interpersonal
communication) of adult English
language learners . . .
– and is designed with current
assessment needs in mind.
Objectives for BEST Plus
development
• Respond to the assessment and
accountability needs of adult ESL
programs.
– Produce a test that is short and
practical.
– Assess learner language for a
variety of purposes and
stakeholders.
– Increase accuracy in measuring
oral language proficiency.
– Provide multiple forms for
pretesting and post testing.
BEST Plus uses current
measurement techniques to
improve assessment.
Measurement is based on Item
Response Theory (IRT).
All questions assess listening and
speaking ability on the same
underlying measurement scale.
Multiple scores are awarded for
each question.
IRT ensures that results of
pretests and post tests are
comparable.
BEST Plus provides a large
item pool.
• The item pool is large enough
that examinees are
administered different sets of
questions with each
assessment.
– The ability to memorize test
questions is minimized.
– BEST Plus can be used for
pretesting and post testing.
(No “forms”)
Currently, the
item pool
contains 256
questions
What does BEST Plus look like?
(Computer-adaptive version)
• Test items for the test
administrator to ask appear on
the computer screen.
• If a test item requires a picture,
examinees view the picture on
the computer screen.
• Test administrators enter
scores directly into the
computer.
Picture Cue
“Tell me about this picture.”
Beginning level test question
Advanced level test question
What is the content of BEST Plus?
Content: 13 specific domains
nested in three general domains
Personal
Occupational
Personal
Getting a job
identification
On the job
Health
Family/Parenting
Consumerism
Housing
Recreation/
Entertainment
Public
Civics
Community
services
Transportation/
Directions
Weather/Seasons
Education
ESL Scope and Sequence of Skills
These topics are
representative of
the ones covered
in the ESL
Scope and
Sequence of
Skills.
How is BEST Plus scored?
The test administrator uses the BEST
Plus Scoring Rubric to guide scoring
decisions.
The examinee’s response to each item
is scored on three aspects of language:
– Listening Comprehension
– Language Complexity
– Communication
Three aspects of language
• Listening Comprehension: How
well did the examinee understand
the setup and question?
• Language Complexity: How did
the examinee organize and
elaborate the response?
• Communication: How clearly did
the examinee communicate
meaning?
Path through the computeradaptive version
• Following fixed warm-up questions,
examinees are asked questions drawn
from several thematically based folders.
• The test administrator scores each
question after the examinee responds.
• After each set of scores is entered, the
computer updates its estimate of the
examinee’s ability and chooses folders
and questions as appropriate.
• The test gives the minimum number of
items to ascertain an examinee’s
proficiency level.
• Users can immediately receive a
complete score report.
What do we know about the reliability
and validity of BEST Plus?
• Rigorous development
procedures
– Feasibility study (1999-2000)
– Initial development (2000-2001)
– Pilot, small-scale field test, initial
reliability study (2001)
– Revisions (2001-2002)
– Pilot, full-scale field test, reliability
study, standard- setting study (2002)
– Finalization of training materials,
ancillary materials, further refinements
(2003-2004)
– Field test for test update (2011-2012)
Full involvement of stakeholders
•
•
•
•
Oversight from U.S. Department
of Education, Office of
Vocational and Adult Education
(OVAE)
Technical Working Group
(TWG), comprised of
researchers, state directors, and
local program directors and
practitioners
Item writers, comprised of
experienced adult ESL teaching
professionals
Instructors and students in the
field
Full-scale field test
• Participants:
– Eight states (DE, FL, IL, MA, MD,
OR, PA, VA) and the District of
Columbia
– 23 programs
– 40 test administrators
– 2,420 examinees
Reliability study
• 32 adult ESL students
• Two testing rooms (A, B)
– Test Administrator (project staff
member)
– Observer/Co-Scorer (project
staff member)
– Observer/Co-Scorer (novice
scorer)
• Each student was tested in Room
A or B, then immediately retested
in the other room.
Test/Retest reliability
Between two test administrations
(different rooms)
.89
Average interrater
agreement
Within one administration
(same room)
Room A
(3 raters)
Room B
(3 raters)
.98
.97
Ability estimation
After each question, the computer
program underlying BEST Plus
estimates the examinee’s ability
based on scores awarded on the
current and all previous questions.
With each estimation, the
accuracy of the measurement
increases.
Goal: To level off in estimation
with an acceptable level of
accuracy.
Convergence
1.750
Student: Example A (ID#25 )
Administrator: A
1.625
Administrator: B
1.500
1.375
1.250
1.125
1.000
0.875
Ability Estimates
0.750
0.625
0.500
0.375
0.250
0.125
0.000
-0.125

-0.250


-0.375


-0.500
-0.625







M4
M5
-0.750





M7
M8


















M14
M15
M16
M17
M18
-0.875
-1.000
-1.125
M1
M2
M3
M6
M9
M10
Questions
M11
M12
M13
The BEST Plus score report
• Information includes
– BEST Plus scale score
– Student Performance Level
(SPL)
– Diagnostic information
BEST Plus and NRS Functioning Levels
NRS Level Descriptor
Scaled Score
Beginning ESL Literacy
400 and below
Low Beginning ESL
400-417
High Beginning ESL
418-438
Low Intermediate ESL
439-472
High Intermediate ESL
473-506
Advanced ESL
507-540
How is BEST Plus used?
• BEST Plus allows programs to assess
the everyday listening and speaking
skills of English language learners for
a variety of purposes:
–
To determine placement into a
program.
–
To track progress.
–
To provide feedback to learners and
teachers to improve instruction.
To evaluate program effectiveness.
–
–
To report outcomes (e.g., pretesting
and post testing to meet NRS
requirements).
BEST Plus Score Management
•
The Score Management Software
program provides users with the
ability to manage data contained
within the BEST Plus Scores
Database.
•
Features include
– Importing and exporting student data.
– Backing up the scores database.
– Producing a complete score report
for the print-based version of the test.
– Purchasing more test
administrations.
For detailed
information on this
topic, please refer
to the “Score
Management
Software User
Guide” located in
the BP Test
Administrator
Guide
BEST Plus:
Print Based Version
• May be administered without a
computer; useful for sites without
computer access or in the event of a
computer malfunction.
• However, a computer is necessary to
determine a scaled score for an
individual student.
• To get a scaled score for the print
based version, the test administrator
must access the “Print-based
scoring” tab on the BEST Plus
testing CD. They must then input
scores from each test item to get a
scaled score.
Path through the printbased version
• The test administrator administers and
scores the locator test (fixed warm-up
items and two high-end discriminators).
• Examinees answer questions from the
appropriate level test based on the chart
in the examinee test booklet.
• The test administrator totals the raw
scores and finds the approximate SPL
range.
• The raw scores are entered into the
computer via the BEST Plus Score
Management Software to obtain a
complete score report.
What does BEST Plus look like?
(Print-based version)
• Test
– Three forms
– Within each form: locator test +
three level tests
• Materials
– Picture cue book
– Examinee test booklet (scripts
and score sheet combined)
Print-based version
BEST Literacy
• BEST Literacy assesses a student’s
English reading and writing skills.
• Life skills based.
• BEST Literacy may be administered
to a group or to an individual.
• Again, no formal training is required
to administer this test; Test
Administrator must study the BEST
Literacy training manual to learn to
score this test.
Also aligns
with the ESL
Scope and
Sequence of
Skills
BEST Literacy
• Comes in three parallel forms
(B,C and D).
• Should (generally) only be
administered to students with
oral proficiency (on BEST
Plus) of Level 3 or higher.
• Takes 60 minutes to complete.
• Must be hand-scored.
Why use BEST Literacy with
your ESL students?
• Use to assess reading and writing
instruction in the ESL classroom.
• Use as a method to show reading
and writing progress in ESL
students.
• Use with students who test at Level
7 or higher on BEST Plus to retain
them as ESL students.
• Aligns with instruction in the ESL
classroom better than TABE.
Pre- and Post-Testing
• Students should be pre-tested on
BEST Plus (and BEST Literacy, if
necessary/desired) PRIOR to
attending 12 hours of class.
• Students must attend a MINIMUM
of 60 hours before post-testing
(different than the 40 hour
requirement in ABE)
• Pre- and post-test must use the
same assessment (either BEST
Plus OR BEST Literacy)
ESL Student Assessment
Progression
• BEST Plus
1
• If Level 7 or above (OR as an alternative
assessment), then…..
• BEST Literacy
2
• If Level 7 or above, then…..
• TABE
3
• Student now classified as ABE student.
BEST Assessments and LACES
1.Choose or
enter a student
and then click
“Assessments”
2. Click “Add
New”
1.
2.
BEST Assessments and LACES
To input BEST Literacy Scores:
1. Click BEST Literacy
2. Select a form (B,C, or
D)
3. Click (not applicable)
When the Continue button
becomes active, click Continue.
1
2
3
BEST Assessments and
LACES
1.Input scores and
date
2. Click save
1.
2.
Practical Information
• You must purchase tests directly from CAL
(see sample order form).
• These tests are added to your program’s
“virtual bank.”
• Approved Test Administrators are given a
unique username and password to access
the “bank.”
• Tests are activated in groups of 20.
• Tests are connected to the computer on
which they are activated.
• BEST Plus tests cost approximately
$1.50/test (depending on quantity ordered).
• Print-based tests are the same price and
use one “banked” test (the same as a
Computer Adaptive test).
Practical Information, cont’d
• To order BEST Literacy tests,
you also must order directly
from CAL (see sample order
form).
• Each test includes a
consumable test booklet and
answer form.
• Tests cost about $2.50/test.
Online Orders
For expedited and
credit card orders,
go to:
www.calstore.cal.org
Still have questions?
Contact your Regional ESL
Consultant:
Region 1
• Monica Rodriguez [email protected]
Region 2
• Karen Hodges [email protected]
Region 3
• Rebecca Carter [email protected]
Region 4
• Jan Camp [email protected]
Region 5
• Wendy Griffin [email protected]
For more information
Call:
1-866-845-BEST (2378)
Email:
[email protected]
Write:
Center for Applied Linguistics
Attn: BEST Plus
4646 40th Street, NW
Washington, DC 20016-1859
Visit the Web site: www.best-plus.net
Questions/
Comments?
Resources for this
presentation
• Center for Applied Linguistics
(www.cal.org)
• SC Adult Education
Assessment Policy Manual
• BEST Plus Test Administrators
Guide