Transcript CAWT Assessment Training - Duval County Public Schools

CAWT
Assessment
Training
March 7, 2013
4:30-6:30pm
Cline Auditorium
Welcome
Purpose:
To train team leads on the Assessment
Developing and Item Writing Component of
the CAWT process.
Illustrate the connection between the
Assessment Component and the Curriculum
component of the CAWT process.
Format:
• Powerpoint
• Embedded videos to refer back to
Welcome
Expected Outcomes: Team Leads should be
able to:
• develop weekly agendas for team members
for assessment tasks
• complete the Testing Blueprint Template, the
Assessment Alignment Template and
Develop Item Specifications for necessary
courses.
• Demonstrate proficiency in item writing and
reviewing.
• Turnkey the information to team members.
Norms
•
•
•
We are all colleagues. Let’s support each
other with a win-win positive attitude.
Everyone participates; no one person
dominates.
Be present and pay close attention.
Vision
• Interim Benchmarks are being replaced by CGAs
• Everyone has a Baseline diagnostic test with roughly 32 - 40
items
• 3 Quarterly assessments for core courses with around 30
items
• 1 Mid-year assessment for non-core with around 40 items
• State assessed courses will use that test as their Post
• All courses that are not State assessed will reuse the
Baseline test as their Post test (thus not released unless a
“Form B” exists)
• Item types: MC, Gridded, and some short constructed
response (auto-graded)
• Items from past district assessments will be available for
repurposing
• 40/60 split
• Zero tolerance policy for plagiarism or copyright violation
Types of CAWTs
•
Explanation of CAWT Types
• Handout: Curriculum & Assessment Writing
Teams
Assessment Deliverables
•
•
We will need you to make and update some
supporting documents, such as Test Item
Specs and Status sheets
We are going to request that you complete
specified overall tasks by certain dates
• Handout: CAWT Assessment Deliverables
Planning Details
•
Revise Weekly Agendas to incorporate
assessment deliverable tasks
•
Handout: CAWT Weekly Agenda Tasks Assessment
Supporting Documents
•
•
•
Handout: CAWT Test Item Specifications
Handout: CAWT Test Blueprint Summary
Handout: CAWT Assessment Alignment
Insight – DAT
CAWT website:
www.duvalschools.org/ResEval/DAT/CAWT.asp
•
•
Video: Logging in to Learning Station Insight
for CAWT
Video: CAWT Item Creation
Developing Assessments
•
•
•
•
Obvious mistakes
Handout: Checklist
Copyright/plagiarism review process
Test Item Security
•
Hard copy of tests available for sign-out on
March 14th downtown on 3rd floor
Developing Assessments
•
•
•
•
„ ave you ever seen a truly awful multipleH
choice test question?
„One that is so defective that the correct answer
is either obvious, debatable, obscure, or
missing altogether?
„One that makes you wonder what the test
writer had in mind when he or she constructed
it?
„What were they thinking?
Poor Example
„ echnicle advances in farm equipment:
T
„
„A. encourage urbanization because fewer people
live on farms
„B. higher food prices
„C. revolutionizd the industry
„D. never occurs rapidly
„E. both a and c
„F. none of the above
Better Example:
„Which of the following is the best explanation of why
technical advances in farm equipment led to an
increase in urbanization?
„
„A. Fewer people were needed to run the farms.
„B. Fewer people were qualified to operate the
equipment.
„C. More people could live in the city and commute to
the farm.
„D. More people went to work at the equipment
manufacturing plants.
Difficulty Of Constructing
Multiple-Choice Items
• „Good multiple-choice test items are generally
•
•
•
•
more difficult and time-consuming to write than
other types of test items!
„They tend to favor simple recall of facts (low
complexity)
„They depend on teachers' writing ability
„They require content knowledge and skill to
create plausible distractors
„This skill may be increased through study,
practice, experience, and PRACTICE!!
Parts of a Multiple-Choice Item
„Stem: asks question that introduces the
problem
„Options: four responses, which include 1
correct answer and 3 distractors
„
„Stimulus/Text is included in the stem. This
includes artwork, graphics, tables, reading
passages, and scenarios
Keep The Options Mutually
Exclusive
Poor Example: Options “B” and “D” overlap, as do options
“A” and “C”.
„
„ ow long does an annual plant generally live?
H
„
„A. It lives for many years.
„B. It dies after the first year.
„C. It lives for more than one year.
„D. It needs to be replanted each year.
Keep The Options Mutually
Exclusive
„Better Example: The options have been rewritten to
be mutually exclusive and the stem now includes the
word “maximum”.
„Generally, what is the maximum amount of time that
an annual plant lives?
„ . one year
A
B. two years
C. one month
D. two decades
Keep The Options Homogeneous
In Content
„Poor Example: contains options testing knowledge of
state agriculture, physical features, flags, and nicknames.
If the student misses the item, it does not tell the teacher in
which of the four areas the student is weak.
„
„ hat is Idaho widely known as?
W
„
„A. The location of the tallest mountain in the United
States.
„B. The largest producer of potatoes in the United States.
„C. The state with a beaver on its flag.
„D. The Treasure State.
Keep The Options Homogeneous
In Content
„Better Example: All of the options refer to state
agriculture. So if the student misses the item, it tells the
teacher that the student has a weakness in that area.
„
„Idaho is widely known as the largest producer of which of
the following crops?
„
„A. apples
„ . corn
B
„C. potatoes
D. wheat
Cognitive Complexity
(DOK)
• „Based upon Dr. Norman L. Webb’s Depth of
Knowledge (DOK) levels
• „Assigning Cognitive Complexity to assessment items
is an essential requirement
• „There are four levels of DOK used for all content
areas.
• „However, we will assign levels 1 through 3 for our
items
o „Low- Level 1
o „Moderate- Level 2
o „High- Level 3
The Power of the Verb
• „Depth
of Knowledge is NOT
determined by the verb, but the
context in which the verb is used and
the depth of thinking required.
• „Remember DOK 1 + DOK 1 + DOK 1
= DOK 1.
• „DOK is about complexity, not difficulty.
Same Verb—Three DOK
Levels
„ igh/DOK 3- Which correctly describes a model that you
H
might use to represent the relationships that exist within
the rock cycle? (requires deep understanding of rock cycle
and a determination of how best to represent it)
„Moderate/DOK 2- Which accurately describes the
difference between metamorphic and sedimentary rocks?
(requires cognitive processing to determine the differences
in the two rock types)
„Low/DOK 1- Which list correctly describes three
characteristics of metamorphic rocks? (simple recall)
How Can This Item Be
Improved?
Developing Assessments
•
•
•
•
•
Obvious mistakes
Handout: Checklist
Items from the interview performance task
(screenshot or slide)
Copyright/plagiarism review process
Test Item Security
•
Hard copy of tests available for sign-out on
March 14th downtown on 3rd floor
Google Drive
•
Drive holds all
sorts of documents
•
•
•
Word, Excel,
Adobe PDF,
images, and
Google Docs
Google Docs are
collaborative
How to get to Help
Google Drive
Questions?
Contact the Assessment Experts at:
390-2976
Julian Davenport [email protected]
Tavy Chen
[email protected]
Malinda Bachelor [email protected]
Shawna Bejelis
[email protected]
Janis Browning