Transcript Progress Monitoring Goal Setting/Graphing Administration

Progress Monitoring
Goal Setting/Graphing
Administration of ORF/Maze
Keith Drieberg, Director of Psychological Services
John Oliveri, School Psychologist
Cathleen Geraghty, School Psychologist
Overview
(Continued)
Comparison of Terms Between Systems
Term
AIMSWEB
District
Grade
Level
Frequency
I/II/III
Levels
Tiers
III (Red Zone)
Progress
Monitor
Intensive
Instructional
Weekly
II (Yellow Zone)
Strategic
Targeted
Assigned
1 x Monthly
I (Green Zone)
Benchmark
School-wide
Screenings
Assigned
3 Times a Year
Screening
 Screening allows for quick identification of
students that are at-risk for academic
difficulties
 Different Screening Measurements
 When to do Screenings
 Who is Screened?
Screening When and Why?
 1st Screening - identification
 2nd Screening - prediction and
identification
 3rd Screening - goal attainment
Which Measures
 Kindergarten
• Fall - Letter Naming Fluency (LNF)
• Winter - LNF & Phoneme Segmentation Fluency (PSF)
• Spring - LNF & PSF
 1st Grade
• Fall - Nonsense Word Fluency (NWF) & Oral Reading
Fluency (ORF)
• Winter - NWF & ORF
• Spring - NWF & ORF
 2nd Grade
• Fall - ORF
• Winter - ORF
Spring - ORF
Administration for Early Literacy
Letter Naming Fluency
Example: Letter Naming Fluency
QuickTime™ and a
H.263 decompressor
are needed to see this picture.
Phoneme Segmentation Fluency
Example: Phoneme Segmentation Fluency
QuickTime™ and a
H.263 decompressor
are needed to see this picture.
Nonsense Word Fluency
Example: Nonsense Word Fluency
QuickTime™ and a
H.263 decompressor
are needed to see this picture.
Who is Screened?
 Ideally, all students are screened in the
Fall, Winter, and Spring
 Students scoring below the 25th
percentile are considered at-risk and
should be progress monitored
What Is & Why Progress Monitor
 What is Progress Monitoring
• Formative assessment tool
 Why Progress Monitor
• Tell us whether students are profiting from
the curriculum, and whether or not an
intervention is effective for that particular
student
 Some Everyday Examples of How We
Use Informal Progress Monitoring
What is Progress Monitoring

Technically adequate
•

Capacity to model growth
•

instructionally eclectic so the system can be used with any type of
instruction or curriculum
Capacity to inform teaching
•

scores should change when students are learning
Independence from specific instructional techniques
•

able to represent student achievement growth within and across academic
years
Treatment sensitivity
•

reliability and validity
should provide information to help teachers improve instruction
Feasibility
•
must be doable
Why Progress Monitor
 Screening is not enough for some students because they may
be in ineffective programs for too long.
 Progress monitoring allows for individualized goals to be
written and determination of a feasible amount of time for the
goal to be reached.
 Allows for an analysis of student need and resources for
determining progress monitoring frequency.
• Programs that are more intensive (e.g., special education), should
monitor student outcomes more frequently that 3x per year.
More Frequent Evaluation
Progress Monitoring Plan
 Progress Monitoring Plan
• No Learning Center
 Monitor at-risk students, once a month at
grade level
• Learning Center
 Monitor at-risk students
• once a month at grade level
• every week at instructional level
 Collect data on Wednesday or
Thursday
Goal Setting - General
 Each student should have a year-long goal
(you want the student to be performing at the
50th percentile on grade level material)
 Every goal should have:
•
•
•
•
Time frame (when the goal should be reached)
Behavior (what the desired level of performance is)
Condition (which measure and where you obtained it from)
Criterion (which grade level passage you are using - if
applicable)
Sample ORF Goal
 Oral Reading Fluency (Fluency)
• In (#) of weeks (student name) will
read (#) word correct in 1 minute as
measured by a (grade ___ ) (DIBELS or
AIMSWeb Oral Reading Fluency
Measure).
Norms and Growth Rates
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Year-Long Goal
 Expected weekly growth
• If the student is not at-risk use the
average rate of growth
• If the student is at-risk use the ambitious
rates of growth (e.g., below the 25th
percentile)
 Multiply the growth rate by 36 (weeks
of school) and add it to the baseline
level of performance
Year-Long Goal Example
 In the fall, a 4th grade student’s
instructional level is 2nd grade and the
student reads 22 words correct per
minute. Ambitious growth for 2nd grade
is 2 words per week
• 22 + (2 x 36) = 94
• The year-long goal would be 94 words
correct per minute
Short-Term Goal
 All students being progress monitored
need shorter-term goals
 These goals should use ambitious
rates of growth to help the students
‘catch-up’
Short-Term Goals
 Determine what level of performance is
needed to move up a percentile rank (e.g.,
10th to 25th; 25th to 50th)
 Take the difference in performance (e.g.,
how many words correct the student will
need to move from the 10th to 25th
percentile), and divide by the ambitious rate
of growth
• This product will be the approximate number of
weeks needed for the student to reach the next
percentile rank
Short-Term Goals
 If using early literacy measures (LNF,
PSF, NWF) move up instructional level
until the student has mastered that
skill, then move up to the next skill
Literacy & Writing Measures - Suggested Timeline for Administration
M AZE (Comprehension)
Oral Reading Fluency
Spelling
Written Expression
Nonsense Word Fluency
Phoneme Segmentation Fluency
Letter Sound Fluency
Letter Naming Fluency
Beg
Initial Sound Fluency
M id
End
Beg
Preschool
M id
Kindergarten
End
Beg
M id
First Grade
End
Beg
M id
Second Grade
End
Beg
M id
End
Third Grade and Abov e
Short-Term Goals
 In the Winter, a 3rd grade student’s
instructional level is 2nd grade and the
student reads 29 words correct per
minute.
• What short-term (progress monitoring)
goal would you write for this student?
Charting Progress
 Once you set the goal, you need to chart progress to see if the
student is responding to the change in instruction
Graphing Progress Monitoring Charts:
What Goes on a Graph:
General Conventions
Corresponding Chart Information
Label for the Horizontal Axis (X-axis)
Lavel for the Vertical Axis (Y-axis)
Progress Monitoring Dates
Number Frequncy Value
Line separating base line data from intervention data; also
used when intervention is modified
The first 'x' is placed at median base line data point. Second
'x' is placed at the point where the student is expected to
be performing at after a certain period of time or at the
end of an intervention.
A line connecting the 'x''s stated above represents studetn's
expected or desired rate of progress. In general the goal
line slants up concerning academics and slants down when
behavior is an issue.
Represents student's estimated rate of actual progress
based on charter performance data
Phase Change Line
Goal
Goal Line or Aimline
Trend Line
Some Basic Procedures:
For Base Line / Current Level of Performance, Use Median (Middle) Score of the Three Data Points
Minimum of Six to Eight Progress Monitoring Response to Intervention (RtI) Data Points Needed
Evaluating Goal Attainment
 Process of assessing student achievement during instruction
to determine whether an instructional program is effective for
individual students.
• When students are progressing, keep using your
instructional programs.
• When tests show that students are not progressing, you
can change your instructional programs in meaningful
ways.
• Has been linked to important gains in student achievement
(Fuchs, 1986) with effect sizes of .7 and greater.
Trendline
Goal Line
Trendline
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Goal Line
Goal line
Trend Line
Special Education
How can Progress Monitoring be
used in Special Education?
Educational Benefit
 Progress monitor at both instructional
and grade level
• Grade Level Goals
 How is the student performing relative to
his/her peers?
• Instructional Level Goals
 How is the student’s performance changing as
a function of the current instruction /
intervention?
• Instructional level progress monitoring is sensitive
to growth.
General Goal Setting
 Long-Term Goal
• The long-term goal should be either to
move the student:
 Up a full grade level (Ex. 50th percentile on a
1st grade probe to 50th percentile on a 2nd
grade probe)
 Up to a higher skill (Ex. Letter Naming Fluency
to Letter Sound Fluency)
General Goal Setting (continued)
 Short-Term Goals (Benchmark 1 & 2)
• The benchmark goals should be to move
the student up a percentile level within
their instructional level (could be the same
as grade level). (Ex. Move from the 25th to
the 50th percentile).
First Example
(Topic Two )
The Movie “City Slickers”
The Cattle Drive----
That Great Line:
 “Don’t Know Where We Are At;
 Don’t Know Where We Are
Going;
 But We Are Sure Are Making A
Lot Of Progress”
That Great Line:
A Case For Level I
Bench Mark Screenings At Assigned Grade Levels For
All Students
Don’t Know Where
We Are At
Don’t Know Where
We Are Going
Not Sure Of
Progress
No Base Line
Point
No Goal
Point
No Goal Line
or Aimline
What We Already Know:
A Case For Level II Strategic/ Targeted
Assigned Grade Level Progress Monitoring On a Regular Basis
The Earlier We Start
Interventions; The Less
Behind the Student is in
Comparison to Their
Grade Level Peers; and
The Sooner A Student
Starts to Makes Progress
Their Peer Growth Rate Is
High And Student “Catches
Up Quickly” (within about a
year) Because They are Not
Too Far Behind
(Assigned Grade Level
Progress Monitoring /
Strategic/Targeted – ideal 1 x
Monthly)
What We Need to Think More About:
A Case For Level III Progress Monitor/Intensive
Instructional Level Progress Monitoring
Higher Grade Levels
(Especially 4th Grade +)
Student Significantly
Farther Behind Assigned
Grade Level Peers
Need Long Term Goals
(Takes Longer Than One
Year to “Catch Up”)
Kids Give Up On Themselves
(Highest Drop Out Rate 9th Grade)
We Expect Less of Students at
Higher Grade Levels and Blame
Others
(Instructional Grade Level
Progress Monitoring To Know If
Intervention is Workng; Appreciate
Progress Sooner; and Receive
Frequent Feedback )
What We Need to Think More About:
Instructional Grade Level Progress Monitoring
Higher Grade Levels
(Especially 4th Grade +)
Student Significantly
Farther Behind Assigned
Grade Level Peers
Need Long Term Goals
(Takes Longer Than One
Year to “Catch Up”)
In Addition,
Peer Growth Rate Is Slower at Higher
Grade Levels
If We and the Student Continue
Ambitious Goals Each Year
The Student Will Make Significant
Gains and Eventually “Catch Up”(Instructional Grade Level Progress
Monitoring)
Why We Need Both Assigned Grade
Level and Instructional Grade Level
Goals and Progress Monitoring
ASSIGNED GRADE LEVEL
TESTING SUMMARY SHEET
Last Name4
First Name4
Teacher2
Primary
Intervention2 Tutor2
Advancing
Alpha
Anaximander
Andrew
Alice
Albert
Smith
Smith
Applegate
Learning Cent
Learning Cent
Bef School
Helpful
Helpful
Goodguy
Last Year
Quintile
3
2
1
Beginning/Fall
Middle/Spring
End/Spring
ORF 1 Screening
ORF 2 Screening2
ORF Screening3
79
65
60
95
90
75
110
105
90
6th Grade 90 WC
7th Grade 120 WC
8th Grade 150 WC
End of Year
Goal
109
95
90
Administration of ORF and
MAZE
ORF (R-CBM) Administration
 Student’s read the passage aloud for 1
minutes
 Number of words read correct and
number of errors are counted
• WRC/errors
Area Assessed
Timing
Test Arrangement
What is Scored
CBM Oral
Reading Fluency
(R-CBM)
1 minute
Individual
# of Words Read
Correct (WRC)
and the # of
Errors
Administration and Scoring of
R-CBM
 What Examiners Need to Do
• Before testing students
• While testing students
• After testing students
Things You Need Before Testing
Standard Reading
Assessment Passage
Student Copy:
• No numbers
• Between 250-300
words (exception:
1st grade)
• An informative first
sentence
• Same font style and
size
• Text without
pictures
Things You Need Before Testing
(Continued)
Standard Reading
Assessment
Passage Examiner
Copy:
Pre-numbered so they
can be scored quickly
and immediately.
R-CBM Standard Directions for 1
Minute Administration
1) Place the unnumbered copy in front of the student.
2) Place the numbered copy in front of you, but shielded so the student cannot see
what you record.
3) Say:
When I say ‘Begin,’ start reading aloud at the top of this page. Read
across the page (DEMONSTRATE BY POINTING). Try to read each
word. If you come to a word you don’t know, I will tell it to you. Be
sure to do your best reading. Are there any questions? (PAUSE)
4) Say “Begin” and start your stopwatch when the student says the first word. If
the student fails to say the first word of the passage after 3 seconds, tell them
the word, mark it as incorrect, then start your stopwatch.
5) Follow along on your copy. Put a slash ( / ) through words read incorrectly.
6) At the end of 1 minute, place a bracket ( ] ) after the last word and say, “Stop.”
7) Score and summarize by writing WRC/Errors
“Familiar” Shortened Directions
When students are assessed frequently and know the
directions.
Say:
When I say ‘Begin,’ start reading aloud at the top of this page.
Items to Remember
• Emphasize Words Read Correctly
(WRC). Get an accurate count.
• 3-Second Rule
• No other corrections
• Discontinue rule
• Be polite
• Best, not fastest
• Interruptions
What is a Word Read Correctly?
• Correctly pronounced words within context
• Self-Corrected Incorrect Words within 3 seconds
What is an Error?
• Mispronunciation of the Word or Substitutions
• Omissions
• Stops or struggles with a letter for more than 3 seconds
(examiner provides correct word)
What is not Incorrect? (Neither a WRC
or an Error)
• Repetitions
• Dialect Differences
• Insertions (Consider them Qualitative Errors)
Calculating and Reporting Scores
• Count the total number of words the student read
• Count the number of errors and subtract
• Report in standard format of WRC/Errors (72/3)
Example of Calculating Scores
• Juan finished reading after 1 minute at the 145th word, so
he read 145 words total
• Juan also made 3 errors
• Therefore, his WRC was 142 with 3 errors
Reported as 142/3
Practice Exercise 1: Let’s Score
QuickTime™ and a
H.263 decompressor
are needed to see this picture.
Practice Exercise 1: Answer Key
This student read 72 WRC/8 Errors
Practice Exercise 2: Let’s Score
QuickTime™ and a
H.263 decompressor
are needed to see this picture.
Practice Exercise 2: Answer Key
This student read 96 WRC/4 Errors
Practice Exercise 3: Let’s Score
QuickTime™ and a
H.263 decompressor
are needed to see this picture.
Practice Exercise 3: Answer Key
This student read 141 WRC/2 Errors
Curriculum Based Measurement Reading Maze
CBM Maze is designed to provide educators a more complete
picture of students’ reading skills, especially when comprehension
problems are suspected.
Area
CBM Maze
Reading
Timing
3 minutes
Test Arrangements
What is Scored?
Individual, Small
Group, or Classroom
Group
# of Correct Answers
Curriculum Based Measurement Reading
Maze (Continued)
• Maze is a multiple-choice cloze task that students complete
while reading silently.
• The students are presented with 150-400 word passages.
• The first sentence is left intact.
• After the first sentence, every 7th word is replaced with three
word choices inside a parenthesis.
• The three choices consist of,
1) Near Distracter
2) Exact Match
3) Far Distracter
An example of CBM Maze
Administration and Scoring of CBM Maze
What Examiners Need to Do . . .
• Before testing students
• While testing students
• After testing students
Items Students Need Before Testing
What the Students
Need for Testing:
• CBM Maze practice
test (optional)
•
Appropriate CBM
Maze passages
• Pencils
Items Administrators Need Before Testing
What the Tester Uses
for Testing:
• Stopwatch
• Appropriate CBM
Maze Answer Key
• Appropriate
Standardized
Directions
Setting up Assessment
Environment
Assessment environments are flexible and could include…
• The classroom if assessing the entire class.
• A cluster of desks or small tables in the classroom for
small group assessment.
• Individual desks or “stations” for individual assessment.
Things You Need to do While Testing
Follow the Standardized Directions
• Attach a cover sheet that includes the practice test so that
students do not begin the test right away.
• Do a simple practice test with younger students.
• Monitor to ensure students are circling answers instead of writing
them.
• Be prepared to “Prorate” for students who may finish early.
• Try to avoid answering student questions.
• Adhere to the End of Timing.
CBM Maze Standard Directions
1)
2)
Pass Maze tasks out to students. Have students write their names on the
cover sheet, so they do not start early. Make sure they do not turn the page
until you tell them to.
Say this to the student (s):
When I say ‘Begin’ I want you to silently read a story. You will
have 3 minutes to read the story and complete the task. Listen
carefully to the directions. Some of the words in the story are
replaced with a group of 3 words. Your job is to circle the 1 word
that makes the most sense in the story. Only 1 word is correct.
3)
Decide if a practice test is needed. Say . . .
Let’s practice one together. Look at your first page. Read the
first sentence silently while I read it out loud: ‘The dog, apple,
broke, ran after the cat.’ The three choices are apple, broke, ran.
‘The dog apple after the cat.’ That sentence does not make
sense. ‘The dog broke after the cat.’ That sentence does not
make sense. ‘The dog ran after the cat.’ That sentence does
make sense, so circle the word ran. (Make sure the students circle
the word ran.)
CBM Maze Standard Directions
(Continued)
Let’s go to the next sentence. Read it silently while I read it out
loud. ‘The cat ran fast, green, for up the hill. The three choices
are fast, green, for up the hill. Which word is the correct word
for the sentence? (The students answer fast)
Yes, ‘The cat ran fast up the hill’ is correct, so circle the correct
word fast. (Make sure students circle fast)
Silently read the next sentence and raise your hand when you
think you know the answer. (Make sure students know the correct
word. Read the sentence with the correct answer)
That’s right. ‘The dog barked at the cat’ is correct. Now what do
you do when you choose the correct word? (Students answer
‘Circle it’. Make sure the students understand the task)
That’s correct, you circle it. I think you’re ready to work on a
story on your own.
CBM Maze Standard Directions
(Continued)
4)
Start the testing by saying . . .
When I say ‘Begin’ turn to the first story and start reading silently.
When you come to a group of three words, circle the 1 word that
makes the most sense. Work as quickly as you can without
making mistakes. If you finish a/ the page/first side, turn the page
and keep working until I say ‘Stop’ or you are all done. Do you
have any questions?
5)
6)
7)
8)
9)
Then say, ‘Begin.’ Start your stopwatch.
Monitor students to make sure they understand that they are to circle only 1
word.
If a student finished before the time limit, collect the student’s Maze task and
record the time on the student’s test booklet.
At the end of 3 minutes say: Stop. Put your pencils down. Please close
your booklet.
Collect the Maze tasks.
CBM Maze Familiar Directions
1)
After the students have put their names on the cover sheer, start the testing by
saying . . .
When I say ‘Begin’ turn to the first story and start reading silently.
When you come to a group of three words, circle the 1 word that
makes the most sense. Work as quickly as you can without
making mistakes. If you finish a/ the page/first side, turn the page
and keep working until I say ‘Stop’ or you are all done. Do you
have any questions?
2)
3)
4)
5)
6)
Then say, ‘Begin.’ Start your stopwatch.
Monitor students to make sure they understand that they are to circle only 1
word.
If a student finished before the time limit, collect the student’s Maze task and
record the time on the student’s test booklet.
At the end of 3 minutes say: Stop. Put your pencils down. Please close
your booklet.
Collect the Maze tasks.
CBM Maze Scoring
What is correct?
The students circles the word that matches the correct word
on the scoring template.
What is incorrect?
An answer is considered an error if the student:
1) Circles an incorrect word
2) Omits word selections other than those the student
was unable to complete before the 3 minutes
expired
Making Scoring Efficient
1) Count the total number of items up to the last circled
word.
2) Compare the student answers to the correct answers on
the scoring template. Mark a slash [/] through incorrect
responses.
3) Subtract the number of incorrect answers from the total
number of items attempted.
4) Record the total number of correct answers on the cover
sheet followed by the total number of errors (e.g., 35/2).
CBM Maze Prorating
If a student finishes all the items before 3 minutes, the score
can be prorated .
1) When the student finished must be recorded and the number
of correct answers counted. For example, the student may
have finished in 2 minutes and correctly answered 40 items.
2) Convert the time taken in seconds. (2 minutes = 120
seconds)
3) Divide the number of seconds by the number correct. (120/40
= 3)
4) Calculate the number of seconds in the full 3 minutes. (3
minutes = 180 seconds)
5) Divide the number of full seconds by the calculated value
from step 3. (180/3 = 60)
Practice Exercise 1: Let’s Score
Practice Exercise 2: Let’s Score
Interrater Reliability
 Why do we need to calculate this?
• Reliability of reporting for student data
• Protect against observer drift
 Agreements / (Agreements +
Disagreements) x 100 = Interrater
reliability
Practice Administration and
Reliability Checks
 Page 21 in the Administration and
Scoring of Reading Curriculum-Based
Measurement (R-CBM) Training
Workbook
• Break into groups of three
 1 person read
 2 people score (primary scorer/observer) calculate Interrator reliability among the two
scorers
 Rotate until every person has been a reader, a
primary scorer, and an observer