Transcript PMRN Reports - The Florida Center for Reading Research

Florida Assessments for
Instruction in ReadingOverview of the K-2 System
Barbara R. Foorman, Ph.D.
Florida Center for Reading Research
Florida State University
FAIR Facts






Florida Center for Reading Research (FCRR) began
development in 2007 at request of FL Dept. of Ed
FCRR conducted item tryouts in 2007-08 and
implementation study in 25 FL schools in 2008-09
Launched statewide in fall 2009: voluntary except for
FLKRS; over 1.6 million students assessed
Administered three times a year with additional ongoing
measures available
Assists Florida in its efforts to provide additional
computer-based assessment in the future
Offered free of charge to Florida school districts
2
K-2 Teacher Survey Results
70%+ of 150 teachers in implementation study responded
•
67% completed K-2 FAIR in 5 days or less
•
89% found instructions clear and EST easy to use
•
90% agreed or strongly agreed that FAIR results helpful
in identifying strengths and weaknesses of students
•
78% said useful in forming instructional groups
•
80% would recommend FAIR to another teacher
•
86% found FAIR had comprehensive approach to reading
•
75% agreed results inform parents about child’s reading
3
Benefits of using FAIR



Valid and reliable tool for monitoring student
progress
Strong psychometric test properties form the
basis for all assessments
Broad Screen tasks are directly related to
Florida standards and predict to end of year
outcome measures (SESAT, SAT-10, and
FCAT)
4
Benefits of using FAIR




Access to the Progress Monitoring and Reporting Network
(PMRN) for recording student scores (by hand or scoring tool
in K-2, automatically in 3-12)
Access to the PMRN reports that assist teachers in analyzing
student data to better inform instruction
Links to instructional resources for teachers (PMRN reports,
search tool, program specific professional development)
Passages equated for difficulty to allow for accurate progress
monitoring and comparison of student performance
5
FAIR Administration
 Internet-based options
 Teacher Administered in K-2
 Electronic Scoring Tool – paper and pencil
scoring also available
 Computer-administered in grades 3-12
 Adaptive assessment provides precise
ability placement for reading
comprehension tasks to monitor students at
their instructional level
6
Reading Priorities in Florida
Six Components
4 Types of
Assessment
High-Quality
Initial Instruction
Immediate
Intensive
Intervention
Oral Language
Screening
Background
Knowledge
Extended Time
Phonological
Awareness
Progress
Monitoring
Motivation
Flexible Grouping
Phonics
Diagnosis
Explicit, Systematic Accommodations
Fluency
Outcome
Differentiated,
Scaffolded
Vocabulary
Reading/Writing
Connection
Comprehension
Print-rich
More frequent
Progress
Monitoring
7
Converging Evidence
Valid and reliable predictors of risk for
reading difficulty are:






Print concepts (preK; early K)
Letter name knowledge (early K)
Phonological awareness and letter sounds (K-1)
Rapid naming of letters (end of K to early G1)
Word recognition (G1 and beyond)
Vocabulary
(Fletcher et al., 2002; NELP, 2009; Scarsborough, 1998; Torgesen, 2002)
8
Comprehensive Assessment System
K-2
1.
Phonemic Awareness
2.
Phonics
3.
Fluency
4.
Vocabulary
5.
Text Comprehension
6.
Orthographic Skills
(Spelling)
3-12
*
* embedded in text comprehension
9
The K-2 “Big Picture” Map
Broad Screen/Progress Monitoring Tool
(BS/PMT)
“All” students
• Letter Naming & Sounds
• Phonemic Awareness
• Word Reading
Broad Diagnostic Inventory
(BDI)
“All” students
“Some” students for vocabulary
• Listening Comprehension
• Reading Comprehension
• Vocabulary
• Spelling (2nd grade only)
Targeted Diagnostic Inventory
(TDI)
“Some” students
• K = 9 tasks
• 1st = 8 tasks
• 2nd = 6 tasks
Ongoing Progress Monitoring
(OPM)
“Some” students
•K – 2 = TDI tasks
•1 – 2 = ORF
10
Predictive Validity of Broad Screen
1
0.9
0.8
0.7
0.6
0.5
NPP
PPP
0.4
0.3
0.2
0.1
0
K
1st
AP1
2nd
K
1st
AP2
2nd
K
1st
AP3
2nd
Reliability of Broad Screen
Table 9: IRT Precision Estimates for Kindergarten and Grade 1 Broad Screen
Tasks by Time of Assessment
Grade
Task
AP 1
AP 2
AP 3
Kindergarten
Letter Names
0.51*
-
-
Letter Sounds
0.87
0.86
-
Phonological Awareness
-
0.86
0.87
Word Reading
-
-
0.86
Word Reading
0.86
0.86
0.86
Grade 1
*The Letter Sounds task was more reliable than Letter Names at AP 1; however, due to the restricted range for
high risk, a policy decision was made to use the Letter Name task in order to better capture the floor of the
distribution. Because the Broad Screen in G2 is a timed task, precision estimates are not reported; however,
test-retest reliability was strong (.79-.84).
The K – 2 “Score” Map
BS/PMT PRS = Probability of Reading Success
BDI
LC = Listening Comprehension
 Total questions correct (implicit/explicit)
RC = Reading Comprehension
 Total questions correct (implicit/explicit),
Fluency, Percent Accuracy
 Target Passage

VOC = Vocabulary
 Percentile Rank
SPL = Spelling
 Percentile Rank
TDI
ME = Meets Expectations
BE = Below Expectations
OPM
ORF = Adjusted Fluency
OPM TDI Tasks = ME or BE and Raw Score
13
Probability Score – Probability of
Reading Success (PRS)

Score calculated based on the BS/PMT

Probability is…the relative possibility that an event will occur
or is likely to occur.

A .70 means that we expect this student to have a 70% chance
of scoring at or above the 40th percentile at the end of the
year on the SESAT or SAT-10.

The PMRN Reports have already converted the probability
score into a percentage so you will see .70 reported as 70%
14
A Teacher’s
View of the
Class Data Class Status
Report (K-2)-
©2009 Florida Center for Reading Research
Student Score
Detail Box (K-2)
Excellent report to
include in a student’s
cumulative folder
©2009 Florida Center for Reading Research
K-2:
School
Status
Report
Success Zones - Probability Scores
GREEN
YELLOW
RED
85% or better chance of
scoring at or above the 40th
percentile at the end of the
year on the SESAT or SAT-10
16-84% chance of scoring at
or above the 40th percentile at
the end of the year on the
SESAT or SAT-10
15% or less chance of scoring
at or above the 40th percentile
at the end of the year on the 18
SESAT or SAT-10
Probability of Reading Success

What percentage of students are in each success zone at the
various grade levels?


High Success (Green Zone)
Moderate Success (Yellow Zone)




It is very important to look at the grade summary report to see the
distribution within the yellow zone
Are most of the students in the yellow zone towards the 84% chance end
of the zone or closer to the 16% chance
Low Success (Red Zone)
What percentage of students are in each success zone within
specific teachers’ classrooms?


Is there a specific class that does not seem consistent with the rest of
the grade?
Click on the + sign next to the grade level summary to extend and see
the data on individual teachers at that grade level
Probability of Reading Success



Look for trends within
grade levels, within
teachers, and across
grades.
Let’s look across
grades in our sample
data.
What do you notice
when you compare
across grades?
85% &
above
(Green)
16%15% &
84%
below
(Yellow) (Red)
K
28%
61%
11%
1st
36%
56%
8%
2nd 25%
70%
5%
Median Score

The median score = the middle score

The value which 50 percent of the scores fall above and 50
percent fall below

The median score will be used throughout the PMRN on
percentiles and PRS to show the middle score for your group
of students.


On the School Status Report the Median Percentile Score is given for
Vocabulary and Spelling (2nd only)
On bar graph form it is represented by the T and in table form
it is represented as a gray shaded box.
Percentile Rank




Percentile ranks are derived scores that are used to rank
one student’s performance in relation to a specific group.
On the Florida Assessments for Instruction in Reading,
the specified group was a representative sample of
Florida students.
Percentile rank is NOT a percentage score identifying
how many items were answered correctly.
Percentile Rank is the score given on Vocabulary and
Spelling (in 2nd grade only)
22
K-2:
School
Status
Report
Target RC Passages for Grades 1 and 2 (BDI)
©2009 Florida Center for Reading Research
K-2:
School
Status
Report
Target Passage – What does it tell me?

At the School Level, the Target Passage column tells you
how many students were able to read at or above the Target
Passage with 90% accuracy.

The calculation includes all students – even those who ended
up in Listening Comprehension so you can answer the
question…
 How many of my students are accurately decoding at the
level expected of this grade level at this assessment period
(AP).

To help remember this, write (Accuracy) in parentheses under or next to
the column header Target Passage.
Target Passage


In first and second grade, this percentage can provide you
with one piece of information that will help you determine if
your Core Reading Program is addressing the decoding
strategies required for accurate grade-level reading.
Let’s look at the sample data

1st grade – 33.6% of my 143 first graders were able to read at or
above the Target Passage which is Passage 1 (66.4% were placed in
Listening Comprehension)

2nd grade – 57.5% of my 122 second graders were able to read at or
above the Target Passage which is Passage 1 (42.5% were placed in a
lower 2nd grade passage, a 1st grade passage or 9 were placed into
Listening Comprehension)
Total Questions Correct

How many total questions correct (out of 5) did the student
answer in either a reading passage or listening
comprehension.


At the student level, you can get a break down of explicit (out of 3)
and implicit (out of 2) questions.
The reading comprehension (RC) graph represents all
students who took a RC passage not only those who were at
or above the target passage, so…what does this tell you?

When they read a passage with at least 90% accuracy (their
instructional level), how many comprehension questions can they
answer correctly.

To help remember this, write (Comprehension) in parentheses under or
next to the column header Total Questions Correct.
K-2:
School
Status
Report
Total Questions Correct - RC

Let’s look at the First Grade Sample Data:

33.6% of 143 were at or above Target Passage
(Passage 1) = 48 students






4 students = Answered 0 questions correct
7 students = Answered 1 question correct
13 students = Answered 2 questions correct
8 students = Answered 3 questions correct
12 students = Answered 4 questions correct
4 students = Answered 5 questions correct
Total Questions Correct - LC

Let’s look at the First Grade Sample Data:

66.4% of 143 were below Target Passage
(Passage 1) = 95 students






9 students = Answered 0 questions correct
13 students = Answered 1 question correct
26 students = Answered 2 questions correct
19 students = Answered 3 questions correct
14 students = Answered 4 questions correct
14 students = Answered 5 questions correct
Questions to Answer Using the
School Status Report




Where should my resources be allocated (which
grades and classes are struggling in certain areas)?
Where should professional development efforts be
focused?
What percentage of students is at each success zone
in each class and grade?
How many students need additional support?


Are these students clustered in one grade/one teacher?
What types of intervention do the students need?


Where should I focus my PD?
Are there any Programs that I need to purchase?
School
Grade
Summary
Report
(K-2)
School Grade
Summary
Report (K-2)
Questions to answer using the School
Grade Summary Report

What is the distribution of scores in a particular
grade level?



The School Status told us the Median Score for
Vocabulary was the 24th percentile, but now I see the full
distribution and can see that the majority of my students
fell within the 11th – 30th percentile range.
This tells me that overall, the First Grade group is below
average in their general vocabulary knowledge
I would want to see if this is true in Kindergarten and
Second Grade as well (on the School Status Report)



Kindergarten Median Vocabulary = 29th percentile
Second Grade Median Vocabulary = 41st
I might look to see how the Core Reading Program addresses
vocabulary and if we have any additional programs addressing
vocabulary (maybe we are using something different or
additional at second grade?)
Questions to answer using the School
Grade Summary Report

What is the median score in the particular grade
level on particular score types?


Median PRS in First Grade = 65% chance of success
What is the distribution of the ‘yellow’ zone (1684%)?

Of my 56% of First Graders who fall in the yellow zone,
are more of them at the high end or the low end of the
zone?

It is a relatively even distribution across the zone until we reach
65% and then we see that 26 students out of the 80 students in
the yellow zone have a PRS of 65-74% chance of success
Resources & Conclusions
Web Resources

FCRR (Florida Center for Reading Research), www.fcrr.org

Just Read, Florida!, www.justreadflorida.com

LEaRN (Literacy Essentials and Reading Network), www.justreadflorida.com/LEaRN

FOR-PD (Florida Online Reading Professional Development)
http://forpd.ucf.edu/strategies/archive.html

Florida Response to intervention (RtI), www.florida-rti.org

Adolescent Literacy, www.adlit.org

Center on Instruction, www.centeroninstruction.org

LD Online, www.ldonline.org

Colorin’ Colorado, www.colorincolorado.org/
40
For questions regarding the FAIR system:

Visit the resources found on the FAIR website:


Access the FAIR FAQ document found at the link below or
accessed from the above website:


http://www.fcrr.org/fair/index.htm
http://www.fldoe.org/faq/Default.asp?Dept=4&submit=GO
Contact:

[email protected]

Please note the email address is not case sensitive
For more detailed information regarding
the PMRN:

The PMRN User’s Guides located at:



Complete PowerPoints can be found on the PMRN website:
www.fcrr.org/pmrn.htm



www.fcrr.org/pmrn/userguides.htm
There is a section that provides an Overview of Reports and a section
that provides an Explanation of Score Types
Topics include:

K-2 Data Entry

K-2 Electronic Scoring Tool

3-12 Web-based Assessment Module (WAM)

School Level Users 1, 2, 3
Phone: (850) 644-0931 or 866-471-5019
Email: [email protected]
Here is what you see if you click on Kindergarten, Phonemic
Awareness, Broad Screen/Progress Monitoring Tool
In summary, remember…




FAIR was designed to inform instruction.
FAIR data are just one part of the puzzle. Teachers
bring information from daily instruction that
provides additional, important information.
As FCRR gains more experience with the data and
collects data statewide, we will be able to provide
guidelines on what are ‘typical’ scores at each grade
level or what is a ‘good’ percentage to have in the
high success zone (green zone).
We will make comparisons at the close of each
assessment period to the State Median scores and
percentages
Thank you for your time and attention!
Any Questions?
47