Transcript Slide 1

2007 Accountability
Annual Meeting
Evaluation and Reporting Office
Florida Department of Education
September 6, 2007
Agenda











Welcome and Introductions
Responsibilities for Evaluation and Reporting
2006-07 School Grade Changes, Results, and Outlook
2006-07 AYP Results, Benchmarks, and Safe Harbor
Impact of Growth Model
AYP in 2007 and Beyond
SV23 Matching/Updating Process
New Website
Existing Website Improvements
School Grade Issues for 2007, Going Forward
School Improvement Ratings for Alternative Schools Rule
Development Workshop
1
Evaluation and Reporting Staff
Juan C. Copa, Bureau Chief

Accountability Programs – Ed Croft, Director

State Accountability Programs – Tracy Selman,
Program Director


Federal Accountability Programs – Reneé Bruno,
Program Director




Analysts – Sue Klos; Gambhir Shrestha, Ph. D.
Analyst – Kiersten Farmer
Support Staff – Toye Coxe, Staff Assistant
Phone – 850-245-0411
Email – [email protected]
2
Evaluation and Reporting Staff
(cont.)

Research and Evaluation – Marcus Mauldin,
Ph. D, Director




Analysts – Martha Miller, Ph. D; Jennifer Blalock
Support Staff – Tria Parsons, Administrative
Secretary
Phone – 850-245-0429
Email – [email protected]
3
School Grades and AYP

School Grades Under A+ Plan




School grades were first issued in 1999.
Since 1999 there have been many changes
to the calculation of school grades.
We continue to look for ways to improve the
calculation.
Adequate Yearly Progress (AYP)



The federal accountability indicator in the
No Child Left Behind Act of 2001.
AYP has been calculated since 2003.
The growth model was approved in 2007.
4
Schools in Need of Improvement



Required by No Child Left Behind (NCLB).
Evaluation and Reporting works with the
Division of K-12 Public Schools to determine
Schools in Need of Improvement.
Only Title I schools subject to sanctions.
5
Alternative Schools Accountability



Part of the A++ Legislation passed during the
2006 Legislative Session (s. 1008.341, F.S.).
Provides Alternative Schools the option of
receiving a traditional school grade or a school
improvement rating.
If an alternative school chooses to receive a
school improvement rating, the performance of
the alternative school’s students will be included in
the calculation of the rating and the school grade
of the students’ home school.
6
School Improvement Ratings for
Alternative Schools – 2006 and 2007


Alternative Schools were identified based on
their Primary Service Type as reported on the
Master School Identification (MSID) file and
district input and adjustment.
Those schools were given the option of receiving
a “Points Only” designation or a school grade.

“Points Only” refers to the calculation of a school
grade without the assignment of a letter grade.
7
School Improvement Ratings for Alternative
Schools – Activities (2006-07)



Collected information to identify students who
are statutorily to be excluded from the
calculation
Collected information on a student’s zoned
school and district (element description adjusted
for 2007-2008 data collection)
Held rule development workshops on March 20
and August 30 in Tallahassee

Workshop Scheduled for this afternoon (3:305:30pm)
8
School Improvement Ratings for Alternative
Schools – Timeline





April 2007 – Provided districts the opportunity to verify
the list of alternative schools in their district (as
identified in 2005-06)
May 2007 – Provided alternative schools the option of a
“Points Only” designation or a school grade
August - September 2007 – Conduct rule development
workshops on the calculation of school improvement
ratings
October/November 2007 – Final passage of rule before
State Board of Education
June 2008 – Release school grades, school improvement
ratings for alternative schools, and AYP
9
Summary of Issues Raised at Workshops

Major concerns raised

Potential exclusion of choice option schools from the
definition of “alternative school”


Crediting of students back to the “home school”


Some charters serve at-risk populations and have students
referred to the school
What limitations? Only referred students? Can middle
school student performance be credited back to an
elementary school?
Lack of a full-year enrollment requirement for the
school improvement rating

Statute specifies Survey 2 and/or Survey 3
10
Calculation Issues to be Determined

Crediting of students back to the “home school”



Magnitude of Learning Gains needed to define the
categories of improvement – Improving, Maintaining,
Declining



Limit to only those students referred?
Limit to only those students whose “home schools” reflect the
same grade configuration as the alternative school?
Improvement of at least 1%, 5%, 10% over the previous
performance?
Examine other performance measures (e.g., QA for DJJ
schools) and determine their compliance with the statutory
requirements.
Cell Size


Unlike school grades, statute outlines that students enrolled in
either membership count (fall and/or spring) are to be
included.
At least 30 students in either count? More than 10 students?
11
Voluntary Pre-Kindergarten (VPK)
Accountability


Evaluation and Reporting worked with the
Department’s Office of Early Learning to
calculate Readiness Rates for VPK Providers
Readiness Rates can be found at
https://vpk.fldoe.org/
12
Voluntary Pre-Kindergarten
(VPK) Accountability
Sample School Information
13
Merit Award Program (MAP)


Evaluation and Reporting developed Value Tables
that districts may use to measure improved
student achievement for teachers in performance
pay calculations.
Value Tables were provided to measure
performance in Reading and Math in Grades 4-10.




FCAT
NRT (Grades 4 and 5)
A total of 19 Value Tables were made available to
districts (grade-level and school type-level).
Value Table Points were also attached to student
records and made available through the INDV file.
14
Supplemental Educational Services
Accountability

Evaluation and Reporting is working with
the Division of K-12 Public Schools to
develop an accountability calculation for
effectiveness of SES providers.
15
Research, Analysis and Evaluation

Research, Analysis, and Evaluation.

Evaluation and Reporting is tasked with a
number of analyses, focusing mainly on
examining the impact of programs and
policies on student achievement.
16
SAT/ACT/PSAT/Plan, Teacher Projections
and Projected High School Graduates

SAT/ ACT/ PSAT/ PLAN and Teacher Projections


Multiple results and trends are published on
our website.
Teacher Projections

Multiple trends and statistical reports
pertaining to teacher data are on our website.





New hires
Critical teacher shortage areas
Florida teacher retention
Supply of New Teachers
Projected High School Graduates
17
2006-07 School Grade
Changes (Recap)

Writing



Science


Added as an eighth component.
Revised School Grading Scale


Added as a seventh component.
Math, Lowest 25 Percent


Remained at 3.5, no increase to 4.0.
Essay only.
800 point scale changed from current 600 point scale.
Retakes of Grade 11 and 12

High Schools earned ten bonus points when half of all
11th and 12th graders retaking the FCAT met the
graduation requirements in reading and math.
18
History of School Grades
19
Raising Standards and Higher
Accountability Lead to Increased Student
Performance
Red bars indicate years when accountability requirements were increased.
The count of "F" schools drops the year after each major increase in requirements.
140
Lowest Performing Schools
120
100
80
83
78
78
64
49
60
35
40
21
20
4
0
0
1999
2000
2001
2002
2003
2004
2005
2006
2007
20
School Grades 2007: What is Measured
Additional requirements applied AFTER a school’s points are calculated:
• Adequate Progress for Lowest 25%—required to earn grade based on calculated points. If a school
does not meet this requirement, the school’s grade is lowered one letter grade.
• Participation Requirement (Percent Tested)—required to earn grade based on calculated points.
Schools must test at least 95% of their students to earn an “A”, at least 90% to earn any other grade.
21
Performance Components 2007
22
Students Included in
Performance Components
Students who are enrolled all year (i.e.,
enrolled in the fall and spring terms through
the testing period) and who are
standard curriculum students*
* Including speech-impaired, gifted, hospital homebound, and English language
learners in an ESOL program two or more years.
23
Performance Component
Criterion:
Are students achieving at or above
grade level?
24
Meeting Performance
Criteria:
Grade level performance for math, reading, and
science = FCAT achievement level 3;
Grade level performance for writing (essay) =
FCAT score of 3.5;
One point is awarded for each percent of
students scoring at or above grade level.
25
Performance Components Points
2007
READING
MATH
WRITING
SCIENCE
100 possible
pts.
100 possible
pts.
100 possible
pts.
100 possible pts.
Schools receive one point for each percent of students meeting the
performance criteria.
26
Overall Learning Gains Components
READING
MATH
WRITING
SCIENCE
Performance
Performance
Performance
Performance
(new in 2007)
Learning Gains
Learning Gains
Learning Gains
of Lowest 25%
Learning Gains of
Lowest 25%
(new in 2007)
27
Students Included in
Learning Gains Components
All students (including students with
disabilities and ELL students) who are
enrolled all year (i.e., enrolled in the fall and
spring terms through the testing period) and
who have both current and prior-year FCAT
scores.
28
Learning Gains Criterion:
Are students making at least one year’s
worth of progress in a year’s time?
Schools receive one point for each percent of
students making learning gains.
29
Meeting the Learning Gains
Criterion:
a. Improving by one or more FCAT achievement
levels – e.g., from 1-2, 2-3,
3-4, 4-5;
b. Maintaining FCAT achievement levels 3, 4, or 5;
c. For non-retained students at achievement levels
1 or 2: showing more than one year’s growth on
FCAT developmental scale scores (DSS).
30
Students Included in Learning Gains
for the Lowest Performing 25%
Of students included in the learning
gains calculation, the count of the
lowest 25% is based on prior-year FCAT
developmental scale scores.
31
Learning Gains for the Lowest
25%: Who Is Included
The count of the lowest performing 25% is
restricted to students at FCAT achievement
levels 1, 2, and 3.
32
2007 School Grades Points
Components, Total
READING
MATH
WRITING
SCIENCE
Performance
100 possible pts.
Performance
100 possible pts.
Performance
100 possible pts.
Performance
100 possible pts.
Learning Gains
100 possible pts.
Learning Gains
100 possible pts.
Learning Gains of
Lowest 25%
100 possible pts.
Learning Gains of
Lowest 25%
100 possible pts.
PLUS 11th and 12th grade retakes for
possible bonus points (10)
33
Additional Criteria


Percent Tested (Participation
Requirement)
Adequate Progress of Lowest
Performing Students
34
Percent Tested

What percentage of eligible students
who should have been tested on the
FCAT were actually tested?

Grade A schools - 95% or more tested

All other grades - at least 90% tested

Grade I schools - fewer than 90% tested
35
Adequate Progress for the
Lowest 25 % of Students

Criterion: At least half (50% or more) of
the lowest performing students must
show learning gains in reading and
math.

Penalty for missing adequate progress =
drop one letter grade.
36
Impact of Adequate Progress
Requirement


In 2007, a total of 136 Schools dropped a letter grade due to
the Adequate Progress requirement:

66 schools dropped from an “A” to a “B”

22 schools dropped from a “B” to a “C”

48 schools dropped from a “C” to a “D”
Impact by School Type


28 elementary schools dropped a letter grade (2% of all
elementary schools)
2 middle schools dropped a letter grade (less than 1% of all middle
schools)

101 High schools dropped a letter grade (26% of all high schools

5 combination schools dropped a letter grade
37
School Grades - 1999-2007
38
A/B Schools Compared to
D/F Schools - 1999-2007
39
School Grades by Type 2007
40
School Grades: 2007 Compared to
2006
41
School Recognition: Schools Maintaining
an “A” or Improving Grades
42
2007 Grades for 2006 “F” Schools
43
22 High Schools That Earned the Bonus
Points Improved a Letter Grade
44
2007-08 Outlook





Adequate Progress of Low 25%
Writing Plus
Science and Writing (Use of District
Averages in the Calculations)
School Grading Scale
Rating System for Alternative Schools
45
No Child Left Behind
Adequate Yearly Progress
2007 vs. 2006
46
2007 School Grades
Compared to AYP
47
Subgroups Classifications
AYP calculations are based upon nine subgroups.
1. Total School
2. White (W)
3. Black (B)
4. Hispanic (H)
5. Asian (A)
6. Native American (I)
7. Economically Disadvantaged Students
8. English Language Learners (ELL)
9. Students With Disabilities (SWD)
48
39 Components of AYP







Participation Rate in Reading for the 9
subgroups
Participation Rate in Math for the 9 subgroups
Reading Proficiency of 9 subgroups
Math Proficiency of 9 subgroups
Change in School Writing Proficiency
Change in School Graduation Rate
School Grade
49
2006-07 State Targets for AYP







95% Participation Rate in Reading
95% Participation Rate in Math
51% of Students Proficient in Reading
56% of Students Proficient in Math
1% Improvement in Writing Proficiency
or 90% at 3 and above
1% Improvement in Graduation Rate
or graduation rate of 85% or higher
School Grade not a D or F
50
AYP Benchmarks
Reading
Mathematics
2001-02
31
38
2002-03
31
38
2003-04
31
38
2004-05
37
44
2005-06
44
50
2006-07
51
56
2007-08
58
62
2008-09
65
68
2009-10
72
74
2010-11
79
80
2011-12
86
86
2012-13
93
93
2013-14
100
100
51
Determine if Reading/Math Proficiency
Targets were Met

Three Ways to Compute:
 First Method: Did the school and ALL subgroups meet the
reading/math proficiency target?
 If Yes: Proficiency Targets are met
 If Not: Check Second Method
 Second Method: Did the school and/or ALL subgroups that
did not meet reading/math proficiency targets meet Safe
Harbor requirements?
 If Yes: Reading/Math proficiency targets are Met
 If Not: Check Third Method
 Third Method: Did the school and/or ALL subgroups that
did not meet Safe Harbor requirements meet Growth Model
“on track to be proficient” requirements?
 If Yes: Reading/Math proficiency targets are Met
 If Not: Reading/Math proficiency targets are Not Met
52
When can a School Use Safe
Harbor?

The Safe Harbor provision can be applied
only if a school has met all four of the
requirements below:




At least 95% of students in total and in each
subgroup have participated in testing.
The school has met writing criteria.
The school has met graduation rate criteria.
The school grade is not D or F.
53
What are Requirements for Safe
Harbor?

Safe Harbor requires that for each subgroup
being evaluated:

The percent of non-proficient students decrease by
at least 10% from the preceding year

Graduation Rate Criteria is met

Writing Criteria is met
54
Safe Harbor: 10% Reduction in Non-Proficient Students
55
When can a School Use the Growth
Model provision?

The Growth Model provision can be
applied only if a school has met all four
of the requirements below:




At least 95% of students in total and in
each subgroup have participated in testing.
The school has met the writing criteria.
The school has met the graduation rate
criteria.
The school grade is not D or F.
56
What are Requirements for the
Growth Model?

The Growth Model requires that for each subgroup being
evaluated:



The percent of students “on track to be proficient” in three years
or less is at least 51% in reading and 56% in math and
The group has met the writing criterion (the increase in writing
proficiency is at least 1% or the school has a writing proficiency
rate of 90% or better) and
The group has met the graduation rate criterion (the increase in
graduation rate is at least 1% or the school has a graduation rate
of 85% or better).
57
AYP Growth Model
Year In State
Tested Grade
Decrease From Baseline Assessment In
Performance Discrepancy
1
33% of original gap
2
66% of original gap
3
Student must be proficient
Grade
3
4
5
6
Student’s Actual Reading
Developmental Score
1001
1325
1450
1635
Required DSS Score for Proficiency
1198
1456
1510
1622
Cut score needed to be “on track to
be proficient”
NA
Is student “on track to be proficient”
No
33% of 66% of
621
621
Yes
Yes
100%
of 621
Yes
58
Impact of the Growth Model




675 schools met all proficiency benchmarks.
114 schools met requirements using Safe
Harbor.
185 schools used the Growth Model or a
combination of the Safe Harbor and the
Growth Model.
113 schools needed the additional help of
the mathematical adjustment to meet AYP.
59
SV2/3 Match Process

Two Critical Processes Performed by
local MIS in Conjunction with EIAS




Matching Process
Data Update Process
EIAS matches records by district, school
and SID/AID
Must be done through NWRDC – not a
web process
60
Purpose of Matching

For AYP and School Grading Process:

Identify students meeting the requirement
of being in school all year.


In Survey 2 and Survey 3.
Identify students needing updates to data
critical to the school grades and AYP
calculations.






Race.
Grade Level.
SWD Code.
ELL Code and Entry Date.
FRL Code.
Withdrawals.
61
The Five Files on NWRDC

F70550 Unmatched Problem Records






Students whose Prior School Status (PSS) shows same
school, same district.
Students for whom the demographic record is missing a
matching PSS record.
Students whose PSS withdrawal code is other than
W01, W02, or ZZZ.
Students who did not match between Svy2 and Svy3.
Students with duplicated alias numbers.
F70551 Unmatched Expected Records


Most likely these students are new to the state
between surveys.
Withdrawal code of ZZZ and Entry/re-entry date after
Svy2.
62
The Five Files on NWRDC (cont.)

F70549 Deleted Survey 3 Records and
F70586 Deleted Survey 2 Records



McKay, Home Ed, & Private School
students.
Be sure each student is identified
correctly.
F70548 Matched Records
63
2007-08 Timeline

Survey 3, 2007-08.



Survey Week: February 4-8, 2008.
State Processing: February 11 – March 7, 2008.
Correct errors and submit corrections by
4:00 p.m. EST on Friday, March 7, 2008.
64
Need for Improvement

Missing and Incorrect Data




Race, Gender, SWD, ELL, etc. not always
updated.
ELL dates often missing.
Incorrect Grade Levels.
Students enrolled in a school where they
do not receive the majority of their
instruction.
65
To Remember

Submit all four records, as appropriate:






Student Demographic,
Exceptional Student (SWD)
English Language Learner (ELL)
Student Course Schedule
Students must have a course record to be
included.
Data should reflect Friday of FTE Week
(February 8).
66
Tentative Timeline for Corrections
Processes






School Types – January/February 2008
Prior Year Data – February  April 2008
Student Data Updates – March 13  April
2008
Retakes – March 13  April 2008
Assessment Corrections – June 2008
Appeals – Mid July to Mid August 2008
67
Aids for Gathering Prior Year
Data

Student Locator



Use CICS system at NWRDC
Locate student IDs and prior FL public
schools attended
FASTER method of sending records
electronically.



Keep Reading and Math Litho Codes
locally
Send to new school when student moves
Required data elements as of 1/1/07
68
Prior Year Data Search Engine




Allows a search of FCAT assessments.
Search for Litho Codes.
Search by last year’s District, School, Last Name,
First Name, Grade level.
An expanded Litho Code was necessary in 2007
to assure each record was unique.




8 digit Litho Code +
2 digit grade +
1 digit Retake indicator—R=Retake; 0=Not a
Retake.
Example: 12345678040 or 3456789810R.
69
Overall Improvements






The site will be made more secure.
Pages will be made more aesthetically
pleasing.
Brief descriptions will be available (mouse
over?) for clarification.
Links will be larger, logically arranged, and
consistent among pages and applications.
Links for downloading views into Excel.
Links will be added where necessary to
enhance navigation.
70
More Improvements






Error messages will be more meaningful and
consistent among applications.
A “View All Errors” page will be added to
appropriate applications.
Page Titles and messages will be clear and
appropriately placed.
Highlighting will be made more visible.
Instructions will be more user friendly.
Views will contain records unique to that
view where appropriate.
71
Page Specific Improvements

Login Page.


Upload Page


A link to download the formats in Excel.
District and School Home Pages


DOE Contact information will be put on this
page.
School page will have a note to call district if a
reset is needed.
Student Edit page


The back button may be disabled.
There will be a reset button that will reset all
data entry cells to blank.
72
Application Specific
Improvements

School Types


Prior Year Data Corrections



The “met graduation requirements” flag will be
recoded.
Withdrawal flag will be recoded.
Assessment Data Corrections


School dropdowns will be populated appropriately.
Student Data Updates


Add a way of indicating new schools.
A “view all records” page will be added.
Appeals

Data will be entered via the web rather than
uploaded.
73
Issues for 2007, Going Forward




Adequate Progress of Low 25%
Writing Plus
Science and Writing (Use of District
Averages in the Calculations)
School Grading Scale
74
Contact Information
Evaluation and Reporting Office
Florida Department of Education
325 West Gaines Street, Room 445
Tallahassee, FL 32399
Website: http://www.fldoe.org/evaluation/
Email: [email protected]
75