Transcript Slide 1
Building an Effective Evaluation of Your Flexible Learning Program (FLP)
Presenter: Sue Hawks Foster, FLP Education Program Specialist Georgia Department of Education [email protected]
(404) 656-2636 Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
1
Agenda – FLP Evaluation
Federal Regulation Requires Evaluation of the Title I Programs FLP Evaluation Requirements Federal, State, and District Building an Effective Program Evaluation LEA Plan for District FLP Evaluation Performance Based Observation of FLP Most Common Findings FY13 Questions and Answers References
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
2
FEDERAL REQUIREMENTS FOR EVALUATING TITLE I PROGRAMS
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
3
Federal Regulation for Title I Programs
• Federal regulations requires that a school operating a Title I program annually evaluate the implementation of, and results achieved by, the Title I program • ESEA Title I program compliance requires written program evaluation procedure and program evaluation summary
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
4
Federal Requirements for Evaluation
The Title I program evaluation is an annual review of Title I activities to determine if they are contributing to the desired outcomes: • Improved student achievement • Effective family engagement • High quality professional development
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
5
Federal Requirements for Evaluation
• • • • •
Evaluation Components:
Measures efficacy and impact of the Title I FLP program Documents impact of Title I FLP services on student learning Informs school and district planning (cycle of continuous improvement) Is an advocacy tool Is conducted at end of program year and used for the next years planning
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
6
Federal Requirements for Evaluation
• • • •
Description of:
Data used Constituents consulted Process used to arrive at evaluation findings – Previous data – Surveys How findings are utilized for planning and improvement
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
7
Federal Requirements for Evaluation
Summary of:
• Program strengths • Program weaknesses • Subsequent program changes • Reallocation of funds
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
8
Needs Assessment & Program Evaluation
•
What are the differences and how do they relate to each other?
Program evaluation – Evaluates Title I program-FLP specifically – Completed after the fact (end of year) – Helps to plan Title I program for next year including FLP
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
9
Needs Assessment & Program Evaluation
•
Needs assessment
– Looks at all school data, including Title I • Schoolwide data in all subject areas • Targeted Assistance data in all subject areas • FLP data – Completed before beginning of school year – Assesses priority needs in district/schools and determines where Title I FLP fits in, as well as, other resources
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
10
FLP EVALUATION REQUIREMENTS
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
11
FLP
Evaluation
Requirements
• State Evaluation –
Participant Feedback
• Evaluation Question: What is the overall experience of participants with program?
• Data Source: Student and Parent Surveys –
Service Delivery
• Evaluation Question: Is the LEAs and programs in compliance with laws and regulations?
• Data Source: Annual monitoring data, Program documentation, Federal reporting, Public reporting, Technical Assistance, etc.
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
12
State Evaluation cont’d
•
State Evaluation
–
Effectiveness
• Evaluation Question: Are programs contributing to increased student academic achievement and performance on state education standards?
• Data Source: Student performance on state tests, Pre-post assessment measures of state standards and academic skills targeting by programs, Performance Flag data, and student growth in schools offering FLP • Evaluation results would be shared with stakeholders and the public and used to inform ongoing program improvement
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
13
State Evaluation
cont’d
• LEA Evaluation – LEAs will be required to evaluate outcomes of FLP interventions implemented at schools within their district that have been identified – LEAs will submit their evaluation plan as part of their overall FLP plan for approval by GaDOE Title I Specialists
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
14
State Evaluation cont’d
• LEA Evaluation The LEA must submit the following components of an evaluation plan with their FLP plan in the CLIP for each Priority, Focus, or Alert school within their district required to implement FLP: – Effectiveness target or overall quantifiable goal of the program – Assessment instrument(s) that will be used to measure this target/goal – Plan for collecting and maintaining data – Data analysis plan
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
15
State Evaluation cont’d
• What do the LEA evaluation requirements mean?
– Focused, purposeful intervention – Tailor evaluation needs of program to match program goals and student needs – Expand on broadband state assessments to quantify benefits of FLP participation – Collect meaningful information for decision making and continuous program improvement
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
16
BUILDING AN EFFECTIVE PROGRAM EVALUATION PLAN
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
17
Building Effective Program Evaluation
• Determining Purpose of Evaluation – Why should we evaluate our program?
– What is our program trying to do?
– What information do our funders expect?
– What questions do we want to answer?
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
18
Building Effective Program Evaluation
• • Performance Measures – Measures of effort – Measures of effect Strong performance measures should: – – – – Have strong ties to program goals, inputs, and outputs Be compatible with age and stage of your program Consider if the data you need are available/accessible Yield useful information to the program
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
19
Building Effective Program Evaluation
• Assessing Capacity for Evaluation – Who should be involved in the process?
– What resources must be in place?
– Who will conduct our evaluation?
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
20
Building Effective Program Evaluation
• Selecting Evaluation Design – What type of evaluation should we conduct?
• Formative to understand the changes that have occurred • Summative demonstrate they have achieved the standard required – What type of data should we collect?
• Qualitative • Quantitative
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
21
Building Effective Program Evaluation
• Selecting Evaluation Design (cont.) – What is our time frame for collecting data?
– Causal • Experimental • Quasi-experimental – Non-experimental • Descriptive • Pre-experimental
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
22
Building Effective Program Evaluation
• Choosing evaluation instruments – Construct – Audience – Format – Reliability – Validity
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
23
Building Effective Program Evaluation
• • Analyzing data – How will data be analyzed and shared?
– Quantify central evaluation question (and supporting questions) Communicating results – How?
– Where?
– Who?
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
24
LEA PLAN FOR DISTRICT FLEXIBLE LEARNING PROGRAM (FLP) EVALUATION
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
25
LEA Plan for District FLP Evaluation
• • List the effectiveness target(s) or overall quantifiable goal(s) of your programs What are the measureable outcomes that your intervention is designed to improve? Must include at least one academic achievement indicator
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
26
LEA Plan for District FLP Evaluation
• List the assessment instrument(s) that will be used to measure each listed effectiveness target/goal • Include a plan for administering assessment instruments and how you will collect and maintain data
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
27
LEA Plan for District FLP Evaluation
• • • When will you administer pre- and post assessments? How will you maintain student assessment information to ensure confidentiality?
How will you determine program effectiveness based on program goals and measurement instruments listed above ?
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
28
LEA Plan for District FLP Evaluation
• • • FLP Program Evaluation – Includes Academic Data and Narrative FLP Program Summary Chart Parent, Staff and Student Summary Results
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
29
True Test for FLP
Are graduation rates improving?
Are CRCT scores increasing?
Are EOCT pass rates increasing?
Is our CCRPI increasing?
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
30
PERFORMANCE BASED OBSERVATION (PBO)
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
31
Performance
Based Observation
• • FY13 PBO in Review – 12 Districts were selected for PBO as a pilot – 16 schools were observed FY14 Review – 26 Districts have been selected for review – 26 School will be observed
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
32
Performance
Based Observation
•
Components of Performance Based Observation(PBO )
– Review of Priority, Focus, and Alert School List – Select number of districts to be reviewed based on the ESEA Flexible Waiver – Selection of schools in district to be monitored
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
33
Performance Based Observation
•
Components of PBO cont’d
– Notifications Sent to LEAs • Call to Title I Director • Follow up Letter to LEA – Superintendent, Title I Director and Principal of selected schools with brief outline of expectations
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
34
Performance Based Observation
•
Components of PBO cont’d
• Agenda Developed by Title I Office • Follow up call to LEA by PBO Team Lead before visit – Addresses and list of staff reviewed – Selection of Teachers/Tutors for Interview
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
35
Performance
Based
Observation
•
Actual Visit to LEA
– Interview with LEA Staff • Superintendent • Title I Director • • • • School Principal School FLP Coordinator School FLP Instructors/Tutors (at least 2) Other vital staff as needed
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
36
Performance Based Observation
•
Actual Visit to LEA
– Classroom Observation -20 minutes per classroom – Talk with students and staff in classroom – Observe student and FLP staff interactions – Depart Building- Inform building staff we are leaving the building
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
37
Performance Based Observation
• Follow-up Letter – Review of the PBO findings for the district – Non-Compliance • If a LEA is found out of compliance the district will be notified immediately by Program Manger • If corrective action is required it will be provided in writing and followed up with the FLP Specialist and/or the Program Director for FLP
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
38
FY13 FLP MOST COMMON FINDINGS
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
39
FLP’s Most Common Findings
• Models used for FLP – After school programs were used by Middle and High Schools – Elementary and Middle schools extended the day or rotated the scheduled day – High schools used Saturday sessions – Summer Sessions were held when extra funds were available to extend the learning year
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
40
FLP’s Most Common Findings cont’d
• Technology Usage Increased • Increase Professional Learning • Increased Student Instruction Time • More contact with parents • Reported Increase in Test Scores
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
41
FLP’s Most
Common
Findings cont’d
• • • • Frustration of operating the program by some districts because it was so new Large amounts of carryover because of FLP funds were not used last year Special Education Departments not connecting to FLP in various districts Lack of staff in some districts to serve students
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
42
FLP’s Most Common Findings in Plans
• • • • • Lack of understanding of FLP purpose Did not provide performance goals for plans Determining the needs assessment was not fully understood to determine subject area Many did not identify all subgroups within the schools Schools did not understand providing data for more than one year in order to provide a trend
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
43
FLP’s Most Common Findings in Plans
• • • • • Special Education Departments were not really aware of what was going on with Students with Disabilities (SWD) in FLP.
Student ratio needs to be smaller Individual Learning Plans needed to be developed and followed up Documentation of Multiple Selection Criteria was lacking in some districts Lack of understanding Professional Learning
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
44
QUESTIONS AND ANSWERS
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
45
• • • • •
References
Afterschool Evaluation 101: How to Evaluate an Expanded Learning Program, Erin Harris, Harvard Family Research Project, 2011 http://www.hfrp.org/ Doing What Works Clearinghouse, USED http://ies.ed.gov/ncee/wwc/ Preschool for California's Children [email protected]
Harvard Family Research Project, 2013 http://www.hfrp.org/evaluation/overview Georgia Department of Education, FLP Report http://www.gadoe.org/School-Improvement/Federal Programs/Pages/default.aspx
Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
46
Contact Information
Sue Hawks Foster FLP Education Specialist [email protected]
404-656-2636 Dr. John D. Barge, State School Superintendent
“Making Education Work for All Georgians”
www.gadoe.org
47