Transcript Slide 1

Using High-Quality
Data to Monitor
Student Performance
Mark Baird, Ph.D.
Division of Career and Adult
Education
AECP Leadership Institute
December 5, 2012
The Impact of Fee and
Residency Policy Changes on
2011-12 AGE Enrollment
Compared to Previous
Year
School Districts
Headcount: -33.9%
FTE: -31.6%
Florida College System
Headcount: -31.0%
FTE: -29.4%
School District Enrollment
(Headcount) Changes
Program
Academic Skills for Adult ESOL Learners
Adult Basic Education (ABE)
Adult English for Speakers of Other Languages (ESOL)
Adult General Education for Adults with Disablitites Educational Plan
Adult High School
Adult High School Co-Enrolled
Applied Academics for Adult Education
Citizenship
English Literacy for Career and Technical Education (ELCATE)
General Educational Development (GED)
Literacy Skills for Adult ESOL Learners
Pre-Applied Academics for Adult Education
Pre-General Education Development (GED)
Workplace Readiness Skills for Adult ESOL Learners
2010-11
2,545
78,914
94,402
809
13,074
57,070
11,973
6,874
4,386
31,722
4,882
2,386
5,310
562
2011-12 Difference
1,849
-27.3%
51,802
-34.4%
59,215
-37.3%
994
22.9%
10,545
-19.3%
36,134
-36.7%
8,814
-26.4%
6,855
-0.3%
6,326
44.2%
21,051
-33.6%
2,411
-50.6%
1,781
-25.4%
3,806
-28.3%
577
2.7%
Florida College System
Enrollment (Headcount)
Changes
Course
Adult Basic
Adult Secondary
EAP Literacy
EAP Vocational Prep.
GED Prep.
Vocational Prep.
2010-11 2011-12 Difference
31,396 22,123
-29.5%
6,436
5,034
-21.8%
14,916 10,070
-32.5%
3,883
3,333
-14.2%
10,646
6,944
-34.8%
5,018
2,398
-52.2%
Enrollment Decline More
Dramatic When Summer
Excluded (District Data)
2010-11
2011-12
Decline
Percent Decline
Winter/Spring
All Terms
Terms
268,663
229,507
177,559
132,840
91,104
96,667
33.9%
42.1%
Early Indication That
Completion Rate Increased
District AGE Program
Adult Basic Education (ABE)
Adult High School
General Educational Development (GED)
Adult English for Speakers of Other Languages (ESOL)
English Literacy for Career and Technical Education
(ELCATE)
Academic Skills for Adult ESOL Learners
Workplace Readiness Skills for Adult ESOL Learners
Citizenship
Adult General Education for Adults with Disablitites
Educational Plan
Pre-General Education Development (GED)
Literacy Skills for Adult ESOL Learners
Pre-Applied Academics for Adult Education
Applied Academics for Adult Education
TOTAL
2010-11, No Summer
2011-12, No Summer
Completion to
Completion to
Headcount Ratio
Headcount Ratio
0.52
0.65
2.25
2.49
1.23
1.27
0.41
0.45
0.44
0.23
0.53
0.31
0.29
0.29
0.47
0.41
0.00
0.49
0.40
0.32
0.55
0.63
0.26
0.48
0.50
0.35
0.66
0.72
The Central Questions
How do you ensure the quality of
your data?
 How do you use this high-quality
data to improve your programs?

Questions of Program
Quality

Can you answer these questions without
using anecdotal evidence?





How good are your programs?
How good are your teachers?
How much are your students learning?
Are your programs a good investment for
Florida taxpayers?
Are your programs as effective and efficient
as they should be?
Four Key Areas of
Accountability Measurement
Enrollment
Progress
Completion
Outcomes
The Necessity of Locally
Generated Reports




Performance reports from the state will
ALWAYS be significantly lagged
Not all useful data are reported to the
state
Timely local response to student
performance data feedback is critical
The timeliest and most granular
performance data will always come from
local data systems
Local Reports





Local data needs to be updated on a
regular basis (daily if possible)
Store data in a back-end database
Use front-end tool to slice and dice data
(e.g. Access, Excel, SPSS)
Create reports that can be distributed to
stakeholders
Liberate your data!!!
NRS Database
State Database
Local Data System
The Key
Feedback Loop
If this is not flowing,
you have a
problem
Local Program Office
Classrooms & Testing Centers
The Data Roach Motel
Problem
Data go in,
but it
doesn’t get
out.
NRS Database
Local Data System
State Database
The Citrus
Solution
Local Program Office
Program
Database
Classrooms & Testing Centers
NRS Accountability
Reports
Your check engine Light
What You Can Get Out of NRS
Data


ABE & ESOL: Percentage of students who
complete a functioning level
ASE low: Percentage of students
completing courses or GED subtests.
ASE high: Percentage of students
earning diploma
Average instructional hours
Comparisons of above between posttested and non-post-tested students.
®



What do WE look at?






Percentage of students reported as posttested
Percentage of students earning LCP
Average LCPs per student
Average instructional hours per LCP
Percentage of ASE students earning
diplomas
Transition of diploma earners to
postsecondary education and
employment



Transition of ABE High completers to ASE
or diploma
Transition of ESOL students to ABE, ASE,
diploma or postsecondary
Percentage of GED Prep students taking
all five subtests and of those, percentage
earning diploma
®
DOE Reports Provided to
District and College AGE
Directors






LEA-Level NRS Report
Student Progression (NRS)
Student Instructional Hours (NRS)
Student Demographics
Student Performance
Student Transition
New State Data Tools




Florida is blessed with copious data
Complex systems of data collection and
management are in place
Next step: Getting data into the hands of
administrators and teachers
Reports and tools need to be intuitive
and easy to use; if not, formal training
required
Statewide Longitudinal Data
System (SLDS) Projects







Upgrade state data systems and consolidate silos into a
comprehensive Education Data Warehouse v 2.0.
Customizable report-building tools and dashboards for
stakeholders
Access to state data via single sign-on gateway (“onestop shopping”).
Data mining tool for FLDOE researchers
Minimum standards for local data systems, a forum for
information exchange, and financial support to small
and rural districts.
Statewide unique identifier to all students and staff.
State research agenda and automated delivery of data
to researchers.
Local Student Data
Management Requirements in
AEFLA Grant

AEFLA grantees must



Collect data from instructional and testing
sites on a monthly basis at a minimum and
store the data in a relational format
Produce monthly reports and make them
available to instructional and testing sites for
review and correction
Produce monthly reports, as requested
locally, for program improvement and
monitoring
Use Your Local Data: Examples
of Local Report Content


Instructional site administrators should have
access to timely classroom-level data on:
 Attendance (e.g. average monthly)
 Completions (e.g. completion rate)
 Contact hours between test
administrations
 Ratio of headcount and contact hours to
completions
Disaggregation by race, gender, and other
relevant variables, e.g. day/evening classes
Test Data


Beginning in 2010-11, the state required
school districts and colleges to report
student-level literacy test data for adult
education students
Use the data for your own analysis by
comparing pre- and post-test scale
scores
Limitation of Measurement by
Functioning Level
No learning gain recorded despite
growth
Is this a better teacher?
Level 2 Floor
Test Scores
Post-test
Pre-test
Level 1 Floor
Time
LOOK OUT! Comprehensive
Courses: AKA The BLOB!
Impossible to
determine the
success rate of
students in
comprehensive ABE
and GED courses by
skill/subject area
and functioning level
NRS Changes for 2012-13

Adult student goal will no longer be used to
identify follow-up cohorts (Table 5)



Enter employment and job retention based
on employment status at entry
Enter postsecondary cohort will be all
diploma earners
Obtain secondary credential


Adult high school: Number of ASE high students
who earn diploma
GED: Number of students taking all subtests who
earn GED (pass rate)
NRS Changes for 2012-13

Participant Status (Table 6)

Highest degree or level of schooling
completed



U.S.-based schooling
Non-U.S.-based schooling
Personnel (Table 7)


Teachers’ years of experience
Teacher certification
2011-12 Data Reporting





Approximately half of all districts had extensive
problems with their data on the first load attempt
Two districts were required to submit supplemental
files to correct instructional hours because of
pervasive problems
Districts with over 1% of AGE course records invalid
 Supplemental files accepted to validate records
and include instructional hours in funding
calculations
Work on 2011-12 WDIS data ended on October 15,
52 work days after the close date.
Numerous reporting issues on the college side
Data Quality Issues

Problems we saw in 2011-12 WDIS
reporting




Invalidated course records
Over-reporting of instructional hours
Under-reporting of instructional hours
Large changes in enrollment or average
instructional hours compared to previous year

Some district reporting staff are


Not reviewing data before it is submitted
Not reviewing feedback reports on submitted
data to investigate and fix errors


Files of student records flagged with validation
errors are available for download and analysis
State MIS staff do not have the time to fix
local data problems or perform analysis
that local staff should handle
WDIS Submission Periods,
2012-13
Survey
Period Opens
Required Load
Period Closes
F/G Summer &
Prelim Fall
September 4,
2012
September 13,
2012
October 18, 2012
W/X Fall & Prelim
Winter
January 7, 2013
February 7, 2013
March 7, 2013
S
June 3, 2013
July 3, 2013
July 11, 2013
N/A
August 1, 2013
S Update Window July 15, 2013
College Data Submission
Periods, 2012-13
Submission Period
Period Opens
Required Load
Period Closes
Summer End/Fall
Beginning
August 27, 2012
September 24,
2012
October 8, 2012
Fall End/WinterSpring Beginning
January 7, 2013
February 4, 2013
March 4, 2013
Winter-Spring End
April 22, 2013
May 13, 2013
June 17, 2013


The time between the required load date
and the close of the submission period
allows for corrections to be made and
data to be resubmitted
Some are missing the load deadline and
loading data at the close of the
submission period, allowing no time for
corrections.
Developing a Local Data
Quality Assurance System



Identify your Reports Coordinator
Build reporting timelines into your planning calendars
Establish with Reports Coordinator a data review
committee that includes program, data, and budget
staff



Determine what pre-load reports the committee should
review
Ensure that the committee has access to validation and
mid-survey reports provided by FDOE showing data you
have submitted
For each survey, set up times for the committee to meet
and review data twice:


Before the data are first loaded
After data are loaded but before the close of the survey
The Submission Review
Process
1.
2.
3.
4.
5.
6.
7.
Review data
Pre-Load
Fix problems
Load data
Review edit and validation reports
Review mid-survey reports
Fix problems
Certify data before deadline
Post-Load
Pre-Load – Review Local
Reports



You should have a set of local reports to
review, by program, before data are
loaded into WDIS
If your system is incapable of producing preload data reports, this is an unacceptable
situation
Ideally your system should run all WDIS
validation edits locally BEFORE the data are
loaded

CCTCMIS can provide COBOL code that runs the
data through the edits
Pre-Load – What to Look for In
Local Reports



If your system runs edits, number and
percentage of valid records
Average instructional hours by program;
school and program
LCPs reported by program; school and
program
Another Fine Mess
Not reviewing your
data before you
load it is like not
scraping out the
lasagna dish
before putting it in
the dishwasher.
Post-Load – Review Reports from
Mainframe and Clean Data

When your data are loaded, the system
generates reports upon request



How many records were accepted, rejected
with critical errors, and flagged with noncritical errors
Lists of student records with error codes
This information should be used to clean
up submitted data


The mainframe verification reports are
accessible online, but are secured
requiring login and password information
Your Reports Coordinator must have
access to these reports– it is possible that
is not the case
Post-Load - Review Mid-Survey
Reports to Detect Anomalies

CCTCMIS produces a set of mid-survey WDIS
reports that are useful for detecting anomalies
in your data



These reports are available on the CCTCMIS
restricted access website:
http://www.fldoehub.org/CCTCMIS/Pages/default.a
spx
College reports on enrollments, FTE, and
completions available on mainframe
Your Reports Coordinator must have a
username and password from CCTCMIS for
authorized login
Post-Load – Key WDIS MidSurvey Reports

Enrollment and Instructional Hour
Comparison with Previous Year






01 – By District
03 – By Program
05 – By School
07 – By School and Program
07A – By School, Program, and Course
11 - Valid Course Records

Completer Counts




14 – By District
15 – By Program
16 – By School
17 – By School and Program
State MIS Advisory
Committees




The Workforce Education District Data Advisory
Committee (WEDDAC) is an group comprising
representatives from school district workforce
education data reporting units
The Management Information Advisory Task
Force (MISATFOR) is the corresponding college
organization
The groups meet jointly with FDOE staff
approximately three times a year to discuss
data issues, including changes to the state’s
databases
Make sure your district/college is represented!
Consequences of
Poor Data Quality
Funding
Accountability
Credibility
Audit
Risk
Contacts
Accountability & Reporting
[email protected] , 850-245-9059
[email protected] , 850-245-9060
State Budget
Districts – [email protected] ,
850-245-9002
State Colleges – [email protected] ,
850-245-9764
Federal Grants
Your regional program manager