Transcript Child Outcomes Data 2013-2014

A National Picture:
Child Outcomes for
FFY2013-14
Abby Winer
Cornelia Taylor
September 16, 2014
On Today’s Call
• Learn about the
latest national child
outcomes data data
• Examine data quality
patterns in the
national data
– Completeness of
data
– State-to-state
variation
– Change over time
2
State Approaches to Measuring Child
Outcomes – FFY 2013-14
Approach
Part C
Preschool
(N=56)
(N=59)
COS 7 pt. scale
42/56 (75%)
37/59 (63%)
One tool statewide
8/56 (14%)
9/59 (15%)
Publishers’ online
analysis
1/56 (2%)
6/59 (10%)
Other
5/56 (9%)
7/59 (12%)
3
State Approaches: Part C
State Approaches: Part B Preschool
Method for
Calculating National Estimates & Critieria
• Weighted average of states that met minimum quality
criteria
• Minimum quality criteria for inclusion in national
analysis:
– Reporting data on enough children
• Part C – 28% or more of exiters
• Part B Preschool – 12% or more of child count
– Within expected patterns in the data
• category ‘a’ not greater than 10%
• category ‘e’ not greater than 65%
6
Number of States that Met Criteria for
Inclusion in the National Analysis (N=51)
50
40
30
20
10
0
08-09
Part C
19
Part B Preschool
15
09-10
29
33
10-11
39
36
11-12
33
39
12-13
41
41
13-14
47
43
Part C: Reason States Were Excluded from
Analyses (out of 51)
Reason Part C State Was Excluded
State is sampling
Missing data
(less than 28% of reported exiters)
‘a’ and ‘e’ patterning
(Had at least one outcome with
category a greater than 10% or
category e greater than 65%)
Missing data
AND
Patterning
States included in the analysis
2011-12 2012-13 2013-14
2
1
1
6
4
0
5
3
1
4
2
2
33
41
47
Part B Preschool: Reason States Were
Excluded from Analyses (out of 51)
201112
2
2
0
201213
3
1
1
201314
3
0
1
4
1
1
‘a’ and ‘e’ patterning
(Had at least one outcome with category a
greater than 10% or category e greater than
65%)
3
4
2
Missing Data
AND
‘a’ and ‘e’ patterning
States included in the analysis
0
0
1
39
41
43
Reason Part B 619 State Was Excluded
State is sampling
No progress category data reported
No child count data available
Missing Data
(Reported outcomes data on less than 12%
of child count)
Part C: Greater than Expected Growth
100
90
77
80
70
70 71 68
66 66 66
74 73 72
71 72
76 76
73 73 71 73
60
50
40
30
20
10
0
Social Relationships
Knowledge and Skills
Action to Meet Needs
2008-9 (19 states)
2009-10 (29 states)
2010-11 (39 states)
2011 - 12 (33 states)
2012-13 (41 states)
2013 - 14 (47 states)
Part C: Exited within Age Expectations
100
90
80
70
60
61 62 61 60 61 61
50
54 54 55 52 52 52
61 60 59 59 59 59
40
30
20
10
0
Social Relationships
2008-9 (19 states)
Knowledge and Skills
2009-10 (29 states)
2011 - 12 (33 states)
2012-13 (41 states)
Action to Meet Needs
2010-11 (39 states)
2013 - 14 (47 states)
Part B Preschool: Greater than Expected
Growth
100
90
80
83 83 81 81
80
77
83 82 81 81
80 79
82 82 81 80 80
77
70
60
50
40
30
20
10
0
Social Relationships
Knowledge and Skills
Action to Meet Needs
2008-9 (15 states)
2009-10 (33 states)
2010-11 (36 states)
2011 - 12 (39 states)
2012-13 (41 states)
2013 - 14 (43 states)
Part B Preschool: Exited Within Age
Expectations
100
90
80
70
60
59 59 60 59 59 57
50
67 67 66 66 65
63
51 52 53 53 53 52
40
30
20
10
0
Social Relationships
Knowledge and Skills
Action to Meet Needs
2008-9 (15 states)
2009-10 (33 states)
2010-11 (36 states)
2011 - 12 (39 states)
2012-13 (41 states)
2013 - 14 (42 states)
What We See
• Continuing to see consistency over time
• Increasing number of states who meet minimum quality
criteria for national analysis
• Increasing number of children in the child outcomes
data
14
Current Emphasis of State Requests
• Data Quality
– Increasing the number of children/families in the
data
– Pattern checking to identify data quality issues
– Training, guidance, supervision, etc.
• Using Data for Program Improvement
– Identifying trends in the data
– Identifying areas of low and high performance
– Identifying meaningful differences
Have a child outcomes TA request?
Email your ECTA state contact or one of us!
15
Part C: Percent of States by Completeness of
Child Outcomes Data*
100
90
80
70
65
63
55
60
65
59
50
40
30
20
10
33
12
20
24
16
14
2010-11
(N=51)
2011-12
(N=51)
20
22
22
14
0
2009-10
(N=49)
< 34%
34% - <70%
2012-13
(N=51)
2013-14
(N=51)
70% or more
* Completeness = (total with outcomes data/total exiters)
14
Part B Preschool: Percent of States by
Completeness of Child Outcomes Data
100
90
80
70
60
53
49
50
39
40
30
20
10
45
20
49
41
49
43
39
27
12
14
8
10
0
09-10 (N=49) 10-11 (N=51) 11-12 (N=49) 12-13 (N=49) 13-14(N=50)
<12%
12 - <33%
>33%
* Completeness = (total with outcomes data/child count)
15
State Level Variation and Patterns
18
Part C: State Variation: Exited within Age
Expectations – Knowledge and Skills, 20132014, All States
100%
90%
80%
National: 52%
72%
70%
60%
50%
40%
30%
20%
10%
0%
9%
100%
Part B Preschool: State Variation: Exited within
Age Expectations – Knowledge and Skills, 20132014, All States
90%
National: 52%
80%
70%
60%
50%
40%
30%
20%
10%
0%
10%
74%
Part C: Exited within Age Expectations by
State Percent of Exiters Not Eligible for Part B*
2013-14 - All States
100%
Average Percent of Children
90%
80%
70%
60%
60%
61%
60%
53%
50%
45%
51%
54%
60%
50%
40%
30%
20%
10%
0%
Social Relationships
<20% (N=14)
Knowledge and Skills
20% to <30% (N=27)
Actions to Meet Needs
30% or greater (N=10)
* Not eligible for Part B defined as the sum of the percent of children exiting as No longer eligible for Part C prior to reaching
age three, Not eligible for Part B, exit with referrals to other programs, and Not eligible for Part B, exit with no referrals.
Part C: Average Percentage Who Exited within
Age Expectations by State Percent Served*, 201314 – All States
100%
Average Percent of Children
90%
80%
70%
60%
64%
64%
61%
55%
54%
50%
53%
61%
52%
45%
40%
30%
20%
10%
0%
Social Relationships
<2.5% (N=24)
Knowledge and Skills
2.5% to <3.9% (N=16)
Action to Meet Needs
3.9%+(N=11)
*http://www2.ed.gov/programs/osepidea/618-data/state-level-data-files/
22
Part C: Average Percentage Who Exited within
Age Expectations by ITCA Eligibility Category,
2012-13, All States
100%
Average Percent of Children
90%
80%
70%
60%
59%
63%
54%
47%
50%
50%
54%
56%
57%
62%
40%
30%
20%
10%
0%
Social Relationships
C - Restrictive (N=19)
Knowledge and Skills
B - Medium (N=17)
Action to Meet Needs
A - Broad (N=15)
23
Part B Preschool: Average Percentage Who
Exited within Age Expectations by State Percent
Served, 2013-14, All States (N=50)*
100%
Average Percent of Children
90%
80%
66%
70%
60%
58%
58%
58%
50%
50%
52%
59%
54%
64%
40%
30%
20%
10%
0%
Social Relationships
<5.7% (N=21)
Knowledge and Skills
5.7% to <7.5% (N=16)
Action to Meet Needs
7.5%+(N=13)
*child count data was not available for 1 state to compute percent served
24
Variation Over Time
25
What Types of Change are Important?
Types of Change
2008-09
2013-14
Small variations from year to
year are expected
Large consistent increases
are good news particularly
when linked to
programmatic changes
Large consistent decreases
require explanation (e.g.
changing population)
Large up and down
changes are an indicator of
questionable data quality
and require explanation
26
Part C: Trends Over Time: Greater Than Expected
Growth– Social Emotional, 2013-2014, All States
100
Percent of Children
90
80
70
60
50
40
30
20
10
0
2010-11
2011-12
2012-13
Year
2013-14
Part C: Longitudinal Patterns
States Included in National Estimate, Last 3 Years
Outcome 1 Summary Statement 1 (31 states)
100
90
Percent of Children
80
70
60
50
40
30
20
10
0
2011-12
2012-13
Year
2013-14
29
Part B Preschool: Trends Over Time: Greater Than
Expected Growth– Social Emotional, 2013-2014,
All States
100
90
Percent of Children
80
70
60
50
40
30
20
10
0
2010-11
2011-12
2012-13
Year
2013-14
Part B 619: Longitudinal Patterns States
Included in National Estimate, Last 3 Years
Outcome 1 Summary Statement 1 (34 states)
100
90
Percent of Children
80
70
60
50
40
30
20
10
0
2011-12
2012-13
Year
2013-14
32
Part C: Statistically Significant* Change between
2012-13 and 2013-14: All States (N=51)
Statistically
Significant OC1SS1
Change
Negative
None
Positive
OC2SS1
OC3SS1
OC1SS2
OC2SS2
OC3SS2
2
9
4
10
7
8
44
36
39
34
41
39
5
6
8
7
3
4
*p<.05
31
Part B Preschool: Statistically Significant* Change
between 2012-13 and 2013-14: All States (N=51)
Statistically
Significant OC1SS1
Change
OC2SS1
OC3SS1
OC1SS2
OC2SS2
OC3SS2
9
7
7
14
11
13
None
31
34
35
30
32
33
Positive
10
9
8
6
7
4
Negative
*p<.05
32
Conclusions
• The data continue to be used by the federal
government to justify funding.
• Results Driven Accountability is shining a spotlight on
each state’s child outcomes data.
• States can expect more scrutiny around data quality.
– Data quality is not as simple as yes/no, there is a
continuum of quality and multiple criteria.
– The criteria used for the national analyses do not set
a high bar for data quality.
33
How We Can Help!
• State data quality profiles for FFY 2013-14 were sent out
to C/619 coordinators
– Email Abby Winer with questions:
[email protected]
• Contact us for help with data quality analysis and
quality assurance activities
• Contact us for help with program improvement
planning and data analysis
34
State Child Outcomes Data Quality Profiles
FFY 2013-14
Updated profiles were sent to C/619 coordinators
38
Part C Specific Updates: Completeness
State
38
Part C Specific Updates: Progress
Categories
Determinations - Anomalies
ECTA Expected Patterns
38
Updated National Graphing Template
http://www.ectacenter.org/eco/pages/summary.asp#summarygraph
38
Child Outcomes Summary (COS)
Process Module
http://dasycenter.org/child-outcomes-summary-cos-processmodule-collecting-using-data-to-improve-programs/
40
Data System Framework
Child Outcomes Measurement Framework
ECTA System Framework
Additional Resources
• Data quality: Pattern checking
– http://www.ectacenter.org/eco/assets/pdfs/Pattern_Checking_Table.
pdf
• Training materials on looking at data:
– http://www.ectacenter.org/eco/assets/ppt/LookingAtData_rev
ised.ppt
• Additional data quality resources
– http://www.ectacenter.org/eco/pages/quality_assurance.asp
• Data analysis for program improvement
– http://www.ectacenter.org/eco/pages/usingdata.asp
• SSIP-related Resources
– http://ectacenter.org/topics/ssip/ssip.asp
41
How to Stay in Touch
• Email
– Abby Winer [email protected]
– Cornelia Taylor [email protected]
• Visit the ECTA and DaSy websites
– http://ectacenter.org
– http://dasycenter.org
The contents of this presentation were developed under a grant from the U.S.
Department of Education, # H373Z120002, and a cooperative agreement,
#H326P120002, from the Office of Special Education Programs, U.S.
Department of Education. However, those contents do not necessarily
represent the policy of the U.S. Department of Education, and you should not
assume endorsement by the Federal Government. DaSy Center Project
Officers, Meredith Miceli and Richelle Davis and ECTA Center Project Officer,
Julia Martin Eile.