Transcript ECO Longitudinal - OSEP Leadership Mtng

National Data on Child Outcomes:
Are We There Yet?
Prepared for the ECO Advisers Meeting
February 2, 2011
Computing national numbers
• Department of Education needs
national numbers for GPRA for Part C
and Part B preschool
• Ideally, we would add up the numbers
from each of the states and get the
national numbers but….
Early Childhood Outcomes Center
2
Child Outcomes: What we know
• States are in various stages with regard to
building their child outcomes
measurement systems
• Some phrased in
• Some changed approaches
Early Childhood Outcomes Center
3
Producing National Numbers:
The Issues
• Missing data?
– Which states are reporting data on all the children
who stayed 6 months and exited in 08-09?*
• Data quality
– Which states have invested sufficient resources to be
producing credible data?
– Which states have been producing outcomes data
long enough to have credible data?
*A very few states are sampling.
Early Childhood Outcomes Center
4
Missing/Incomplete Data: Part C
St.
A
B
C
D
E
N for Outcomes Data
Exiters
N for
outcomes/
Exiters
473
797
389
3014
1765
14962
4160
1225
5529
2457
3.2%
19.2%
31.8%
54.5%
71.8%
Aver=35.7%; 31 Sts. > 33%
Early Childhood Outcomes Center
5
Missing/Incomplete Data:
Part B Preschool
N for Outcomes Data
Child Count
N for
outcomes/
child count
A
1230
33796
3.6%
B
693
9967
953
2348
5944
36997
3083
6170
11.7%
26.9%
30.9%
38.1%
St.
C
D
E
Aver= 21.1%; 22 Sts. > 25%
Early Childhood Outcomes Center
6
Data Quality: Outliers
• Some data quality issues are obvious
• States whose data look VERY different
from other states or from what would be
expected.
Early Childhood Outcomes Center
7
What would we expect?
Part C: Variation in Category a, Outcome 1
70
60
50
40
%
30
20
10
0
1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435363738394041424344454647484950
Early Childhood Outcomes Center
8
What would we expect?
Part B Preschool: Variation in Category a, Outcome 1
18
16
14
12
10
%
8
6
4
2
0
1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536373839404142434445464748
Early Childhood Outcomes Center
9
What would we expect?
Part C: Variation in Category e, Outcome 3
90
80
70
60
50
%
40
30
20
10
0
1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435363738394041424344454647484950
Early Childhood Outcomes Center
10
What would we expect?
Part B, Preschool: Variation in Category e, Outcome 3
80
70
60
50
% 40
30
20
10
0
1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637383940414243444546474849
Early Childhood Outcomes Center
11
Producing National Estimates
1. Divide the states into those with large, medium,
and small child counts.
2. Identify the states that appear to have the best
data.
3. Weight by child count and calculate the overall
OSEP percentages using the “best data” states
in each category.
4. Use the OSEP percents and total child count
for each category to get the national estimates.
Early Childhood Outcomes Center
12
1. Divide the states into those with large,
medium, and small child counts.
Size of Child Count
States
Large
CA, NY, PA…..
Medium
TN, CO, AZ…
Small
MT, ND, DE…..
Early Childhood Outcomes Center
13
How Weighting Works
STATES w/
USEABLE DATA
a
%
b% c%
e%
Child Count
New #
for e
A
30
2000
600
B
25
2400
600
C
20
1500
300
D
30
3000
1000
E
33
2400
800
11,300
3,300
Total
Re-calculated a-e for this set of states
Early Childhood Outcomes Center
d%
29.2%
14
And now the results….
Early Childhood Outcomes Center
15
Estimated National Data for Part C, 2008-09
40%
35%
30%
25%
20%
15%
10%
5%
0%
a
b
c
d
e
Social relationships
1.4%
18.4%
18.8%
27.9%
33.4%
Knowledge and skills
1.2%
18.6%
26.3%
37.3%
16.6%
a
b
c
d
Action to meet needs
1.3%
17.3%
20.9%
38.1%
22.4%
e
Note: Analysis based on data from 19 states. Data were weighted by child count to represent the
nation.
Early Childhood Outcomes Center
16
Estimated National Summary Statement Percentages for
Part C, 2008-09
100%
80%
60%
40%
20%
0%
SS1
SS2
Social relationships
70.2%
61.2%
Knowledge and skills
76.2%
53.9%
SS1
Action to meet needs
76.0%
60.5%
SS2
Note: Analysis based on data from 19 states. Data were weighted by child count to represent the
nation.
Early Childhood Outcomes Center
17
Estimated National Data for Part B Preschool, 2008-09
40%
35%
30%
25%
20%
15%
10%
5%
0%
a
b
c
d
e
Social relationships
2.1%
11.0%
28.2%
34.5%
24.2%
Knowledge and skills
2.0%
12.5%
34.4%
34.9%
16.3%
a
b
c
d
Action to meet needs
1.8%
10.8%
20.2%
36.0%
31.2%
e
Note: Analysis based on data from 15 states. Data were weighted by child count to represent the nation.
Early Childhood Outcomes Center
18
Estimated National Summary Statement Percentages
Part B Preschool, 2008-09
100%
80%
60%
40%
20%
0%
SS1
SS2
Social relationships
82.7%
58.7%
Knowledge and skills
82.7%
51.2%
Action to meet needs
81.6%
67.2%
Note: Analysis based on data from 15 states. Data were weighted by child count to represent the nation.
Early Childhood Outcomes Center
19
How trustworthy are these
numbers?
• One possible test: What do we get when
we compare this method with other
methods for calculating the national
numbers?
Early Childhood Outcomes Center
20
3 Approaches to National Numbers
1. Use data from states with the best data
(least missing, not an outlier on “a” or “e”).
Weight their data to represent states of
similar size.*
2. Use data from all states. Average across
the percentages. Each state weighted as
one.
3. Use data from all the states. Weight each
state according to their child count.
*We did this twice with a smaller set of states and a slightly larger set.
Early Childhood Outcomes Center
21
Comparing the three methods
• See handout
Early Childhood Outcomes Center
22
Our Questions for You
• What do these data say to you about the
outcomes of Part C and Part B Preschool?
• What method makes the most sense for
generating the calculations?
• What would you use as indicators that the
national data are valid and meaningful?
• When would you trust the data?
– When all (most?) states can demonstrate valid data?
– When the alternate methods converge more closely?
Early Childhood Outcomes Center
23