Transcript ECO Longitudinal - OSEP Leadership Mtng

Highs and Lows on the Road to
High Quality Data
Kathy Hebbeler and Lynne Kahn
ECO at SRI International and ECO at UNC
American Evaluation Association
Anaheim, CA
November, 2011
What we will cover
• Review of the timeline for national
reporting
• Share the national data
• Describe how the national data were
computed
• Discuss the quality of the national data
• Discuss the meaning of the numbers
Early Childhood Outcomes Center
2
Timeline
When
Critical Event
January 2004 – January 2005
Stakeholder input gathered on 3
child outcomes
July 2005 (revised September 2006)
OSEP releases reporting
requirements for state programs
February 2008
States submit first data on 5
progress categories: Children who
exited between July 1, 2006 to June
30, 2007
February 2010
States establish baseline and set
targets on the Summary Statements
for first time.
February 2011
States submit data on 5 categories
for the 4th time.
Early Childhood Outcomes Center
3
Request from the
U.S. Department of Education
• Analyze the data for possible inclusion as
a GPRA indicator.
• Also, to use in President’s budget
justification for Part C and Preschool 619
funding.
• Initial request received in 2010, repeated
in 2011.
Early Childhood Outcomes Center
4
The Dilemma
• Variations in quality of state data
– Some states started earlier
– Some states had devoted more attention to
improving quality
• What would be the impact of state
variation in data quality on the national
number?
Early Childhood Outcomes Center
5
Our Response
• Compute the analyses several ways
1. Identify the states with the highest quality
data and use only their data. Stratify by
number of children served and weight data
to produce national estimate.
2. Use data from all states. Weight data to
represent the nation.
•
Weighting necessary because a few states are
sampling. Also, many states not reporting data
on all children.
Early Childhood Outcomes Center
6
Estimated Data for Part C, 2009-10
40%
35%
30%
25%
20%
15%
10%
5%
0%
a
b
c
d
e
Social relationships
1.7%
18.0%
18.6%
29.4%
32.4%
Knowledge and skills
1.5%
20.0%
24.8%
36.9%
16.8%
Action to meet needs
1.5%
17.5%
21.4%
37.1%
22.6%
Note: Based on 29 States with highest quality data
Early Childhood Outcomes Center
7
OSEP Reporting Categories
Percentage of children who:
a. Did not improve functioning
b. Improved functioning, but not sufficient to move nearer
to functioning comparable to same-aged peers
c. Improved functioning to a level nearer to same-aged
peers but did not reach it
d. Improved functioning to reach a level comparable to
same-aged peers
e. Maintained functioning at a level comparable to sameaged peers
3 outcomes x 5 “measures” = 15 numbers
Early Childhood Outcomes Center
8
Illustration of 5 Possible Paths
70
60
Score
50
40
30
20
10
0
1
6
11
16
21
26
31
36
41
46
51
56
Age in Months
Maintained functioning comparable to age peers
Achieved functioning comparable to age peers
Moved nearer functioning comparable to age peers
Made progress; no change in trajectory
Did not make progress
Estimated Summary Statement Data for Part C, 2009-10
80%
70%
60%
50%
40%
30%
20%
10%
0%
SS1
SS2
Social relationships
71.0%
61.8%
Knowledge and skills
74.2%
53.8%
Action to meet needs
75.6%
59.7%
Note: Based on 29 States with highest quality data
Early Childhood Outcomes Center
10
The Summary Statements
1. Of those children who entered the program below
age expectations in each outcome, the percent who
substantially increased their rate of growth by the
time they turned 3 [6] years of age or exited the
program.
2. The percent of children who were functioning within
age expectations in each outcome by the time they
turned 3 [6] years of age or exited the program.
Early Childhood Outcomes Center
11
Estimated National Data for Early Childhood Special
Education, 2009-2010
40%
35%
30%
25%
20%
15%
10%
5%
0%
a
b
c
d
e
Social relationships
1.7%
11.5%
28.2%
34.3%
24.3%
Knowledge and skills
1.8%
13.4%
32.5%
34.4%
17.9%
Action to meet needs
1.6%
10.8%
20.9%
35.6%
31.0%
Note: Based on 33 States with highest quality data
Early Childhood Outcomes Center
12
Estimated National Summary Statements for Early
Childhood Special Education, 2009-2010
100%
80%
60%
40%
20%
0%
SS1
SS2
Social relationships
82.6%
58.7%
Knowledge and skills
81.5%
52.3%
Action to meet needs
81.9%
66.7%
Note: Based on 33 States with highest quality data
Early Childhood Outcomes Center
13
Criteria for States with Quality Data
1. Low percentage of
missing data
2. No odd patterns in “a”
or “e” categories
3. Did not use
questionable data
collection methods
Early Childhood Outcomes Center
14
Calculating Missing Data for Part C
Proxy for missing data =
Number with data for C3/
Exiting Data (618)
• Do not expect this number to be
100%
• ..but we don’t expect it to be 10%
either
Early Childhood Outcomes Center
15
Part C: Percent of Exiters included
in Outcomes Data
08-09
09-10
<10% = 10*
10- 20% = 4
20- 30% = 8
30- 40% = 11
40- 50% = 8
50- 60% = 8
60- 70% = 4
70- 80% = 2
>80% = 1
<10% = 5*
10- 20% = 4
20- 30% = 6
30- 40% = 8
40- 50% = 5
50- 60% = 11
60- 70% = 9
70- 80% = 1
>80% = 0
*3 states are sampling for Part C.
Cut off was > 27%.
Calculating Missing Data for 619
Proxy for missing data =
Number with data for B7/
Child count
• Do not expect this number to be
100%
• ..but we don’t expect it to be 10%
either
Early Childhood Outcomes Center
17
Percent of Child Count included in
Outcomes Data for ECSE
08-09
09-10
<10= 11*
10- 20%= 15
20- 30%= 12
30- 40%= 12
40-50% =1
>50% = 2
<10= 6*
10- 20%= 11
20- 30%= 12
30- 40%= 16
40-50% =4
>50%= 0
*4 States are sampling for 619
Cutoff was > 11%.
Odd Patterns in a or e
• a = % of children who show no new skills
– Except this to be very small.
• e = % of children who maintained
functioning comparable to age
expectations.
– Don’t expect this to be large.
• Quality defined as <10% in a and <65% in
e.
Early Childhood Outcomes Center
19
Percent Reported in "a“ for Knowledge and Skills for
ECSE by State
16%
14%
12%
10%
8%
6%
4%
2%
0%
1
2
3
4
5
6
7
8
9
10
11
12
Early Childhood Outcomes Center
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
20
Can we trust
these data?
Early Childhood Outcomes Center
21
Pattern checking for validity
• Checking across years
– How do the 2009-10 data compare to the
data for 2008-09?
• Checking across methods
– How do the data for all states compare to
states with highest quality data?
Early Childhood Outcomes Center
22
Part C, Outcome A:
Social Relationships
SS1: % who Increased
Growth Rates
80
SS2: % who Exited at
Age Expectations
80
71
70.2
64.5
67
64
61.3
60
60
40
40
20
20
0
0
All states
19 best
FFY 08-09
All states
29 best
FFY 09-10
64.7
61.8
All states 19 best All states 29 best
FFY 08-09
FFY 09-10
Part C, Outcome B:
Knowledge and Skills
SS1: % who Increased
Growth Rates
76.6
80
70.4
SS2: % who Exited at
Age Expectations
74.2
80
68.1
60
60
40
40
20
20
0
All states
19 best
FFY 08-09
All states
29 best
FFY 09-10
59
53.9
55.8
53.8
0
All states 19 best All states 29 best
FFY 08-09
FFY 09-10
Part C, Outcome C:
Meets Needs
SS1: % who Increased
Growth Rates
80
70
76
69.7
SS2: % who Exited at Age
Expectations
75.6
68.1
80
70
60
60
50
50
40
40
30
30
20
20
10
10
0
All states 19 best All states 29 best
FFY 08-09
FFY 09-10
64.6
60.5
61.5
19 best
All states
59.7
0
All states
FFY 08-09
29 best
FFY 09-10
Part B Preschool:
Social Relationships
SS2: % who Exited at Age
Expectations
SS1: % who Increased
Growth Rates
90
80
82.7
77.2
79.7
82.6
70
90
80
70
60
60
50
50
40
40
30
30
20
20
10
10
0
All states 15 best All states 33 best
FFY 08-09
FFY 09-10
61.7
58.7
62.1
58.7
0
All states 15 best All states 33 best
FFY 08-09
FFY 09-10
Part B Preschool:
Knowledge and Skills
SS1: % who Increased
Growth Rates
SS2: % who Exited at Age
Expectations
100
100
82.7
80
76.2
78.3
81.5
80
60
60
40
40
20
20
55.8
51.2
55.5
52.3
0
0
All states
15 best
FFY 08-09
All states
33 best
FFY 09-10
All states
15 best
FFY 08-09
All states
33 best
FFY 09-10
Part B Preschool:
Meets Needs
SS2: % who Exited at Age
Expectations
SS1: % who Increased
Growth Rates
100
100
81.6
80
75.3
78.1
81.9
80
67.8
60
60
40
40
20
20
0
0
All states
15 best
FFY 08-09
All states
33 best
FFY 09-10
67.2
66.7
66.7
All states 15 best All states 33 best
FFY 08-09
FFY 09-10
Possible interpretation of the data
• Nationally, a high proportion of children
who receive Part C and ECSE services
are showing greater than expected
progress
• Nationally, many (over half) are exiting the
program functioning like same age peers
in at least one of the outcomes.
Early Childhood Outcomes Center
29
Would you agree?
Early Childhood Outcomes Center
30
Should each state’s data look like
the national data?
• Probably not
• More important that each state continue to
focus on the quality of its own data
– Getting outcomes data on all children who exit
– Working with programs whose data look
unusual to address possible data quality
issues
Early Childhood Outcomes Center
31
Additional information
For information on state activities related to
improving data quality and using data for
program improvement
www.the-eco-center.org
Early Childhood Outcomes Center
32