2011 OSEP Leadership Mega Conference

Collaboration to Achieve Success from Cradle to Career 2.0

The Results are In: Using Early Childhood Outcome Data

Kathy Hebbeler and Lynne Kahn

ECO at SRI International and ECO at UNC

August, 2011

2

What we will cover

• • • • • Quick review of the reporting requirement and state approaches Share the national data Describe how the national data were computed Discuss the quality of the national data Discuss the meaning of the numbers Early Childhood Outcomes Center

3

OSEP Reporting Requirements: Child Outcomes

• Positive social emotional skills (including positive social relationships) • Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) • Use of appropriate behaviors to meet their needs Early Childhood Outcomes Center

OSEP Reporting Categories

4 Percentage of children who: a. Did not improve functioning b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same-aged peers but did not reach it d. Improved functioning to reach a level comparable to same-aged peers e. Maintained functioning at a level comparable to same aged peers 3 outcomes x 5 “measures” = 15 numbers Early Childhood Outcomes Center

Illustration of 5 Possible Paths 70 60 50 40 30 20 10 0 1 6 11 16 21 26 31 36 Age in Months 41 46 51 56 Maintained functioning comparable to age peers Achieved functioning comparable to age peers Moved nearer functioning comparable to age peers Made progress; no change in trajectory Did not make progress

6 The Summary Statements 1. Of those children who entered the program below age expectations in each outcome, the percent who substantially increased their rate of growth by the time they turned 3 years of age or exited the program.

2. The percent of children who were functioning within age expectations in each outcome by the time they turned 3 years of age or exited the program.

Early Childhood Outcomes Center

State Approaches to Outcomes Data

Approach

COS 7 pt. scale One tool statewide Publishers’ online analysis Other

Part C (56 states/jurisdictions)

41/56 (73%)

Preschool (59 states/jurisdictions)

37/59 (63%) 7/56 (13%) 3/56 (5%) 9/59 (15%) 6/59 (10%) 5/56 (9%) 7/59 (12%)

Estimated Data for Part C, 2009-10

40% 35% 30% 25% 20% 15% 10% 5% 0% a b c d e Social relationships 1,7% 18,0% 18,6% 29,4% 32,4% Knowledge and skills 1,5% 20,0% 24,8% 36,9% 16,8% Note: Based on 29 States with highest quality data Early Childhood Outcomes Center 8 Action to meet needs 1,5% 17,5% 21,4% 37,1% 22,6%

Estimated Summary Statement Data for Part C, 2009-10

80% 70% 60% 50% 40% 30% 20% 10% 0% SS1 SS2 Social relationships 71,0% 61,8% Knowledge and skills 74,2% 53,8% Action to meet needs 75,6% 59,7% Note: Based on 29 States with highest quality data Early Childhood Outcomes Center 9

Estimated National Data for Early Childhood Special Education, 2009-2010

40% 35% 30% 25% 20% 15% 10% 5% 0% a b c d e Social relationships 1,7% 11,5% 28,2% 34,3% 24,3% Knowledge and skills 1,8% 13,4% 32,5% 34,4% 17,9% Note: Based on 33 States with highest quality data Early Childhood Outcomes Center 10 Action to meet needs 1,6% 10,8% 20,9% 35,6% 31,0%

100%

Estimated National Summary Statements for Early Childhood Special Education, 2009-2010

80% 60% 40% 20% 0% SS1 SS2 Social relationships 82,6% 58,7% Knowledge and skills 81,5% 52,3% Action to meet needs 81,9% 66,7% Note : Based on 33 States with highest quality data Early Childhood Outcomes Center 11

Criteria for States with Quality Data

12 1. Low percentage of missing data 2. No odd patterns in “a” or “e” categories 3. Did not use questionable data collection methods Early Childhood Outcomes Center

Calculating Missing Data for Part C

Proxy for missing data = Number with data for C3/ Exiting Data (618) 13 • Do not expect this number to be 100% • ..but we don’t expect it to be 10% either Early Childhood Outcomes Center

Part C: Percent of Exiters included in Outcomes Data 08-09 <10% = 10* 10- 20% = 4 20- 30% = 8 30- 40% = 11 40- 50% = 8 50- 60% = 8 60- 70% = 4 70- 80% = 2 >80% = 1 09-10 <10% = 5* 10- 20% = 4 20- 30% = 6 30- 40% = 8 40- 50% = 5 50- 60% = 11 60- 70% = 9 70- 80% = 1 >80% = 0 *3 states are sampling for Part C. Excluded states with <28%.

Calculating Missing Data for 619

15 Proxy for missing data = Number with data for B7/ Child count • Do not expect this number to be 100% • ..but we don’t expect it to be 10% either

Percent of Child Count included in Outcomes Data for ECSE 08-09 <10= 11* 10- 20%= 15 20- 30%= 12 30- 40%= 12 40-50% =1 >50% = 2 09-10 <10= 6* 10- 20%= 11 20- 30%= 12 30- 40%= 16 40-50% =4 >50%= 0 *4 States are sampling for 619 Excluded states with <12% of child count

17

Problem with Missing Data

• • We don’t know how well the data the state has represent the entire state.

If the data are representative, the percentages for the a to e Progress Categories and the Summary Statements won’t change as data are added on more children.

Early Childhood Outcomes Center

Percent Reported in " a“ for Knowledge and Skills for ECSE by State

16% 14% 12% 10% 8% 6% 4% 2% 0% 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 1 2 3 4 5 6 7 8 Exclusion criteria: a<10% in any outcomes e<65% in any outcomes 18 Early Childhood Outcomes Center

19

Can we trust these data?

Early Childhood Outcomes Center

Pattern checking for validity

• Checking across years – How do the 2009-10 compare to the data for 2008-09?

• Checking across methods – How do the data for all states compare to states with highest quality data?

Early Childhood Outcomes Center 20

Part C, Outcome A: Social Relationships

SS1: % who Increased Growth Rates

80 70 60 50 40 30 20 10 0 64,5 70,2 64 71 All states 19 best FFY 08-09 All states 29 best FFY 09-10

SS2: % who Exited at Age Expectations

80 70 60 50 40 30 20 10 0 67 61,3 64,7 61,8 All states 19 best FFY 08-09 All states 29 best FFY 09-10

Part C, Outcome B: Knowledge and Skills 80 70 60 50 40 30 20 10 0

SS1: % who Increased Growth Rates

76,6 70,4 68,1 74,2 All states 19 best FFY 08-09 All states 29 best FFY 09-10

SS2: % who Exited at Age Expectations

80 70 60 50 40 30 20 10 0 59 53,9 55,8 53,8 All states 19 best FFY 08-09 All states 29 best FFY 09-10

Part C, Outcome C: Meets Needs 80 70 60 50 40 30 20 10 0

SS1: % who Increased Growth Rates

76 69,7 68,1 75,6 All states 19 best All states 29 best FFY 08-09 FFY 09-10

SS2: % who Exited at Age Expectations

80 70 60 50 40 30 20 10 0 64,6 60,5 61,5 59,7 All states 19 best FFY 08-09 All states 29 best FFY 09-10

Part B Preschool: Social Relationships 30 20 10 0 90 80 70 60 50 40

SS1: % who Increased Growth Rates

77,2 82,7 79,7 82,6 All states 15 best FFY 08-09 All states 33 best FFY 09-10

SS2: % who Exited at Age Expectations

30 20 10 0 90 80 70 60 50 40 61,7 58,7 62,1 58,7 All states 15 best FFY 08-09 All states 33 best FFY 09-10

Part B Preschool: Knowledge and Skills 90 80 70 60 50 40 30 20 10 0

SS1: % who Increased Growth Rates

76,2 82,7 78,3 81,5 All states 15 best FFY 08-09 All states 33 best FFY 09-10

SS2: % who Exited at Age Expectations

90 30 20 10 0 80 70 60 50 40 55,8 51,2 55,5 52,3 All states 15 best FFY 08-09 All states 33 best FFY 09-10

Part B Preschool: Meets Needs

SS1: % who Increased Growth Rates

30 20 10 0 90 80 70 60 50 40 75,3 81,6 78,1 81,9 All states 15 best All states 33 best FFY 08-09 FFY 09-10

SS2: % who Exited at Age Expectations

90 80 70 60 50 40 67,8 67,2 66,7 66,7 30 20 10 0 All states 15 best All states 33 best FFY 08-09 FFY 09-10

Part C

Part C

29

What to these data tell us?

• • Nationally, a high proportion of children who receive Part C and ECSE services are showing greater than expected progress Nationally, many (over half) are exiting the program functioning like same age peers in at least one of the outcomes.

Early Childhood Outcomes Center

30

Should your state data look like the national data?

• • Probably not More important that each state continue to focus on the quality of its own data – Getting outcomes data on all children who exit – Working with programs whose data look unusual to address possible data quality issues Early Childhood Outcomes Center

Supporting States in Building a Child Outcomes Measurement System

32

Two Frameworks

• • Child Outcomes Measurement System Family Experiences and Outcomes Measurement System Early Childhood Outcomes Center

33

Child Outcomes Measurement System

• The set of components a state needs to have in place to make full use of child outcomes data.

• NOT just a data system or a data collection method.

Early Childhood Outcomes Center

Purpose of the Framework

• Provide a common language for ECO and other TA providers to use in discussing COMSs with states.

• Provide a organizing structure of categorizing resources and state examples related to implementation of a COMS.

• Serve as the organizing structure for the state self assessment Early Childhood Outcomes Center 34

Framework and Self-Assessment

• •

F RAMEWORK

– Set of components and quality indicators – Provides the structure for the self assessment

S ELF ASSESSMENT

– Scale that provides criteria for levels of implementation within each quality indicator – Rating assigned based on level of implementation within each indicator Early Childhood Outcomes Center 35

Process for Framework Development

• • • Built off what we had learned from ECO work with states Literature review Repeated discussion and review internally and with 7 Partner States Early Childhood Outcomes Center 36

Framework Partner States

Part C State

California Colorado Delaware Maine Minnesota New York Ohio Early Childhood Outcomes Center X X X X X X

619

X X X X X X 37

Purpose Data Collection and Trans mission Analysis Reporting Using Data Cross-system Coordination

38

Evaluation

Early Childhood Outcomes Center

39

Quality Indicator

• Provides additional detail as to what constitutes quality implementation of the component.

• 18 quality indicators across the 7 components Early Childhood Outcomes Center

Components Purpose Data Collection and Transmission Analysis Reporting Using Data Evaluation Cross-System Coordination 40 Quality Indicators 1.

2.

Purpose

State has articulated purpose(s) of COMS.

Data Collection and Transmission

Data collection procedures are carried out efficiently and effectively. 3.

4.

5.

6.

7.

Providers, supervisors, and others involved in data collection have the required knowledge, skills, and commitment.

State's method for entering, transmitting, and storing data is effective and efficient.

Analysis

State identifies accountability and program improvement questions related to child outcomes.

Local programs identify accountability and program improvement questions related to child outcomes.

State agency analyzes data in a timely manner.

8.

9.

Local programs analyze data in a timely manner.

State agency ensures completeness and accuracy of data.

Reporting

10. State agency interprets, reports, and communicates information related to child outcomes. 11. Local programs interpret, report, and communicate information related to child outcomes.

Using Data

12. State agency makes regular use of information on child outcomes to improve programs. 13. Local programs makes regular use of information on child outcomes to improve programs.

Evaluation

14. State evaluates its COMS regularly.

Cross-system Coordination

15. Part C and 619 coordinate child outcomes measurement.

16. Child outcomes measurement is integrated across early childhood (EC) 17.

programs statewide.

Child outcomes measurement is aligned with state’s early learning guidelines/standards.

18. State has a longitudinal data system to link child outcomes data from EC program participation to K –12 data.

Early Childhood Outcomes Center Elements a. State has… b. State has… c. State agency..

d. Representative..

e. State agency… f. State …… g. State provides… h. State has..

Early Childhood Outcomes Center 41

Purpose of the Self Assessment

• Provide guidance to states on what constitutes a high quality child outcomes measurement system.

• • Assist states in setting priorities for improving their COMS Provide information to assist states in advocating for resources for systems development Early Childhood Outcomes Center 42

• Each QI has multiple elements.

• Evidence for the extent of implementation for each element is provided.

• Each element is rated as • NY = Not Yet or Don’t know • IP = In Process, or • IF = Fully Implemented • The QI is given a rating based on the ratings of the elements.

Early Childhood Outcomes Center 43

Back-up for Each Element

• • • • • Live link from the element Describes the element Describes what “fully implemented looks like” • Provides examples of what “In process” might look like Provides examples of how states are addressing the element Provides additional resources related to the element Early Childhood Outcomes Center 44

The Scale for the Quality Indicators

Implementation of Elements

All elements are fully implemented Nearly all elements are fully implemented and the rest are in process Most of the elements are fully implemented and the rest are in process.

At least one element is fully implemented and the rest are in process All of the elements are in process Some of the elements are in process None of the elements are yet in process Early Childhood Outcomes Center

Quality Indicator Score

7 6 3 2 1 5 4 45

46 Early Childhood Outcomes Center

47 Working with the Self Assessment Early Childhood Outcomes Center

Possible Process

48 1. State identifies the highest priority QI(s).

2. Stakeholders convened – Priority QIs reviewed and rated – Plan developed to address elements not yet fully implemented.

– Progress reviewed with stakeholders at regular intervals.

3. State identifies next set of QI(s), etc. Early Childhood Outcomes Center

Additional information

49 For information on improving data quality and using data for program improvement

www.the-eco-center.org

Measuring and Improving Child and Family Outcomes Conference, September 18-21, 2011 in New Orleans, LA

Early Childhood Outcomes Center