Transcript Wyoming DDD Monitoring Manual

The Current Status of States'
Early Childhood Outcome
Measurement Systems
Kathy Hebbeler, SRI International
Lynne Kahn, FPG Child Dev Inst
October 17, 2009
DEC's 25th Annual International Conference on Young
Children with Special Needs and Their Families
1
We’ve
come a
long way,
BABY!
2
In the Beginning- Federal
Accountability
• 1994 Government Performance and
Results Act (GPRA)
• 2002 Performance Assessment Rating
Tool (PART)
– Both Part C and Part B Preschool categorized
as “Results Not Demonstrated” due to lack of
outcome data.
– OMB recommended OSEP develop a strategy
for collecting outcome data
The Development of Outcome
Statements
• 2004-early 2005: ECO generates
discussion and gathers input on child and
family outcomes
• Summer 2005: OSEP announces the child
and family outcomes States must report
on through their SPP/APRs
Early Childhood Outcomes Center
4
Three Child Outcomes
Percent of children who demonstrate
improved:
– Positive social emotional skills (including
positive social relationships)
– Acquisition and use of knowledge and
skills (including early language/
communication [and early literacy])
– Use of appropriate behaviors to meet
their needs
5
Early Childhood Outcomes Center
Key Concepts to Communicate
• Outcomes are functional–
term has multiple
definitions
• Each outcome is integrated
across domains
• Challenge: Reporting
language does not clearly
define the outcomes
Early Childhood Outcomes Center
6
Reporting Child Progress
• December 2005: States
submit plans via SPP on
how they will collect
outcome data
• September 2006: OSEP
finalizes the child outcome
reporting categories
Early Childhood Outcomes Center
7
Child Progress- the % of
children who
a. did not improve functioning
b. improved functioning but not sufficient to move nearer to
functioning comparable to same-aged peers
c. improved functioning to a level nearer to same-aged
peers but did not reach it
d. improved functioning to reach a level comparable to
same-aged peers
e. maintained functioning at a level comparable to sameaged peers
Early Childhood Outcomes Center
8
Key Concepts to Communicate
• Progress categories
require 2 data points for
each child,
• are based on growth
trajectories,
• compare a child to him
or herself over time, and
also compare each child
to age expectations
Early Childhood Outcomes Center
9
Entry
Exit
Early Childhood 10
Outcomes Center
States Submit Data to OSEP in
their Performance Reports
•
•
•
Feb 2007: “status at entry”
data
Feb 2008: child progress
data for children who
exited 7/1/06 through
6/30/07
Feb 2009: child progress
data for children who
exited 7/1/07 through
6/30/08
Early Childhood Outcomes Center
11
State Approaches to Measuring
Child Outcomes
Part C
(56 states/jur)
7/56 (13%)
Preschool
(59 states/jur)
9/59 (15%)
Publishers’ online
analysis
3/56 (5%)
6/59 (10%)
COSF 7 pt. scale
Other
41/56 (73%)
5/56 (9%)
38/59 (64%)
7/59 (10%)
Approach
One tool statewide
Number of Children Included in
Feb ‘09 SPP/APR Data
Part C (56)
Range: 5-6452
<30 = 3
30-99= 10
100-499= 25
500-999= 6
1000+ = 12
Preschool (59)
Range: 3-10157
<30 = 3
30-99 = 9
100-499= 14
500-999= 10
1000+ = 22
Key Concepts to Communicate
• Meaningfulness of the
progress categories
• Establishing expectations
for how the percentages
“should” look
• Exploring relationships
between state eligibility
definitions and
expectations for outcomes
Early Childhood Outcomes Center
18
Setting Targets for Improving
Child Outcomes
• 2008 and 2009: State input into summary
statements to be the basis of target setting
• Currently: summary statements have
been out for public comment, and are in
draft form in SPP Measurement tables
• February 2010: Baseline data and target
setting on summary statements
Summary Statement Data
• Required Summary Statement 1:
Of those children who entered or exited the program
below age expectations in each Outcome, the
percent who substantially increased their rate of
growth by the time they exited the program.
• Required Summary Statement 2:
The percent of children who were functioning within
age expectations in each Outcome by the time they
exited the program.
20
Key Concepts to Communicate
• The meaning of the
summary statements
• Ensuring that data is
clean before making
program changes
based on it
• Choosing/ developing
strategies that will
improve child outcomes
Early Childhood Outcomes Center
21
So where
are the
states
now?
States and Outcomes
Measurement
• Many states have embraced the ideas of
measuring outcomes
– Want information for own purposes
– Program improvement
• The power of GOOD outcomes data is
widely recognized
States and Outcomes
Measurement
• States have invested resources and time in
building an outcomes measurement system.
–
–
–
–
Involving stakeholders
Identifying an approach
Training providers in approach
Building a data system
And collecting data!
States have made incredible
progress
• States started at various
times
• Have varying amounts of
data
• Examining their data (and
procedures)
– The Quality Question??
This is VERY DIFFICULT work
Some of the Challenges
• Need for good assessment tools
– Need for more providers skilled in assessment
• Need strong grounding in typical child development
• Adequate training statewide in data collection
procedures
– Understanding and commitment
• Resources for new data systems
• Capability to analyze, report and interpret data
– Generating reports for local programs
• Resources in general
Next steps:
Putting it all
together
Making Good Use of Data:
Some Necessary Ingredients
• Set of good questions
• Data set with the required elements
• Analytic capability to analyze the data to
address the question
• Commitment to use the information for
program improvement
Early Childhood Outcomes Center
28
Examining and Tweaking the
Service System
Reflect
Is there a problem?
Are we where we
want to be?
Is it working?
Why is it happening?
What should be done?
Check
Plan (vision)
(Collect and
analyze data)
Program characteristics
Child and family outcomes
Implement
Is it being done?
Information Infrastructure:
Data Needed for Program Improvement
WHO
SERVICES
OUTCOMES
QUALITY
Early Childhood Outcomes Center
COST
30
Where is your state (or program) now?
WHO
SERVICES
OUTCOMES
What do you have?
How much is linked?
Early Childhood Outcomes Center
31
Tracking
outcomes
over time
Grade 2
Grade 1
Kindergarten
Preschool
An outcomes
measurement system
is more than hardware,
software, and data
Framework Development
• ECO is working with 7 partner states over
the next two years to develop a framework
for outcomes measurement.
• Framework provides the basis for a self
assessment to allow states to track their
own progress in building an outcomes
measurement system.
Areas of DRAFT Outcomes
Measurement System Framework
Vision for What Is to Be Learned
Data Collection
and
Transmission
Analysis
Evaluation
Information
Sharing
Action
Data Collection and
Transmission
• State has a method capable of accurately
capturing outcomes information
• State has effective procedures for
professional development around data
collection.
• State has effective procedures for moving
data from the local to the state level
• Etc.
State Self Assessment
• Is each area of the outcomes
measurement system implemented?
• Is the area implemented in a way
that reflects high quality?
We’ve
come a
long way,
BABY!
39
…and we still have a long way to
go…
...but won’t it
be great when
we get there?
For more information
(and for a copy of this presentation)
www.the-eco-center.org