Transcript Early Childhood Outcomes: Using Data for Program Improvement Kathy Hebbeler ECO at SRI International Robin Rooney and Christina Kasprzak ECO at Frank Porter Graham Institute Illinois October, 2008

Early Childhood Outcomes:
Using Data for Program
Improvement
Kathy Hebbeler
ECO at SRI International
Robin Rooney and Christina Kasprzak
ECO at Frank Porter Graham Institute
Illinois
October, 2008
Objectives




Understand the purposes of the child outcomes
data collection
Be familiar with key considerations related to
accurately completing the Child Outcomes
Summary Form
Be able to evaluate child outcomes data for
accuracy as part of supervision
Understand the use of child outcomes data for
program improvement, including for instruction,
resource allocation, and professional
development
Early Childhood Outcomes Center
2
Why are we doing this?
Early Childhood Outcomes Center
3
Keeping our eye on
the prize:
High quality services
for children and
families that will lead
to good outcomes.
Early Childhood Outcomes Center
4
Goal of early childhood
special education
“…To enable young children to be active
and successful participants during the early
childhood years and in the future in a
variety of settings – in their homes with
their families, in child care, in preschool or
school programs, and in the community.”
(from Early Childhood Outcomes Center,
http://www.fpg.unc.edu/~eco/pdfs/eco_outcomes_4-13-05.pdf)
Early Childhood Outcomes Center
5
High Quality Data on Outcomes

Data are a piece of a system that
helps to achieve overarching goals for
children and families

Data yield

Evidence that allows you to make an
inference that should lead to specific
actions to improve the system.
Early Childhood Outcomes Center
6
System for Producing Good Child and
Family Outcomes
Adequate funding
Good
Federal
policies and
programs
Good State
policies and
programs
Good Local
policies and
programs
Strong Leadership
Prof’l Development
•Preservice
•Inservice
High quality
services and
supports for
children 0-5
and their
families
Good
outcomes
for
children
and
families
The Vision: Using Data as a Tool for
Program Improvement

Illinois will have quality data available
on an ongoing basis about multiple
components of the system

Outcomes for children and families

Programs and services provided

Personnel (types, qualifications, etc.)

Etc.
Early Childhood Outcomes Center
8
Driving Force for Data on Child Goals
Comes from the Federal Level



Government Performance and Results
Act (GPRA)
Program Assessment Rating Tool
(PART)
Individuals with Disabilities Education
Act (IDEA)
Early Childhood Outcomes Center
9
Government Performance and Results
Act (GPRA) passed in 1993
 Requires goals and indicators be
established for IDEA
 Indicators and data collection further along
for school age population than for EC
 Previously, for early childhood data had
been collected on:
• Number of children served
• Settings
Early Childhood Outcomes Center
10
PART evaluation results (2002)
 130 programs examined in 2002; 50%
programs had no performance data
 Programs looking at inputs, not results
 Part C and Section 619

No long-term child outcome goals or data

Need to develop a strategy to collect
annual performance data in a timely
manner
Early Childhood Outcomes Center
11
Federal Funding for Preschool Special
Education
Total U.S.
Illinois
2004
$387,699,000
$17,943,956
2005
$381,385,691
$17,812,936
2006
$380,751,030
$17,650,453
Early Childhood Outcomes Center
14
Early Childhood Outcomes Center
15
Individuals with Disabilities Education Act
SEC. 616. <<NOTE: 20 USC 1416.>>
MONITORING, TECHNICAL ASSISTANCE, AND
ENFORCEMENT. ``(a) Federal and State Monitoring.…..
…..
``(2) Focused monitoring.--The primary focus of
Federal and State monitoring activities described in
paragraph (1) shall be on-- ``(A) improving educational
results and functional outcomes for all children with
disabilities;
Early Childhood Outcomes Center
16
How Office of Special Education
(OSEP) responded

Required states to submit outcome
data in their Annual Performance
Report (APR)

Funded the Early Childhood Outcomes
(ECO) Center to do research, make
recommendations, and assist states
Early Childhood Outcomes Center
17
Where are we now:
Federal reporting requirements
Early Childhood Outcomes Center
18
OSEP Reporting Requirements:
the Outcomes

Positive social emotional skills
(including positive social
relationships)

Acquisition and use of knowledge and
skills (including early language/
communication [and early literacy])

Use of appropriate behaviors to meet
their needs
Early Childhood Outcomes Center
19
OSEP Reporting Categories
Percentage of children who:
a. Did not improve functioning
b. Improved functioning, but not sufficient to move
nearer to functioning comparable to same-aged
peers
c. Improved functioning to a level nearer to sameaged peers but did not reach it
d. Improved functioning to reach a level comparable
to same-aged peers
e. Maintained functioning at a level comparable to
same-aged peers
3 outcomes x 5 “measures” = 15 numbers
Early Childhood Outcomes Center
20
Reporting Schedule



Due February 2008
Data in reporting
categories at exit for all
children who have been
in the program for at
least 6 months
Must be reported for the
year beginning July 1,
2006
Repeat with next year’s
data in 2009, etc.



2010
States must set targets
Summary statements
for targets
recommended; not yet
official*
States must report
data to public by
school districts for
these target numbers
*See http://www.fpg.unc.edu/~ECO/meetings.cfm for more information on
recommended targets
Early Childhood Outcomes Center
21
Where are we now:
State decisions and activities
Early Childhood Outcomes Center
22
WHY?
(State Version)
Purpose
To respond to
federal
reporting
requirements
Early Childhood Outcomes Center
To have data for
program
improvement
and
to respond to
federal reporting
requirements
23
WHY?
(Local Version)
Purpose
To provide
data to the
state
Early Childhood Outcomes Center
To have data for
program
improvement
and
to provide data to
the state
24
State approaches

Most states have embraced outcomes
measurement and are collecting outcomes
data for their own purposes.

Many states are building bigger systems
than needed to produce the federal data.

Go to www.the-eco-center.org for more
information about what other states are
doing
Early Childhood Outcomes Center
25
How are states collecting child
outcomes data?
Possible state approaches to collection of
child data

Child Outcomes Summary Form (COSF)

Publisher’s online assessment system

Single assessment statewide

Other approaches
Early Childhood Outcomes Center
26
State Approaches to Measuring
Child Outcomes
Approach
One tool
statewide
Multiple
Publishers’ online
tools
COSF 7 pt. scale
Other
Part C
Preschool
(56 states)
(59 states)
8/56 (14%)
13/59 (22%)
2/56 (4%)
3/59 (5%)
40/56 (71%)
36/59 (61%)
6/56 (11%)
7/59 (12%)
Checking in – How is it going?
Implementing the Child Outcomes
Summary Form Process

What is going well?

What is not going well?

Are there other issues or concerns that
need to be addressed about the
process?
Early Childhood Outcomes Center
28
Let’s Review:
Child Outcome
Summary Form
Early Childhood Outcomes Center
29
Learning from Each Other
1.
2.
Did you attend a training on the COSF?
Have you participated in a COSF team rating
process?

3.
4.
5.
For more than 10 children? 20? 30?
Have you reviewed COSF forms completed by
others?
Have you provided training on the COSF?
How confident are you in your knowledge of the
COSF process?
Early Childhood Outcomes Center
30
Essential Knowledge for Completing
the Child Outcomes Summary Form
Between them, team members must:
1. Know about the child’s functioning across settings
and situations
2. Understand age-expected child development
3. Understand the content of the three child outcomes
4. Know how to use the rating scale
5. Understand age expectations for child functioning
within the child’s culture
Early Childhood Outcomes Center
31
Important point

It is not necessary that all team
members be knowledgeable in all 5
areas

Especially, no expectation that parents
understand the rating scale or typical
child development

But the professionals have to!
Early Childhood Outcomes Center
32
Essential Knowledge for Completing
the Child Outcomes Summary Form
Between them, team members must:
1. Know about the child’s functioning across
settings and situations
2.
Understand age-expected child development
3.
Understand the content of the three child outcomes
4.
Know how to use the rating scale
5.
Understand age expectations for child functioning within the
child’s culture
Early Childhood Outcomes Center
33
1. Know about the child’s functioning
across settings and situations
How we learn about
the child’s functioning
across settings and
situations:
Good
assessment
Early Childhood Outcomes Center
34
DEC* recommended practices
for assessment

Involve multiple sources


Examples: family members, professional team
members, service providers, caregivers
Involve multiple measures

Examples: observations, criterion- or curriculumbased instruments, interviews, norm-referenced
scales, informed clinical opinion, work samples
*Division for Early Childhood
Early Childhood Outcomes Center
35
Assessment practices appropriate for
outcomes measurement: ASHA*
ASHA recommended practices:
 Gather information from families, teachers,
other service providers
 Collect child-centered, contextualized,
descriptive, functional information
(*American Speech-Language-Hearing Association)
Early Childhood Outcomes Center
36
Assessment instruments


Assessment the tool
vs. assessment the
process
Assessment tools can
inform us about
children’s functioning
in each of the three
outcome areas
Early Childhood Outcomes Center

Challenge:
There is no
assessment tool
that assesses the
three outcomes
directly
37
Essential Knowledge for Completing
the Child Outcomes Summary Form
Between them, team members must:
1.
Know about the child’s functioning across settings and
situations
2. Understand age-expected child
development
3.
Understand the content of the three child outcomes
4.
Know how to use the rating scale
5.
Understand age expectations for child functioning within the
child’s culture
Early Childhood Outcomes Center
38
Resources for understanding ageexpected child development

ECO link
http://www.fpg.unc.edu/~eco/pdfs/Age-expected_child_dev_9-5-07.pdf
(under “ECO Tools”)

New course coming soon

Watch ECO web site
www.the-eco-center.org
Early Childhood Outcomes Center
39
Essential Knowledge for Completing
the Child Outcomes Summary Form
Between them, team members must:
1.
Know about the child’s functioning across settings and
situations
2.
Understand age-expected child development
3. Understand the content of the three
child outcomes
4.
Know how to use the rating scale
5.
Understand age expectations for child functioning within the
child’s culture
Early Childhood Outcomes Center
40
Outcomes Jeopardy
Pointing to the
cabinet
$100for
cereal
Reading the
letter$100
“S” on the
Stop sign
Washes hands
$100
before lunch
Biting
$200
Plays by himself
$200
in the classroom
Plays with
$200
rhyming words
Building a castle
from blocks
$300 with
a friend
Problems
$300
sleeping
Sharing a cookie
$300
at lunchtime
Early Childhood Outcomes Center
41
Children have positive social
relationships

Involves:

Relating with adults

Relating with other children

For older children, following rules related to
groups or interacting with others
Includes areas like:

Attachment/separation/autonomy

Expressing emotions and feelings

Learning rules and expectations

Social interactions and play
Early Childhood Outcomes Center
42
Children acquire and use knowledge
and skills

Involves







Thinking
Reasoning
Remembering
Problem solving
Using symbols and language
Understanding physical and social worlds
Includes:





Early concepts—symbols, pictures, numbers
Imitation
Object permanence
Expressive language and communication
Early literacy
Early Childhood Outcomes Center
43
Children take appropriate action to meet
their needs

Involves:





Taking care of basic needs
Getting from place to place
Using tools (e.g., fork, toothbrush, crayon)
In older children, contributing to their own health
and safety
Includes:



Integrating motor skills to complete tasks
Self-help skills (e.g., dressing, feeding, grooming,
toileting, household responsibility)
Acting on the world to get what one wants
Early Childhood Outcomes Center
44
Essential Knowledge for Completing
the Child Outcomes Summary Form
Between them, team members must:
1. Know about the child’s functioning across settings
and situations
2. Understand age-expected child development
3. Understand the content of the three child outcomes
4. Know how to use the rating scale
5. Understand age expectations for child functioning
within the child’s culture
Early Childhood Outcomes Center
45
The two COSF questions
a. To what extent does this child show ageappropriate functioning, across a variety of
settings and situations, on this outcome?
(Rating: 1-7)
b. Has the child shown any new skills or
behaviors related to [this outcome] since the
last outcomes summary? (Yes-No)
Early Childhood Outcomes Center
46
7 – Completely



The child shows behaviors and skills
expected in all or almost all everyday
situations that are part of the child’s life
 Home, store, park, child care, with
strangers, etc.
The child’s functioning is considered
appropriate for his/her age
No one has significant concerns about the
child’s functioning in this outcome area
Early Childhood Outcomes Center
47
6 – Between completely
and somewhat


The child’s functioning generally is
considered appropriate for his or her
age
but there are some significant
concerns about the child’s functioning
in this outcome area
Early Childhood Outcomes Center
48
5 – Somewhat



The child shows functioning expected for
his/her age some of the time and/or in
some situations
The child’s functioning is a mix of ageappropriate and not appropriate functioning
The child’s functioning might be described
as like that of a slightly younger child
Early Childhood Outcomes Center
49
4 – Between a 5 and a 3


Child shows some age appropriate
functioning some of the time or in some
situations or settings but most of the child’s
functioning would be described as not yet
age appropriate.
The child’s functioning might be described
as like that of a younger child
Early Childhood Outcomes Center
50
3 – Emerging



The child does not yet show functioning
expected of a child his/her age in any
situation
The child’s behaviors and skills include
immediate foundational skills on which
to build age-appropriate functioning
The child’s functioning might be described
as like that of a younger child
Early Childhood Outcomes Center
51
2 – Between 3 and 1



The child does not yet show functioning
expected of a child his/her age in any
situation
The child’s behaviors and skills does have
some the immediate foundational skills on
which to build age-appropriate functioning
but these are not displayed very often
The child’s functioning might be described
as like that of a younger or even much
younger child
Early Childhood Outcomes Center
52
1 – Not yet




The child does not yet show functioning
expected of a child his/her age in any
situation
The child’s skills and behaviors also do not
yet include any immediate foundational
skills on which to build age-appropriate
functioning
The child’s functioning might be described as
like that of a much younger child
Children with 1 ratings still have skills, just not
yet at an immediate foundational level
Early Childhood Outcomes Center
53
Rating Scale Jeopardy
Age appropriate
functioning
$100– no
concerns
Mix of age
appropriate and not
age $100
appropriate
functioning
No age appropriate
functioning – not yet
$100
showing
immediate
foundational skills
Some age
appropriate
$200
functioning
but very
little
No age appropriate
functioning – lots of
$200
immediate
foundational skills
Age appropriate
functioning
$200– some
concerns
Rarely shows age
appropriate
$300
functioning
No age appropriate
functioning – some
$300
immediate
foundational skills
Age appropriate
$300
functioning
Early Childhood Outcomes Center
54
Early Childhood Outcomes Center
55
Essential Knowledge for Completing
the Child Goals Outcome Form
Between them, team members must:
1. Know about the child’s functioning across settings
and situations
2. Understand age-expected child development
3. Understand the content of the three child outcomes
4. Know how to use the rating scale
5. Understand age expectations for child
functioning within the child’s culture
Early Childhood Outcomes Center
56
Point of clarification

Process is NOT about comparing
groups of children – it IS about asking
how close children are to being able to
do what is expected at their age

Early learning guidelines

Kindergarten and access to the
general curriculum
Early Childhood Outcomes Center
57
What we are learning nationally
The process of training for child
outcomes data collection has
uncovered other areas of
significant need related to
professional development.
Early Childhood Outcomes Center
58
Providers need to know more about:

Assessment

How to gather assessment data to reflect
functioning across settings and situations,
especially how to gather child functioning
information from families

Understanding the results of the assessment

Sharing assessment results sensitively and
honestly with families
Early Childhood Outcomes Center
59
Providers need to know more about:

Functional outcomes




What are they?
How do they differ from outcomes organized
around domains?
What do they mean for how professionals from
different disciplines operate as a team?
Typical child development

What are the functional expectations for
children at different ages with regard to each of
the 3 goal statements?
Early Childhood Outcomes Center
60
Let’s meet Ava
Early Childhood Outcomes Center
61