Transcript ECO Overview New Mexico

The How and Why of Functional Child
Outcomes in an Outcomes
Measurement System for Part C and
Part B Preschool Programs
Donna Spiker
Kathy Hebbeler
Steve Bagnato
Conference on Research Innovations in
Early Intervention
San Diego, California
February 2008
In this session, we will:






Provide a brief history of accountability systems
for Part C and Part B preschool special
education.
Describe the current Federal outcomes reporting
requirements.
Discuss the 3 OSEP functional child outcomes;
how and why we arrived at them.
Describe assessment dilemmas posed.
Present on Research Foundations on
Assessment for Early Childhood Intervention.
Pose questions for dialogue about research and
practices about assessment for early childhood
intervention regarding functional outcomes in
an accountability system.
2
A provocative quote…..
“
You can’t test four-year-old kids –
it’s unreliable.”
Comment by Ed Zigler in the Jan/Feb 2008 issue
of Mother Jones magazine about the Head
Start National Reporting System
3
Challenges in measuring child
outcomes for children with disabilities




Outcomes are individualized; no
consensus on general outcomes.
Extreme diversity in functional levels in
the population of children served.
Variety of assessment tools used because
no single tool appropriate for the entire
population.
Children with severe disabilities will show
poor outcomes – would policy-makers
conclude the programs were ineffective?
4
And a follow-up quote…..
“Head Start spends $7 billion a year.
Taxpayers and Congress have the
right to know whether it’s working.
But there can be good accountability
and lousy accountability.”
Comment by Ed Zigler in the Jan/Feb 2008
issue of Mother Jones magazine about the
Head Start National Reporting System
5
SO……the accountability question
and dilemma
How do we produce valid
data for accountability
on programs for young
children with
disabilities, while
remaining true to what
we know about best
assessment practices?
6
Accountability for EI and preschool special
education programs: Background
7
Critical events in accountability for
programs for young children with
disabilities

1993

2002

2003

2005

2008
GPRA (Government Performance and
Results Act) passed
PART finds there are no data on
outcomes for Part C or 619
OSEP begins to ask states for EI child
outcome data (and funds the
ECO Center)
OSEP releases federal reporting
requirements
All states begin reporting first set of
outcomes data to OSEP
8
PART Findings:
“Results Not Demonstrated” (2002)
Part C: “While the program has met its goal
relating to the number of children served, it
has not collected information on how well
the program is doing to improve the
educational and developmental outcomes of
infants and toddlers served.”
Part B preschool: “The Department has no
performance information on preschool
children with disabilities served by the
program.”
9
GPRA to PART: Intervening years

Special Education (K-12)



National study found poor
outcomes.
Push to include students with
disabilities in statewide
assessment systems.
Early Childhood


Debate about whether child
outcomes should be
measured at all.
Much discussion of the many
problems in trying to measure
outcomes for young children
with disabilities.
10
Measuring child outcomes for
Part C and Part B preschool (619)


The PART findings in 2002
put an end to the debate
about whether or not to
do it.
Unfortunately, almost no
progress had been made
in the intervening years as
to HOW to do it.
11
OSEP’s response to PART findings


Required states to submit
outcome data in their Annual
Performance Report (APR).
Funded the Early Childhood
Outcomes (ECO) Center in Sept
2003 to build consensus, make
recommendations, and assist
states in developing systems to
measure outcomes.
12
What is the ECO Center?
5-year OSEP-funded project
Collaboration among:
 SRI International
 Frank Porter Graham Child Development
Institute
 Juniper Gardens Children’s Project
 RTI International
 University of Connecticut
13
What was the goal of the ECO
Center?


Promote the development and
implementation of child and family
outcome measures for infants, toddlers
and preschoolers with disabilities that can
be used in national and state
accountability systems:
 Outcomes data that can be aggregated
across all states and territories.
 Outcome systems in states that meet
the individual state’s need for data.
For more information, see: www.the-ecocenter.org
14
Federal government is the driving force
behind collection outcome data…….BUT…

Many state agencies and local
programs understand the value of
outcomes data for their own purposes:

To document program effectiveness


Increase in funding?
To improve programs:
Identify strengths and weaknesses.
 Allocate support resources, such as
technical assistance.

15
Were there any parameters from
OSEP about state reporting?

Givens from OSEP:



States would not be required to use one
instrument.
Data on % of IFSP/IEP goals attained would
not be acceptable.
No additional resources
would be provided for
data collection.
16
What stakeholder input did ECO
receive?







Reflect vision of what programs are trying to
accomplish; consistent with IDEA
One set of outcomes 0-5 and across types of
disabilities.
Use them to drive practice forward - consistent
with best practices; no harm.
Avoid domain organization.
Implementation issues to consider:
 Children participate in multiple accountability
systems.
 Providers are overburdened.
Statements should be easy to explain/understand.
Importance of family outcomes as well (similar,
but separate process).
17
Overarching goal of programs

To help children become active and
successful participants


Now and in the future.
In variety of settings:
 in
their homes with
their families,
 in childcare or school
programs, and
 in
the community.
18
The three OSEP functional child
outcomes



Children have positive socialemotional skills (including social
relationships).
Children acquire and use knowledge
and skills (including early
language/communication
[and early literacy]).
Children use appropriate behaviors
to meet their needs.
19
Children have positive
social relationships

Involves:




Relating with adults
Relating with other children
For older children, following rules
related to groups or interacting with
others
Includes areas like:




Attachment/separation/autonomy
Expressing emotions and feelings
Learning rules and expectations
Social interactions and play
20
Children acquire and use
knowledge and skills

Involves:






Thinking and reasoning
Remembering
Problem solving
Using symbols and language
Understanding physical and social worlds
Includes:





Early concepts—symbols, pictures,
numbers, classification, spatial relationships
Imitation
Object permanence
Expressive language and communication
Early literacy
21
Children take appropriate action
to meet their needs

Involves:





Taking care of basic needs
Using tools (e.g., fork, toothbrush, crayon)
Getting from place to place
In older children, contributing to their own
health and safety
Includes:



Integrating motor skills to complete tasks
Self-help skills (e.g., dressing, feeding,
grooming, toileting, household responsibility)
Acting on the world to get what one wants
22
Outcomes reflect global functioning

Each outcome is a snapshot of:




The whole child
Status of the child’s current
functioning
Functioning across settings
and situations
Rather than:



Skill by skill
In one standardized way
Split by domains
23
OSEP child outcomes reporting
categories
Percentage of children who:
a. Did not improve functioning
b. Improved functioning, but not sufficient to
move nearer to functioning comparable to
same-aged peers
c. Improved functioning to a level nearer to
same-aged peers but did not reach it
d. Improved functioning to reach a level
comparable to same-aged peers
e. Maintained functioning at a level comparable
to same-aged peers
3 outcomes x 5 “measures” = 15 numbers
24
Implemented on a very fast
timeline……


Aug. 2005 - OSEP released child
outcome reporting requirements (no
additional funding provided)
First data due Feb. 2007


Status data from children July 1, 2005June 30, 2006
Progress data entry to exit for children
in program at least 6 months – due the
following year and thereafter
25
Major state dilemmas

How to measure 3 functional outcomes with
data from domain-organized assessments that
often focus on more discrete skills?
 What kind of assessment(s) to use?
 Use of 1-to-1 standardized tools or
observational approaches?
26
Problem: Domains assessed are
interrelated

Children tend to develop in multiple areas
simultaneously.





Language, cognition, motor skills march
forward more or less together.
Even though development has been divided
into domains for assessment and research,
much of development is intertwined.
Performance on a given task often requires
using skills that cross several domains.
These interconnections present challenges for
obtaining a “pure” domain score.
These interconnections present challenges for
obtaining valid scores when a child has delays
in development in one or more domains.
27
What do we mean by functional
outcomes?
Important behaviors acquired and displayed
by young children in everyday settings
that cut across domains:


Integrated behaviors that are meaningful to
the child in the context of everyday living
Integrated patterns of behaviors or skills
that allow the child to achieve important
everyday goals
28
Functional outcomes are NOT


A single behavior
The sum of a set of discrete
behaviors or splinter skills
such as…..
*Knows 10 words
*Pincer grasp
(picks up a
raisin)
*Smiles at mom
*Stacks 3 blocks
*Goes up and
down stairs with
one foot on each
29
Functional outcomes




Not domains-based, not separating
child development into discrete areas
(communication, gross motor, etc.)
Refer to behaviors that integrate skills
across domains
Can involve multiple domains
Emphasize how the child is able to
carry out meaningful behaviors in a
meaningful context
30
Thinking functionally
(within age-expected bounds)
Isolated
skill/behavior



Knows how to
imitate a gesture
when prompted by
others
Uses finger in
pointing motion
Uses 2-word
utterances
Functional
skill/behavior



Watches what a peer
says or does and
incorporates it into
his/her own play
Points to indicate needs
or wants
Engages in back and
forth verbal exchanges
with caregivers using 2word utterances
31
How well do current assessments
measure functioning for young children?
32
Problem: Nature of the young child makes
accurate and reliable assessments difficult

Performance varies
from






day to day,
place to place,
person to person.
Don’t perform well
for strangers or on
demand.
Growth is sporadic
and uneven.
Not well suited to a
standardized testing
situation.
33
Problem: Difficulty of assessment
increases with special populations

Concerns with the accuracy
of assessments multiply with
children:



from diverse cultures/new
immigrants.
who are English language
learners.
with disabilities,
developmental delays, or
other special needs.
34
Problem: Issues with assessing
English language learners




Do they understand the directions?
Is the assessment tapping cognition or
language?
Do norms exist for
the population being tested?
Do cultural differences affect the child’s
interactions with the assessor in ways
that make the child’s performance an
inadequate indication of his/her
competence?
35
Problem: Issues with assessing
children with disabilities or delays




Assessment assumes that child can see,
hear, and understand spoken language,
point, etc.
Few assessments include
accommodations, nor were children with
disabilities included in the norming
sample.
Very little data on validity of
accommodations with young children.
Are other behavioral or attentional factors
influencing performance?
36
Recommended best practices for
assessing young children




Instruments only used for
intended purposes.
Assessors are well trained in
early child development and
assessment principles.
Assessors are knowledgeable
about child’s culture, and
assess children in their
dominant language.
Assessments should include
multiple sources of evidence
gathered over time.
37
More on best practices
from Steve Bagnato……
38
Questions for discussion

How do we reconcile:



Best practices in early childhood
assessment?
The need to measure functional outcomes
(and limitations of existing instruments
available)?
The need to measure for purposes of
accountability?
39
Applied issues for future research
and practice……





Design national measurement field-validation
research for specific EI purposes.
Design measures with universal-design
features.
Develop efficient and functional measures or
mapping procedures for accountability.
Change from domain-based to functional
whole-child focus.
Conduct research on authentic and alternative
assessment procedures that balance internal
and external validity.
40
Applied issues for future research
and practice….. (cont)



Reduce burden on providers by enabling
use of CBA measures already used by
programs or efficient probes for
accountability purposes mapped to
OSEP/state standards.
Collect and link child outcome measures to
measures of programmatic, family, &
contextual dimensions.
Consider advantages of applying an
overarching interdisciplinary framework for
disability outcome research, such as WHOICF-CY.
41
New Pew Charitable Trusts report
about early childhood accountability
Full report is available at:
http://www.pewtrusts.org/uploadedFiles
/wwwpewtrustsorg/Reports/Prek_education/task_force_report1.pdf
42
Additional references about EC
assessment




Where We Stand?—Curriculum, Assessment, &
Program Evaluation (NAEYC, 2002)
DEC Recommended Practices (Sandall et al,
2005; Neisworth & Bagnato, 2005)
DEC—Promoting Positive Outcomes—
Recommendations for Curriculum, Assessment,
and Program Evaluation (DEC, 2007)
Child Outcomes: Measuring Success in Head
Start (Children & Families, NHSA, 2003)
43
Contact information


Donna Spiker – [email protected]
Kathy Hebbeler –
[email protected]
Additional resources on
assessment and
accountability systems
for programs serving
young children: see
www.the-eco-center.org
44