Transcript ECO Longitudinal - OSEP Leadership Mtng

Catch 22s in Using Assessments in
Accountability Systems for
Young Children
Lynne Kahn
Scientist, Frank Porter Graham Child
Development Institute
The Early Childhood Outcome Center
What I’ll talk about
• The experiences of the ECO Center in
determining how to measure outcomes for
young children 0-5 with disabilities and
their families
• Focusing in on some of the dilemmas
posed by the current state of “assessment”
of young children
Early Childhood Outcomes Center
2
Context: Thinking on assessment and
accountability has changed in the early
childhood community
• Old position: We don’t
want to test young
children
• New position: Ongoing
assessment is part of a
high quality early
childhood program
• New reality: We HAVE to
be accountable for results
3
First dilemma: what outcomes are
we accountable for?
• What are the underlying
purposes of the program?
• Does “measurable” mean
identifying outcomes that
match assessment tools?
• Or do we measure those
we value- hard to
measure or not?
Early Childhood Outcomes Center
4
Next dilemma: how to measure the
intended outcomes
• What assessment tools were
already in use?
• What types of assessment
tools and approaches are
appropriate for young
children
• IT DEPENDS ON WHO YOU
ASK!
Early Childhood Outcomes Center
5
The evolving context of assessment
• Assessment tools have
become increasingly used
over the last 15 years for
Early Childhood
– Curriculum-based
assessments, e.g.,
Creative Curriculum>
GOLD, Work Sampling,
etc.
– Tools for 3-5 came first;
then tools for 0-3
6
What changed?
• The purpose of assessment was redefined
• Not about: sorting, labeling, using to deny
access
• Now about: Getting a rich picture of what
children can do and can’t do and using that
information to help them acquire new skills
– “progress monitoring”
7
What we found
• A large number of
different assessment
instruments are currently
being used in programs
for young children
• Most of these instruments
were developed for and
are being used for
purposes other than
accountability
8
Another dilemma: Can we use these
assessments for multiple purposes?
• Can assessments already
being used by programs for
other purposes (whatever
they are) be used for
accountability purposes?
• Does this apply to all tools
or only some categories of
tools?
9
Catch 22: Validity vs. credibility
dilemma for accountability
Strangers can’t elicit valid data on
young children’s performance
capabilities in a testing situation
BUT
can data produced by those who
know the child and whose programs
are being evaluated, be credible in an
accountability system?
10
Problem: Nature of the young child
• Not well suited to a standardized testing
situation
• Performance varies from day to day, place
to place, person to person
• Don’t perform well for strangers or on
demand
• Growth is sporadic and uneven
11
Problem: Psychometric properties of existing
instruments
• Some of the most
common instruments are
being used with limited or
no reliability and validity
data
• None have validity or
reliability data related to
use for outcomes and
accountability
12
Response: the most “reasonable”
forms of assessment
• Growing recognition that the only way to get a
valid picture of what a child can do/does to is
look at performance over a variety of settings
and people including what the child does
spontaneously with familiar adults
• Can’t base conclusions about child’s capabilities
on elicited responses alone
• “Authentic assessment”
13
Response: Use multiple sources of
information (best practice)
“A single test, person, or occasion is not a
sufficient source of information. This means that
we must gather information from several
sources, instruments, settings and occasions to
produce the most valid description of the child’s
status or progress”
• ---DEC Recommended Practices
14
Yet another dilemma
• How do we synthesize
information from multiple
sources?
• Are there current
assessment tools designed
to collect information from
multiple sources?
15
Issue: Role of families in the
assessment process
• Families provide a unique
perspective on the child’s
functioning
• Few assessment tools have
procedures for incorporating
that perspective
• Need good tools/procedures
for learning about child from
the family
16
For children with disabilities
• We developed a process
for summarizing data from
multiple sources
• A rubric linked to age
expectations
• A common metric with
flexibility for multiple data
sources
Early Childhood Outcomes Center
17
Accountability for family
outcomes?
• What are the underlying
purposes of the
programs for families?
• What’s measurable?
• What do we value?
• How do we measure it?
Early Childhood Outcomes Center
18
Family Assessment Dilemma
• Assessing families with direct
assessments or using family
self report?
• Direct assessment (e.g.
pre/post) is often perceived as
intrusive, non-collaborative
• Family self report of gains or
changes are not always seen
as credible
Early Childhood Outcomes Center
19
Alternate Approaches
• Various approaches to measuring
coaching outcomes
– Review and coding videos
– Coaching logs
• BUT- in early childhood programs for
young children with disabilities and their
families, self report is used almost
exclusively
Early Childhood Outcomes Center
20
In summary: this isn’t a win-win
• There are no perfect answers
• Accept the disadvantages you can live
with
• Implement what you choose with special
attention to approaches to improving the
quality of the data.
Early Childhood Outcomes Center
21
Thank you
For more information go to
www.the-eco-center.org
Early Childhood Outcomes Center
22