Transcript Using Research to Advance the Practice and Impact of Early Education and Care Board of Early Education & Care: Presentation January 8, 2013

Using Research to Advance the
Practice and Impact of Early
Education and Care
Board of Early Education & Care:
Presentation
January 8, 2013
1
Thank you to Carol Craig O’Brien for her
hard work on the Board of Early
Education and Care and on the Planning
and Evaluation Committee.
2
Definition of Quality

3
EEC has defined quality through the 5
areas of the QRIS
 Curriculum and Learning
 Safe, Healthy Indoor and Outdoor
Environments
 Workforce Development and
Professional Qualifications
 Family and Community Engagement
 Leadership, Administration and
Management
EEC funded Research since 2009
4

Teacher Quality
 MA Professional Development System Study
 Project of the CAYL Institute
 Literacy/Social Emotional/Numeracy/Digital Strategies

Screening and Assessment
 Massachusetts Kindergarten Readiness Assessment System
 Massachusetts Common Metric
 Child Assessment in Universal Pre-Kindergarten

Program Quality
 Evaluation of Summer 2010 Out-of-School Time Literacy &
Learning Promotion Grant
 Institutions of Higher Education Mapping
 QRIS Participation Study
 QRIS Pilot Evaluation Phase I
 QRIS Pilot Evaluation Phase II
 QRIS Validation Study
EEC funded Research con’t

5
Community and Family Engagement
 Early Childhood Information Systems
 State Advisory Needs Assessment
 Home Visiting Grant Evaluation
 Limited English Proficiency
 MA Child Care Resource and Referral Agencies (CCR&R) System
A Study Circle
 Family Survey on Waitlist
 Massachusetts Head Start Collaboration Needs Assessment
Survey Report
 Early Education and Care Policies and Guidelines for Children
Whose Home Languages are other than or in addition to
English
 Massachusetts Market Price Survey
 Waitlist Analysis, Program Access Analysis, & Continuity of Care
Study
Examples of Key
Findings/Recommendations

Teacher Quality- MA Professional Development Study











6
Invest in supporting regional partnerships to implement the professional
development system
Continue to define the scope, limits, and purpose of regional authority and
responsibility particularly as it relates to innovation.
Assure a timely, accurate and high quality information flow from EEC to regional
partnerships
Assess outreach efforts to Priority A, dual language and family educators.
Continue progress made to foster reciprocal communication, or communication
loops, between EEC and regional partnerships to promote sharing of information
and perspectives and to mutually inform policy and practice.
Share efforts by EEC to address compensation issues and engage regional
partnerships in initiatives and efforts focused on compensation and incentives.
Implement utilization of MOU’s between educators and program directors, and
collect/share data regarding MOU use to inform policy and practice development.
Advance the regions’ understanding about specific strategies for aligning
professional development with QRIS.
Increase regional capacity around effective practices for assessing educator
competencies and individual pathways, and use educator/provider assessment in a
continuous cycle to informal professional development service planning and
delivery.
Continue efforts to provide lead agents with access to relevant and useful EEC data
on educators and providers.
Assess options for increasing the use of evidence-based professional development
approaches, and support regions in evaluating the effectiveness of professional
development services.
Examples of Key
Findings/Recommendations

Screening and Assessment- Child Assessment
Universal Pre- Kindergarten










7
Using web-based (or on-line) submissions as a way of collecting data should be a
requirement of UPK grantees
EEC web-based licenses should indicate if they are a UPK grantee and specify the
program’s UPK children.
While UPK grantees should be allowed to assess all of the children in their
programs through the on-line license, they must delineate the children that are
enrolled in UPK.
Increased and efficient communication with family child care grantees that belong
to a family child care system needs to be systematically planned in order to raise
submission rates.
Domain completion rates could be increased with professional development.
Professional development on how best to utilize the web-based system of assessing
children should be offered annually
To determine specific professional development needs for teachers as related to
early childhood assessment, EEC will need to depend on director interviews,
teacher interviews and/or focus groups or observation.
Professional development coupled with technical assistance should be a systematic
part of offering UPK grants and incentives.
Provide professional development and technical assistance to UPK grantees on how
they can use their aggregated child assessment data at the local level.
Massachusetts should continue its practice of encouraging UPK programs to assess
children using evidence-based assessment tools through grants and incentives to
track child outcomes.
Examples of Key
Findings/Recommendations

Screening and Assessment- Child Assessment
Universal Pre-Kindergarten con’t




8
Massachusetts should consider whether it wants to add additional tools to it
approved list, or whether it would in fact be better to narrow the choices to one or
two tools that are most often used.
Examine whether High Scope Child Observation Record should remain one of the
four EEC- approved child assessment tools due to low use.
Investigate new versions of the ASQ and the Creative Curriculum Developmental
Continuum to determine what, if anything, grantees need to incorporate them into
their data collection systems.
Consider whether ASQ should remain an approved tool.
Examples of Key
Findings/Recommendations

9
Program Quality- QRIS Participation Study

Non-Participating programs
• Verify and update program contact information
• Increase communication with programs
• Ensure communication is language-accessible
• Consider ways to streamline the process for hard-to-reach programs
• Provide outreach to programs not currently participating to gauge interest
and need

Participating programs
• Engage in additional public education regarding the QRIS
• Financial support and professional coverage
• More training
• Clarify how to access a QRIS contact person/unit/consultant at EEC
• Help programs acquire rating tools
• Glossary of terms and FAQ
• List of resources
• Consider changes to the QRIS rating system
• Provide follow-up
• Consider ways to provide QRIS support to programs that need it
Examples of Key
Findings/Recommendations

Community and Family Engagement- State Advisory
Needs Assessment









10
EEC may want to continue or expand initiatives to assist educators in
understanding the importance of key features in the state’s QRIS standards.
EEC may want to consider these findings as it formulates future policies that
promote degree attainment and increased competencies among educators.
EEC may want consider these findings as it formulates policies to assist programs
and educators in supporting the needs of English language learners.
EEC may want to consider this finding as it formulates policies to promote and
support inclusive opportunities for children with special needs or disabilities.
EEC may want to continue initiatives designed to help educators manage
challenging classroom behaviors.
EEC may want to consider options for replicating the educator survey on an ongoing basis so that survey results can be linked to quality data from the QRIS and
data from periodic family surveys to determine if programs and educators are
meeting child and family needs, and overtime, to help evaluate the effectiveness of
policy initiatives.
EEC may want to consider using data collected from both the representative sample
and the public version of the survey to model the level of effort that would be
required to make improvements to the early education workforce in different
regions and to estimate the cost of related quality supports and incentives for
programs and educators.
EEC may want to continue or expand community engagement and outreach
initiatives that will help families understand important quality considerations and
their connection to school achievement.
EEC may want to consider these differences in tailoring outreach strategies to
promote the QRIS and other quality-related initiatives.
Examples of Key Findings

Community and Family Engagement- State Advisory
Needs Assessment con’t








11
EEC may want to expand strategies to support programs and educators in the
inclusion of children with special needs.
EEC may want to keep in mind these findings and the need that some parents may
have for more flexible scheduling as it considers changes to eligibility policies for
families and as it considers options for rate reform.
EEC may want to consider tailoring certain policy strategies to target families that
do not use formal early education and care programs, especially in promoting the
expansion of QRIS.
EEC also may want to keep this finding in mind as it considers strategies to ensure
that all families have access to translated materials that provide information on
early education programs, literacy initiatives and other programs important to
children and families.
EEC may want to consider financial assistance strategies and incentives that help
make high-quality programs more accessible to low-income families and encourage
them to make selections based on program quality.
EEC may want to keep this finding in mind when considering future support for
family engagement initiatives, especially those focused on helping hard-to-reach
families gain access to information on and connections to comprehensive supports
and services.
EEC may want to continue or expand strategies to strengthen early literacy and
language development and further expand efforts to engage hard-to-reach families
that may not be connected to formal early education.
EEC may want to keep this finding in mind as it considers strategies to help connect
children and families to additional supports and services that promote healthy child
development.
Examples of Key Findings

Community and Family Engagement- State Advisory
Needs Assessment con’t



12
EEC may want to keep this finding in mind as it considers strategies to help connect
children and families to additional supports and services that promote healthy child
development.
EEC may want to consider options for replicating the family survey on an on-going
basis that would allow survey results to be linked to quality and educator data from
the QRIS to determine if programs are meeting child and family needs, and over
time, to help evaluate the effectiveness of policy initiatives.
EEC may also want to consider qualitative methods – e.g., focus groups or family
interviews - for evaluating the needs of families to augment the quantitative
assessment.
CURRENT STUDIES
Professional Development Study
Literacy/Social-Emotional/Numeracy/Digital
Strategies
Common Metric
QRIS Validation Study
Home Visiting Evaluation
13
Current Studies
Most of the studies conducted have looked at either
policy or practice change and creates new research
or provides more research for planning purposes
 Current studies include






14
Home Visiting Evaluation which creates new research.
Literacy/Social-Emotional/Numeracy which improves
practice.
Common Metric which provides more research for
planning purposes
Professional Development System Study which
improves practice
QRIS Validation Study which improves policy.
Teacher Quality

Professional Development System Study

Research Questions
•
•
•
•
•
•

What are the the characteristics of individuals who have enrolled in the Professional
Qualifications Registry.
What are the characteristics of professional development courses offered to early
educators.
What are the professional development experiences of individual educators
attended.
What are the core competencies that are targeted in these professional
development experiences.
Are the three content areas (social-emotional development, literacy and numeracy)
included in the professional development experiences of educators.
Determine how are professional development experiences, with respect to taking
college versus CEU coursework and in relation to the core competencies and content
areas covered, being distributed across early childhood programs.
Methodology
•
A database was developed by merging data from the following databases. The data
was analyzed to answer the above research questions.
•
•
•
•
•
15

PQR
EPS Grantee PD Attendance Records from FY12
CSEFEL Training Attendance List
EEC Course Catalog FY12
EEC Program Priority List
Funded through ARRA funds.
Teacher Quality

Literacy/Social-Emotional/Numeracy/ Digital
Strategies

Research Questions- Social Emotional
•
•
•
•
•
•
•
•
16
Which classroom practices best support the healthy social/emotional
development of children birth to age 5?
How does the rate of expulsion relate to educators’ effectiveness across all
learning domains with emphasis on competency in meeting the
social/emotional needs of children?
How does the rate of expulsion relate to the extent of parental engagement
with the program?
Which children remain most at risk for expulsion?
What program characteristics are most associated with a healthy
social/emotional climate and fewer expulsions?
How do educators in the program create a classroom climate that is
conducive to healthy social emotional development?
What is the relationship between educators’ access to supports, such as
intentional consultation on a regular basis and related professional
development, and the success of children with challenging behaviors in
programs?
How do programs that have supports compare to those that do not in terms
of expulsion rates, behavior problems, parental involvement, and observable
classroom social-emotional climate?
Teacher Quality
•
Research Questions- Literacy
•
•
•
•
•
•
•
Research Questions- Numeracy
•
•
•
•
•
•
17
Which classroom practices best support the mastery of literacy in children birth to
age 5 including those who are bi-lingual or whose primary language is not English?
How does educator competency in curriculum areas addressing literacy affect the
outcomes for children?
What educator competencies help high needs children achieve desired literacy
outcomes?
What program characteristics are most associated with children achieving the desired
literacy outcomes?
What is the relationship between educators’ access to instructional supports, such as
coaching and intentional professional development, and children achieving the
desired literacy outcomes?
What external factors affect programs’ and educators’ ability to achieve desired
outcomes for the children in their care?
Which classroom practices best support the mastery of numeracy concepts by infant,
toddler, and preschool children?
How does educator competency in curriculum areas addressing numeracy affect the
outcomes for children?
What educator competencies help high needs children achieve desired numeracy
outcomes?
What program characteristics are most associated with children achieving the desired
numeracy outcomes?
What is the relationship between educators’ access to instructional supports, such as
coaching and intentional professional development, and children achieving the
desired numeracy outcomes?
What external factors affect programs’ and educators’ ability to achieve desired
outcomes for the children in their care?
Teacher Quality
•
Research Questions- Digital Strategies
•
•
•
•
•
•
•
•
•
•

Methodology
•

18
Of the digital strategies examined, which are most effective in increasing teacher
competency?
Of the digital strategies examined, which are most effective in enabling parents to
more effectively support their children’s development?
How are digital resources best integrated into, and aligned with, the program’s
curriculum?
Which areas of development do these digital strategies support?
How can the use of digital resources best reinforce and advance instruction?
How do the characteristics of the early education setting interact with the digital
strategies’ effectiveness?
Are certain strategies more effective than others in supporting children’s healthy
social, emotional development or the development of literacy and numeracy skills?
What is the relationship between educators’ access to digital resources and
accompanying instructional supports and children achieving the desired outcomes?
How could the effectiveness of the digital strategies be increased?
What external factors affect programs’ and educators’ ability to use digital strategies
to help achieve desired outcomes?
AIR will use extant data collection from the state, regional offices, and local
providers, Classroom observations (CLASS. ELLCO, COEMET) using validated,
evidence-based tools in a stratified random sample of early childhood sites,
Teacher and administrator surveys, Teacher assessments of children’s socialemotional development and literacy and numeracy skills to collect data to
answer the research questions.
Funded through ARRA funds.
ASSESSMENT AND
SCREENING
Common Metric Project
19
Assessment and Screening

Common Metric
 Research Questions
• What are the commonalities across each of the domains within each
of the different assessment tools (Teaching Strategies GOLD,
Pearson Work Sampling and High Scope COR) including a separate
analysis for preschool and kindergarten data.
• What is the feasibility of developing a statistical methodology to
answer the question of what are the baseline skills, knowledge and
abilities children enter with or bring to Massachusetts preschool and
kindergarten children.

Methodology
• Through analysis of the assessment data, identify common items
with the domains.
• Using data analysis explore the items and the factor structure to
determine reliability, inter-item consistency.
• Conduct confirmatory factor analyses and structural equation
modeling.

20
Funded through Race to the Top funds
PROGRAM QUALITY
QRIS Validation Study
21
Program Quality

QRIS Validation

Research Questions
•
22
Validating the Five Key Components of Quality and the Associated
Standards of the MA QRIS
• How do the participating early education and care providers vary along
the five key components of quality and their associated Standards?
• What does available evidence show regarding the link between the
Standards in the five components of quality and effects on child
outcomes? Of the evidence-based Standards linked to child
development outcomes which methods are used to measure these
Standards and is there a difference between self-assessment, as
opposed to external verification, in relation to child outcomes?
Program Quality

QRIS Validation Study
 Research Questions
•
23
Assessing Differences in Quality Across Quality Levels and
Changes Over Time
• Do the QRIS levels or groups of levels (e.g., 1–2 vs. 3–4)
represent true distinctions in program quality?
• Do the two groups of quality levels differ as expected in their
underlying components of quality and their associated
Standards? Are some quality components and associated
Standards more important than others in distinguishing quality
differences among the two groups of QRIS levels?
• Do the differences in quality between the QRIS levels or groups
of levels (e.g., 1–2 vs. 3–4) vary according to the method in
which the Standards are measured (self-assessment vs. external
verification)? Is self-assessment a valid method for distinguishing
between the QRIS levels or groups of levels?
• Is there an improvement in provider’s quality levels across time?
If so, what factors contributed to this improvement? What is the
contribution of state quality improvement efforts to improvement
in providers’ quality levels across time?
Program Quality

QRIS Validation Study
 Research Questions
•
24
Relating Quality Levels to Children’s Developmental Outcomes
• Do children enrolled in providers with QRIS levels 3–4 exhibit
developmental gains over time and more optimal growth
trajectories when compared with children enrolled with providers
at quality levels 1–2 across the five domains of development
(language, cognitive, social, physical, approaches to learning)?
• Which components and Standards of the QRIS are most strongly
associated with developmental outcomes and growth
trajectories? What provider characteristics are most strongly
associated with these child outcomes?
• The MA QRIS comprises five key components of quality including
Curriculum and Learning; Safe, Healthy Indoor and Outdoor
Environments; Workforce Development and Professional
Qualifications; Family and Community Engagement; Leadership,
Administration, and Management, as described in Exhibit 1.
Within each component, the Standards are specified for each
quality level that is appropriate to the provider setting (e.g.,
center-based, school-based, family child care, and afterschool/out of school).
Program Quality

QRIS Validation Study
• Methodology
• The study will employ a mixed method crosssectional and longitudinal study design.
• Provider data will be collected at 2 time
periods (Fall 2013 and Spring 2015) and child
outcome data will be collected 3 times (Fall
2013, Spring 2014 and Spring 2015).
• A Pilot Study will be conducted in Spring 2013.
• Funded through Race to the Top –
Early Learning Challenge Grant.
25
COMMUNITY AND FAMILY
ENGAGEMENT
Home Visiting Evaluation
26
Community and Family Engagement

Home Visiting Evaluation
 Research Questions
•
•
•
•
•
•
•
27
How can we understand the community capacity sphere of the ecological model of child
development?
• What is community capacity?
• What constitutes community capacity? What components, practices, and resources are
markers of community capacity?
• How do capable communities mobilize to support child development?
• How do families and children connect with resources in their communities? By what
pathways do such connections occur?
What are the key components of community capacity, as described in relevant literature and
from the perspective of state- and community-based stakeholders?
In what areas do specific communities demonstrate particular strength or weakness in
relation to components of community capacity?
In a subset of closely studied communities, what systems are in place to support child
development?
How accessible and well-utilized are systems/services?
To what extent are services available well-aligned with the community’s demonstrated needs?
Are services available in the correct proportions with regard to need?
What are the implications for policy and practice?
Community and Family Engagement

Home Visiting Evaluation
 Methodology
•
Develop a definition of community capacity through engaging with
community stakeholders as well as existing research
•
•
Engage in community-specific data collection and assessment
Develop data collection instruments and implement community-level data
collection plans
Work with community partners to deploy data collection efforts in relevant
communities
Monitor data collection efforts
Gather, organize, and analyze community data; possibly engage in smallerscale data collection
Identify, organize, and analyze secondary data
On an ongoing basis, develop summaries of emergent findings, such that they
can contribute to system planning and mid-course adjustments in response to
community-level data. Sequence phases of data collection in such a fashion as
to be complementary to other interdependent work pursued through the
overall study.
•
•
•
•
•

28
Funded through an ISA with DPH