Transcript (ECMHC) Evaluation - National Technical Assistance Center for

Maryland Early Childhood
Mental Health Consultation
(ECMHC) Evaluation
Webinar Presentation
by Deborah F. Perry Ph.D.
Georgetown University
October 27, 2010
Overview of Presentation
 Overview of




statewide ECMHC
evaluation in
Maryland
Maryland’s approach
Logic model and
components
Levels of data
collection and other
features
Future directions
Maryland ECMHC Evaluation
 Funding:
Maryland State Department of Education (MSDE),
Division of Early Childhood
Involves 12 sites across Maryland
Three year evaluation (2008-2011), with two option
years
Principal Investigators:
 Sharon Stephan: University of Maryland
 Bruno Anthony, PhD & Deborah Perry, PhD:
Georgetown University
Minority Business partner: CKD Communications, LLC





Early Childhood Mental Health Consultation
(ECMHC)
 Collaborative, capacity building services
provided to early care and education (ECE)
providers and family members
 Can be initiated in response to concerns about
an individual child’s social-emotional or
behavioral development
 The impact can be measured in the domains of
child, parent, staff, and family
Approach to Evaluation
 Collaborative
 Participatory
 Comprehensive
 Sensitive to respondent burden
 No site comparisons
 Technical assistance related to
evaluation and data collection provided
Funded Projects and Evaluation Liaisons
Evaluation Logic Model
Early Childhood Mental Health Consultation
Skills and Qualifications
Satisfaction
Relationship
Parenting
Early Care and Education
- Behavior and Attitudes
- Attitudes, Beliefs, and Stress
- Stress
- Classroom Environment
Young Children
↓Challenging Behavior
↑Social Skills
Expulsions
Evaluation Components
Implementation
Factors
Program Directors
Model
Description,
Quarterly
Interview
Consultants
Knowledge
& Skills,
Service Log,
Exiting Child Form,
Roster, Preschool
Climate Scale
Core Outcomes
ECE Providers
SDQ
TOS
RQS
DECA
Parents
PSI
PBI
DECA
ECE Directors
GAS
Exiting Child
Study
Consultants,
ECE Directors,
ECE Providers,
and Parents
Qualitative
Interviews
Consultants
Climate
RQS
Note: instruments in pink will only be collected for child-specific referrals
Levels of Data Collection
 Program-level outcomes
Measures assess changes in: ECE provider behaviors and
beliefs; job stress; structural and interpersonal characteristics of
the ECE setting; and the quality of the relationship between the
MHC and ECE providers.
 Child-/Family level outcomes
 For child-specific referrals:
 Measures assess changes in the child’s behavior (rated
by teacher and parent), parenting stress and behaviors;
and the quality of the relationship between the MHC and ECE
providers.

ECMHC Program Directors
Tools
Construct
Frequency
Model Description
Program
Structure
Baseline,
Annual
Quarterly Site Interviews
Program
Changes
Quarterly
Mental Health Consultants
Tools
Construct
Frequency
Knowledge and Skills Inventory
Consultant
Characteristics
Baseline, Annual
Service Log
ECMHC Services
Ongoing
Classroom Roster
Administrative
Tool
Ongoing
Preschool/Infant MH Climate Scale
Classroom
Environment
Baseline, every 4
months (ongoing)
Consultant Final Perceptions Form
Exit Study (Qualitative Interviews)
Quality of
Relationship
Perceptions and
predictors of
expulsion
4 months, or at
Discharge
Following an exit
due to behavior
issues
Preschool/Infant Climate Scale
 These tools are completed by the mental health
consultants for every child-specific and programmatic
consultation they initiate.


Preschool Mental Health Climate Scale (Gilliam, undated)
Infant Climate Scale (FSU, 2009)
 The instruments assess multiple dimensions of the
classroom environment and ECE providers’ behaviors.
 Consultants complete these scales at baseline, at 4
months or at discharge; and if ongoing, every 4 months
thereafter.
Child Care Directors
Tools
Construct
Frequency
Goal Achievement Scale
Opinions and Baseline, 4
Attitudes
months or at
discharge
Exit Study Interviews
Perceptions of When child
Expulsion
exits due to
behavior
Early Care and Education Providers
Tools
Construct
Frequency
Strengths and Difficulties Questionnaire
Child
Functioning
Baseline, 4
months or at
discharge
Teacher Opinion Survey
Teacher
Attitudes
Baseline, 4
months or at
discharge
Provider Job Stress Index
Job Stress
Baseline, 4
months or at
discharge
Teacher Final Perceptions Form
Quality of
Relationship
Baseline, 4
months or at
discharge
Exit Study Interviews
Perceptions of
Expulsions
When child exits
due to behavior
Parents
Tools
Construct
Frequency
Parenting Stress Index-Short Form
Parent stress
Baseline, 4
months or at
discharge
Parent Behavior Inventory
Parenting
practices
Baseline, 4
months or at
discharge
Parent Final Perceptions Form
Quality of
relationship
Baseline, 4
months or at
discharge
Devereux Early Childhood Assessment
Child
functioning
Baseline, 4
months, 12* & 24*
months
Exit Study Interview
Perceptions of
expulsions
If their child is
expelled
* For those who consent to long-term follow along
Exit Study Interviews
 Motivated by MSDE’s interest in this policy-relevant (but
infrequent) outcome
 Always interview the consultant; seek the additional
perspectives of ECE Director, provider and parent
 Includes open-ended questions such as:
 What behaviors led to the child’s exit?
 What was done to try to maintain the child in the
placement?
 Did you expect this child to be expelled? Why or why
not?
Professional Activity Units
 Working with MSDE,
established process
for ECE Directors
and Providers to get
professional activity
units when complete
baseline and followup measures for
evaluation
 Increase participation
and help with followup data
Current Status and Future
Directions
 Data collection in the field: good
participation from all sites
 Exit interviews completed with
consultants, and a few ECE providers
 Comparison group
 Addition of CLASS as objective measure
of change
 Paper on Preschool Mental Health
Climate Scale with Walter Gilliam