Transcript Early Home Visitation for At

Evidence-Based Home Visiting Models
to Prevent Child Maltreatment –
Assessing and Addressing Fidelity
February 12
Anne Duggan, ScD
Research Supported by:
NIH, CDC, HRSA MCHB, ACF
Annie E. Casey Foundation, Robert Wood Johnson Foundation,
David & Lucile Packard Foundation
Hawaii Department of Health, Hawaii Family Support Institute,
Alaska State Health Department, Alaska Mental Health Trust,
Family League of Baltimore City; Safe & Sound Initiative; United
Way of Central Maryland
Today’s Talk
• Describe Evidence-based Models of
Home Visiting
• Identify basic components of models
and associated fidelity measures
• Introduce and demonstrate use of a
conceptual framework for research
to increase home fidelity and impact
Evidence-based Practices
An evidence-based practice, also
called EBP, refers to an approach to
prevention or treatment that is
validated by some form of
documented research evidence.
Rand Corporation Promising Practices
Criteria for Research Evidence
Proven
Promising
Program directly impacts
indicators of interest
Pgm impacts intermediary outcomes
Effect Size
1 or more outcomes
changed by > 20% or > .25 sd
Change in outcome
>1%
Statistical
Significance
p<.05
p<.10
Comparison
Groups
RCT or quasi-experimental
Comparison group,
but weaker
Sample Size
N > 30, each group
N > 10, each group
Types of
Outcomes
Three Major HV Models
PAT
Pilot (1981)
 MI Statewide Scale Up (1985)
 National Scale Up (1986)
 RCT (1993)
NFP
RCTs (1977, 1987, 1994)
 National Scale Up (1996)
 Nonprofit Replication Org. (2003)
HHS/
HFA
Demonstration Pilot of HHSP (1985)
 HFA National Scale Up (1992)
 HHSP RCT (1992, 1993)
 HI Statewide Scale Up (2001)
Designations of the Three Models
PAT
NFP
HFA
Pending
Pending
Not Submitted
OJJDP
PROMISING
EXEMPLARY
EFFECTIVE
Rand Corp.
PROMISING
PROVEN
PROVEN
SAMHSA
Adoption of PAT, NFP and HFA
Home Visiting Models
Red= All 3 programs
Blue= 2 programs
Green= 1 program
PAT: >3300 sites, 50 states
NFP: 118 sites, 25 states
HFA: 430 sites, 35 states
What are the Basic Components of
a Home Visiting Model?
• Families to be targeted
• Outcomes to be achieved
• Causal chain from inputs to outcomes
PAT, NFP, HFA Target Different Families
PAT
All pregnant women and
families with child <6 months old
NFP
First time mothers, <26 wks gestation,
and <19 years old or single or low SES
CAN Risk per Family Stress Checklist >25
HHS/
HFA
Hx of abuse as child; SU, PMH, Criminal Hx; Low selfesteem, poor coping ability; multiple life stressors;
violence potential; unrealistic expectations of child;
harsh punishment of child; sees child as
difficult/provocative
PAT, NFP, HFA Focus on Similar Outcomes
PAT
Increase parenting knowledge and behavior
Identify developmental delays and health issues
Prevent CAN
Increase school readiness and success
NFP
Improve pregnancy outcomes;
Promote child health and development;
Strengthen families’ economic self-sufficiency
Improve family functioning
HHS/
Prevent CAN
HFA
Promote child health and development
Framework for Influence of Home Visiting on
Family and Child Outcomes
Process
Program Model
Outreach
Trust Building
Crisis Intervention
Case Management
Parenting Education, Role
Modeling, Reinforcement
Screening &
Assessment
enrollment
Direct Home
Visiting Services
linkages
Medical Home
Other Needed
Community
Services
Family Functioning
Parent Mental Health, SU, IPV
Social Support
Economic Sufficiency
Parenting
Knowledge/Attitudes/Skill
Parent-Child Interaction
Environment for Learning
CAN
Child Outcomes
Health and Development
Measures to Assess Fidelity to the
Basic Components of an EBHV Model
• How well families are targeted
• How well outcomes are achieved
• How well each element in the causal
chain is carried out
What Do We Know about Home
Visiting as a Preventive Intervention?
1. Home visiting can improve outcomes,
but effects tend to be modest and
variable.
From Meta-Analytic Studies:
HV can be effective, but effect sizes are small.
Domain
ES
Cognitive Development (41 studies)
Socio-emotional Development (24)
.18*
.10*
ES Key
CAN Prevention (7)
.32
Small .20
CAN – Prevention of Potential Abuse (13)
Parenting Stress (4)
.24*
.21
Medium .50
Large .80
Parenting Behavior (37)
Parenting Attitudes (15)
Maternal Education (5)
.14*
.10*
.13*
*p<.05
Maternal Employment (7)
.02
Public Assistance (3)
-.04
Sweet & Applebaum,
2002
From single studies, we see that:
Effects can vary over time, & across subgroups.
Birth – 2 Yrs
2-4 Yrs
Birth – 15 Yrs
All Families
10% vs. 5%, NS
All Families
No group
difference
All Families
0.73 vs. 0.44
p<.05
Poor, Unmarried
Teens
19% vs. 4%
p=.07
Poor, Unmarried
Teens
No group
difference
Violent Families
No group
difference
What Do We Know about Home
Visiting as a Preventive Intervention?
1. Home visiting can improve outcomes,
but effects tend to be modest and
variable.
2. Programs like HSP/HFA target
–
the right families and individuals,
–
at the right time,
–
focusing on the right outcomes
HHS/HFA Home Visiting Model Who is Targeted, and When
• WHO IS TARGETED
– Caregivers in Families with Multiple Malleable
Risks
– Kempe Family Stress Checklist (“at risk” > 25)
• WHEN ARE THEY TARGETED?
– Prenatally, at Child’s Birth, Shortly Thereafter
HHS/HFA targets the right families – those with
multiple, malleable risks for poor parenting.
Outcomes in 1st Grade
13%
41%
41%
34%
18%
24%
19%
1%
CAN
Mother
Child Language
Poor Classroom
Depressed
Delay
Concentration
Not-at-Risk Families do better than
At-Risk Controls
HV Targets Right Individuals
Caregivers = Primary Influence in EC
Genotype
Self-Regulation
Child’s Developmental
Trajectory
Environment
Risk & Protective
Factors
Proximal 
Distal 
Communicating
and Learning
Making Friends &
Getting Along
Brain Development
Caregivers
Peers
Exposure down
with age
Exposure up
with age
Community
e.g. Socioeconomic and EC SERVICES
Source: Adapted from Tremblay, R. E. 2006. www.excellence-earlychildhood.ca
HSP/HFA Targets Caregivers at the Right Time:
Early Parenting Tracks into Grade School
AOR*
p
Nonviolent Discipline
Psychological Aggression
Minor Assault
Severe Assault
4.5
5.9
5.1
7.6
<.01
<.01
<.01
<.01
Neglectful Behavior
3.7
<.01
Confirmed CPS Report
6.7
<.01
*AOR for Later Use if Tactic Used Birth–3 Years
Home Visiting Focuses on the Right Outcomes:
Parenting and Its Determinants
Partner Relations
Parent’s
Developmental
History
Personality /
Relationship
Capacity
Stresses
Adaptation of Belsky’s Framework
Social Network
PARENTING
Child
Outcomes
Child Attributes
The quality of parent-child interaction is most vulnerable
to maternal relationship insecurity
under conditions of stress.
9
Continuous
Secure
8
7
5
Earned
Secure
4
Insecure
6
3
2
1
0
Low Stress
High Stress
Mean and 95% CI
Phelps JL, Belsky J
and Crnic K.
1998
HHSP/HFA Targets the Right Outcomes:
Association of Maternal Depression & IPV with
Severe Physical Abuse of Child, Birth – 3 Years
Depression
None
Possible (CES-D 16-23)
Probable (CES-D >23)
Intimate Partner Violence
Neither injured
Only mother injured
No partner
Both partner & mother injured
AOR
1.0
1.8
3.7
95% CI
Ref.
0.8, 3.8
1.9, 7.2
1.0
0.8
4.9
6.4
Ref.
0.1, 6.0
2.4, 10.0
2.9, 14.1
HHSP/HFA Targets the Right Outcomes:
Depression and IPV  Other Parenting Behaviors
Parenting
Behaviors
Depressive
Symptoms
Intimate Partner
Violence
OR
p
OR
p
Poor HOME Score
1.8
.05
1.2
.38
Poor NCAST
2.2
.02
1.1
.60
Neglect
1.7
.10
2.5
<.001
Assault on Esteem
2.4
<.01
2.8
<.001
Harsh Parenting
3.2
.01
2.3
<.01
What Do We Know about Home
Visiting as a Preventive Intervention?
Home visiting can improve outcomes, but
effects tend to be modest and variable.
1.
2. Programs like HSP/HFA target
–
–
–
the right families and individuals,
at the right time,
focusing on the right outcomes
3. But even if the model seems right,
desired outcomes might not be achieved.
Overall HSP/HFA Impact was Negligible
for Most Outcomes
Hawaii
Alaska
AOR
p
AOR
p
Maternal Depression
Physical IPV
0.97
0.83
.84
.19
0.66
0.82
.16
.43
Poor HOME Score
Poor NCAST Score
0.87
0.86
.39
.29
0.51
0.79
<.001
.31
So what’s going on? Wrong model?
Type III Error
Dobson and Cooke, Evaluation & Program Planning, 1980
• Unless fidelity of implementation is
determined, it is not possible to
determine whether negative impact is due
to:
– Inadequacies in the model or
– Departures from the model
“Every system is perfectly
designed to achieve exactly the
results it gets.”
Donald M Berwick, M.D.
Institute for Healthcare Improvement
• Let’s look at how home visiting services were
provided – let’s look at coverage, duration and
frequency of visits, visit content……
Percent of Families Screened, by Hospital
89
84
88
82
67
64
% Screened
H1
H2
H3
H4
H5
H6
HSP Screening & Assessment Rates
Number of Births
Percent Screened
Percent Screen +
Percent Assessed
Percent Assess +
Percent Referred
Overall
(FY00-FY07) FY 05 FY06 FY07
153,403
18,023 18,364 19,069
72%
77%
74%
76%
52%
50%
50%
52%
81%
85%
83%
83%
50%
47%
47%
47%
70%
74%
82%
77%
Attrition was higher than expected and
there was substantial, unintended variation
across sites. (Hawaii)
Percent of Families Active
120
Percent
100
80
60
40
20
Agency B
Agency A
Agency C
0
0 4 8 12 16 20 24 28 32 36 40 44 48 52
Weeks of Age
Mean Number of Visits
Active
All
Agency Families Families
B
A
C
22
19
28
16
11
12
p
<.01
<.01
Visit Content also Differed from the Model:
HVers Often Failed to Respond to Parenting Risks
32%
21%
11%
14%
14%
8%
Poor Maternal Mental
Health
Domestic Violence
Maternal Substance
Use
All Families with Risk
All Families with Risk and High Dose
Programs Varied Substantially in
Provision of Core Services
Hawaii
All
Range
Sites
Alaska
All
Range
Sites
Had 1st IFSP on Time
44%
28%-62%
24%
0% - 53%
Developmental Screenings
Discussion of Risks
Poor Mental Health
Domestic Violence
36%
23%-67%
48%
38% - 58%
15%
22%
6%-30%
13%-40%
45%
24%
9% - 88%
6% - 33%
15%
0%-33%
30%
11% - 42%
Substance Use
Framework of Determinants of Integrity
(Carroll et al., 2007)
Intervention
Model
Factors for
Integrity
Model Complexity
& Clarity
Service Integrity
• Fidelity =
Outcomes
Coverage
Duration, Frequency
Visit Content
• Competence
Implementation
System
Component Analysis
Participant
Responsiveness
To identify essential
components of the model
Quality of Delivery
We need to understand how family attributes and
the implementation system moderate impact.
Program Model
Outreach
Trust Building
Crisis Intervention
Case Management
Parenting Education, Role
Modeling, Reinforcement
Factors for Integrity
•Implementation System
(Hiring, Training,
Supervision, Curricula,
Protocols, Monitoring,
Linkage Agreements)
•Family Attributes
Process
Screening &
Assessment
enrollment
Direct Home
Visiting Services
linkages
Medical Home
Other Needed
Community
Services
Family Functioning
Parent Mental Health, SU, IPV
Social Support
Economic Sufficiency
Parenting
Knowledge/Attitudes/Skill
Parent-Child Interaction
Environment for Learning
CAN
Child Outcomes
Health and Development
Influence of Home Visiting Model
Complexity and Clarity (Hawaii)
Program impact was compromised …. by drift in
the model itself that had arisen in taking the
model to scale.
Original CAN
Prevention Program
Designation as an EI
Program for Children at
Risk for CHCN due to
Environmental Factors
Case Plan focused on
Risks that Made
Families Eligible
Parent-driven philosophy;
IFSP with family as
decision-makers in setting
goals & strategies
Influence of HV Model Clarity
Staff and Recipient Understanding of PAT
– Similarity
• Belief that a strong relationship was
important and beneficial to the parents.
– Differences
• Perception of home visitor’s expertise
• Perception of purpose of home visitor
showing the child a new activity
- From Hebbeler & Gerlach-Downie, 2002
Influence of Implementation System Training
SB6 HVers lacked basic knowledge of child development.
Language
SocialEmotional
9
39
24
52
30
46
Selected Age Too Young
Selected Correct Age Range
Selected Age Too Old
.
Tandon et al. Success by 6 Evaluation, Baltimore, 2004.
Influence of Implementation System Training
Training improved staff knowledge, at least short term.
Lang-2004
9
Lang-2006
5
SE-2004
SE-2006
39
52
65
24
19
30
30
46
49
Selected Age Too Young
Selected Correct Age Range
Selected Age Too Old
32
Comparison of Study Results with
ICMQ Validation Study Results*
12 Months
• % Agreement
• Sensitivity
Validation Studies
This study
91%
88%
89%
74%
Validation Studies
72%
0%
94%
7%
92%
90%
83%
89%
This study
• Specificity
24 Months
Validation Studies
This study
*Bricker D and Squires J, the Effectiveness of Parental Screening
of At-Risk Infants: The Infant Monitoring Questionnaire.
Overall sensitivity 4-36 months = 63%; overall specificity = 91%
The implementation system must include
SKILLS TRAINING & FEEDBACK & COACHING.
Knowledge
Shown in
Training
Setting
Skill Shown in
Training
Setting
Skill
Observed
in Practice
Didactic & Discussion
10%
5%
0%
Skill Demonstrated in
Training
30%
20%
0%
Practice/Feedback in
Training
60%
60%
5%
Coaching in Practice
Setting
95%
95%
95%
Joyce and Showers, 2002
RCT of Coaching to Promote
Fidelity & Impact
• Statewide random assignment of home visiting teams
• Data sources: parent interviews; record review;
observation of home visitors; surveys of and in-depth
interviews with home visitors and supervisors.
15 Teams Stratified
by Caseloads and
Retention Rates,
then Randomly
Assigned to Three
Study Group
Usual HSP Services
Training in HFT +
Usual Supervision
Training in HFT +
Enhanced Supervision
Influence of Participant Responsiveness
Program outreach - most effective in a subset of mothers.
Percent Active in Program
100
80
60
40
20
Assertive Outreach
Relaxed Outreach
0
16
32
48
64
80
Maternal Anxiety Score
96
An assertive
outreach
policy
promoted
retention of
mothers with
high
relationship
anxiety.
Home Visiting Impact Was Pronounced in
Mothers with High Relationship Anxiety
Example: Depressive Symptoms
68%
57% 58%
31%
34% 32%
19% 19%
ALL AR
Neither
High
Anxiety
High
33%
26%
Avoidance
High
Both High
Where We Go from Here…
– Learn What Works, for Whom
– Improve Fidelity
• Clarify existing models
• Build implementation system infrastructure
• Understand providers and recipients
• Build basic skills
– Enhance Home Visiting Models
Where We Are Going in Hawaii…
5-year ACF-funded project
 Staff training and supervision
 CQI capacity
 Targeting of families
 Enhancements to the model
 Reconciliation of funding incongruities