Transcript 18th National Conference on Child Abuse & Neglect April 18

18th National Conference on Child Abuse &
Neglect
April 18 – 20
Washington, D.C.
Understanding Decision-Making
Donald Baumann
Saint Edwards University
John Fluke & Katherine Casillas
American Humane Association
Understanding Decision-Making
Overview
•
•
•
•
•
•
•
•
Advances Outside Child-Welfare
Assumptions in the Decision Sciences
Heuristics and Biases
Signal Detection Theory
Decision-making Errors
Cumulative Prospect Theory
The Decision-Making Continuum
Models in Child Welfare
Understanding Decision-Making Frameworks:
Advances Outside Child-Welfare
• Engineering Psychology and Economics
–
–
–
–
Theories of riskless and risky decision-making (decision choice models)
Game Theory (mathematical models of conflict and cooperation)
Bounded rationality (doing “well enough”)
Optimization (maximizing outcomes)
• Artificial Intelligence
– Network analysis (systems for determining causality)
– Expert systems (complex decision-making system that learn)
• Medicine
– Fuzzy trace theory (understanding the “gist”, values and inference)
• Psychology, Statistics and Psychophysics
–
–
–
–
–
–
Heuristics and bias (errors made in decision-making)
Normative approaches (the standards are probability & statistics)
Signal detection theory (analysis of threshold differences)
Fast and frugal reasoning (based on probability cues)
Implicit and explicit judgments (intuition and logic)
Cumulative prospect theory
Understanding Decision-Making Frameworks:
Advances in Economics and Psychology
• Economics and Exchange Theory
– Assumption of rationality
– People weigh the costs and benefits of a decision and optimize
• Herbert Simon and Satisfysing
– Bounded rationality- we do “well enough”
– Reason is limited and we do not optimize
• Tversky, Kahneman and Meehl
– Reason is very limited and we are poor decision-makers
– Under conditions of uncertainty we use heuristics (rules of thumb)
and make errors
– Thinking fast and slow under different circumstances
– Cumulative Prospect Theory (choices based on gains and losses)
• Gigerenzer
– heuristics work a lot of the time
– Fast and frugal reasoning (based on probability cues)
– We make the choice with the best cue
Understanding Decision-Making
Frameworks: Assumptions
• Assumptions
– Decisions have psychological (cognitive, motivational and emotional)
properties and are at the individual level
– Decisions have a context
– People have different thresholds for different decisions
– People make errors when they make decisions
– Sources of error and accuracy can be empirically understood and
improved upon
Understanding Decision-Making Frameworks:
Heuristics and Biases
• Overconfidence phenomenon (We overestimate the accuracy of
our beliefs and incompetence feeds overconfidence)
• Confirmation bias (We look for evidence to justify beliefs rather
than falsify them)
• Representativeness heuristic (The belief that things belong to a
group that is typical)
• The availability heuristic (judging the likelihood of an event by how
easily it come to mind e.g., vivid instances)
• Illusory thinking (The search for order in random events)
• Illusory correlation (The perception of a relationship where none
exits – usually confirmatory)
• Illusion of control (The belief that chance events are subject to our
control)
• Moods and judgment (good and bad moods bring to mind good
and bad associations)
Understanding Decision-Making Frameworks:
Implicit and Explicit Judgments
• Implicit (Intuitive) Judgments
– Powers of intuition (fast thinking): Somewhat instantaneous making for efficiency
– Limits of intuition: The speed can make us error prone when we need
to slow down and think about things more
• Explicit Judgments
– Powers of explicit judgments (slow thinking): The length of processing
can make us less error prone
– Limits of explicit judgments: Cognitively labor intensive and inefficient
A CHILD WELFARE VALUES EXERCISE
(DALGLEISH)
Signal Detection Theory and
Decision Errors
Child Welfare and the Problem of False
Positives- Receiver Operator
Characteristics
Sensitivity (true positive)
N = 210,642
•Accuracy in
assessing risk is
generally not
very good, and
false positives are
very likely
•Furthermore
almost all
predictive power
is tied to prior
occurrences
Child Welfare and the Problem of False
Positives
(assume actual prevelance is 10%)
Sensitivity (true positive)
N = 210,642
Low threshold
Effect of Thresholds on False Positives
The assessment has an Area Under the Receiver Operator Curve = 63%:
Prevalence assumed to be 10%: Applied to 100,000 children
LOW THRESHOLD
Child Welfare and the Problem of
False Positives
Sensitivity (true positive)
N = 210,642
High threshold
Low threshold
Effect of Thresholds on False Positives
The assessment has an Area Under the Receiver Operator Curve = 63%:
Prevalence assumed to be 10%: Applied to 100,000 children
HIGHER THRESHOLD
Child Welfare and the Problem of
False Positives
Sensitivity (true positive)
N = 210,642
High threshold
Effect of Thresholds on False Positives
The assessment has an Area Under the Receiver Operator Curve = 63%:
Prevalence assumed to be 10%: Applied to 100,000 children
LOW THRESHOLD
HIGHER THRESHOLD
Cumulative Prospect Theory (Tversky &
Kahneman, 1992)
• A psychological theory for explaining non-rational
decisions under uncertainty
• Principles
– We make choices based on change in gains and losses
relative to a reference point; a reference point based on
what we have or know. Child Welfare
• Example: a child is safe at home.
– Given the choice of a large sure loss compared to a chance
that we either might not have a loss or have a large loss,
we tend to take the risker option; we take a sure gain and
do not take a risk even when a risk might increase our
gains. We dislike losing more than we like winning.
Cumulative Prospect Theory (Tversky &
Kahneman, 1992)
• Principles
– Given the equal choice of a larger gain and smaller
loss we tend to choose neither; asymmetric loss
aversion.
• Example: There is a 50/50 chance If we provide a specific
service a child will be safe for 6 months, or the child will
certainly be maltreated again in 4 months.
– We tend to make choices based on very unlikely
events as if they are more likely; overweighting
unlikely events.
• Example: Involving the court in a child maltreatment case is
a large worry.
Relationships of Gains and Losses in
Prospect Theory
EXERCISE IN PROSPECT THEORY
The Continuum of Child Welfare
Intervention
ASSESSMENT
Child protection Screening Assessment Placement Reunification
Decisions/Actions
Assessments and decisions are made at key points along the child
protection continuum
Each key decision point requires a specific decision and action
Models of Decision Making in
Child Welfare
Models of Child Welfare Decision
Making
• Structures and Systems of Decision Making
– Stein T and Rzepnicki T. (1984). Decision Making in
Child Welfare Services: Intake and Planning. Boston:
Kluwer-Nijhoff Publishing.
– Gleeson J. (1987). Implementing structured decisionmaking procedures at child welfare intake. Child
Welfare 66 (2) 101-112.
– Wells, S. J. (1985). How we make decisions in child
protective services intake and investigation.
Washington, DC: American Bar Association.
System Models of Observed Child
Welfare Decision Making Behavior
•
Rossi, P. H., Schuerman, J. & Budde, S. (1999). Understanding decisions about child
maltreatment. Evaluation Review, 23(6) 579-598.
•
Baumann, D., Kern, H., & Fluke, J. (1997). Foundations of the decision making
ecology and overview. In Kern, H., Baumann, D. J., & Fluke, J. (Eds.). Worker
Improvements to the Decision and Outcome Model (WISDOM): The child welfare
decision enhancement project. The Children’s Bureau, Washington, D.C.
•
Dalgleish, L.I. (2003). Risk, needs and consequences. In M.C. Calder (Ed.)
Assessments in child care: A comprehensive guide to frameworks and their use.
(pp. 86-99). Dorset, UK: Russell House Publishing.
•
Munro, E. (2005) Improving practice: Child protection as a systems problem.
Children and Youth Services Review, 27, 375-391.
•
Baumann, D.J., Dalgleish, L., Fluke, J., & Kern, H.(2011).The decision-making
ecology. Washington, DC: American Humane Association.
DECISION-MAKING
ECOLOGY (DME)
Decision Making Ecology/General
Assessment and Decision Making Model
(Baumann, Dalgleish, Fluke &Kern, 2011)
Case Factors
Organizational
Factors
Decision
Making
Outcomes
External
Factors
Decision Maker
Factors
Influences
Decisions
Outcomes
GENERAL ASSESSMENT AND
DECISION MAKING (GADM)
MODEL: THE PROCESS OF
DECISION-MAKING
Risk assessment and decision making
• In many jurisdictions risk assessment is used
as a way to summarise the case information.
• How is this assessment turned into a
decision about a course of action?
• In general, the risk of harm has to be
sufficient to warrant taking protective
action.
Assessments and thresholds are
influenced by different factors
• The Risk Assessment • From Theory, the
derives from case
Threshold for Action
information on:
derives from the
the Child: the Family and the
experiences and history
nature of the current and
of the worker.
past concerns.
• Information organized
into operationally
defined factors.
– Possible consequences
for the different
stakeholders.
– How the worker values
the consequences.
Assessment and decision making is a
difficult task
• Assessments and
decisions are based
on information that
is often unclear,
noisy and uncertain.
• Sometimes made
under time pressure
in a highly
emotional
atmosphere.
• There are structural
and resource
constraints, media
interest,
unpredictability of
outcomes.
• This is:
Decision making
under uncertainty.
Crucial points:
The general model for assessment and decision
making.
Separates: The assessment of the situation.
From: The decision to something about it.
– Qualitatively different factors influence
assessment and decision making.
Distinguishes: The person’s ability to detect the
need to take action (how good they are).
From: The person’s willingness to take action
(their threshold).
A General Model for Assessing the Situation
and Deciding what to do about it - Dalgleish
Assessment Dimension:
e.g. Risk or ‘Level of Concern’
HIGH
Factors
Influencing
Assessment.
Information from
Current situation
being Assessed.
The Case
Factors.
Assessment
Threshold
Factors
Influencing
Threshold for
Action
Information from
Experiences and
Organizational
Factors)
LOW
If the Assessment is ABOVE the Threshold, then ACTION is taken.
If the Assessment is BELOW the Threshold, then NO ACTION is taken.
The Process of Decision Making:
The Threshold Concept
High
Threshold
W2
W2
Assessment.
• If threshold low, W1
needs little evidence
before taking action.
No
 If threshold high, W2 needs
W1
Assessment. much evidence before
taking action.
Yes
Threshold
W1
Low
Assessed level of risk or need

Even if they agree on the
assessment,

they disagree about taking
action.
*From Len Dalgleish, 2000
Transition
slide subtitle
The
Structured
Decision
Making System (SDM)®
The National Council on Crime and
Delinquency, founded in 1907, is a nonprofit
organization that creates just and innovative
solutions to complex social problems, and
works to improve the lives of all people
through research, public policy
What is the SDM system?
A research and
evidence-based
decision-support
system
What is SDM?
Screening &
Response
Priority
Assessment
Safety
Assessment
Risk
Assessment
Family
Strengths &
Needs
Assessment
Reunification
Assessment
Risk
Re-assessment
Case Process
TOOL DESIGN
One decision:
one tool
How can
research
inform tool?
Clear
presumptive
decision
Overrides
(measure!)
Kinds of Evidence
• Validity
• Construct validity
Do the tools
• Inter-rater reliability
work?
• Completion rates
• Process changes
Do people
use the tools? • Worker/ family surveys
Do outcomes
change?
• Outcomes
What is SDM?
Screening &
Response
Priority
Assessment
Safety
Assessment
Risk
Assessment
Family
Strengths &
Needs
Assessment
Reunification
Assessment
Risk
Re-assessment
Case Process
PRACTICE SKILLS
Use content
(DEFINITIONS) to
focus information
gathering
Think through with
family, team
Discuss results with
family, team
Plan collaboratively
Risk assessment is about likelihood and
probability
The SDM Risk Assessment Answers this questions:
Based on the family’s characteristics, how likely are they
to abuse and neglect their children in the next 24
months?
California Risk Study Results
N = 2,511 investigations conducted in 1995, followed for two years.
California Risk Assessment Validation: A Retrospective Study, 1998
© 2012 by NCCD, All Rights Reserved
“Final decisions
should be left to
formulas, especially
in low-validity
environments.”
p. 225
Typical risk assessment presumptive
decisions
What is the
probability of
future abuse or
neglect?
LOW
close
MODERATE
HIGH
open
VERY HIGH
Risk Level by Initial Safety Assessment
2.1%
Unsafe
5.0%
Conditionally Safe
22.9%
30.7%
Safe
33.9%
67.2%
0%
20%
Low
N = 67,140
45.9%
36.3%
61.0%
40%
Moderate
29.7%
40.8%
60%
High
24.4%
80%
100%
Very High
2011 California Combined Report
© 2012 by NCCD, All Rights Reserved
Important Distinctions
Danger/ safety
• Immediate
• Serious
• Guides
decisions
about
placement
Risk
Need
• There is time
to change the
course
• Any level of
abuse/neglect
• Guides
decisions
about case
open/close
• Conditions
that appear to
be barriers to
increased
safety and
reduced risk
• Focus of
intervention
Key Concept
Threat of Danger?
+
Vulnerable child?
Protective Capacity?
=
“unsafe child”
Present danger
• Present danger is an immediate, significant and
clearly observable family condition occurring in
the present tense, already endangering or
threatening to endanger a child. This
phenomenon is also referred to as immediate
harm; immediate severe harm; or imminent
harm. It is important to understand that the
primary criterion that qualifies present danger is
what is happening that endangers a child is
happening now; it is currently in process of
actively placing a child in peril.
Impending danger
• Impending danger is associated with a child living or being in a state
of danger; a position of continual danger. Danger may not exist at a
particular moment or be an immediate concern but a state of
danger exists. Impending danger is not necessarily active in the
sense that a child might be hurt immediately like is true of
immediate, present danger. When a child lives in impending danger
one can expect severe harm as a reasonable eventuality.
• Impending danger refers to threats to a child’s safety that exist; are
insidious; but are not immediate, obvious, or active at the onset of
CPS intervention.
• Impending danger refers to threats that eventually are identified
and understood upon more fully evaluating and understanding
individual and family conditions and functioning.
Safety (Danger) versus Risk
Safety  concerned about imminence and severe
consequences due to things being out of control
Risk  broad concept regarding whether something
might occur if there is not intervention; risk may be low,
moderate, high.
Vocabulary: Safe and Unsafe Child
• Safe child
– “Vulnerable” children are safe when there are no
“threats of danger” within the family or home OR
when the caregivers possess sufficient “protective
capacity” to manage or control any threats.
• Unsafe child
– Children are unsafe when they are “vulnerable,”
there are “threats of danger” within the family or
home AND the caregivers have insufficient
“protective capacities” to manage or control the
threats, making outside intervention necessary..
Three Types of Protective Capacity
Cognitive
• knowledge
• understanding
• perceptions
Behavioral
• actions
• activities
• performance
Emotional
• feelings
• attitudes
• identification
In home safety plan
Safety Plans
combination
Out of home safety plan
Safety Plan
actions and services
that will
temporarily
substitute for
lacking parental
protective
capacity to
control the
threat of danger
Current decision-making tools
• Intake (Minnesota intake screening guidelines) North Carolina
Structured Intake report (very extensive)
• Safety assessment (first contact)
• Safety plans (first contact-Alaska example)
• Safety assessment (end of investigation and life of case); Safety
assessment Nebraska (includes information standards; threshold
criteria and supervisory review of safety assessment decision and
process)
• Safety plan (not first contact)
• Risk assessment
• Risk re-assessment
• Protective capacity assessments
• Case plan
Current decision-making tools
• Family assessments (often for Differential
Response) (Ohio example)
• Concurrent planning (Idaho) includes ICWA
considerations; 3 month intervals up to 22
months.
• Educational neglect (home schooling) Idaho
• Pediatric symptom checklist (Maine)
Current decision making tools
• Maine signs of safety
• UNCOPE- screening tool for drugs/alcohol
(Maine)
• Abandonment screening tool (Nebraska)
• Dependency screening tool (Nebraska)
• Idaho safety decision tree
Current Decision-making tools
New resource – NRCCPS website
www.nrccps.org
http://nrccps.org/information-dissemination/1249-2/
18th National Conference on Child Abuse &
Neglect
April 18 – 20
Washington, D.C.
What We Know and Current Challenges
Tamara Fuller
University of Illinois
Donald J. Baumann
Saint Edwards University
What We Know and Current Challenges:
Overview
• Tying Tools and Practices into Improved
Outcomes
• Implementation and Decision-Making
• What do we Need to Know: Challenges for
Research
What We Know and Current Challenges: Tying
Tools and Practices into Improved Outcomes
• The Assessments
– Safety
– Risk
– Family
• What are the key decisions?
•
•
•
•
•
•
•
Investigation
Substantiation
Service provision
Placement into care
Reunification
Relative Care
Adoption
• The Instruments
–
–
–
–
CERAP
SDM
Concept Guided (Texas and Georgia)
Other
Illinois Child Endangerment Risk
Assessment Protocol
• Safety assessment instrument
• 14 safety factors that focus on readily observable
and harmful behaviors
• Family strengths and mitigating circumstances
• Safety decision
– Safe: No children likely to be in immediate danger of
moderate to severe harm
– Unsafe: A safety plan must be developed OR one or
more children must be removed from the home
because without the plan they are likely to be in
immediate danger of a moderate to severe nature
CERAP
• Safety Plan
– What actions have or will be taken to protect each
child in relation to current safety concerns?
– Who is responsible for implementing each plan
component?
– How will the plan be monitored and by whom?
– What must happen in order to terminate the plan?
– What time frames have been imposed by this plan?
CERAP
CERAP intended as a “life of case” instrument, and
should be completed at several case milestones:
• Initial investigation
• Investigation closure/case opening
• Ongoing safety plan monitoring
• Unsupervised visits
• Reunification
• Case closure
What We Know and Current Challenges: Reviewing
of the Status of Decision-Making Research
• Factors that Relate to Different Decision Points
•
•
•
•
Case characteristics
Organizational characteristics
External characteristics
Decision-maker Characteristics
What We Know and Current Challenges: Reviewing
of the Status of Decision-Making Research
• Factors that Relate to Different Decision Points
– Where are the decision-making errors
• The threshold shift from child protection (more false positives) to family
preservation (more false negatives)
• The middle of the safety and risk continuums probably has greater
threshold variability than the extremes
• The Harm Evidence model for substantiation decision (Drake, 1998
suggests we must have: (1) sufficient risk of harm coupled with (2)
evidence to show it. This would suggest a high threshold, more false
positives and fewer false negatives
• Some evidence suggests the decision-making threshold for reunification
is higher than for placement. This would indicate the possibility of more
protection, more false positive and fewer false negative errors.
– The fallibility of decision aids (e.g., risk and safety
instruments)
• They only predict a small amount of the variability in decisions
• They don’t tell us what the errors are but they do tell us how much
• Important to focus our attention on understanding and practical utility
What We Know and Current Challenges:
Issues With Instruments
• Evaluating Risk and Safety Assessment Instruments
– Scientific Integrity
• Face validity
• Content validity
• Construct validity
– Practical Utility
• Influence
• Accessibility
• Efficiency
Evaluating the CERAP
• Implementation evaluations have examined
how workers using the CERAP
• Outcome evaluations have examined whether
CERAP implementation has impacted child
safety
Impact evaluation
• The CERAP was implemented statewide in
December 1995
• Therefore, a true experimental design to
assess its impact on safety was not possible
• A time series analysis was used to examine
differences in child safety before and after
CERAP implementation
• Additional analyses attempted to rule out
alternative hypotheses
Impact evaluation
• The outcome of interest is child safety, defined
in these evaluations as the number and
percentage of children with an indicated
report of maltreatment within 60 days of an
initial report
60-day recurrence rates (1995-2001)
2.5
2.06
2.0
1.75
1.55
1.58
1.5
1.44
1.25
1.19
1.0
Implementation
Year
0.5
0.0
12/94-11/95
12/95 - 11/96
12/96 - 11/97
12/97 - 11/98
12/98 - 11/99
12/99 - 11/00
12/00 - 11/01
Extended Secular Trend Analysis
To strengthen the validity of the inference about
CERAP effectiveness, the trend analysis was
extended several years before CERAP
implementation to assess whether the decline in
recurrence rates was a reversal of an earlier
pattern or a continuation of past trends.
60-day recurrence rates (1986-2002)
3.5 3.36
3.0
2.5
2.0
1.5
1.0
2.81
2.62
2.30
2.42
2.14
1.99 2.06 2.00
2.05
1.75
1.55 1.58
1.44
1.25 1.19
1.00
0.5
0.0
12/85- 12/86- 12/87- 12/88- 12/89- 12/90- 12/91 - 12/92 - 12/93 - 12/94 - 12/95 - 12/96 - 12/97 - 12/98 - 12/99 - 12/00 - 12/01 11/86 11/87 11/88 11/89 11/90 11/91 11/92 11/93 11/94 11/95 11/96 11/97 11/98 11/99 11/00 11/01 11/02
CERAP Completion Rates by Milestone
Milestone
Within 24 hours after the
investigator first sees the victim
Within 5 days of case opening or
transfer
Every six months for intact family
cases
Commencement of unsupervised
visits
Before an administrative case
review
Prior to returning home
Prior to closing a case
When child’s safety is in jeopardy
Intact Family
Substitute Care
(n=273)
(n=288)
93%
73%
67%
n/a
50%
n/a
87%
90%
69%
45%
n/a
48%
77%
50%
100%
81%
CERAP Section Completion Rates
CERAP section
Plan identifies specific
actions
Intact Family
94%
90%
95%
90%
73%
Substitute Care
86%
88%
91%
78%
63%
Plan identifies who will
implement
73%
59%
Plan identifies who will
monitor
38%
39%
Safety Factor Checklist
Family Strengths
Safety Decision
Safety Protection Plan
Summary
CERAP completion by workers in the field is
consistently high during the investigation
(97%), moderately high at case transfer and
case closing (70%), and much lower at other
milestones. When completed, the CERAP
tends to be completed in its entirety, although
safety plans may not be completed according
to instruction.
18th National Conference on Child Abuse &
Neglect
April 18 – 20
Washington, D.C.
Group Decision-Making Process
Donald Baumann
Saint Edwards University
Lisa Merkel-Holguin
American Humane Association
Group Decision-Making Processes
Overview
• Principles of Group Decision-Making
• Family Involvement Models Used in Child
Welfare Decision-Making Lisa will discuss
later this afternoon
Principles of Group Decision-Making:
Assumptions
• Our decisions are influenced by groups: We
have evolved to function in groups
• More hands makes for light work
• Interactions between group members will
move initial extreme positions toward the
middle
• Coming to consensus is a good way for groups
that are already cohesive to make decisions
• Groups make better decisions than individuals
Principles of Group Decision-Making:
Understanding Sources of Errors in Groups
• The Influence of Groups on Decisions:
– Groups have influence even by their mere presence: We are
affected by the presence of others, even unconsciously. We do
better in the presence of others when we have learned how to
do something well. When we have not, we do worse
• More Hands Makes for Lighter Work
– Many hands can make for social loafing: Collective efforts result
in less than the sum of individual effort under some
circumstances (additive task, unchallenging tasks, not being held
accountable, etc.)
• Interactions Make for Compromise
– Interaction between group members with different positions
produces group polarization: Interaction tends to intensify
opinions for the like- and unlike-minded: They shift to more
extreme positions because informational and normative
influences.
Principles of Group Decision-Making:
Understanding Group Influence
• Consensus Decisions
– Consensus can produce groupthink: High cohesiveness, group
insulation, the lack of procedures for appraisal and directive
leadership leads groups to poorer decisions. Group preferences
for supportive vs. challenging information, as well as the need
for acceptance or approval, helps facilitate groupthink.
• Group vs. Individual Problem Solving
– Though people feel more productive when solving problems in a
group, people working alone or in two’s generate more good
ideas. Large brainstorming groups are especially ineffective.
– Among the ways this can be overcome are to combine group
and solitary brainstorming, or have members first write then
read (as in networked computers), rather than speak, then
listen.
A Brief Exercise in Setting Action
Thresholds
Case Vignette
Mother tested positive for marijuana at the birth of the child.
The child was not tested. Mother admits to using marijuana weekly.
Mother and the biological father reside together and have all items to
meet the baby’s basic needs. Mother has two other children who are
not in her care. They were committed to the legal custody of mother’s
mother. That arrangement was done privately with no Agency court
involvement. The Agency did substantiate Neglect in that case, but the
grandmother filed on her own. Mother has been diagnosed with Major
Depressive Disorder. Mother is not receiving counseling services nor is
she on medication. Mother states she uses marijuana to help with her
mental health. Mother has been uncooperative with the Agency in the
past, but has now agreed to do a drug and alcohol assessment. Mother
has resided out-of-state and has just returned to their home town in
November. Father is employed and is cooperative with the Agency. He
participated in a drug and alcohol assessment after admitting to past
use of marijuana. Mother would be the primary caretaker of the child
while the father is at work. WOR’s home visit revealed no concerns
about provisions for the baby, but some concerns about caring for the
baby given mother’s mental health and history of drug usage.
The Process of Decision Making:
The Threshold Concept
High
Threshold
W2
W2
Assessment.
• If threshold low, W1
needs little evidence
before taking action.
No
 If threshold high, W2 needs
W1
Assessment. much evidence before
taking action.
Yes
Threshold
W1
Low
Assessed level of risk or need

Even if they agree on the
assessment,

they disagree about taking
action.
*From Len Dalgleish, 2000
Vocabulary: Safe and Unsafe Child
• Safe child
– “Vulnerable” children are safe when there are no
“threats of danger” within the family or home OR
when the caregivers possess sufficient “protective
capacity” to manage or control any threats.
• Unsafe child
– Children are unsafe when they are “vulnerable,”
there are “threats of danger” within the family or
home AND the caregivers have insufficient
“protective capacities” to manage or control the
threats, making outside intervention necessary..
Transition slide subtitle
Implementation
Science:
an introduction to key concepts
www.nccd-crc.org
National implementation Research Network
www.fpg.unc.edu
From Paper to Performance
The “what”
The “how”
Effective
Not effective
Effective
Not effective
Good
implementation
Poor
implementation
Good outcomes
Poor outcomes
Good
implementation
Poor
implementation
Poor outcomes
Poor outcomes
Fixsen et al (2005), p. 69
www.nccd-crc.org
Implementing Best Practices
Outcomes/ evaluation
•
•
•
•
•
•
Selection
Training
coaching
Systems intervention
Facilitative administration
Decision support data system
INTEGRATED &
COMPENSATORY
Technical
Leadership
Adaptive
Fixsen & Blase , 2008
Stages of implementation
Exploration
Design/
installation
Initial
implementation
Full
implementation
Innovation
Sustainability
Fixsen et al (2005),
West Virginia Safety Assessment
and Management System
• Purpose: To implement a new (CPS)
safety assessment and intervention
system that:
– Utilizes a change management approach
to fully implement the system
– Permanently changes practice in WV
– Results in better outcomes for children ad
families
SAMS Mission
• The successful implementation of
SAMS resulting in an improved system
of intervention based on consistent
standards; focused and efficient
information collection; and a familycentered approach that will improve
caregiver and family functioning and
increase safety, permanency and wellbeing.
SAMS
• A comprehensive CPS assessment process
from initial intake to case closure and a
more precise way to determine safety and
respond to unsafe children
• Will improve safety outcomes for children
and families by:
– Focusing services more on safety
– Focusing on the protective capacities of
caregivers
– Improving family engagement
SAMS: Key to Success
• Fidelity – using the decision-making
model as designed. Blame cannot be
placed on deficiencies of the model if it
is not used as designed
• Understanding a form cannot change
practice by itself
• Understanding that learning takes time.
West Virginia’s Application of
NIRN Drivers
Practitioner Selection
• All CPS supervisors and staff implementing the SAMS
model according to the Implementation Schedule.
• Bureau decision to utilize the concept of “Special Forces.”
A group of Program Managers, Supervisors, and Trainers
who are the primary purveyors of the model and responsible
for building practice competence with staff and supervisors.
Members selected to represent the best and brightest of
West Virginia’s Child Welfare workforce.
• Working on long-range plans for staff recruitment,
selection and allocation including interviewing questions
based on SAMS competencies.
West Virginia’s Application of
NIRN Drivers
Training
• Intense focus on staff training and professional
development.
– NRCCPS/ACTION trains and mentors SAMS Special Forces
– ACTION and SAMS Special Forces train supervisors
– SAMS Special Forces and supervisors train, mentor and
guide staff
– Targeted training prior to SAMS training on identified issues
that could negatively impact SAMS implementation.
– Targeted training after SAMS implementation on issues
that need more clarification or understanding.
- Ongoing skills training to improve interviewing and
family engagement skills.
West Virginia’s Application of
NIRN Drivers
Supervision and Coaching
• Major focus: build the knowledge, skills, and
abilities of CPS supervisors to supervise and coach
their staff.
– Receive SAMS training first that includes building
supervisor competencies related to SAMS.
– Attend SAMS training with their staff and take a lead
role in case discussions.
– Attend Action field consultation to discuss
implementation of the model with actual cases.
– Trained on the Action Supervisor Guide this summer
that discusses consultative supervision.
– Developing “mini-trainings” for supervisors to use in
staff meetings.
West Virginia’s Application of
NIRN Drivers
Performance Assessment
• Mastery of a new system of decision
making and delivery occurs over time with
focused consultation, supervision and
quality improvement.
– Fidelity monitoring integrated into clinical
case consultation through the use of a
standardized review instrument.
– Low fidelity areas addressed through Local
Maintenance Plans and Special Forces
consultation.
– Ongoing evaluation to be integrated into
OPQI reviews.
West Virginia’s Application of
NIRN Drivers
Systems Intervention
• Initially not enough attention paid to
stakeholder perception and
understanding.
– Internal stakeholders
• BCF Managers, Supervisors, and Staff (Child
Welfare as well as other program staff); DHHR
Secretary and Other Bureaus; Special Forces;
Implementation and Steering Committees
West Virginia’s Application of
NIRN Drivers
Systems Intervention
– External stakeholders
• Legislature; Court System; Education System;
Health Care System; Private Agencies; Service
Providers; Advocacy and Professional
Organizations
West Virginia’s Application of
NIRN Drivers
Systems Intervention
• Implemented activities in a variety of venues for increasing
stakeholder buy-in for SAMS.
– Communication with the Legislature including SAMS pocket cards.
– SAMS training for the legal community conducted by Regional
Attorneys.
– Presentations at statewide conferences and events, like NASW.
– Joint trainings with service providers and staff on safety services.
– Presentations at the Court Improvement Board and judicial crosstraining events.
– Presentations at professional and community group meetings.
– Web-based course providing an introduction to SAMS for all staff.
– Discussion on SAMS required at all district and regional staff
meetings.
West Virginia’s Application of
NIRN Drivers
Facilitative Administration
• Made changes to policies and procedures to align
with SAMS, including creation of transitional
policies.
• Organizational structure refined to support SAMS.
–
–
–
–
Director and Coordinator, funded through ACCWIC.
SAMS Implementation Team.
CPS Steering Team.
Feedback from the field brought back to
Implementation and Steering to address barriers and
facilitate implementation.
– Restructuring of office procedures, unit procedures,
and staff assignments at the local level.
West Virginia’s Application of
NIRN Drivers
Facilitative Administration
• State, regional, and district administrators prepared for
SAMS through statewide meetings and training sessions.
• Pre- and post-implementation meetings held with every
district administrator.
• Pre-implementation meetings with supervisors and
administrators, conducted by Action.
• Post-implementation consultation with supervisors and
administrators, conducted by Action and Special Forces.
• State, regional, and district administrators attend SAMS
training and consultation with their supervisors and staff.
West Virginia’s Application of
NIRN Drivers
Data Decision Systems
• West Virginia’s SACWIS system, FACTS,
currently making changes to the system to
support the SAMS assessments.
• Because of delays to FACTS implementation,
decision made to use a paper assessment
system until FACTS changes could be made.
• Staff still responsible for entering NCANDS and
AFCARS information into the current system
until FACTS changes are complete.
West Virginia’s Application of
NIRN Drivers
Data Decision Systems
• SAMS Fidelity Monitoring System developed
with ACCWIC to collect practice-relevant
fidelity data to inform implementation.
– Data system currently housed with ACCWIC
during development but will transition to OPQI.
– Data discussed at Implementation, Management,
and Special Forces meetings to identify gaps and
determine necessary actions to take.
– Fidelity data will be integrated into dashboards
along with federal outcomes data.
West Virginia’s Application of
NIRN Drivers
Data Decision Systems
• Dashboards developed with TA from NRC on
Data and NRC Organizational Improvement.
– Will measure CFSR outcomes and fidelity data by
region, county, and judicial district.
– Will show information on cases that do not meet
standards for managers to use to evaluate and
improve performance.
• West Virginia has initiated a bureau-wide
effort to make better use of data in
management decisions.
West Virginia’s Application of
NIRN Drivers
Leadership
• One of West Virginia’s strengths is the involvement of key
leaders in the implementation of SAMS.
– Commissioner and Deputy Commissioner take a lead role
and actively participate in Implementation and Steering.
– Reports on SAMS progress provided at each Leadership
Team and Field Operations Management Team meeting.
– SAMS discussion required at all regional and district
meetings.
- Commissioner, Deputy Commissioner, and Regional
Directors visiting districts that are implementing and
meeting with staff.
– Community Services Managers attending training with their
supervisors and actively involved in community education.
What do we need to know: challenges
for research?
 Basic Research on Decision Making in Child
Welfare
 Improved Assessment
 Improvements Decision Making Information
 Improving the Utility of Decision Making Supports
 Workforce Development
 Improved Implementation
Katherine Casillas
The Kempe Center for the Prevention and Treatment of Child Abuse and Neglect
Children’s Hospital Colorado
Denver, CO
[email protected]
What effects decision making in child
welfare, and what can we do to change
negative influences?
Effects on the Decision Threshold
Case
Factors
Individual
Factors
Decision
Threshold
Outcomes
Organizational
Factors
Influences
Intervention
Outcomes
Case Factors
 The Family
 Level of Risk
 Income
 Number of
Children
 Marital Status
 Race/Ethnicity
• The Child
– Gender
– Age
– Behavioral
Problems
– Medical
Problems
– Race/Ethnicity
Effects on the Decision Threshold
Case
Factors
Individual
Factors
Decision
Threshold
Outcomes
Organizational
Factors
Influences
Intervention
Outcomes
Individual Factors
 Decision Making Skills (e.g., good
judgment)
 Case Skills (e.g., fact finding)
 Knowledge of Placements
 Cultural Sensitivity (e.g., awareness)
Effects on the Decision Threshold
Case
Factors
Individual
Factors
Decision
Threshold
Outcomes
Organizational
Factors
Influences
Intervention
Outcomes
Organizational Factors
 Supervision
 Agency Policy
 Availability of Placements
 Workload
The influence of disproportionality and disparities
 Are racial and ethnic groups disproportionately
represented in Child Protective Services (CPS)?
 Can the process be changed?
The Continuum of Intervention
ASSESSMENT
Child protection Screening Assessment Placement Reunification
Decisions/Actions
Assessments and decisions are made at key points along the child
protection continuum
Each key decision point requires a specific decision and action
Understanding Trends

National level data can be misleading when attempting to understand
and improve child welfare at the local level.

Racial disproportionality and disparate outcomes data must be analyzed
and understood at the local level.

Disparity-related dynamics can be very different from one jurisdiction to
the next, even when the jurisdictions share similar characteristics.
127
128
Enumerating Disparities and
Disproportionality in Decision Points
 Population Based Denominator Ratios
 Based on data from one child welfare decision (e.g.,
new placements/population)
 Easiest to obtain
 Decision Based Denominator Ratios
 Based on data from at least two child welfare
decisions (e.g., new placements/opened cases)
 Disparity: Relative to another group
Colorado Child Population /
Race & Ethnicity Legend
Black
White
Hispanic
Asian
AI_AN
NH_OPAC
Multi
Black alone, not Hispanic or Latino
White alone, not Hispanic or Latino
Hispanic or Latino
Asian alone, not Hispanic or Latino
American Indian and Alaska Native Alone, Not Hispanic or Latino
Native Hawaiian and other Pacific Islander, not Hispanic or Latino
Two or more races, not Hispanic or Latino
2010
Population
Under 18
49967
710280
374225
32225
7298
1557
47285
FY 2011 (Jul, 1 2010 - June, 30 2011)
Children in Referrals
Type of Analysis
Children in Referrals
Ethnicity
Child Population
Disparity Indices
Rate Per
% of (Total Number % Of (Total
1000 of
Children % of Eligible Minus of Eligible
Minus
Eligible
Missing) Children Missing)
Children
Total
126734
Black
8471
17.0%
6.7%
49967
4.1%
169.53
2.57
White
46857
6.6%
37.0%
710280
58.0%
65.97
1.00
Hispanic
33791
9.0%
26.7%
374225
30.5%
90.30
1.37
Asian
628
1.9%
0.5%
32225
2.6%
19.49
0.30
AI_AN
430
5.9%
0.3%
7298
0.6%
58.92
0.89
NH_OPAC
180
11.6%
0.1%
1557
0.1%
115.61
1.75
3014
6.4%
2.4%
47285
3.9%
63.74
0.97
Multi
Missing
33363
1225609
Compared with
White
*
103.40
FY 2011 (Jul, 1 2010 - June, 30 2011)
Children in Assessments
Type of Analysis
Children in Assessments
Ethnicity
Child Referred
Disparity Indices
Rate Per
% of (Total Number % Of (Total
1000 of
Children % of Eligible Minus
of Eligible
Minus
Eligible
Missing) Children Missing)
Children
Total
64657
Black
4572
54.0%
7.1%
8471
6.7%
539.72
0.96
White
26246
56.0%
40.6%
46857
37.0%
560.13
1.00
Hispanic
19887
58.9%
30.8%
33791
26.7%
588.53
1.05
Asian
381
60.7%
0.6%
628
0.5%
606.69
1.08
AI_AN
244
56.7%
0.4%
430
0.3%
567.44
1.01
NH_OPAC
105
58.3%
0.2%
180
0.1%
583.33
1.04
1714
56.9%
2.7%
3014
2.4%
568.68
1.02
11508
34.5%
17.8%
33363
26.3%
344.93
0.62
Multi
Missing
126734
Compared with
White
510.18
FY 2011 (Jul, 1 2010 - June, 30 2011)
Children in Substantiated Cases
Type of Analysis
Children in
Substantiated Cases
Ethnicity
Child Assessed
Disparity Indices
Rate Per
% of (Total Number % Of (Total
1000 of
Children % of Eligible Minus of Eligible
Minus
Eligible
Missing) Children Missing)
Children
64657
Compared with
White
Total
10713
16.6%
165.69
Black
920
20.1%
8.6%
4572
7.1%
201.22
1.07
White
4956
18.9%
46.3%
26246
40.6%
188.83
1.00
Hispanic
4035
20.3%
37.7%
19887
30.8%
202.90
1.07
Asian
73
19.2%
0.7%
381
0.6%
191.60
1.01
AI_AN
73
29.9%
0.7%
244
0.4%
299.18
1.58
NH_OPAC
17
16.2%
0.2%
105
0.2%
161.90
0.86
Multi
392
22.9%
3.7%
1714
2.7%
228.70
1.21
Missing
247
2.1%
2.3%
11508
17.8%
21.46
0.11
FY 2011 (Jul, 1 2010 - June, 30 2011)
Type of
Analysis
Ethnicity
Total
Black
White
Children Hispanic
in
Removed Asian
Within 90 AI_AN
days
NH_OPA
C
Multi
Missing
Children in Removed Within 90
Disparity
days
Children Served
Indices
% of
% Of
Rate Per
Number
% of
(Total
(Total
1000 of
Compared
Children
of Eligible
Eligible
Minus
Minus
Eligible with White
Children
Missing
Missing Children
2415
245
1114
899
9
31
40.4%
35.5%
35.5%
23.1%
50.8%
3
110
4
42.9%
43.1%
2.6%
3.6%
16.4%
13.2%
0.1%
0.5%
6796
607
3141
2534
39
61
20.1%
18.9%
20.3%
19.2%
29.9%
403.62
354.66
354.78
230.77
508.20
1.14
1.00
1.00
0.65
1.43
0.0%
1.6%
0.1%
7
255
152
16.2%
22.9%
2.1%
428.57
431.37
26.32
1.21
1.22
0.07
FY 2011 (Jul, 1 2010 - June, 30 2011)
First Placement Congregate Care
Type of Analysis
Ethnicity Children
Disparity Indices
Rate Per
% of (Total Number % Of (Total
1000 of Compared with
Minus of Eligible Minus
Eligible
White
Missing) Children Missing)
Children
Total
207
Black
32
13.1%
0.5%
245
20.1%
130.61
1.46
White
100
9.0%
1.5%
1114
18.9%
89.77
1.00
71
7.9%
1.0%
899
20.3%
78.98
0.88
Asian
9
19.2%
AI_AN
31
29.9%
3
16.2%
110
22.9%
36.36
0.41
4
2.1%
Hispanic
First Placement
Congregate Care
% of
Eligible
Children Removed
2415
NH_OPAC
Multi
Missing
4
3.6%
0.1%
FY 2011 (Jul, 1 2010 - June, 30 2011)
First Placement Foster Care
Type of Analysis
First Placement
Foster Care
Ethnicity Children
% of
Eligible
Children Removed
Disparity Indices
Rate Per
% of (Total Number % Of (Total
1000 of Compared with
Minus of Eligible Minus
Eligible
White
Missing) Children Missing)
Children
Total
1266
Black
124
50.6%
1.8%
245
20.1%
506.12
0.97
White
581
52.2%
8.5%
1114
18.9%
521.54
1.00
Hispanic
460
51.2%
6.8%
899
20.3%
511.68
0.98
Asian
4
44.4%
0.1%
9
19.2%
444.44
0.85
AI_AN
26
83.9%
0.4%
31
29.9%
838.71
1.61
1
33.3%
0.0%
3
16.2%
333.33
0.64
66
60.0%
1.0%
110
22.9%
600.00
1.15
4
100.0%
0.1%
4
2.1% 1000.00
1.92
NH_OPAC
Multi
Missing
2415
FY 2011 (Jul, 1 2010 - June, 30 2011)
First Placement Other Care
Type of Analysis
First Placement
Other Care
Ethnicity Children
% of
Eligible
Children Removed
Disparity Indices
Rate Per
% of (Total Number % Of (Total
1000 of Compared with
Minus of Eligible Minus
Eligible
White
Missing) Children Missing)
Children
Total
57
2415
Black
6
2.4%
0.1%
245
20.1%
24.49
1.09
White
25
2.2%
0.4%
1114
18.9%
22.44
1.00
Hispanic
20
2.2%
0.3%
899
20.3%
22.25
0.99
Asian
1
11.1%
0.0%
9
19.2%
111.11
4.95
AI_AN
1
3.2%
0.0%
31
29.9%
32.26
1.44
3
16.2%
110
22.9%
36.36
1.62
4
2.1%
NH_OPAC
Multi
Missing
4
3.6%
0.1%
FY 2011 (Jul, 1 2010 - June, 30 2011)
First Placement Relative Care
Type of Analysis
Ethnicity Children
% of
Eligible
Children Removed
Disparity Indices
Rate Per
% of (Total Number % Of (Total
1000 of Compared with
Minus of Eligible Minus
Eligible
White
Missing) Children Missing)
Children
Total
870
Black
83
33.9%
1.2%
245
20.1%
338.78
0.95
White
399
35.8%
5.9%
1114
18.9%
358.17
1.00
Hispanic
342
38.0%
5.0%
899
20.3%
380.42
1.06
4
44.4%
0.1%
9
19.2%
444.44
1.24
4
12.9%
0.1%
31
29.9%
129.03
0.36
2
66.7%
0.0%
3
16.2%
666.67
1.86
36
32.7%
0.5%
110
22.9%
327.27
0.91
4
2.1%
First Placement Relative Asian
Care
AI_AN
NH_OPAC
Multi
Missing
2415
What can we do?:
Explaining Removals at the Worker Level
Percent on
Caseload
Disparity Index
Workload
(worker)
-
Skills (AA
only)
+
AA
Only
Removals
+
Community
Resources
(worker)
Family
Poverty
+
Family Risk
Level
Organizational Interventions and the Removal Rates for
African American Children Pre and Post
Relative Rate Indices for Removals of African American
Children in the Five Original Sites
2.00
1.50
1.00
Pre
Post
0.50
Harris
Dallas
Tarrant
Travis
Jefferson All Texas
Counties
0.00
• In four of the five counties where the effort has been most
intense have lowered African American removal rates
Discussion
 Are there key decision points where we should be focusing
our attention?
 What are the explanatory factors that we should attend to?
 What are some suggested policy revisions that could
mitigate factors in child welfare?
 How do we make progress in integrating and
improving clinical/professional judgment in the
assessment process?
 What are the best ways to influence decision actions?
Decision Making Café
• Please Self Select a Discussion Topic
– Table 1 – Tools
– Table 2 – Policy and Workforce
– Table 3 – Implementation
– Table 4 - Research
Family Group Decision Making and
Family Engagement in Decision Making
Description of the “It”
What it is:
An opportunity for the family group to gather all the needed information
about the agency’s concerns in order to make well-informed decisions.
• Family-driven decision making planning
• Family as the expert
What it is not:
Mediation
Conflict resolution process
An opportunity for families to come together to hear agency professionals’
solutions
Guidelines for Family Group Decision Making in Child Welfare, 2010
143
Terminology
Family Involvement and Decision Making Models:
– Family Group Conference or FGC
(New Zealand, 1989)
– Family Unity Meeting Model or FUM
(Oregon, 1990)
– Family Team Conferencing (FTC) as developed by
the Child Welfare Practice and Policy Group
– Team Decision Making (TDM)
– Family Team Meetings (FTM)
– Rapid Case Planning Conference
– Hybrid Models that combine
FGC, FTC, FUM, and other models
Slide 144
Five Core Elements of FGDM
1.
An independent Coordinator
2.
3.
Family group as key decision making partner—resources put
towards finding and preparing
Private family time
4.
When plan meets agency concerns, preference to family plan
5.
Services and resources available to meet agreed upon plans
American Humane Association (2008)
145
Family Involvement Continuum1
Family Voice in Decision Making
Families, along with their
support network, craft initial
plans that are subsequently
shared with the professionals
who work collaboratively with
the family to ensure it is
attainable and meets the
highest standards for achieving
the goals of safety,
permanency, and well-being.
Families are part of the
decision making team.
In these instances,
families partner with
professionals to create
consensual decisions
acceptable to all
parties.
System Voice in Decision Making
Families have a
genuine voice at the
meetings. Their ideas,
needs, perspectives,
and other inputs are
sought at the meetings,
but the decision
making rests with
professionals.
Families are
present at
meetings where
decisions will be
made about their
children.
Families are not
included in
meetings or other
forums where
decisions are
made about their
children.
1Taken
from: Merkel-Holguin, L. and Wilmot, L. (2005). Analyzing family involvement approaches in J. Pennell & G. Anderson (Eds.), Widening the circle: The
practice and evaluation of family group conferencing with children, young persons, and their families. Washington, DC: NASW Press.
May be reproduced and distributed with appropriate citation.
146
Participation Satisfaction and
Communication Benefits
• Family satisfaction
– Conflict decreased
– Felt empowered
• Worker satisfaction
– Felt children would be better protected and greater
confidence in child safety
• Improved relationships with child welfare agency
– Felt listened to
– Better understood agency’s concerns
147
Resources for Families
• More quickly linked to
services
• Received more individualized
resources
• More likely to receive
counseling for children and
mental health and parenting
services for adult caregivers
148
Child welfare outcomes
• Placements
– More stable
– Less likely to be placed in institutions
– More likely to be placed with extended family
• Beneficial impacts: more stability, better adjustment, ties to
culture and traditions
– Fewer temporary moves
– More likely to maintain connections to siblings
– Achievement of permanency sooner and less time in outof-home care
• Improved child safety
• Reduced disproportionality
149