Transcript FRIENDS Template

Evaluating Outcomes of
Prevention Programs
FRIENDS National Resource Center for
Community Based Child Abuse Prevention
A Service of the Children’s Bureau
www.friendsnrc.org
Overview of the Day
• Review of Logic Models: a template for planning,
evaluating, and reporting
• Using data to develop and maintain a CQI
environment
• Administering the Protective Factors Survey
(PFS)
• Identifying other methods for measuring
outcomes
• Managing and reporting on outcomes
Tell me, why evaluate?
If you measure it, it’s more likely to get done.
If you don’t measure results, how do you know
you’ve succeeded?
And, how do you tell others what you’ve done?
How do you learn what needs improvement?
We also need to show funders, staff, &
parents that their time, money & efforts are
well spent.
Plus, with limited funding opportunities . . . .
Good evaluation
could lead to better
chances of funding?
Well. . . yes
Evidence-based
and evidenceinformed
programs require
evaluation
I’m convinced.
. . How do I
get started?
It all begins with a logic
model . . .
What is that fancy one-paged
chart?
Is it a program map? A tool for
planning evaluation? A report
template? Is it a requirement for
grant applications?
It’s all of that and
more! Friends, it’s a
Logic Model!
A Program
Map? !
Yes!
A logic Model outlines:
 Where you want to go.
 How you plan to get there
 What will you see that tells you you’ve
arrived at your destination
If we do this . . . Then this will happen . . .
A tool for
planning
evaluations?
Yes! A logic model can specify
the tools that will be used to
measure to what degree you
achieved your desired outcomes
A report
template?
Yes!
When you’re ready to report on your
services, pull out your logic model and
use it as a template to organize your
report.
A requirement for
grant applications?
Yes!
Most grant applications are asking for
a logic model as part of the proposal.
Don’t leave home without one.
Logic Model Components
• Vision
• Population
• Services
• Assumptions
• Resources
• Outcomes
• Indicators
• Measurement Tools
Remember:
These
components
sometimes
go by
different
names.
Vision (Goal, impact, long-term outcome)
A broad statement of well-being; the long-term impact
on social, economic or environmental conditions.
Examples:
* Children &Youth are Nurtured, Safe & Engaged
* Families are Strong & Connected
* Communities are Caring and Responsive
* From Pathways to the Prevention of Child Abuse &
Neglect, Pathways Mapping Initiative , Schorr, & Marchand,
2007
http://www.dss.cahwnet.gov/CDSSWEB/entres/pdf/Pathway.pdf
Population
(consumers, participants, audience)
• Who are your participants?
• What are their characteristics and
demographics?
• What are their needs?
• How do you know?
– Needs assessments
– Focus groups
Services (outputs, Activities)
• Based on the needs assessment of your
target population– What services are you providing/will you
provide?
(You will revisit this section as you move
further in your logic model)
– What will they look like?
– How frequently will they occur?
Assumptions
(theory of change, research base)
•What assumptions are you making that
suggest your services will be successful?
•Services should be based on what is known
to be effective.
•Your research into assumptions should
result in understanding the degree to which
you are providing evidence-based services.
Assumptions, continued
If uncertain of the empirical foundations of
your approach, now is the time to stop and
investigate what works. This link will help
get you started:
http://www.childwelfare.gov/preventing/pr
ograms/whatworks/index.cfm
The FRIENDS website has additional
guidelines on Evidence-Based Practices.
Please check the site regularly
http://www.friendsnrc.org
Services
• Twelve 1 hour parenting education sessions using
the ”Safe Homes” curriculum.
• Access to concrete materials (such as fencing, stair
gates, car seats, etc.)
Assumptions
Two evaluations of the Safe Homes curriculum (Fisher,
2000, Hale & Carter, 2003), each conducted over a 3year period, found that children of parents who had
participated in the program were 50% less likely to
experience preventable injuries than children in
comparison groups.
Resources (inputs)
Do you have the infrastructure needed?
• Based on the services you identify,
– What resources do you need?
– If you don’t have them, can you get them?
The Implementation Plan worksheet on the
FRIENDS web-site can help.
http://www.friendsnrc.org/download/dtapp
endices.pdf
page 45
Outcomes
(Goals, objectives)
Movement towards or attainment of a
change in attitude, belief, behavior.
Ask yourself: “If you conduct your
program’s activities, and succeed
with participants, what do they
believe, know, have or do as a result?
Short-Term Outcomes
(engagement, knowledge)
• Engagement outcomes: First steps:
gaining trust, interest and involvement
• Learning outcomes: Changes in
knowledge, attitudes, beliefs
Intermediate Outcomes
(practice)
Intermediate
• Moving from learning to action
• Practicing and refining new skills
• Adopting or changing behaviors
Long-Term Outcomes
(integration)
Long-term
• When participants have integrated new
behaviors into their lives over time.
• Consistently practicing new skills and
behaviors
Writing Outcome Statements
Who
(example: Participants)
Will Do
(example: will describe)
What
(example: a child-safe home)
The FRIENDS Evaluation toolkit has examples
of outcomes a click away!
http://www.friendsnrc.org/outcome/toolkit/ou
tin.htm
Formatting Outcomes
• Short-term: Participants demonstrate
knowledge of how to appropriately
supervise their children.
• Intermediate: Participants provide
appropriate supervision of their children.
• Long-term: Participants provide
appropriate supervision according to the
changing needs of the children as they
grow and develop.
Outcomes need indicators
Desired
Change
Evidence of
Change
Indicators
(performance indicators)
• Indicators are specific signs needed to
track progress
• Indicators answer the question: What
would I see or hear that would tell me
that the outcome was being achieved?
• Can spell out a level of achievement to
attain in #’s or %’s
Participants appropriately manage
child behavior.
Participants
encourage positive
behaviors
Participants
consistently enforce
their household rules.
Participants set realistic,
age-appropriate household
rules.
Participants enforce rules
without coercion or
physical punishment
Participants encourage positive behaviors
Participants state
expectations clearly
Participants use words
of encouragement to
their children
Participants look for their
children’s “good” behavior
Participants model
appropriate behavior
Setting Benchmarks
• You may need to identify performance
targets (benchmarks) for each indicator).
• If possible, set benchmarks based on
earlier evaluations.
• If you need to set benchmarks before you
provide service, consult with peers to set
realistic benchmarks and adjust them
based on your evaluation findings.
Parents demonstrate knowledge of how to
provide a safe home environment for
their children.
80% of the participants
name 6 features of a
child-safe house.
80% of the participants
identify at least 6
common household
hazards.
80% of participants
describe how to
eliminate or reduce at
least 6 household
hazards.
How will you know if you’ve achieved
targeted outcomes?
Outcomes
Indicators
Measures
Measurement Tools “test” whether
the indicator was achieved.
Outcome
Student is proficient in 4th grade math.
One Indicator:
Student knows multiplication tables up to 12
One Measure:
Student scores 90% in the Tedious Tables in
the Nuts about Numbers text book
Participants foster their infant’s
social/emotional development
Parents correctly identify their infants
verbal and non-verbal cues
Subscale 3 on the NCAST PCI
Feeding and Teaching Scales.
Selecting Measurement Tools
Look for tools that are
• linked to outcomes
• culturally appropriate
• valid and reliable
• practical (think time & money)
Compendium of annotated tools:
http://www.friendsnrc.org/outcome/toolkit
/annot.htm
Simple Counts & Checklists
Children receiving a dental check-up
Parents who attended a parent/teacher
conference
Mothers who received pre-natal care
Number of fathers who maintained
regular contact with their children
Example Counts and Checklists
Is your home safe for children?
Bathroom:
All medications are kept out of reach and sight of
my children.
Chemicals I use for cleaning sinks, toilets and
floors are inaccessible to children.
Public Agency Records
• Police Reports
• CPS Records
• Court Records
• School Records
A potentially good data source but there’s
a lot to consider before using them.
Informed consent, reliability of data,
accessibility . . .
You may choose to construct
your own tools
This should be undertaken very
cautiously. Check out the FRIENDS
Evaluation Toolkit for information.
friendsnrc.org
Qualitative Data
Captures information that evaluation
tools can’t always capture. Can be
gathered through
Surveys, interviews, focus groups, key
informants, case notes, observations
For more information:
FRIENDS Guide to Qualitative Evaluation:
http://www.friendsnrc.org/download/outcomeresour
ces/qualitativedata.pdf
“The horror of that
moment,” the King
went on, “I shall
never, NEVER forget!”
“You will, though,”
the Queen said, “if
you don’t make a
memorandum of it.”
Lewis Carroll,
Through the Looking
Glass
Continuous Quality Improvement is.
a process to ensure programs are
systematically and intentionally
increasing positive outcomes for the
families they serve. It is about getting
better and better.
Instead of “if it ain’t broke, don’t fix
it,” CQI emphasizes that even if
programs aren’t broken, they can
always be improved.”
FRIENDS National Resource Center for Community Based Child Abuse Prevention
A Service of the Children’s Bureau
Plan, Do, Study, Act Cycle
PLAN services that are
 responsive to consumer needs &
desires
 evidence based or evidence informed
 mapped in a logic model
Develop a manual
Plan your evaluation methods
FRIENDS National Resource Center for Community Based Child Abuse Prevention
A Service of the Children’s Bureau
Do
Provide services
– Hire, train and supervise staff
– Document activities
– Monitor fidelity
– Collect data
–formally
–informally
FRIENDS National Resource Center for Community Based Child Abuse Prevention
A Service of the Children’s Bureau
Study
Study your data
• formally, in the course of staff
supervision, full staff meetings, board
meetings, and
• informally, through daily discussions with
staff and participants; self-assessment
of job performance, observation of dayto-day participant progress and
satisfaction.
FRIENDS National Resource Center for Community Based Child Abuse Prevention
A Service of the Children’s Bureau
Act
Act by adjusting practices
•formally, at the agency level by adopting
new practices, programs, policies and
procedures based on findings
•informally, by making personal
adjustments to improve job performance
Plan, Do, Study, Act may begin as a linear
process but soon all steps are happening
all the time.
Safe Environment
What Drives CQI?
•
•
•
•
•
•
•
System Wide
Consumer Involvement
Logic Model
Manual
Staff
Evaluation
Data Driven Decision Making
CQI Self Assessment Tool
• What CQI activities are occurring in
your agency?
• How you would like to strengthen the
atmosphere of CQI in your agency?
The Protective Factors Survey (PFS)
It is difficult to show that a program
prevented negative things from happening
Increasing protective factors minimizes the
risk for maltreatment
The Protective Factors Survey is one way to
measure the effectiveness of prevention
programs.
Protective Factors
•There is no single cause of child
maltreatment.
•a number of risk factors or
attributes commonly associated
with maltreatment.
•a number of protective factors
to reduce the occurrence of child
maltreatment.
Risk Factors
Parent Factors – mental health, substance
abuse, attitudes and knowledge, age
Family Factors – economics, stress,
domestic violence, household size,
Child Factors – Age, disabilities,
temperament
Environmental Factors – poverty,
unemployment, lack of social support, and
community violence
Understanding the Risk Factors
Even though certain factors are
present where maltreatment occurs,
this does not mean that the presence
of these factors will always result in
child abuse and neglect.
Most people living in poverty do not
harm their children.
Protective Factors
Factors that can protect families
and promote resilience.
A Protective Factors framework
focuses on prevention strategies
based on building strengths with
families rather than exclusively on
risks and deficits.
Strengthening Families & the PFS
CSSP
Parental Resilience
Social Connections
Parenting Knowledge
Concrete Support in
Times of Need
Social & Emotional
Competence of Children
CBCAP/PSF
Family
Functioning/Resiliency
Social Emotional Support
Knowledge of Parenting/
Child Development
Concrete Support
Nurturing and Attachment
The relationship between the PFS
& other measures
• All four subscales of the PFS were:
– significantly negatively correlated with child
abuse potential and stress
– significantly positively related to optimism
and the adaptive coping strategy of positive
reframing
• All subscales except Concrete Support were
significantly negatively related to depression
and positively related to positive affect
Predictive validity of the PFS
• Time One PFS subscales were negatively
related to Time Two stress and depression
• Time One PFS subscales were positively
related to all six subscales of the Rand
Health Survey at Time Two
Survey Administration
Staff preparation
Setting
Individual or group
Informed Consent
Instructions
Scripts
Paraphrasing
Data Management
Enter data
Store raw data in secure location
Destroy surveys within prescribed
time frames
FRIENDS PSF Database
Resources
• Survey materials
– Training manual, PFS technical
information handout, survey, reports all
available at www.friendsnrc.org
– FRIENDS database
How can I find other evaluation
tools?
The FRIENDS Evaluation Toolkit and
Logic Model Builder includes a
compendium of evaluation tools.
Access it at:
http://www.friendsnrc.org/outcome/to
olkit/annot.htm
Reporting on Outcomes
The report should include:
• Executive Summary
• Body of the Report
– Use your logic model as a map
for developing this major
section of the report
• Conclusion
The Body of the Report
Population Targeted/Population Served
Services Planned/Services Provided
(include info on Implementation & Fidelity)
Assumptions/Underlying theory
Outcomes Projected
Evaluation Methodology
Outcome Achievement
Analysis of Results
Future Planning
Contact Information
Casandra Firman
Training and Technical Assistance Coordinator
FRIENDS National Resource Center for Community Based
Child Abuse Prevention
(360) 769-7167
[email protected]
FRIENDS National Resource Center for Community Based Child Abuse Prevention
A Service of the Children’s Bureau