Transcript Examining Evidence

NASCSP March 1, 2012
• “Success should be judged by results, and
data is a powerful tool to determine results.
We can’t ignore facts. We can’t ignore data.”
President Barack Obama
July 24, 2009
• “The test of a performance management
system is whether it’s actually used….Federal
managers and employees at all levels must
use performance goals and measures to set
priorities, monitor progress, and diagnose
problems.”
Chief Performance Officer Jeffrey Zients
October 29, 2009
2

Two key points:
◦ Results are supported by data
◦ Data from performance measurement systems must
be put to use.


In recent OCS guidance, several “models” and
references to “evidence” have been identified.
Here is a quick review of these terms.
3



Approaches to prevention or treatment that are
validated by some form of documented scientific
evidence.
This could be findings established through
scientific research, such as controlled clinical
studies, or other comparable and rigorous
methods.
Identifies proven effectiveness, supported by
objective and comprehensive research and
evaluation.
4


Approaches that use the best available
research and practice knowledge to guide
program design and implementation within
context.
This informed practice allows for innovation
and incorporates the lessons learned from
the existing research literature.
5


An innovative and consistently applied policy,
process, practice, or procedure that takes a
comprehensive approach to developing and
implementing activities using strategies that
are related to the intended service recipients
and community.
This practice model is culturally competent,
data-driven, measurable, and replicable and
incorporates a method for documenting
programmatic results.
6

A practice with at least preliminary evidence
of effectiveness in small-scale interventions,
or for which there is potential for generating
data that will be useful for making decisions
about taking the intervention to scale the
results diverse populations and settings.
7


A program, activity or strategy that has been
shown to work effectively and produce
successful outcomes
This is supported to some degree by
subjective and objective data sources
8
What do we know about our measurement tools?
Are they
 Accessible
 Reliable
 Valid (appropriate)
 Affordable
 Scalable (able to measure change)
9









Bank accounts
Copy of Diploma or
Certificate
Employment records
Escrow accounts
Financial reports
Health or nutrition records
Inspection results
Lease agreements
Legal documents









Loan monitoring reports
Mortgage documents
Observation log
Pre-post tests
Progress reports
Questionnaire
Rent receipts
Scales and Matrices
Survey
Testing results
10
IDENTIFY PROS AND CONS OF EACH SOURCE







Agency database
Case notes
Centralized database
Computer spreadsheets
File cabinets
Individual case records
Manual tallies






Public database
School records
Specialized database
Tax Assessor database
Training center
Work plan reports
11
 If
no one is assigned the task,
no one will do it.


Consider various agency staff responsibilities
Include partner reports
12




Home visits
Office appointments
Phone follow up
Review of case notes/progress reports
(scales?)
13







Daily
Weekly
Monthly
Quarterly
Biannually
Annually
Upon incident
14
15
Data about community needs and resources
This can identify the scope of community issues
Used to prioritize community needs for agency intervention
Data used during strategic planning
Identify resources ($, facilities, staff, etc) the agency has/needs
What results does the agency expect to achieve?
Data collected during implementation of services
How many individuals, families, communities are projected to be served?
How many are projected to achieve results?
How many individuals, families, communities were actually served?
Who were they?
Data collected to identify achievement of results
How many made movement toward their goals?
How many achieved the end results previously identified?
16
Evaluation Data
Used in comparing CAA performance with a
“control” group to determine the quality of
the results
Done by an independent reviewer
Used to validate that results actually impact
on identified community needs
17
Source: Child Trends
18


The collection of NPI data for a report to
OCS is a comprehensive effort by the
CSBG Network to identify the results of
the efforts of local CAAs.
This is sometimes called the ROMA
report, but we have just seen that ROMA
data includes more than these indicators.
19





Measure and report performance
Identify patterns and relationships
Consider new actions based on analysis
Create new outcome-focused goals
Develop new resources
20
•
•
•
•
When you have your data, remember that you
have to turn them into information.
They must be interpreted to become evidence
Consider this piece of data and the questions
that arise:
35 individuals got a job.
Is that good?
What is the unemployment rate? Opportunity for
employment? Characteristics of those who got jobs.
Etc.
21



What are the top 10 services provided by your
agency?
What are the top 10 outcomes achieved by
your customers?
What is the connection between these two
lists?
22


Build on ROMA principles to identify ways
to collect credible evidence in systematic
ways (making the connections among
mission, community need, agency
strategies and well documented results)
Find ways to compare CAA performance
with established standards and
performance of other similar programs.
23
•
•
•
We must recognize the value of evidence-based
or evidence-informed interventions…. but our
definition of what counts as credible evidence
should be expanded to allow for continuing
improvement and innovation.
‘‘Evidence-based’’ does not have to mean
experimental-based.
We draw on evidence from many kinds of
research, including program evaluations, and
practice.
From Lisbeth Schorr
24


Lisbeth Schorr tells us to identify the clear,
measurable results sought by a complex
intervention as the essential first step toward
both successful implementation and to a
successful evaluation.
We have a start on this with the NPIs.
25
•
•
It is essential to have some way of comparing
results to establish that the observed change has
a high probability of resulting from the practices.
How you will compare results from your
programs to those who have not been involved in
your programs.
The community-specific nature of place-based
interventions makes it very hard to find a
comparison group that would allow for a clinical
“control group.”
26
It is possible to compare outcomes among the
populations those served by a specific
initiative to
(a) similar populations in the geographic area
before the intervention began for whom
baseline data are available,
(b) current populations that did not receive
similar services and supports, but for whom
data are already available, i.e. does not have
to be collected as part of the evaluation, and
(c) national, state or local norms.
27



The elements of ROMA principles and
practices can help agencies establish
evaluation frameworks that include the
evidence that will be used to support their
results
Establish CSBG industry standards
Remember what baseball did that we
haven’t yet done:
28




Clearly identified the indicator to be
measured
Collect data consistently
Analyze the data against records of
“wins”
Publish the data and the analysis, so the
public recognizes success
29





Support knowledge collection, analyses, and
evidence syntheses that yield a more complete
body of evidence.
Develop network tools and capacities to gather
knowledge at greater scale.
Expand the menu of available evaluative
techniques that can be matched to different types
of interventions.
Combine findings from research, theory, and
practice, for informed decision-making
Promote use of a results framework to strengthen
measurement for accountability and learning
30
•
Real-time learning can be achieved by using a
well developed results framework to track
progress toward those results, and using the
data for real-time learning.
31



As Peter Drucker has pointed out, “The
greatest danger in times of turbulence is not
the turbulence; it is to act with yesterday’s
logic.”
We do not want to rely on fads, hunches,
anecdotes, or good intentions.
Nor are we reluctant to identify and end
support for the efforts that are ineffective.
32
Barbara Mooney
National ROMA Training Project
243 E. High St.
Waynesburg, PA
724-852-2272
[email protected]
33