Transcript Results-Based Accountability

The Charter Oak Group, LLC
Rev. 1 (12/31/11)
I N N O VAT I O N
RBA Overview for Connecticut
Nonprofits and Communities
Results-Based Accountability™
Mark Friedman
Fiscal Policy Studies Institute
Santa Fe, New Mexico
WEBSITES
www.resultsacountability.com
www.raguide.org
Book - DVD Orders
amazon.com
resultsleadership.org
2
2
Objectives
 Understand the basic concepts and language of ResultsBased Accountability™ (RBA)
 Learn how indicators can be used to track progress and
develop strategies
 Understand how you can use RBA to engage a group in
turning the curve on an indicator that is important for the wellbeing of children, families, or communities
 Know the different types of performance measures and how to
use them for accountability and program improvement
 Understand how you can use RBA to help your organization
or community develop a framework for an agenda that can
guide short-term action and long-term planning
3
3
SIMPLE
COMMON SENSE
PLAIN LANGUAGE
MINIMUM PAPER
USEFUL
4
4
Two Key Principles for Achieving
Measurable Community Results
1. Start with ends and work backwards to
means
2. Use data-driven, transparent decision
making
5
5
Results Accountability
Is Made Up Of Two Parts:
Population Accountability
about the well-being of
WHOLE POPULATIONS
For Communities – Cities – Counties – States - Nations
Performance Accountability
about the well-being of
CLIENT POPULATIONS
For Programs – Agencies – and Service Systems
6
6
Results and Performance
Accountability
COMMON LANGUAGE
COMMON SENSE
COMMON GROUND
7
7
The Language Trap
Too many terms. Too few definitions. Too little discipline.
Benchmark
Outcome
Result
Modifiers
Indicator
Measurable Core
Urgent
Qualitative
Priority
Programmatic
Targeted
Performance
Incremental Strategic
Systemic
Measure
Goal
Objective
Target
Lewis Carroll Center for Language Disorders
8
8
Definitions
RESULT
Population
A condition of well-being for
children, adults, families or communities.
Children born healthy, Children succeeding in school,
Safe communities, Clean Environment, Prosperous Economy
INDICATOR
A measure which helps quantify the achievement
of a result.
Performance
Rate of low-birthweight babies, Rate of high school graduation,
crime rate, air quality index, unemployment rate
PERFORMANCE MEASURE
A measure of how well a program, agency or service
system is working.
1. How much did we do?
Three types:
2. How well did we do it?
3. Is anyone better off?
= Customer Outcomes
9
9
From Ends to Means…
From Talk to Action
Population
RESULT
ENDS
Performance
INDICATOR
PERFORMANCE
MEASURE
Customer outcome = Ends
Service delivery = Means
MEANS
10
10
IS IT A RESULT, INDICATOR, OR
PERFORMANCE MEASURE?
RESULT
INDICATOR
PERF. MEASURE
RESULT
INDICATOR
RESULT
INDICATOR
PERF. MEASURE
1. Safe Community
2. Crime Rate
3. Average Police Dept response time
4. A community without graffiti
5. % of surveyed buildings without graffiti
6. People have living wage jobs and income
7. % of people with living wage jobs and income
8. % of participants in job training program who get
living wage jobs
11
11
Connecticut Glossary of RBA Terms
 The Appropriations Committee standardized the terms
we use in Connecticut
 Terms in Connecticut glossary are consistent with
Friedman’s RBA approach
 Everyone in Connecticut– executive branch, legislative
branch, and now communities – is using a common
language and speaking with a common understanding
12
12
For Whole Populations
in a Geographic Area
I N N O VAT I O N
POPULATION
ACCOUNTABILITY
Results
 Results (population results or quality of life results) are conditions of
well-being for children, adults, families or communities, stated in
plain English (or plain Spanish, or plain Korean...).
 They are things that voters and taxpayers can understand. They are
not about programs or agencies or government jargon. Results
include: "healthy children, children ready for school, children
succeeding in school, children staying out of trouble, strong families,
elders living with dignity in setting they prefer, safe communities, a
healthy clean environment, a prosperous economy."
 Definition: A condition of well-being for people in a place, stated as
desired result. "All ______ in ______ _____." e.g. All children in
Connecticut born healthy and developmentally on target from Birth to
3“ or “A clean and healthy Long Island Sound for Connecticut’s
residents” or “All Connecticut citizens secure and free from crime.”
14
14
Results
Population
+
Geographic Area
+
Condition of Well
Being
=
Result
15
15
Connecticut Early Childhood Result
Statements
 Ready By Five, Fine By Nine
 Goal 1: All Children Healthy and Ready For School
Success at Entry To Kindergarten
 Goal 2: All Children Healthy and Achieving School
Success By Age 9
 All Infants and Very Young Children Achieve Optimal
Health and Development In Safe, Nurturing Families and
Environments
16
16
Result Statements Developed by
CT Non-Profits and Funders
 Connecticut children of all races and income levels are
ready for school by age five and are successful learners
by age nine
 Families and individuals live in stable, affordable housing
 All Connecticut residents are healthy throughout their
lives
 All children and youth in Connecticut become resilient,
empowered, productive and engaged citizens
 All Capital Region adults are self-sufficient
 All Capital Region residents enjoy a healthy economy
17
17
Community Outcomes for
Christchurch, NZ
 A Safe City
 A City of Inclusive and Diverse Communities
 A City of People who Value and Protect the Natural
Environment
 A Well-Governed City
 A Prosperous City
 A Healthy City
 A City for Recreation, Fun and Creativity
 A City of Lifelong Learning
 An Attractive and Well-Designed City
18
18
Every Child Matters – Children Act
Outcomes for Children and Young People
Being Healthy
Enjoying good physical and mental health
and living a healthy lifestyle
Staying Safe
Being protected from harm and neglect and
growing up able to look after themselves
Enjoying and
Achieving
Getting the most out of life and developing
broad skills for adulthood
Making A Positive
Contribution
Economic
Well-Being
To the community and to society and not
engaging in anti-social or offending
behavior
Overcoming socio-economic disadvantages
to achieve their full potential in life
19
19
Georgia Policy Council
for Children and Families
Results
 Healthy Children
 Children Ready for School
 Children Succeeding in School
 Strong Families
 Self Sufficient Families
20
20
Criteria for Choosing Indicators as
Primary vs. Secondary Measures
Communication
Power
Does the indicator communicate to a broad
range of audiences?
Proxy Power
Does the indicator say something of central
importance about the result?
Does the indicator bring along the data
HERD?
Data Power
Quality data available on a timely basis.
21
21
Communication Power
 Does the indicator communicate to a broad range of
audiences?
– Public square test: If you briefly describe the indicator and
give your fellow citizens the data (e.g., less than 60% of
third graders are reading at grade level), they would
understand the indicator and its importance to the result
– Elevator test: If you are in the elevator with a legislator and
have that short ride to make your point, the indicator will
quickly highlight an important issue associated with the
result
 Communication power means that the data must be clear
to diverse audiences
22
22
Proxy Power
 Does the indicator say something of central importance
about the result or is it peripheral?
 Does the indicator capture an important aspect of the
plain English statement of well-being? What pieces of
data really get at the heart of the matter?
 Indicators run in herds. If one indicator is going in the
right direction, often others are as well. You do not need
a dozen indicators telling you the same thing.
23
23
Data Power
 Do we have quality data on a timely basis? We need data
that are reliable and consistent. And we need timely data
so we can see progress - or the lack thereof - on a
regular basis.
 Problems with data availability, quality or timeliness can
be addressed as part of the Data Development Agenda.
24
24
Choosing Indicators Worksheet
Safe Community
Outcome or Result_______________________
Candidate Indicators
Measure 1
Measure 2
Measure 3
Measure 4
Measure 5
Measure 6
Measure 7
Measure 8
Communication
Power
H M L
Proxy
Power
Data
Power
H M L
H M L
H
H
H
H
H
L
Data
Development
Agenda
25
25
Three Part Indicator List for
Each Result
Part 1: Primary Indicators
● 2 or 3 or 4 “Headline” Indicators
● What this result “means” to the community
● Meets the Public Square Test
Part 2: Secondary Indicators
● Everything else that’s any good (Nothing is wasted.)
● Used later in the story behind the baseline
Part 3: Data Development Agenda
● New data
● Data in need of repair (quality, timeliness etc.)
26
26
The Matter of Baselines
H
M
OK?
L
Point to Point
History
Turning the Curve
Forecast
Baselines have two parts: history and forecast
27
27
Caution
 Data are not the same as reality. Think of a leaking roof. No
water in the bucket under the leak does not prove that the roof
is fixed. In fact, the ceiling may be about to come down.
 Data are a proxy for the condition of well-being we want. Our
goal is not an empty bucket; it is a fixed roof and a dry house.
 The better the proxy, the closer to reality we get. Having more
than one indicator increases the chance that we have actually
captured reality.
 The rating of headline indicators is not the last word. You must
look at the identified indicators and see if, as a whole, they
encompass the important dimensions of the result statement.
Sometimes you need to select a less powerful headline
indicator in order to be able to talk about an aspect of the
result that is important to your community.
28
28
29
29
REPORT CARDS
Georgia
Santa Cruz, CA
Dayton, OH
Lehigh Valley, PA
30
30
REPORT CARDS
Country
New Zealand
Neighborhood
City
Portsmouth, UK
Kruidenbuurt
Tilburg,
Netherlands
31
31
32
32
33
33
34
34
The Power of RBA Thinking:
Turning the Curve
35
35
Turn-the-Curve Thinking™: Talk to Action
Result or Program:
How are
we doing?
Data
Baseline
Why?
Story behind the baseline
Help?
Partners (with a role to play in turning the curve)
Options?
What Works
Propose
to do?
Strategy (w/ Budget)
Results Leadership Group
36
36
Turn-the-Curve Thinking™: Talk to Action
Result or Program:____________
How are
we doing?
Data Baseline
37
37
Turn-the-Curve Thinking™: Talk to Action
Result or Program:_____________
Data Baseline
Why?
Story behind the baseline
Research Agenda
38
38
Force Field Analysis
Factors Restricting?
Factors Contributing?
39
39
The Story Behind the Baseline
 Root Causes (ask “Why?” five
times)
 Positive and negative
 Prioritize – which are the most
important to address to “turn the
curve” of the baseline?
 Research agenda?
40
40
Turn-the-Curve Thinking™: Talk to Action
Result or Program:_____________
Data
Baseline
Story behind the baseline
Help?
Research Agenda
Partners (with a role to play in turning the curve)
41
41
Partners
 Who are partners who may have
a role to play in turning the
curve?
 Does the story behind the curve
suggest any new partners?
42
42
Turn-the-Curve Thinking™: Talk to Action
Result or Program:______________
Data Baseline
Story behind the baseline
Research Agenda
Partners (with a role to play in turning the curve)
Options?
What Works
Research Agenda
43
43
What Works
 Options for actions to “turn the
curve”?
 Research-based?
 Low-cost/no-cost?
 Off-the-wall ideas?
 Research agenda?
44
44
Turn-the-Curve Thinking™: Talk to Action
Result or Program: _______________
Data Baseline
Research Agenda
Story behind the baseline
Partners (with a role to play in turning the curve)
What Works
Research Agenda
Criteria: Leverage; Feasible; Specific; Values
Propose
to do?
Strategy
45
45
Action Plan
 Leverage: will turn the curve of the
baseline?
 Feasible (a.k.a. “reach”)?
 Specific: who, what, when, where, how?
 Consistent with values?
46
46
I N N O VAT I O N
Turn the Curve Exercise
Turn the Curve Exercise:
Population Well-Being
5 min:
Starting Points
- timekeeper and reporter
- two hats (yours plus partner’s)
5 min:
Baseline
- forecast: Where is the trend line going?
- turn the curve: Is forecast OK or not OK?
30 min: Story behind the baseline
- causes/forces at work
- information & research agenda part 1 - causes
25 min: What works? (What would it take?)
- what could work to do better
- each partner’s contribution
- no-cost / low-cost ideas
- information & research agenda part 2 – what works
Two
pointers
to action
10 min: Report: Convert notes to one page
48
48
ONE PAGE Turn the Curve Report
Result: _______________
Indicator
Baseline
Indicator
(Lay Definition)
Story behind the Baseline
-----------------------------------------------------
(List as many as needed)
Partners
-----------------------------------------------------
(List as many as needed)
Three Best Ideas – What Works
1. --------------------------2. --------------------------3. ---------No-cost / low-cost
4. ---------Off the Wall
Sharp
Edges
49
49
For Services, Agencies and Service Systems
I N N O VAT I O N
Performance Accountability
Population Versus Performance
Accountability
Population Accountability
 About the well being of entire populations, like all young
children in Waterbury
 Not about any program or service system
Performance Accountability
 About the well being of client populations only
 Applies to programs, agencies, or service systems
51
51
Population Versus Performance
Accountability
 At the population level, we first ask what quality of life
we want and then what strategies (collections of activities
or services) we are prepared to pay for to achieve our
quality of life result
 At the performance level, once we have decided to buy a
particular program or service, we want to know how well
it is being implemented and whether anyone is better off
52
52
END
Children Enter School
Ready to Learn
1. Doing the
right things?
Indicator
Comprehensive Strategy/Partners
MEANS
Program A
Interagency
System
Agency/Program
Performance Measures
2. Doing those things right?
Interagency Service System
Performance Measures
Results Leadership Group
53
53
Output
Effect
Input
Effort
Program Performance Measures
Quantity
Quality
How much
service did
we deliver?
How well
did we
deliver it?
How much
change/effect
did we produce?
What quality of
change/effect
did we produce?
54
54
Effect
Effort
Program Performance Measures
Quantity
Quality
How much
did we do?
How well
did we do it?
Is anyone
better off?
#
%
55
55
Effort
Education
Quantity
Quality
How much did we do?
How well did we do it?
Number of
students
Student-teacher
ratio
Effect
Is anyone better off?
Number of
high school
graduates
Percent of
high school
graduates
56
56
Effort
Health Practice
Quantity
Quality
How much did we do?
How well did we do it?
Number of
patients
treated
Percent of
patients treated
in less than
1 hour
Effect
Is anyone better off?
Incidence of
preventable
disease
Rate of
preventable
disease
(in the practice)
(in the practice)
57
57
Drug/Alcohol Treatment Program
How much did we do?
Number of
persons
treated
How well did we do it?
Unit
cost of
treatment
Is anyone better off?
Number of clients Percent of clients
off alcohol/drugs off alcohol/drugs
-at exit
-12 months post-exit
58
58
What Quadrant?
 % participants who got jobs
 staff turnover rate
LR
UR
 # participants who got jobs
LL
 % of children reading at grade level
 cost per unit of service
LR
UR
 # applications processed
 % patients who fully recover
UL
LR
59
59
What Quadrant?
 % of customers satisfied with outcome of service (from
survey) LR
 % of customers satisfied with service quality (from
survey) UR
 % of applications processed within 2 working days
 # on waiting list
UR
UL, UR
 % of teachers with certification
UR, LR
60
60
All Data Have Two Incarnations
Lay
Technical
Definition
Definition
HS Graduation Rate
% enrolled June 1 who graduate June 15
% enrolled Sept 30 who graduate June 15
% enrolled 9th grade who graduate in 12th grade
61
61
Separating the Wheat from the Chaff
Types of Measures Found in Each Quadrant
How much did we do?
How well did we do it?
# Clients/customers
served
% Common measures
# Activities (by type
of activity)
% Activity-specific
measures
e.g. client staff ratio, workload ratio, staff
turnover rate, staff morale, % staff fully
trained, % clients seen in their own language,
worker safety, unit cost
e.g. % timely, % clients completing activity,
% correct and complete, % meeting standard
Is anyone better off?
% Skills / Knowledge
#
(e.g. parenting skills)
#
#
#
Point in Time
vs. Point to Point
Improvement
% Attitude / Opinion
(e.g. toward drugs)
% Behavior
(e.g.school attendance)
% Circumstance
(e.g. working, in stable housing)
62
62
Selecting Headline Performance
Measures
How much did we do?
# Clients/customers served
# Activities
(by type of activity)
\
63
63
Selecting Headline Performance
Measures
How well did we do it?
% Common measures
e.g. workload ratio, staff turnover rate, %
staff fully trained, unit cost
% Activity-specific measure
e.g. % timely intakes, % accreditation
standards met
64
64
Selecting Headline Performance
Measures
How much did we do?
How well did we do it?
Is anyone better off?
#/% Skills / Knowledge
(e.g. cognitive, social, physical)
#/% Attitude
(e.g. toward language, parenting)
#/% Behavior
(e.g. reading to child at home)
#/% Circumstances
(e.g. child care, transportation)
65
65
Effort
Choosing Headline Measures and
the Data Development Agenda
Quantity
Quality
How much did we do?
How well did we do it?
# Measure 1 ----------------------------
% Measure 8 ----------------------------
# Measure 2 ----------------------------
% Measure 9 -----------------------------
# Measure 3 ----------------------------
% Measure 10 ---------------------------
# Measure 4 ----------------------------
% Measure 11 ---------------------------
# Measure 5 ----------------------------
% Measure 12 ---------------------------
# Measure 6 ----------------------------
% Measure 13 ---------------------------
# Measure 7 ----------------------------
% Measure 14 ---------------------------
#3 DDA
#2 Headline
Effect
Is anyone better off?
# Measure 15 ----------------------------
% Measure 15 ----------------------------
# Measure 16 ----------------------------
% Measure 16 ----------------------------
# Measure 17 ----------------------------
% Measure 17 ----------------------------
# Measure 18 ----------------------------
% Measure 18 ----------------------------
# Measure 19 ----------------------------
% Measure 19 ----------------------------
# Measure 20 ----------------------------
% Measure 20 ----------------------------
# Measure 21 ----------------------------
% Measure 21 ----------------------------
#2 DDA
#3 Headline
#1 Headline
#1 DDA
66
66
Effort
Not All Performance Measures Are
Created Equal
Quantity
Quality
How much did we do?
How well did we do it?
Least
Important
Effect
Is anyone better off?
Most
Important
67
67
The Matter of Control
Quality
Effort
Quantity
How much did we
do?
How well did we do
it?
Most
Control
Effect
Is anyone better off?
Least
Control
PARTNERSHIPS
68
68
LR
UR
69
69
70
70
Report Cards for the Connecticut
General Assembly
71
71
Report Cards for the Connecticut
General Assembly
72
72
I N N O VAT I O N
How Population and
Performance Accountability
FIT TOGETHER
THE LINKAGE between POPULATION and PERFORMANCE
POPULATION ACCOUNTABILITY
Healthy Births
POPULATION
RESULTS
Rate of low birth-weight babies
Stable Families
Rate of child abuse and neglect
Children Succeeding in School
Percent graduating from high school on time
PERFORMANCE ACCOUNTABILITY
Child Welfare Program
# Foster
Children
Served
% with
Multiple
Placements
# Repeat
Abuse/Neglect
% Repeat
Abuse/Neglect
Contribution
relationship
Alignment
of measures
Appropriate
responsibility
CUSTOMER
Outcomes
74
74
THE LINKAGE between POPULATION and PERFORMANCE
POPULATION ACCOUNTABILITY
Healthy Births
Rate of low birth-weight babies
POPULATION
RESULTS
Children Ready for School
Percent fully ready per K-entry assessment
Self-sufficient Families
Percent of parents earning a living wage
PERFORMANCE ACCOUNTABILITY
Job Training Program
# persons
receiving
training
Unit cost
per person
trained
# who get
living wage jobs
% who get
living wage jobs
CUSTOMER
Outcomes
Contribution
relationship
Alignment
of measures
Appropriate
responsibility
75
75
76
76
I N N O VAT I O N
Performance Measurement Exercise
Exercise to Develop Performance
Measures
 45 minutes total. Select timekeeper and reporter
 5 minutes. Identify one community-based program that is
a critical partner in your communities
 10 minutes. Identify key customer groups and major
clusters of activities
 25 minutes. Generate as many How Well and Better Off
measures as you can
 5 minutes. Select 5 headline measures, 2-3 How Well
and 2-3 Better Off
78
78
I N N O VAT I O N
Data Display Principles
Important Data Display Principles
 Show the data
 Engage the viewer in thinking about substance rather
than about the method, graphic design, or technology
used to produce the graphic display
 Avoid distorting what the data have to say
 Make large data sets coherent
– Reveal data at several levels of detail, from a broad
overview to a fine structure (drill-down approaches)
 Encourage the eyes to compare different pieces of data
*Adapted from Tufte, 1982
80
80
Some Key Design Principles
 Organization of reports should be driven by the audience
and use of the report
 Level of detail should also vary based on audience and
intended use of report. This includes:
– How many and what kind of indicators and
performance measures to include
– What kinds of comparisons to include
– How much detail is directly accessible in the report
 Don’t bury people in data. For community reports, the
concept of headline indicators and performance
measures is crucial
81
81
Report the Trend, Not a Data Point
NO
YES
Number of Clients Served
4000
3500
3000
2500
2000
1500
1000
500
0
PY 2009 Actual
Number of Clients Served
4000
3500
3000
2500
2000
1500
1000
500
0
2006
2007
2008
2009
82
82
Note Any Changes in Data
Reporting
NO
60%
YES
% Achieving Goal on 3rd Grade
Reading CMT
% Achieving Goal on 3rd Grade
Reading CMT
50%
100%
40%
80%
30%
60%
20%
40%
10%
20%
0%
2004
2005
2006
2007
2008
2009
0%
2004 2005
2006 2007 2008 2009
Note: Testing Format Revised for 2006
83
83
Use 0-100 Scale Whenever
Possible
NO
YES
% Ready For K
% Ready For K
100%
62%
80%
60%
60%
58%
56%
40%
54%
20%
0%
52%
2006
2007
2008
2009
% Ready For K
2006
56%
2007
58%
2008
61%
2009
60%
84
84
Use Chart/Table Combination
Percent of Kids at or Above Goal on 3rd Grade Reading CMT
100.00%
90.00%
80.00%
70.00%
60.00%
50.00%
40.00%
30.00%
20.00%
10.00%
0.00%
all kids
# of kids at or above goal
% at or above goal
2006
41400
22554
54.5%
2007
41652
21784
52.3%
2008
41133
21430
52.1%
2009
39639
21643
54.6%
2010
39005
22272
57.1%
85
85
Always Tell the Story with
the Data
NO
YES
Entry and One Year Parenting Rigidity
Scores
Entry and One Year Parenting
Rigidity Scores
40
40
30
30
20
20
10
10
0
0
2004
2005
2006
2007
2004
2005
2006
2007
Story behind the Baseline: This measure shows program entry
and one year follow-up scores on a parental rigidity assessment.
Parental rigidity is predictive of a number of negative parenting
behaviors , including abuse. The trend shows consistently
lower (better) scores after participation in the program.
86
86
Display Important Disaggregations
If you know this…
Don’t just show this…
% Achieving Goal on 3rd Grade Math CMT
% Achieving Goal on 3rd Grade Math
CMT
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
All Students
Free and
Reduced Lunch
Students
100.0%
80.0%
60.0%
40.0%
2007
59.4%
2008
60.2%
2009
63.0%
2010
62.6%
34.4%
34.6%
37.9%
38.9%
20.0%
0.0%
All Students
2007
59.4%
2008
60.2%
2009
63.0%
2010
62.6%
87
87
Other Critical Comparisons
 To a standard or expectation
 To a previous point in time
 To another place or group of places
 To similar, but not exactly the same, programs
88
88
How RBA Is Being Used in
Connecticut
 Connecticut legislature
 State agencies
 Communities
 Non-profits
 Funders
 Connecticut RBA Practitioners Network
89
89
I N N O VAT I O N
IN CLOSING
“If you do what you always did…
you will get
what you always got.”
Kenneth W. Jenkins
President, Yonkers NY NAACP
91
91
Never Be Afraid To Try Something New.
Remember…
A lone amateur
built the Ark.
A large group of professionals
built the Titanic.
— Dave Barry
92
92
Thank You
Barry Goff
[email protected]
(860) 659-8743
Bennett Pudlin
[email protected]
(860) 324-3555
Ron Schack
[email protected]
(860) 478-7847
www.charteroakgroup.com
93
93