Transcript Document

Overview of Performance Measurement
Overview of
Performance Measurement
Copyright © 2012 by JBS International, Inc.
Developed by JBS International for the Corporation for National & Community Service
Overview of Performance Measurement
Learning Objectives
By the end of the module, you will be able to:
• Describe what performance measurement is,
and how it can be useful in program
management
• Describe how CNCS approaches performance
measurement
• Describe evidence-based interventions
• Identify the approved sources of evidence
2
Overview of Performance Measurement
Systematic Process for Measuring
Outputs and Outcomes
Outputs
• Amount of service
provided
(people served,
products created, or
programs
developed)
3
Overview of Performance Measurement
Systematic Process for Measuring
Outputs and Outcomes
Outcomes
• Reflect the changes or
benefits that occur
• Can reflect changes in
individuals, organizations,
communities, or the
environment
• Address changes in
attitudes/beliefs,
knowledge/skills,
behavior, or conditions
4
Overview of Performance Measurement
Outcomes
Types of Outcomes
Attitude/Belief
Knowledge/Skill
Behavior
Condition
Thought, feeling
Understanding,
know-how
Action
Situation,
circumstance
5
Overview of Performance Measurement
Outcome Examples–
Education
Attitude/Belief
Knowledge/Skill
Behavior
Condition
Increased interest
in school
Improved math
ability
Increased school
attendance
Successful
completion of
High School
6
Overview of Performance Measurement
Why Measure
Performance?
Accountability to funders
and stakeholders
• Tell your story, justify
funding
Recognizing progress;
reflects change
• Reliable information
collected in a
systematic way
7
Overview of Performance Measurement
CNCS National
Performance Measures
Reflect the CNCS Strategic
Plan and its programming
priorities:
• Disaster Services
• Economic Opportunity
• Education
• Environmental Stewardship
• Healthy Futures
• Veterans & Military Families
• Capacity Building
8
Overview of Performance Measurement
CNCS National
Performance Measures
Where do I go to learn about
National Performance
Measures?
https://www.nationalservicere
sources.org/npm/ac
9
Overview of Performance Measurement
Benefits of National
Performance Measures
• CNCS can focus efforts on national priorities where
service has the greatest impact
• High quality data for a strong narrative about National
Service
• Guidance for using national performance measures:
• Definitions and descriptions
• Data collection and aggregation information
10
Overview of Performance Measurement
National Performance
Measure Example
Intervention: Regular support for social-emotional needs of
homebound adults, older adults, and individuals with disabilities;
includes light assistance with housekeeping and errands.
Output
• Number of homebound or older adults and
individuals with disabilities receiving food,
transportation, or other services that allow
them to live independently. (H8)
Outcome
• Number of homebound or older adults and
individuals with disabilities who reported
having increased social ties/perceived
social support. (H9)
11
Overview of Performance Measurement
Evidence: What It Is and
Where to Find It
How evidence helps you select an
effective intervention
Copyright © 2012 by JBS International, Inc.
Developed by JBS International for the Corporation for National & Community Service
Overview of Performance Measurement
Theory of Change Elements
Community
Need
Specific
Intervention
Intended
Outcome
Statistics
Evidence
documenting the
need
•Guides choice of intervention
•Supports cause-effect relationship
Overview of Performance Measurement
Evidence-Based Intervention
Service activities that are
supported by research to
demonstrate a high
likelihood that these
activities will address the
need, and result in intended
changes.
Evidence demonstrates that
the proposed intervention
(design, dosage) is likely to
lead to the outcome.
Overview of Performance Measurement
Everyday Life Example of a Theory
of Change
Problem/Need
Strep Throat
Intervention
Antibiotics
Evidence
 Penicillin
 Amoxicillin
 Tetracycline
Outcome
Healthy
Overview of Performance Measurement
Evidence
EVIDENCE: Information that
supports your choice of a
specific intervention and its
ability to produce your intended
outcome.
It is NOT enough to just say “we
believe” our intervention is
“likely” to be successful. You
need to justify your choice with
evidence.
Overview of Performance Measurement
Lessons from the Evidence
Evidence provides a reality check for theory of
change elements
• For new programs:
•
•
What is the recommended design (specific program
activities) and dosage (frequency, intensity, and
duration) to achieve an intended outcome?
For existing programs:
•
•
•
Is there sufficient evidence for the intervention to
continue its use?
Based on the evidence, are there modifications to
the intervention that should be considered to have
greater impact?
Do you need to choose a new intervention?
Overview of Performance Measurement
Evidence Basis for Interventions
Possible sources of evidence
include:
• Your past performance
measurement outcome data
• Results from an impact
evaluation of your program
• Research that documents the
outcomes of similar programs
• Impact evaluations that
document outcomes of
similar programs
Overview of Performance Measurement
Evidence Source: Your
Performance Measurement Data
Past performance measurement
outcome data:
• What do your past performance
measurement results tell you?
• Can you show positive outcomes
over time?
Overview of Performance Measurement
Assessing Evidence
Criteria include:
• Relevant: Cites comparable intervention with
•
•
•
similar beneficiaries and results
Compelling: Persuasive, shows clear likelihood
of success
Up-to-date: Recently published or most recent
available
Objective: Source with no stake in outcome and
published in a peer reviewed journal or by credible
organization
Overview of Performance Measurement
Evidence Source: Research
Studies and Evaluations
Sources of Strong Evidence
• University or research
organizations (national or
local)
• Known professionals/
experts in the field
• Similar sounding programs/
descriptions
• Articles that review multiple
studies (meta studies)
Overview of Performance Measurement
Web Search Tool
Google Scholar is one web search tool that helps narrow
the search for your keywords to professional journals
http://scholar.google.com
Tip:
Use quotation marks around key terms - may help narrow
the number of search results
Overview of Performance Measurement
Search Result Questions
Is the article:
• Relevant?
• Compelling?
• Up-to-date?
• Objective?
Overview of Performance Measurement
Evidence Continuum
Causality
Low
Preliminary
• Doesn’t show
causality
• Outcome results
from performance
measurement
High
Moderate
• Show causality,
compares
intervention
recipients to nonrecipients
• Results from
studies/impact
evaluations
• Comparison groups:
Quasi-experimental
Design
Strong
• Show causality,
compares
intervention
recipients to nonrecipients
• Results from
studies/impact
evaluations
• Randomly-assigned
control groups:
Experimental Design
Overview of Performance Measurement
Evidence Continuum
Shape Up: afterschool obesity prevention program
Preliminary
Moderate
Strong
Performance
measurement shows
that 75% of girls age 1416 participating in the
Shape Up program
made more healthy food
choices
A 2005 impact
evaluation by internal
evaluators (using a
quasi-experimental
design of girls
participating in a similar
program to Shape Up)
found that after 12
weeks, the girls in the
program made 50%
more healthy food
choices than the
comparison group.
A 2010 impact
evaluation of the GEMS
program by University of
MN using experimental
design/ random
assignment found after
12 weeks, the girls in the
experimental group
made 50% more healthy
food choices than
control group girls
Overview of Performance Measurement
Key Points Summary
• The term “evidence-based intervention” describes
service activities that are supported by research to
indicate a high likelihood that the intervention will
address the need and result in intended outcomes.
• Evidence is information or documentation that a
specific intervention (design and dosage) will
produce your intended outcome.
Overview of Performance Measurement
Key Points Summary
• The four sources of evidence that CNCS has identified
include:
•
•
•
•
•
•
Past performance measurement data
Results from a program impact evaluation
Research studies that document the outcomes of similar
programs
Evaluations that document outcomes of similar programs
When assessing evidence, check to be sure it is
relevant, compelling, up-to-date, and objective.
The strongest evidence clearly proves that the
intervention caused the positive change by using an
experimental design.
Overview of Performance Measurement
Key Points Summary
Activity: Evidence Case Study #1 –
Science Education Program
•Review the Problem, Intervention & Outcome
•Respond to Questions
•Report Out