Transcript Evaluation Plans & Performance Indicators

Evaluation Plans &
Performance Indicators
Office of Research, Evaluation, and Policy Studies
Marcella M. Reca Zipp
November 30, 2010






Necessity of evaluation plan
Types of evaluation plans
Components of an evaluation plan
Performance indicators
Reporting requirements
Sample evaluation plans
Purpose of Evaluation Plan
 Cohesive approach to conducting
evaluation and using results
 Explains what, when, how, why, who
 Documents the evaluation process
 Ensures implementation fidelity
 Promotes a participatory approach
Source: University of Toronto
Source: University of Toronto
3 Levels of Evaluation
 Project-Level Evaluation
 Context
 Implementation
 Outcome
 Cluster Evaluation
 Program and Policymaking Evaluation
Project-Level Evaluation
 Context
 Needs, assets, and resources of community
 Political atmosphere, social and environmental
strengths/weaknesses
 Implementation
 Critical components/activities of project
 Aspects that are strengths and weaknesses
 How do the components connect to goals and
outcomes
 Outcome
 Critical outcomes you are trying to achieve
 Impact on clients, community, etc.
 Unexpected impact
Cluster Evaluation
 Determines how well the
collection of projects fulfills the
objective of systemic change.
 Not a substitute for project-level
evaluation.
 Looks across a group of projects
to identify common themes.
 Information reported in
aggregate form to granting
organization.
EVALUATOR
PROJECT A
PROJECT B
PROJECT C
Program and Policy Making
Evaluation
 Macro form of evaluation.
 Utilizes information gathered from
both project-level and cluster
evaluation to make effective decisions
about program funding and support.
 Supports communities in creating
policy change at local, state, and
federal levels.
Elements of an Evaluation







Introduction
Project Objectives
Logic Model
Partnership Roles and Responsibilities
Intervention Programming/Research
Methodology/Data Collection
Instrumentation Measures
Introduction
 Provides background information for
the evaluation, identifies purpose and
goal, sets the course on the
evaluation road map.
 Evaluation purpose and goals
 What does the evaluation strive to achieve?
 Evaluation team
 Who is the evaluation coordinator?
 Who are the members of the evaluation
team?
Logic Model
 Graphic depiction of the program
description.
 Links needs, objectives, activities,
measurements.
 Provides scope of program.
 Ensures systematic decisions are
made about what will be measured.
 Identifies and organizes indicators.
Objectives
 PIMO method
 The number of objectives will be
determined by purpose (i.e.,
intervention, treatment, prevention).
 Interrelated to your projected
activities (i.e., education, service,
research).
 Feasible to collect, will provide
accurate results.
Partnership Roles and
Responsibilities
 Project partners are expected to provide
certain, unique expertise to your project
activities, either in direct service function or
as advisory units.
 Identify each partner in their role and
responsibility in terms of involvement for
your project.
 Partner-cited activities must be evaluated
formatively and summatively.
Intervention Programming
 Identify one or more intervention strategies
used to support project activities and
anticipated outcomes.
 Cite if the program is on the federal
evidence-based initiative list: EBI
 For market-available programs that require
training and certification of direct service
providers, provide a timetable for acquiring
training before intervention can be used.
Performance Indicators
 Visible, measurable signs of program
performance.
 Relevant, understandable and useful.
 Reflect program objectives, logic
model and evaluation questions.
 Define success
 Reasonable expectations of program
performance.
Source: University of Toronto
Performance Indicators cont.
 Other terms – industry jargon





Key Performance Indicator (KPI)
Performance metric
Performance standard
Balanced Scorecard
Quality indicators
 All are different words for the same
thing: measure performance.
Data Collection
 What methods will be used?
 How often will data be
collected?
 Who will collect the data?
 Validity and reliability of data
sources




Baseline data
Outcomes-based triangulation
Quality assurance
Design (experimental, quasiexperimental, etc.)
Instrumentation Measures
 Tools for data collection
 Only collect the information you need
 Easy to administer and use
 Pilot test tools before use in the
evaluation
 Human Subjects Considerations
 IRB, school board approval
 Data management and storage
 Confidentiality and data quality
Tips & Helpful Hints
 Be realistic
 In your assessment of resources
 In your timeline
 Seek help
 Use templates, tables, or guides that
may be provided in the RFP or model
after past funded proposals.
Reporting and Dissemination
 Dissemination
 How will you disseminate findings?
 Who is responsible?
 How, where, when will findings be used?
 Reporting
 Formative reports – quarterly, biannually
 Summative reports – final report/end of
project
 Project deliverables
Sample RFP Evaluation Plan
 Two examples of an evaluation plan
within an RFP
 General, limited specifications
 Complex, very detailed
Evaluation Resources
 CDC: www.cdc.gov/eval
 University of Toronto: www.utoronto.ca/shp/hcu
 W.K. Kellogg Foundation:
www.wkkf.org/Publications/evalhdbk
 Connell, J.P., Kubisch, A.C., Schorr, L.B., Weiss, C.H.
(1995). New Approaches to Evaluating Community
Initiatives, New York, NY: Aspen Institute.
 Shadish, W.R., Cook, T.D., Leviton, L.C. (1991).
Foundations of Program Evaluation. Newbury Park, CA:
Sage Publications.
 Taylor-Powell, E., Steele, S., Douglas, M. (1996).
Planning a Program Evaluation. Madison, WI: University
of Wisconsin Cooperative Extension.