Transcript Health Program Planning and Evaluation: Overview and Feasibility CHSC 433

Health Program
Planning and Evaluation:
Overview and Feasibility
CHSC 433
Module 1/Chapter 1
UIC School of Public Health
L. Michele Issel, PhD, RN
Learning Objectives
What you ought to be able to do by the end of this module:
1.
2.
3.
4.
5.
Articulate key factors that constitute the
context in which health programs are developed
and evaluated.
Describe the historical background of program
evaluation.
Appreciate evaluation as an aide to program
decision making.
Describe the major phases of the Planning and
Evaluation Cycle.
Explain the relationship of the Pyramid to the
Planning and Evaluation Cycle.
Outline of this ppt
Terminology
 Evaluation definitions
 History and types of evaluation
 Why evaluate
 When not to evaluate
 Standards and principles of evaluation
 Public health pyramid

Terminology
(Write your definitions, some in the text)
Aggregate
Ecological model
Enabling services
Evaluation
Planning
Population
Population services
Program
Project
Research
Service
Evaluation is….

Systematic application of research procedures to
assess the conceptualization, design,
implementation and utility of intervention
programs

(Rossi & Freeman)
Assessing the quality and effect of programs
Evaluation is….

(continued)
Using research methods, measure the
effectiveness against a set of standards
for the purpose of decision making
Weiss)
(Carol
History of Evaluation



Began in field of education
Strengthened during the 1960’s emphasis
on social programs and determining their
effect on society
Further strengthened during the 1990’s
emphasis on outcomes measurement and
quality improvement
Generations of Evaluations




1900’s: Technical, testing
to1960: Descriptive, based on
program objectives
1960’s: Judgement, merit and value as
focus
1980’ and beyond: Negotiation,
responsive, pluralistic
Pluralist Perspectives Led to
Diverse Types of Evaluation

Utilization-focused

Goal free

Theory driven

Participatory

Outcome-focused

Value based
Why Evaluate:
Explicit Reasons

Monitor program implementation

Determine level of need

Make resource allocation decisions

Determine effect of local conditions

Improve quality of current efforts

Generate new knowledge
Why Evaluate:
Implicit Reasons

Form of social and organizational control

As a symbolic action

Delaying and skewing decision

Facilitate program termination

Incentive for planning
Typically,
Evaluations Question:

The need for the program

The program conceptualization

The program operations

The outcomes from the program

The cost and efficiency of the
program
Questions Lead to
Foci of Evaluations

Activities of the program

Inputs and resources used

Processes and interactions involved in doing or
providing the program

Outputs as services

Impact as change in program participants


Outcomes as change in target population toward
goal achievement
Efficiency as results per effort
Steps in Conducting
Evaluations
1. Assess feasibility of doing evaluation
(Evaluatability Assessment)
2. Develop evaluation questions
3. Select evaluation design and methods
4. Collect evaluation data
5. Analyze evaluation data
6. Disseminate findings
Deciding to Evaluate
Depends on:

Presence of Contraindications

Context of evaluation

Feasibility
Contraindications to
Evaluating



The program is very popular
The program is meaningless but has
public support
Changes in the program would be
expensive or dangerous

Too costly to do the evaluation

The program has no clear orientation
Contraindications
(continued)

There are no questions about the program

People can not agree on program objectives

Impossible to evaluate because of technical
problems

Program sponsor or staff uncooperative or
resistant
Context of Evaluations

Environments
• Social norms and values
• Attitudes toward evaluations and evaluators



Policies and Politics
Program Quality
Science
• About evaluation
• About health problem, population, intervention
Standards and
Principles
The professional evaluation association
has established both standards for
high quality evaluations, and principles
for conducting evaluations.
These are values that guide the
practice of evaluation.
4 Standards for Evaluation


The utility standards are intended to
ensure that an evaluation will serve
the information needs of intended
users.
The feasibility standards are
intended to ensure that an evaluation
will be realistic, prudent, diplomatic,
and frugal
Standards…

The propriety standards are intended
to ensure that an evaluation will be
conducted legally, ethically, and with
due regard for the welfare of those
involved in the evaluation, as well as
those affected by its results.
Standards…

The accuracy standards are intended
to ensure that an evaluation will
reveal and convey technically
adequate information about the
features that determine worth or
merit of the program being evaluated.
Principles for Evaluation
Guiding Principles for Evaluators:
A Report from the AEA Task
Force on Guiding Principles for
Evaluators
http://www.eval.org/EvaluationDocuments/aeaprin6.
html
Resulting Principles for
Evaluation
Given the diversity of interests and
employment setting represented on the
Task Force, it is noteworthy that Task
Force members reached substantial
agreement about the following five
principles. The order of these principles
does not imply priority among them;
priority will vary by situation and evaluator
role.
Principle: Systematic Inquiry
Evaluators conduct systematic, databased inquiries about whatever is
being evaluated.
Principle: Competence
Evaluators provide competent
performance to stakeholders.
Principle: Integrity/Honesty
Evaluators ensure the honesty and
integrity of the entire evaluation
process.
Principle: Respect for People
Evaluators respect the security, dignity
and self-worth of the respondents,
program participants, clients, and
other stakeholders with whom they
interact.
Principle: Responsibilities for
General and Public Welfare
Evaluators articulate and take into
account the diversity of interests and
values that may be related to the
general and public welfare.
Good Enough Evaluation
Sufficient and acceptable evaluation
But the
Minimum necessary to provide the
answers to only the most important
questions
Using the least amount of effort.
NOT the Perfect, Ideal….
The Public Health Pyramid
Direct
Health Care
Services
____________________
Enabling Services
___________________________
Population-Based Services
___________________________________
Infrastructure Services
Public Health Pyramid
Useful for:
Keeping a current project within the big
picture.
Remembering to address aggregates and
populations, not just individuals.
Framing analyses of contextual elements
that may be supporting or hindering
program.