Transcript Evaluation Essentials for Health Promotion in Higher

Evaluation Essentials for
Health Promotion in
Higher Education: A Primer
Gina Baral Abrams, MPH, LSW, CHES
Amy Melichar, MEd, CHES
AMERICAN COLLEGE HEALTH ASSOCIATION
ANNUAL MEETING
SAN FRANCISCO, CA
MAY 28, 2009
Learning Objectives
 Describe the uses and purposes of program evaluation




in health promotion.
List conceptual approaches to evaluation.
List the key qualities of measurable evaluation
questions.
List qualitative and quantitative data collection
methods.
Describe key qualities of an effective evaluation report.
Evaluation Defined
“Examining the worth of a program, usually
measuring it against a set of predetermined
objectives or a standard of acceptability.”
Green & Kreuter, 1999.
Uses and Purposes of Program Evaluation
 Determine the rate and level of






attainment of program
objectives.
Assess the strengths and
weaknesses of a program.
Help make decisions.
Monitor standards of
performance.
Establish quality assurance and
control mechanisms.
Determine the generalizability of
an overall program or program
elements to other populations.
Meet the demand for public or
fiscal accountability.
 Improve professional staff skill




in program planning,
implementation, and
evaluation activities.
Promote positive public
relations and community
awareness.
Fulfill grant or contract
requirements.
Contribute to scientific
knowledge.
Indentify hypotheses for future
study.
Conceptual Approaches to Evaluation
 Objectives-based evaluation

Determine extent to which objectives are achieved.
 Management-oriented evaluation

Provide information to aid in making decisions.
 Consumer-oriented evaluation

Provide information about products to aid decisions about purchases
or adoptions.
 Expertise-oriented evaluation

Provide professional judgments of quality.
 Participant-oriented evaluation

Understanding the complexities of programmatic activity,
responding to an audience’s requirements for information.
The Evaluator’s Role
 Objectivity is key when strategically selecting the
evaluator.
 Possible evaluators include:



External evaluator
Internal evaluator
Internal evaluator with external consultant
Basic Types of Evaluation
 Formative evaluation
 To provide information for program improvement.



What is working?
What needs to be improved?
How can it be improved?
 Summative evaluation
 To provide information to make decisions about the program’s
future or adoption.





What results occur?
With whom?
Under what conditions?
With what training?
At what cost?
Levels of Evaluation in Health Promotion
 Process evaluation: to determine feasibility.
“Implementation evaluation”
 “Quality assurance review”




What was delivered and how it was delivered?
Did program follow protocol?
Applies non-experimental designs.
 Impact evaluation: to provide information to make decisions about the program’s
future or adoption.
What were short-term (immediate-24 months) effects of the program on target
behaviors and their predisposing, enabling and reinforcing antecedents or on
environmental factors?
 Applies quasi-experimental and experimental designs.

 Outcome evaluation: to assess changes in overall health status and quality of life
indicators.
What were long-term changes (1-10 years)?
 Applies quasi-experimental and experimental designs.

Types of Evaluation Designs
 Non-experimental


Does not include random assignment or a control group.
Asserts little or no control over confounding factors.
 Quasi-experimental



Includes experimental and control group.
Does not include random assignment.
Offers some control over confounding factors.
 Experimental




Experimental and control group.
Includes random assignment.
Asserts greatest degree over confounding factors.
Evidence is most interpretable, defensible, and definitive.
Evaluation Plan: Key Steps
1. Gather background information
2. Identify stakeholders and audiences
3. Budget for evaluation
4. Develop evaluation question(s)
5. Determine sources of information required
6. Select data collection methods
7. Conduct evaluation (collect, analyze and interpret
data)
8. Communicate findings -- provide final report
Step 1: Gather Background Information
 Why is the program needed?
 General background, research on other programs and
approaches.
 What are the goals and objectives of the program?
 What is the program context?
 What are the program activities?
Step 1: Case Study Example
 Required orientation program for all first-year students.
 One-hour workshop/discussion.
 Delivered by residential advisors in small groups.
 Conducted concurrently in residential setting.
 Resident advisors trained in advance to lead the workshops.
 Program goals:
 Upon completion of the program, participants will understand the
role of drinking motives in decision-making concerning alcohol.
 Upon completion of the program, participants will be aware of the
role of social norms in decision-making concerning alcohol.
 Upon completion of the program, participants will be able to
identify “decision-making” as part of their own drinking behaviors.
Step 2: Identify Stakeholders and Audiences
 Who are the stakeholders and audiences? What are their
roles?
 Stakeholder: anyone who
has a stake in the program to be
evaluated or the evaluation’s results.
 Audiences: groups which have an interest in the evaluation
and receive its results.
 Consider stakeholder and audience roles:





To make policy
To make operational decisions
To provide input to evaluation
To react to
For interest only
Step 2: Case Study Example
To Make
Policy
To Make
Operational
Decisions
To Provide
Input to
Evaluation
Health Services
X
X
X
Student Affairs Office
X
X
X
Residential Life Office
X
X
X
To
React
Resident Advisors
X
X
Student participants
(freshmen)
X
X
For
Interest
Only
Parents and family of
University freshmen
X
University community
X
Other University students
X
University faculty and staff
X
Step 3: Budget for Evaluation
 What resources are available to dedicate to the evaluation?
 Estimate 7 – 10% of overall project budget for evaluation.
 Components may include, but are not limited to:
 Staff salary with benefits
 Consultants
 Travel
 Communications
 Printing and duplication
 Program materials
 Supplies and equipment
 Food
 Incentives
 Components may be paid in-kind, with operational dollars, external
funding, etc. List all regardless of funding source.
Step 3: Case Study Example
Expenses
Staff Salary
Description
Amount
Health Educator
Fringe
.01 FTE
27.60%
Total
$600
$166
$766
Personnel Total
Consultants
Evaluator
Statistician
Focus Group Facilitators
$150/hour x 30 hours
$60/hour x 10 hours
$20/hour x 10 groups (20 hours
total) x 2 facilitators
Outside Consultants Total
$4,500
$600
$800
$5,900
Travel Total
$500
$150
$650
Based on historical average
Printing and Duplication Total
$150
$150
Program Materials Total
$500
$500
Travel
Travel/Meals
Business Meeting Expenses
Printing and Duplication
Printing and Mailing
Program Materials
Survey Administration
Supplies and Equipment
Meeting Rooms
General Supplies
Computer Supplies
Telephone Calls
Based on historical average
Based on historical average
Supplies and Equipment Total
$100
$250
$200
$100
$650
Based on historical average
Food Total
$500
$500
Incentives Total
$500
$250
$750
Food
Catered Meals and Meetings
Incentives
Focus Group Incentives
Survey Incentives
Total Funds Needed to Complete Project
$
9,866
Step 4: Develop Evaluation Questions
 What questions need to be answered?
 Seek input from stakeholders and audiences.
 Consider process, impact and outcome evaluation
needs.
 Divergent phase:

Create a laundry list of potentially important questions
 Convergent phase:
 Select most critical questions from laundry list
Step 4: Case Study Example
 Process Evaluation Question:
 To what fidelity was the program implemented?



Does the content of the workshop cover the intended objectives?
Are resident advisors sufficiently prepared to lead the workshops?
Do first-year students participate in the workshops in the intended
manner?
 Impact Evaluation Question:
 Did the students who participated in the workshop achieve the
desired results?



Do participants understand the role of drinking motives in decisionmaking concerning alcohol?
Do participants understand the role of social norms in decision-making
concerning alcohol?
Do participants identify “decision-making” as part of their own
drinking behaviors?
Step 5: Determine Sources of Information
 What information do you need in order to answer
the evaluation questions?
Process
Health
promotion
program
Peer review,
quality control,
accreditation,
audit,
certification,
satisfaction
Impact
Outcome
Predisposing, enabling
and reinforcing factors
Protective behavior or
environment
Health
Social benefit
Changes in knowledge,
attitudes, beliefs,
skills,
resources, social
support, policy
Changes in frequency,
distribution, timing of
behavior, or quality of
the environment
Changes in
mortality,
morbidity,
disability or risk
factors
Changes in
quality
of life
Adapted from Green & Kreuter, 1999.
Step 6: Select Data Collection Methods
 What data will you collect and how will you collect them?
 Strategies may include, but are not limited to:
 Tests and assessments
 Written questionnaires
 Policy audit
 Interviews with key participants
 Observations of behavior or environmental conditions
 Self-report logs or diaries
 Focus groups
 Community-level indicators of impact
 Service utilization rates
 Case studies
 Document review
Steps 5 & 6: Case Study Example
 Determine information required and source/data collection method for each
question.
 Selected questions appear below. Note that there would be many more data
collection methods if all questions were presented.
Research Question
Information Required
Information Source/
Data Collection Method
Workshop objectives;
workshop components.
Document review: Workshop lesson
plan, resources, handouts,
activities.
Change in knowledge of
drinking motives.
One group, pre-test post-test
design: Web-based self-report
survey of workshop participants.
To what fidelity was the
program implemented?
Does the content of the workshop
cover the intended objectives?
Did the students who
participated in the workshop
achieve the desired results?
Do participants understand the
role of drinking motives in
decision-making concerning
alcohol?
Step 7: Conduct Evaluation Plan
June
2009
Contract with consultants
Meet with program
developers and
stakeholders
Develop data collection
methods
Resident advisor training
Program implementation
and participant
observation
Administer surveys
Conduct focus groups
Data analysis
Deliver final report
July
2009
August
2009
September
2009
October
2009
November
2009
December
2009
Step 8: Communicate Findings
Components of the final report may include:





Front matter (front cover, etc.)
Executive summary
Program background/evaluation description
Present the results
Discussion, conclusion, recommendations
 Report in ways that encourage follow-through by
stakeholders, to increase the likelihood that the evaluation
will be used.
 Re-visit list of stakeholders and audiences to determine
how to communicate findings and to whom.
References
 Fitzpatrick, JL, Sanders, JR & Worthen, BR. (2004). Program evaluation:





Alternative approaches and practical guidelines, 3rd ed. Boston, MA:
Pearson.
Green, LW & Kreuter, MW. (1999). Health promotion planning: An
educational and ecological approach, 3rd ed. Mountain View, CA: Mayfield
Publishing Company.
McDermott, R., & Sarvela, P. (1999). Health education evaluation and
measurement, 2nd ed. The McGraw-Hill Companies.
Modeste, NN & Tamayose, TS. (2004). Dictionary of public health
promotion and education terms and concepts, 2nd ed. San Francisco, CA:
Jossey-Bass.
WK Kellogg Foundation. (1998). The W.K. Kellogg evaluation handbook.
Available at www.wkkf.org.
Windsor, R., Baranowski, T., Clark, N., & Cutter, G., (1994). Evaluation of
health promotion, health education, and disease prevention programs,
2nd ed. Mountain View, CA: Mayfield Publishing Company.
Questions?
 Gina Baral Abrams, MPH, LSW, CHES
Princeton University
[email protected]
 Amy Melichar, MEd, CHES
Marquette University
[email protected]