Transcript Medicaid & State Oral Health Programs

Resources for State Oral
Health Program Evaluators
January 30, 2013
B. J. Tatro, PhD
ASTDD, [email protected]
Cassandra Martin Frazier, MPH, CHES
CDC, [email protected]
1
General Reminders
• This webinar will be recorded and archived on the
ASTDD website
• Questions will be addressed after the speakers are
finished. Please type your question into the “chatbox”
that will appear at the end of the webinar and then
click on the bubble to the right of where you type your
question to send it to the moderator
• Please respond to the polling questions at the
conclusion of the webinar
2
• This presentation was supported by
Cooperative Agreement 5U58DP001695 from
CDC, Division of Oral Health. Its contents are
solely the responsibility of the authors and do
not necessarily represent the official views of
CDC
3
Desired Results
• Increase awareness of the importance of
evaluation and evaluation capacity within state
oral health programs
• Increase familiarity with the scope of public oral
health evaluation topics
• Increase familiarity with and access to a variety of
practical and useful evaluation resources relevant
to public oral health
4
PART 1
Importance of Evaluation and
Evaluation Capacity in Public
Oral Health
5
Evaluation: A Key Component of State
Oral Health Program Infrastructure
Oral Health
for All
Policy and Programs
Planning & Evaluation Capacity
Surveillance Capacity
Leadership & Partnerships
Coalitions & Community Advocates
Resources
6
Status of Evaluation in State Oral
Health Programs
• In 2012, ASTDD completed a report on the status and
factors impacting state oral health infrastructure. The
capacity to conduct evaluation was one of these.
Findings included the following:
– According to CDC, the number of funded states reporting
using evaluation to improve programs had increased from
3 to 11 of the 12 by the end of their second state grant
funding cycle
– In a 2009 ASTDD survey, most states indicated that they
had access to evaluation expertise, but 13 did not
– Currently there is no way to assess and track if and how
non-CDC funded states are evaluating their programs,
partnerships, or policies
7
Key Factors Associated with Program
Success/Improvements in Oral Health
“Evaluation helps build infrastructure and enhance
sustainability when results are used to improve
programs, increase program visibility and
demonstrate program achievements. Strong
programs have evidence-based goals, conduct
routine evaluation and alter their programs based
on evaluation results.”
(ASTDD, State Oral Health Infrastructure and
Capacity: Reflecting on Progress and Charting the
Future, 2012)
8
Evaluation Capacity
Evaluation helps build infrastructure but the
capacity to do evaluation must be present
Evaluation requires:
• Sufficient and appropriate staffing
• Ongoing training
• Access to data
• Access to decision makers
• Financial resources
9
Importance of Evaluation in Public
Oral Health
In the ASTDD Guidelines for State and Territorial
Oral Health Programs, there are 10 Essential
Public Health Services to Promote Oral Health in
the US, one of which is:
Evaluate effectiveness, accessibility, and quality
of personal and population-based oral health
promotion activities and oral health services
10
State Roles in Public Oral Health
Evaluation
• Create a comprehensive evaluation plan
• Evaluate availability, utilization, appropriateness, and
effectiveness of activities and services, with partners
• Provide evaluation consultation to partners
• Validate and share results with community members,
stakeholders, and other partners
• Use results in oral health improvement and planning
• Leverage and allocate sufficient resources for evaluation
11
• Community
partnerships,
leverage
resources,
advocate
• Policies and
plans
Assurance
• Surveillance
• Analyze
determinants
and respond
• Inform,
educate,
empower
Policy Development
Assessment
Essential Public Health Services to
Promote Oral Health in the U.S.
• Laws and
regulations
• Services
• Workforce
• Evaluation
• Research
12
Evaluation of Other Essential
Public Oral Health Services
Virtually all of the other nine Essential Services
have an evaluation component!
13
Evaluation of Other Essential Public
Oral Health Services
Essential Service
Evaluation-related State Role/Activities
Surveillance
Review effectiveness of surveillance systems to track oral
health status and disparities
Inform, educate, empower Evaluate effectiveness and quality of communication and
promotion programs
Community partnerships,
leverage resources,
advocate
Evaluate partnerships and their effectiveness in creating
oral health improvements
Policies and plans
Determine impact of oral health policies
Laws and regulations
Evaluate impact of laws and regulations
Services
Evaluate impact of interventions and advocacy on access
to services
Workforce
Evaluate professional development and dental public
health workforce for their effect on oral health outcomes
Research
Evaluate results of implementing the research agenda
14
Funder Expectations for
Evaluation
• Federal Government: “A central pillar of good government is a
culture where answering such [outcome and impact] questions is a
fundamental part of program design and where agencies have the
capacity to use evidence to invest more in what works and less in
what does not.” (Program Evaluation, in Budget of the United States
Government, Fiscal Year 2011: Analytical Perspectives, 2010, 90–92.
Washington, DC: U.S. Government Printing Office)
• Foundations: “The field of philanthropy is undergoing a
fundamental transition toward more performance-centered and
forward-looking evaluation approaches that provide foundations
and their grantees with timely information and actionable insights.”
(Mark Kramer, From Insight to Action: New Directions in Foundation
Evaluation, April 2007)
15
CDC Structure for Evaluation
• Describe the program
– Develop logic models
– Develop a comprehensive evaluation plan and objectives
• Document what happens in the program
– Maintain accurate and organized records on programs
– Designate a person responsible for evaluation
– Create success stories
• Improve the program
– Use evaluation findings to plan program improvements
• CDC’s Framework for Program Evaluation in Public Health
16
CDC Structure for Evaluation
• Incorporate evaluation into the following
required activities for CDC-funded states:
– Leadership
– The surveillance system
– The state oral health plan
– Partnerships and state oral health coalitions
– Sealant programs
– Policy development
17
Competencies for Public Oral
Health Evaluators
CDC has identified competencies for State oral
health evaluators
Conducting an
Evaluation
3. Evaluation Approach
Conducting
Yourself as an
Evaluator
1. Interpersonal skills
2. Professional Practice
4. Program Description
5. Evaluation Plan
6. Program Monitoring
and Tracking
7. Collect, Analyze, and
Interpret Data
8. Evaluation
Reporting/Communication
9. Evaluation Use
18
Competencies for Public Oral
Health Evaluators
Competencies based on:
• Guiding Principles for Evaluators, American Evaluation
Association
• CDC, Framework for Program Evaluation
• Stevahn L, King J, Ghere G, Minnema J. Establishing
Essential Competencies for Program Evaluators.
American Journal of Evaluation, 2005; 26 (1) (43-59)
• Yarbrough, D, Shulha, L, Hopson, R, and Caruthers, F
(2011). The Program Evaluation Standards: A Guide for
Evaluators and Evaluation Users (3rd ed.). Thousand
Oaks, CA: Sage
• ASTDD Competencies for State Oral Health Programs
19
PART 2
Resources for Public Oral
Health Evaluators
20
Evaluation Planning and
Management
Overview of Evaluation
Resource: Introduction to Program Evaluation for Public Health Programs: A Self-Study
Guide (updated 2011)
Description: Comprehensive “how to” guide for planning and implementing
evaluation activities based on CDC’s Framework for Program Evaluation in Public
Health. Designed to assist evaluators in planning and implementing practical
evaluations. Includes real-world examples, tips, and worksheets.
Excellent tool for beginner evaluators.
Resource: W.K. Kellogg Foundation Evaluation Handbook (1998)
Description: A handbook that provides a broad overview of Kellogg’s
philosophy and expectations for evaluation and a framework for conducting
evaluation projects.
21
Evaluation Planning and
Management
Evaluation Project Management
Resource: Managing and Implementing an Evaluation, Guidelines and Tools for
Evaluation Managers (2008, Catholic Relief Services)
Description: A guidance document on evaluation management, including
management skills, communications, and dealing with the unexpected.
Resource: Evaluation Plans and Operations Checklist (1999, Daniel Stufflebeam)
Description: A checklist for conducting preliminary meta-evaluations of the evaluation
plan or process in order to strengthen the evaluation operations.
Resource: Evaluation Budget Development Checklist (2001, Jerry Horn)
Description: A checklist designed to assist evaluators and others to think through the
development of an evaluation budget. This instrument is divided into several
categories of typical cost items for conducting evaluations.
22
Evaluation Planning and
Management
Evaluation Planning
Resource: Community-Based Oral Health Programs: A Need and Plan for Evaluation
(2005, W.K. Kellogg Foundation)
Description: A report describing the major components that should be included in
evaluations of community-based oral health programs, along with suggestions for
specific measures that could be considered.
Resource: Developing an Effective Evaluation Plan (2011, CDC)
Description: A workbook to help public health program managers, administrators, and
evaluators develop a joint understanding of what constitutes an evaluation plan, why
it is important, and how to develop an effective evaluation plan in the context of the
planning process.
23
Evaluation of Prevention and
Promotion Programs
School-based Sealant Programs
Resource: Seal America: The Prevention Invention, Step 10, Program Evaluation
(Updated 2011)
Description: This manual is designed to assist health professionals in
initiating, implementing, and evaluating a school-based dental sealant
program to help prevent dental caries in children and adolescents. It
includes practical suggestions for evaluating program effectiveness and efficiency.
Resource: Sealant Efficiency Assessment for Locals and States (SEALS) (Updated 2009,
CDC)
Description: An Excel-based program that captures, stores, and analyzes
data from school-based sealant programs. SEALS reports generated at
the school, local, and state levels contain a variety of performance
measures that can be used to assess program effectiveness and economic
impact.
24
Evaluation of Prevention and
Promotion Programs
Fluoridation Program Management
Resource: Community Water Fluoridation Program Review (2004, CDC)
Description: A questionnaire to assist state programs in conducting a self-assessment
of their community water fluoridation program. Based on the Engineering and
Administrative Recommendations for Water Fluoridation guidelines and the ASTDD
Best Practices guidelines on fluoridation. To request a copy of the questionnaire,
please contact Cassandra Frazier or Kip Duchon.
Resource: Oral Health Maps (CDC)
Description: An interactive mapping application that presents indicators of oral health
and preventive interventions for oral health. Can view data by state.
Resource: Fluoride Legislative User Information Database (FLUID)
Description: A comprehensive database of federal and state policies
regarding community water fluoridation. Enables users to access information based on
legal fact and compare their current or proposed policies with others from across the
country.
25
Evaluation of Prevention and
Promotion Programs
Communications
Resource: Communications Plan Template for State Oral Health Programs (2010,
ASTDD)
Description: This tool is designed to help stakeholders plan and evaluate health
promotion and other state oral health communications.
26
Evaluation of Policy Work
Policy Work
Resource: What Gets Measured, Gets Changed: Evaluating Law and Policy for
Maximum Impact. Jamie F. Chriqui, Jean O’Connor, and Frank J. Chaloupka, in Using
Law, Policy, and Research to Improve the Public’s Health, Spring 2011
Description: An article that describes the need for and relevance of
conducting systematic policy evaluation, factors to consider when evaluating
laws and policies, and examples of how policy impact studies have informed
policy discussions and decision making. Provides samples of evaluation questions.
Resource: The Evaluation Exchange, Vol. XIII, No. 1 (Spring 2007, Harvard Family
Research Project)
Description: This volume is dedicated to evaluation of advocacy and
policy change and has several relevant articles.
Resource: The Policy Tool Guidebook (CDHP and CDC, 2009)
The Guidebook includes a section on evaluation, including an assessment
of baseline capacity for use by state oral health directors or other leaders
and a participant survey for those engaged in the policy process.
27
State Plan Evaluation
State Plans
Resource: CDC State Oral Health Plan Evaluation Workbook (~2013, CDC)
Description: This is a comprehensive and practical guide to evaluating the
implementation and impact of state oral health plans. Provides at-a-glance
summaries, tips, and strategies for keeping stakeholders engaged, evaluation
approaches and tools, resources, and action steps. Release expected in 2013.
Resource: Oral Health State Plan Review Index (2003, CDC)
Description: A checklist to help plan and develop the state oral health plan.
Can be used to conduct a review of the existing oral health plan.
Resource: State Oral Health Plan Comparison Tool (2012, CDHP)
Description: A database that provides an overview of the current state of
state oral health plans and facilitates cross-state comparisons.
28
Evaluation of Surveillance
Surveillance
Resource: Updated Guidelines for Evaluating Public Health Surveillance Systems
MMWR, July 27, 2001; 50(RR13) (1-35)
Description: This report provides guidelines for evaluating surveillance systems based
on CDC’s Framework for Program Evaluation in Public Health, research and discussion
of concerns related to public health surveillance systems, and public comments.
29
Evaluation of Leadership and
Leadership Development
Leadership Development
Resource: ASTDD Competencies for State Oral Health Programs (2009, ASTDD)
Description: Lists competencies in seven domains, including Domain 2, Plan and
Evaluate Programs. For each competency, provides four levels from basic to advanced.
There are links on the ASTDD website to associated tools and information
about the availability of technical assistance.
Resource: The National Public Health Leadership Institute: Evaluation of a Team-Based
Approach to Developing Collaborative Public Health Leaders, American Journal of
Public Health, April 2005, Vol. 95, No. 4 (641-644)
Description: Describes an evaluation of the National Public Health Leadership
Institute, which aims to develop collaborative leaders and strengthen
networks of leaders who share knowledge and jointly address public health
problems. Results are based on analysis of interviews conducted 12-18 months after
the program end and project final reports.
30
Evaluation of Partnerships and
Collaboration
Partnerships and Collaborations
Resource: Handbook on Planning, Evaluating and Improving Interagency Collaboration
in State Oral Health Programs (ASTDD, 2012)
Description: The handbook and companion workbook provide a framework for
evaluating collaboration between state oral health programs and other entities.
A variety of tools, as well as oral health examples, are provided.
Technical assistance is available through ASTDD.
Resource: Evaluating Partnerships to Prevent and Manage Chronic Disease , FD
Butterfoss, Preventing Chronic Disease, April 2009, Vol. 6, No. 2
Description: This article describes how to develop a comprehensive
evaluation strategy based on partnership theory, select outcomes
indicators, choose methods and tools, and use results for
accountability and improving effectiveness.
31
Where to Find New Resources
• ASTDD website (slides from this webinar will also be posted on the ASTDD
website)
• CDC evaluation website
• Join the American Evaluation Association (AEA) or a local affiliate
• AEA Linked-In
• University of Wisconsin Extension Center evaluation website
• Western Michigan University, the Evaluation Center
• Online Evaluation Resource Library – National Science Foundation website
• Bureau of Justice Assistance evaluation website
• Conferences, including the AEA Annual Conference, National Oral Health
Conference and preconference workshops, and CDC/AEA Summer Institute
• Online and live trainings, including AEA, CDC, and colleges/universities
32
Next Steps
• Are you interested in:
–
–
–
–
–
–
Future evaluation webinars?
NOHC evaluation roundtables?
NOHC evaluation presentations?
NOHC evaluation pre-sessions?
A evaluation community of practice?
An evaluation blog?
• Are you willing to post evaluation articles, data
sources, tools, or other resources to a shared site,
if it were available?
33