Program Planning, Evaluation and QI OH MY!

PART 1

Kusuma Madamala, PhD, MPH Consultant, NNPHI February 5, 2010

Storyboard Review 7 Components

1. Description of situation/problem 2. Articulation of aim statement 3. Description of proposed intervention or solution to address the situation/problem 4. Description of what was done to address the situation/problem http://nnphi.org/CMSuploads/Storyboard-Guidelines FINAL-05868.pdf

Storyboard Components (continued)

5. Description of the analysis/evaluation of the actions taken to address the situation/problem 6. Description of subsequent action that took place to standardize the improvement 7. Overarching lessons learned

Storyboard Review

• • • Wide variation in the storyboards regarding the 7 components Space allocation challenges Some use of PDSA as a template to address 7 components and to test proposed change

Storyboard Review

Description of the situation or problem – Most storyboards had a clearly defined problem – Problem articulated by team – Utilized Quality Improvement tools and methods – As noted by Joly and colleagues in their review of the MLC(9/18/2009), lack of clarity regarding a QI “project”. – Process vs outcome focused

Description of the situation or problem Process focused storyboards

• • • • Examples: Improve community engagement/representation in coalitions Implement organizational process improvement Implement staff training on a variety of issues Transfer to electronic systems

Description of the situation/problem: Health Outcome/Intervention focused storyboards

• • • • Examples: Reduce heart disease Decrease childhood obesity Increase nutrition education Increase immunization rates

Aim statement

• • • Variation in clearly articulated AIM statement Goals and objectives (process and outcome) Measurable objectives By when, Who, Will Do What by How much?

Child Health Mini-Collaborative

Reduce Incidence of Vaccine Preventable Disease AIM Statement: To maximize the satisfaction, health, and well being of the client population in 8 LHDs by assuring that all clients have access to public health services when they want and need it. By providing timely, efficient, and patient-centered care, the changes made will benefit LHD staff, clinicians, and practice and reduce costs by eliminating waste in the local system.

Child Health Mini-Collaborative Reduce Incidence of Vaccine Preventable Disease

Goals

1) 2) 3) Assist LHDs that provide sick and well child care to Medicaid children through full implementation of Open Access/Office Efficiency principles in their Child Health Clinics; Identify best practice strategies for implementation of efficiency and access principles in public health; Develop NCDPH staff competencies in Quality Improvement to more effectively spread the principles statewide.

Objectives

1. Meet or exceed established standards for customer service access to services within each service area 2. Meet or exceed customer satisfaction rating of 96% (Excellent/Very Good) 3. Maintain expenses at or below approved budget. Maintain or exceed revenue projections of approved budget 4. Meet established standards for productivity (# persons seen and served) within each service area 5. Maintain/reduce unavoidable employee turnover rate of 11%

Storyboard Components

Varied degree of detail provided in each of the storyboards regarding the components • • • • Component 3 - description of intervention or solution to address the problem Component 4 -matching the actions to the problem Component 5 - analysis/evaluation Component 6 - subsequent action- next steps if no improvement or how to sustain improvement

Description of intervention or solution to address the problem

• • • Theory of improvement – varied in what was in it and what it meant – some framed within an “if/then” context – occasional reference to the implemented QI methods Reference to what we may already know – varied use of best or promising practice in public health or other sectors References used differed by process vs. outcome nature of the problem

Matching actions to the problem

• • • Dependent upon clear articulation of the problem – QI tools used to id problem Both easy and difficult to tell how the actions were linked to the problem (tools used guided actions) Wide range and variety of number of implemented actions

Analysis/Evaluation

• • Highly dependent upon: Clear AIM statement (with measurable goals and objectives) Description of intervention or proposed solution to the problem Generally, description of the problem more detailed than in the analysis and evaluation plans

Analysis/Evaluation: Satisfaction

Problems with solely relying on consumer satisfaction: – Measures have a ceiling effect (generally 4.5-4.8) on 1-5 scale – Clients report high satisfaction to minimize dissonance – Dissatisfied rarely reply – Satisfaction says nothing about the impact or outcome of the program

To better understand storyboard components and the variations among them….

Need to explore some differences between Quality Improvement and Program Planning & Evaluation

Quality Improvement is…

• • A set of concepts and methods geared toward improving the ability of a product or services to meet consumer needs.

Involves an understanding of performance problem in system terms, the will to change, the use of data to implement and track changes, and the sequential building of knowledge from testing through implementation.

Basic tools for QI

• • • • • • • • • • • • • • • • Bar Chart Brainstorming Cause and effect diagram (i.e. fishbone diagram) Check sheet Control chart Five whys and five hows Flowchart Force field analysis Histogram Nominal group technique Pareto chart Pie chart Run chart Scatter diagram SIPOC+ CM Solution and effect diagram

Advanced tools of QI

• • • • • • • • • • • Affinity diagram Control and influence diagram Gantt chart Interrelationship diagram Matrix diagram Process decision program chart Prioritization matrix Radar chart SMART matrix SWOT analysis Tree diagram

Evaluation is….

• Systematic application of research procedures to assess the conceptualization, design, implementation and utility of intervention programs (Rossi and Freeman,1993) • Assessing the quality and effect of programs

Program Theory

Current language of explaining the connection between what we do (process and effort) and what difference the program makes (effect, impact/outcome)

Functions of Program Theory

• • • • Guides program staff activities Guides evaluation plan Facilitates explaining what went right or wrong and why program failure occurred Facilitates communication among stakeholders

Steps involved in creating a program theory

• • • • Draw upon the scientific literature to identify which theories and interventions are known to be effective in addressing the health problem Involve selected stakeholders in developing program theory Temporally sequence the causal chain of events Specify “dose” and “mode of administration” of the intervention delivered in the program

Steps

(continued) • • Construct logic model with or without additional diagrams or drawings of the causal chain of events Check the model against – Assumptions held by program staff and stakeholders – Available resources and commitments – The science behind the final intervention, health problem and target audience

• Issel, L.M. Health Program Planning and Evaluation – A Practical, Systematic Approach for Community Health. Jones and Bartlett. 2009.

• Rossi, P. , Freeman, H. & Lipsey, M. Evaluation: A Systematic Approach, Sixth Edition, 1999. Sage Publications. Adapted with permission o f Sage Publications.

Philosophy Purpose Approach Who does it Methods

Process Improvement Methodologies

Organizations can be more effective if they use staff expertise to improve services and products Systems analysis and improvement focus on identified problem areas from point of view of customer needs Team-based approach to identifying and analyzing the problem Staff employees from any or all departments, mid level managers, top level executives Engineering approaches to systems analysis

Program Process Evaluation

Programs need to be justified in terms of their effect on participants Evaluators determine whether a program was provided as planned and if it made a difference to the participants (customers) Evaluator-driven approach to data collection and analysis Evaluators and program manager, with or without participation of employees or stakeholders Scientific research methods Issel, L.M. Health Program Planning and Evaluation – A Practical, Systematic Approach for Community Health. Jones and Bartlett 2009

References

• • • • • • • • • • • • • Deming WE. Out of the crisis. 2nd ed. Boston: MIT Press; 2000.

Gillen, S. McKeever, J. Edwards, K & Thielen, L. Promoting Quality Improvement and Achieving Measurable Change: The Lead States Initiative. Journal of Public Health Management and Practice 2010, 16(1), 55-60.

Institute for Healthcare Improvement: http://www.ihi.org/IHI/Topics/Improvement/ImprovementMethods/HowToImprove/ Accessed January 16, 2010.

Issel, L.M. (2009).Health Program Planning and Evaluation. A Practical, Systematic Approach for Community Health.

Joly, B. Shaler, M, Booth, M, Conway, A. & Mittal, P. Evaluating the Multi-State Learning Collaborative. Journal of Public Health Management and Practice 2010, 16(1), 61-66.

Joly, B., Booth, M., Shaler, G. and Conway, A. “What Have We Learned So Far.” 9/18/2009 MLC presentation Langley GL, Nolan KM, Nolan TW, Norman CL, Provost LP.

Enhancing Organizational Performance .

The Improvement Guide: A Practical Approach to Lotstein D, Seid M, Ricci K, Leuschner K, Margolis P, Lurie N. Using quality improvement methods to improve public health emergency preparedness: PREPARE for Pandemic Influenza. Health Affairs. June 15, 2008;27(5):328-339.

Mark, M. & Pines, E. (1995). Implications of continuous quality improvement for program evaluation and evaluators. Evaluation Practice, 16,131-139.

Quality portal homepage. 2008; www.thequalityportal.com

. Accessed December 10, 2008.

Bialek, Duffy and Moran(2009) The Public Health Quality Improvement Handbook. ASQ Quality Press: Milwaukee, WI The Plan-Do-Study-Act (PDSA) cycle was originally developed by Walter A. Shewhart as the Plan-Do-Check Act (PDCA) cycle. W. Edwards Deming modified Shewhart's cycle to PDSA, replacing "Check" with "Study." [See Deming WE.

The New Economics for Industry, Government, and Education

. Cambridge, MA: The MIT Press; 2000.] Rossi and Freeman (1993). Evaluation: A systematic approach (6 th Publications.

ed). Thousand Oaks, CA: Sage

Program Planning, Evaluation, and QI, Oh My!

PART 2 Mary V. Davis, DrPH, MSPH February 5, 2010

Plan: PH Program & QI

• • • • Same planning phase BUT more rapid in QI Targets, goals, theory of change, change packages (interventions), data needs QI—Emphasis on root causes that can be changed and tested in small cycles PH—Emphasis more contextual, identify the BIG program that can change everything

NC Mini-Collaborative 2

Target: Workforce is assessed to determine its abilities to deliver population-based services and a workforce development plan is crafted. Specific Target (AIM Statement): Improve the retention of public health nurses through a mentoring program that provides nurturing for individual and professional growth to new public health nurses.

1. Description of Situation

• High public health nurse turnover rates in local health departments • Factors – Pay – Nurses entering public health positions lack training in epidemiology, health education, case management (job mismatch and dissatisfaction) • Cost of turnover: 75% of annual salary for position

2. Revised AIM Statement

The overall aim of this collaborative is to determine if a PHN mentoring program can be used to improve the competence, retention and satisfaction of new PHN’s (defined as <2 years or in a new PH nursing role).

2a Goals (set up for Evaluation/Study phases)

• • • • • • By October, 2010… Mentees planning to remain in public health will increase by 25% Mentees who recommend working in an LHA to a colleagues will increase by 25% Mentees and mentor overall job satisfaction will improve by 15% Understanding of population-based services as well as mentee knowledge and understanding of their role will increase by 25% 100% of participating mentors and mentees will complete the 6 month program and all will be satisfied with the process Greater than 90% of mentees and mentors will be willing to serve as a mentor in the future

3/4. Description of proposed intervention / Actions being planned (DO)

• Six month pilot mentoring program for new PH nurses NC LHDs – Mentor training/mentee orientation – Twice monthly mentor/mentee contacts – Monthly TA calls to mentors – TA calls for mentees as needed – Reunion meeting @ NCPHA conference – Pre/post measures of new nurse job satisfaction/ intent to stay in job

Change Package Development

• Nurse mentoring programs designed to job fit – Wayne County NC program written into policy – Georgia District Nurse Mentoring Program – Colorado Leaders Nurse Mentoring Program • Planning Committee chose to adapt and modify Georgia District Nurse Mentoring Program

Theory of Change

Nurse mentoring program for new ph nurses PH Nurse job satisfaction Better job match PH Nurses trained in population based services PH nurse job retention Stable PH Nurse capacity to deliver population based services

Intervention vs QI

• • • • •

Classic PH Intervention

Identified retention problem Identify mentoring practices as solution Tailor or adopt mentoring programs to situation Implement pilot program with set intervention Evaluation • • • • •

QI

Identified retention problem—check root cause Identify mentoring practices as solution Tailor or adopt mentoring programs to situation Implement program with small cycles tests of change Evaluation

So How is this QI?

• • • Small cycle tests of change by mentors Mentoring techniques – Problem solving – Meeting times and venues – Improving job fit – Teaching population-based services Monthly conference calls with faculty/planning committee

5. Evaluation of Actions Taken

(QI: Study )

• Process measures-- number of mentor/mentee meetings, satisfaction with process, challenges and facilitators, mentor best practices • Outcomes—mentees – Population based knowledge – Job satisfaction and fit – Intention to stay in public health

• •

6. Standardizing Improvements

(QI: Act)

• Identify aspects of mentoring program that worked well for most LHDs Create mentoring change package Disseminate statewide

7. Plan Phase Lessons Learned

• Getting to root cause for target with LHDs challenge • Getting right planning committee members (nurse supervisors, health educators) has been key • Engaging QI experts improved planning for QI: Aim statements, change package, applications, training, timeline