Transcript CCME Corporate PPT Template 2 - The Carolinas Center for

Performance Improvement Projects:
Validating Process, Tips, and Hints
Eric Jackson, MA
Research Analyst
October 19, 2009
Overview of the Training
•
•
•
•
•
•
•
Meeting Logistics
Validation Process
CMS Protocol Activities
Tips and Hints
Resources
Questions and Answers
Other Training Topics
Meeting Logistics
• Using GoToWebinar™
• How to ask questions to team during the
presentation
– Through the questions section in the GoToWebinar
application
– You can use the GoToWebinar raise your hand option
• We will review submitted and additional
questions at the end of the presentation
First the Basics of GoToWebinar:
The GoToWebinar Attendee Interface
1. Viewer Window
2. Control Panel
Validation Process for PIP
• Request desk materials
– All PIPs currently in progress or completed since the
last EQR should be sent
– Example: NCQA Activity Forms
• After receiving desk materials, three PIPs are
selected for validation
• Perform CMS Protocol Activities by reviewing
the desk materials we received
CMS Protocol Activities
• Activity One: Assessing the Study Methodology
• Activity Two: Verify Study Findings (optional)
• Activity Three: Evaluating Overall Validity and
Reliability of Study Results
Activity One:
Assessing the Study Methodology
• Ten steps with related review questions and
scoring
– 27 components / questions and 116 possible points
• Specific questions can be found in the CCME PIP Validation
Overview document
Activity One:
Assessing the Study Methodology (cont)
• Each component is scored by the degree they
meet the protocol requirements
– MET: component fully meets the criteria without any issues
– PARTIALLY MET: component meets some but not all of the
criteria
– NOT MET: component fails to meet most or all the criteria
– NA: component does not apply to the project being reviewed
Activity One: Step One
• Review the Selected Study Topic(s)
– How was the topic of the study selected
– Is the topic appropriate
• 3 questions with 7 possible points
Activity One: Step Two
• Review the Study Question(s)
– Is the study question documented
• Single question with 10 possible points
Activity One: Step Three
• Review the Selected Study Indicator(s)
– Are the study indicators objective and clearly defined
– Are the study indicators appropriate
• 2 questions with 11 possible points
Activity One: Step Four
• Review the Identified Study Population
– Is the study population well defined
– Is the population being captured correctly
• 2 questions with 6 possible points
Activity One: Step Five
• Review the Sampling Methods
– Applies only if sampling was used in the project
– Is a valid sampling method being used
– Is the sample large enough
• 3 questions with 20 possible points
Activity One: Step Six
• Review the Data Collection Procedures
– Are data sources clearly specified
– Was a data analysis plan established in the
documentation
– Did the instruments used for data collection provide
consistent, accurate data
• 6 questions with 18 possible points
Activity One: Step Seven
• Assess Improvement Strategies
– Are reasonable interventions being planned and
implemented
• Single question with 10 possible points
Activity One: Step Eight
• Review the Data Analysis and Interpretation of
Study Results
– Was the data analysis plan followed
– Are numerical results presented accurately and
clearly
• 4 questions with 17 possible points
Activity One: Step Nine
• Assess Whether Improvement is “Real”
Improvement
– Is there any documented, quantitative improvement
– Was the same methodology used for baseline and
repeated measurement
• 4 questions with 12 possible points
Activity One: Step Ten
• Assess Sustained Improvement
– Is sustained improvement demonstrated in the
repeated measurements of the study
• Single question with 5 possible points
Activity Two: Verify Study Findings
• Optional Activity
• Requires the plans to produce the data that
generated the results of the PIP
• Review would try to mimic the documented
results from the data received from the plan
Activity Three: Evaluating Overall
Validity and Reliability of Study Results
• Scores are summarized
• Validation Finding is calculated
o VF = (score project received / total possible points)
o Multiply by 100 to report as a percentage
• Final Audit Designation is assigned
Activity Three: Evaluating Overall Validity
and Reliability of Study Results (cont)
• Score ranges for the Final Audit Designation
AUDIT DESIGNATION POSSIBILITIES
High Confidence in
Reported Results
Little to no minor documentation problems or issues that do not lower the confidence in what the
plan reports. Validation findings must be 90%–100%.
Confidence in Reported
Results
Minor documentation or procedural problems that could impose a small bias on the results of the
project. Validation findings must be 70%–89%.
Low Confidence in
Reported Results
Reported Results NOT
Credible
Plan deviated from or failed to follow their documented procedure in a way that data was
misused or misreported, thus introducing major bias in results reported. Validation findings
between 60%–69% are classified here.
Major errors that put the results of the entire project in question. Validation findings below 60%
are classified here.
Tips and Hints
• Remember the Study Questions in your project
documentation!
• Document the numerators and denominators
along with the rates
• Double check rate calculations
Tips and Hints (cont)
• Check your terminology
o Percent change is not the same as percentage point
change
• Example: Baseline = 55% Re-measurement = 95%
• Percent Change = (95% - 55%) / 55% = 73% change
• Percentage Point Change = 95% - 55% = 40 percentage
point change
Resources
• Go to: www.thecarolinascenter.org
– Search for SC EQR
– Training materials should be one of the top results
• CCME PIP Validation Overview document
• CMS PIP Protocol
o “Validating Performance Improvement Projects: A
protocol for use in conducting Medicaid external
quality review activities”
Question and Answers
Additional Training Topics
Please Remember the Evaluation!
It will display after you end the webinar.