Transcript www.myfloridaeqro.com

Performance Improvement Projects
Technical Assistance
Health Maintenance Organizations
& Provider Service Networks
Wednesday, March 28, 2007
3:00 p.m. – 5:00 p.m.
Cheryl L. Neel, RN, MPH, CPHQ
Manager, Performance Improvement Projects
David Mabb, MS
Sr. Director, Statistical Evaluation
Presentation Outline
•
•
•
•
PIP Overall Comments
Aggregate MCO PIP Findings
Aggregate HMO Specific Findings
Technical Assistance with Group Activities
– Study Design
– Study Implementation
– Quality Outcomes Achieved
• Questions and Answers
Key PIP Strategies
1.
2.
3.
4.
Conduct outcome-oriented projects
Achieve demonstrable improvement
Sustain improvement
Correct systemic problems
Validity and Reliability of PIP Results
• Activity 3 of the CMS Validating Protocol:
Evaluating overall validity and reliability of PIP
results:
– Met = Confidence/High confidence in reported PIP
results
– Partially Met = Low confidence in reported PIP
results
– Not Met = Reported PIP results not credible
Summary of PIP Validation Scores
Percentage Score of
Evaluation
Elements Met
90% to 100%
80% to 89%
70% to 79%
60% to 69%
Less than 60%
Total
HMO
NHDP
PMHP
Total
2
13
13
8
17
53
0
0
1
3
16
20
0
6
0
1
1
8
2
19
14
12
34
81
Proportion of PIPs Meeting the
Requirements for Each Activity
Aggregate Valid Percent Met
I
II
III
IV
V
VI
VII
VIII
IX
X
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
1
3
5
7
9
11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53
Evaluation Element Number
HMO Specific Findings
•
•
•
•
53 PIPs submitted
Scores ranged from 5% to 98%
Average score was 65%
Assessed evaluation elements were scored
as Met 64% of the time
Summary of HMO Validation Scores
Percentage Score
of Evaluation
Elements Met
HMO
90% to 100%
80% to 89%
70% to 79%
60% to 69%
Less than 60%
Total
2
13
13
8
17
53
Study Design
Four Components:
1. Activity I. Selecting an Appropriate Study
Topic
2. Activity II. Presenting Clearly Defined,
Answerable Study Question(s)
3. Activity III. Documenting Clearly Defined
Study Indicator(s)
4. Activity IV. Stating a Correctly Identified
Study Population
Activity I. Selecting an Appropriate Study
Topic – HMO Overall Score
100%
90%
96%
80%
96%
94%
96%
4
5
89%
81%
70%
60%
50%
40%
30%
20%
10%
0%
1
2
3
Evaluation Element Number
6
Activity I. Selecting an Appropriate
Study Topic
Results:
• 92 percent of the six evaluation elements
were Met
• 8 percent were Partially Met or Not Met
• None of the evaluation elements were Not
Applicable or Not Assessed
Activity I: Review the Selected Study Topic
HSAG Evaluation Elements:
• Reflects high-volume or high-risk conditions (or was
selected by the State).
• Is selected following collection and analysis of data
(or was selected by the State).
• Addresses a broad spectrum of care and services (or was
selected by the State).
• Includes all eligible populations that meet the study
criteria.
• Does not exclude members with special health care
needs.
• Has the potential to affect member health, functional
status, or satisfaction.
Bolded evaluation elements show areas for improvement
Activity II. Presenting Clearly Defined,
Answerable Study Question(s) - HMO Overall
Score
100%
90%
80%
70%
60%
50%
40%
45%
42%
30%
20%
10%
0%
1
2
Evaluation Element Number
Activity II. Presenting Clearly Defined,
Answerable Study Question(s)
Results:
• 43 percent of the two evaluation elements
were Met
• 57 percent were Partially Met or Not Met
• None of the evaluation elements were Not
Applicable or Not Assessed
Activity II: Review the Study Question(s)
HSAG Evaluation Elements:
• States the problem to be studied in simple terms.
• Is answerable.
Bolded evaluation elements show areas for improvement
HMO Overall Score for Each Evaluation
Element – Activity III
100%
100%
90%
80%
89%
84%
81%
70%
72%
60%
50%
40%
30%
36%
32%
20%
10%
0%
1
2
3
4
5
Evaluation Element Number
6
7
Activity III. Documenting Clearly Defined
Study Indicator(s)
Results:
• 56 percent of the seven evaluation elements
were Met
• 27 percent were Partially Met or Not Met
• 16 percent were Not Applicable or Not
Assessed
Activity III: Review Selected Study
Indicator(s)
HSAG Evaluation Elements:
•
•
•
•
Is well defined, objective, and measurable.
Are based on practice guidelines, with sources identified.
Allows for the study question to be answered.
Measures changes (outcomes) in health or functional status,
member satisfaction, or valid process alternatives.
• Have available data that can be collected on each indicator.
• Are nationally recognized measure such as HEDIS®, when
appropriate.
• Includes the basis on which each indicator was adopted, if
internally developed.
Bolded evaluation elements show areas for improvement
Activity IV. Stating a Correctly Identified Study
Population - HMO Overall Score
100%
90%
80%
70%
60%
66%
50%
40%
56%
43%
30%
20%
10%
0%
1
2
Evaluation Element Number
3
Activity IV. Stating a Correctly Identified
Study Population
Results:
• 53 percent of the three evaluation elements
were Met
• 43 percent were Partially Met or Not Met
• 3 percent of the evaluation elements were
Not Applicable or Not Assessed
Activity IV: Review the Identified Study
Population
HSAG Evaluation Elements:
• Is accurately and completely defined.
• Includes requirements for the length of a
member’s enrollment in the managed care plan.
• Captures all members to whom the study
question applies.
Bolded evaluation elements show areas for improvement
Group Activity
Study Implementation
Three Components:
1. Activity V. Valid Sampling Techniques
2. Activity VI. Accurate/Complete Data
Collection
3. Activity VII. Appropriate Improvement
Strategies
Activity V. Presenting a Valid Sampling
Technique - HMO Overall Score
100%
90%
80%
70%
60%
50%
58%
58%
40%
30%
42%
42%
42%
42%
3
4
5
6
20%
10%
0%
1
2
Evaluation Element Number
Activity V. Presenting a Valid Sampling
Technique
Results:
• 12 out of the 53 PIP studies used sampling.
• 11 percent of the six evaluation elements
were Met.
• 12 percent were Partially Met or Not Met.
• 77 percent of the evaluation elements were
Not Applicable or Not Assessed.
Activity V: Review Sampling Methods
* This section is only validated if sampling is used.
HSAG Evaluation Elements:
• Consider and specify the true or estimated frequency of
occurrence. (N=7)
• Identify the sample size. (N=7)
• Specify the confidence level to be used. (N=5)
• Specify the acceptable margin of error. (N=5)
• Ensure a representative sample of the eligible population.
(N=5)
• Ensure that the sampling techniques are in accordance
with generally accepted principles of research design and
statistical analysis. (N=5)
Populations or Samples?
Generally,
– Administrative data uses populations
– Hybrid (chart abstraction) method uses
samples identified through
administrative data
Activity VI. Specifying Accurate/Complete
Data Collection - HMO Overall Score
100%
90%
89%
80%
70%
60%
72%
65%
63%
58%
50%
40%
58%
58%
58%
43%
42%
30%
20%
26%
10%
0%
1
2
3
4
5
6
7
Evaluation Element Number
8
9
10
11
Activity VI. Specifying Accurate/Complete
Data Collection
Results:
• 34 percent of the eleven evaluation elements
were Met
• 24 percent were Partially Met or Not Met
• 42 percent of the evaluation elements were
Not Applicable or Not Assessed
Activity VI: Review Data Collection
Procedures
HSAG Evaluation Elements:
• Clearly defined data elements to be collected.
• Clearly identified sources of data.
• A clearly defined and systematic process for
collecting data that includes how baseline and
remeasurement data will be collected.
• A timeline for the collection of baseline and
remeasurement data.
• Qualified staff and personnel to collect manual data.
• A manual data collection tool that ensures consistent
and accurate collection of data according to indicator
specifications.
Bolded evaluation elements show areas for improvement
Activity VI: Review Data Collection
Procedures (cont.)
HSAG Evaluation Elements:
• A manual data collection tool that supports
interrater reliability. Clear and concise written
instructions for completing the manual data
collection tool. An overview of the study in the
written instructions.
• Administrative data collection algorithms that
show steps in the production of indicators.
• An estimated degree of automated data
completeness (important if using the
administrative method).
Bolded evaluation elements show areas for improvement
Where do we look for our sources of data?
Baseline Data Sources
•
•
•
•
•
•
•
Medical Records
Administrative claims/encounter data
Hybrid
HEDIS
Survey Data
MCO program data
Other
Activity VII. Documenting the Appropriate
Improvement Strategies - HMO Overall Score
100%
90%
80%
70%
68%
60%
62%
58%
50%
40%
43%
30%
20%
10%
0%
1
2
3
Evaluation Element Number
4
Activity VII. Documenting the Appropriate
Improvement Strategies
Results:
• 28 percent of the four evaluation elements
were Met
• 21 percent were Partially Met or Not Met
• 51 percent of the evaluation elements were
Not Applicable or Not Assessed
Activity Seven: Assess Improvement
Strategies
HSAG Evaluation Elements:
• Related to causes/barriers identified
through data analysis and Quality
Improvement (QI) processes.
• System changes that are likely to induce
permanent change.
• Revised if original interventions are not
successful.
• Standardized and monitored if
interventions are successful.
Bolded evaluation elements show areas for improvement
Determining Interventions
Once you know how you are doing at
baseline, what interventions will produce
meaningful improvement in the target
population?
First Do A Barrier Analysis
What did an analysis of baseline results
show ?
How can we relate it to system
improvement?
• Opportunities for improvement
• Determine intervention(s)
• Identify barriers to reaching improvement
How was intervention(s) chosen?
• By reviewing the literature
– Evidence-based
– Pros & cons
– Benefits & costs
• Develop list of potential interventions -- what
is most effective?
Types of Interventions
•
•
•
•
•
•
Education
Provider performance feedback
Reminders & tracking systems
Organizational changes
Community level interventions
Mass media
Choosing Interventions
• Balance
– potential for success with ease of use
– acceptability to providers & collaborators
– cost considerations (direct and indirect)
• Feasibility
– adequate resources
– adequate staff and training to ensure a
sustainable effort
Physician Interventions: Multifaceted
Most Effective
• Most effective:
– real-time
reminders
– outreach/detailing
– opinion leaders
– provider profiles
• Less effective:
– educational
materials (alone)
– formal CME
program without
enabling or
reinforcing
strategies
Patient Interventions
• Educational programs
– Disease-specific education booklets
– Lists of questions to ask your physician
– Organizing materials: flowsheets, charts,
reminder cards
– Screening instruments to detect
complications
– Direct mailing, media ads, websites
Evaluating Interventions
• Does it target a specific quality indicator?
• Is it aimed at appropriate stakeholders?
• Is it directed at a specific process/outcome of
care or service?
• Did the intervention begin after baseline
measurement period?
Interventions Checklist
 Analyze barriers (root causes)
 Choose & understand target audience
 Select interventions based on cost-benefit
 Track intermediate results
 Evaluate effectiveness
 Modify interventions as needed
 Re-Measure
Group Activity
Quality Outcomes Achieved
Three Components:
1.
2.
3.
Activity VIII. Presentation of Sufficient Data
Analysis and Interpretation
Activity IX. Evidence of Real Improvement
Achieved
Activity X. Data Supporting Sustained
Improvement Achieved
Activity VIII. Presentation of Sufficient Data
Analysis and Interpretation - HMO Overall
Score
100%
96%
90%
80%
70%
70%
60%
50%
78%
73%
67%
60%
63%
60%
50%
40%
30%
20%
10%
0%
1
2
3
4
5
6
Evaluation Element Number
7
8
9
Activity VIII. Presentation of Sufficient Data
Analysis and Interpretation
Results:
• 35 percent of the nine evaluation elements
were Met
• 15 percent of the evaluation elements
Partially Met or Not Met
• 50 percent of the evaluation elements were
Not Applicable or Not Assessed
Activity VIII: Review Data Analysis and
Interpretation of Study Results
HSAG Evaluation Elements:
• Is conducted according to the data analysis plan
in the study design.
• Allows for generalization of the results to the
study population if a sample was selected.
• Identifies factors that threaten internal or external
validity of findings.
• Includes an interpretation of findings.
• Is presented in a way that provides accurate,
clear, and easily understood information.
Activity VIII: Review Data Analysis and
Interpretation of Study Results (cont.)
HSAG Evaluation Elements:
• Identifies initial measurement and remeasurement of
study indicators.
• Identifies statistical differences between initial
measurement and remeasurement.
• Identifies factors that affect the ability to compare
initial measurement with remeasurement.
• Includes the extent to which the study was
successful.
Bolded evaluation elements show areas for improvement
Changes in Study Design?
Study design should be same as baseline
 Data source
 Data collection methods
 Data analysis
 Target population or sample size
 Sampling methodology
If change:
rationale must be specified & appropriate
Activity IX. Evidence of Real Improvement HMO Overall Score
100%
90%
80%
70%
74%
60%
50%
56%
40%
30%
31%
20%
10%
11%
0%
1
2
3
Evaluation Element Number
4
Activity IX. Evidence of Real Improvement
Results:
• 22 percent of the four evaluation elements
were Met
• 29 percent were Partially Met or Not Met
• 50 percent of the evaluation elements were
Not Applicable or Not Assessed
Activity IX: Assess the Likelihood that
Reported Improvement is “Real” Improvement
HSAG Evaluation Elements:
• The remeasurement methodology is the same as
the baseline methodology.
• There is documented improvement in processes
or outcomes of care.
• The improvement appears to be the result of
intervention(s).
• There is statistical evidence that observed
improvement is true improvement.
Bolded evaluation elements show areas for improvement
Statistical Significance Testing
Time
Periods
Measurement
Periods
Numerator
Denominator
Rate or
Results
Industry
Benchmark
Statistical Testing
and Significance
CY 2003
Baseline
201
411
48.9%
60%
N/A
CY 2004
Re-measurement 1
225
411
54.7%
60%
Chi-square = 2.8
P-value = 0.09387
NOT SIGNIFICANT
AT THE 95%
CONFIDENCE
LEVEL
Activity X. Data Supporting Sustained
Improvement Achieved - HMO Overall Score
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
14%
0%
1
Evaluation Element Number
Activity X. Data Supporting Sustained
Improvement Achieved
Results:
• 6 percent of the one evaluation element was
Met
• 36 percent was Partially Met or Not Met
• 58 percent of the evaluation element was Not
Applicable or Not Assessed
Activity X: Assess Whether Improvement is
Sustained
HSAG Evaluation Elements:
• Repeated measurements over
comparable time periods
demonstrate sustained improvement,
or that a decline in improvement is
not statistically significant.
Bolded evaluation elements show areas for improvement
Quality Outcomes Achieved
Baseline
1st Yr
Demonstrable
Improvement
Sustained
Improvement
Sustained Improvement
• Modifications in interventions
• Changes in study design
• Improvement sustained for 1 year
HSAG Contact Information
Cheryl Neel, RN, MPH,CPHQ
Manager, Performance Improvement
Projects
[email protected]
602.745.6201
Denise Driscoll
Administrative Assistant
[email protected]
602.745.6260
Questions and Answers