Transcript HSAG Performance Improvement Training Statistical Testing Presented by Donald Grostic, MS

HSAG Performance Improvement Training
Statistical Testing
Presented by Donald Grostic, MS
Health Services Advisory Group, Inc.
February 14, 2008
Agenda
I.
Background



II.
What statistical test do I use?




III.
Why do statistical testing?
When is statistical testing used?
Which PIP Activities and Evaluation Elements does statistical
testing apply?
Chi-square test of inference
How is the retention rate numerator and denominator used in a
chi-square test?
Website recommendation for Chi-square test
How do I interpret the results of the Chi-square test?
Question and answer session
Background
• Why do statistical Testing?
– The CMS Protocols require that statistical
testing be used to prove that any improvement
in rates is real.
– Without statistical testing, a PIP would not
meet the CMS Protocols.
Background (continued)
• When is statistical testing used?
– After collecting retention rate baseline data (four quarters of data
equaling a year end roll up), completing a causal barrier analysis,
implemented interventions and completing a first re-measurement
(four more quarters of data equaling the second year end roll up).
– At this point (after eight quarters of data has been collected and
rolled up into two year end rates), statistical testing should be
completed to compare the annual roll up baseline retention rate to
the annual roll up first re-measurement rate.
– Statistical testing would also need to be performed between the
annual re-measurement 1 and re-measurement 2.
Background (continued)
• Which PIP Activities and Evaluation Elements
does statistical testing apply?
– Activity VIII Evaluation Element 7: Identifies statistical
differences between initial measurement and remeasurement.
– Activity IX Evaluation Element 4: There is statistical
evidence that observed improvement is true
improvement.
– Activity X Evaluation Element 1: Repeated
measurements over comparable time periods
demonstrate sustained improvement, or that a decline
in improvement is not statistically significant.
Chi-square Test
• What statistical test should I use?
– Retention rate is calculated using a numerator and
denominator. When a rate is calculated, the chi-square
test is used.
– For example if the annual roll up for the baseline year
had 20 clients retained out of 42 total clients, the
retention rate would be 20/42 = 0.47x100 = 47.6 %.
– If in the annual roll up first re-measurement had 60
clients retained out of 78 total clients, the retention rate
would be 60/78 = 0.76.9x100 = 76.9%.
Chi-square Test (continued)
• How is the retention rate numerator and
denominator used in a chi-square test?
Hypothetical Retention Rate data showing chi-square test
Time Frame (X)
Retained Client (Y) Baseline R1
Yes (numerator)
20
60
No
22
18
Total (denominator) 42
78
Total
80
40
120
Chi-square Test (continued)
• Website recommendation for a chi-square test.
– http://faculty.vassar.edu/lowry/tab2x2.html
X
0
1
Totals
Expected Cell
Frequencies per
Null Hypothesis
1
20
60
80
28
52
0
22
18
40
14
26
Y
Totals
42
78
120
Time Frame (X)
Retained Client (Y) Baseline R1
Yes (numerator)
20
60
No
22
18
Total (denominator) 42
78
Note: Only enter numbers with arrows
Total
80
40
120
Chi-square Test (continued)
• Chi-square value = 10.55
• P-value = 0.001162
Chi-Square
Phi
Yates
+0.3
9.27
P
0.002329
Pearson
10.55
0.001162
Chi-square Test (continued)
• How do I interpret the results of the statistical test?
– If the p-value is less than 0.05, then the change in
retention rates is statistically significant.
– Since the p-value is equal to 0.001162 which is less
than 0.05, the conclusions is that there is a
statistically significant improvement in the annual
roll up retention rates between the baseline
(47.6%) and first re-measurement (76.9%).
Questions
?