Transcript The Inter-rater Reliability and Intra-rater Reliability

The Inter-rater Reliability and Intra-rater
Reliability of Bedside Ultrasounds of the
Femoral Muscle Thickness
Daren K. Heyland, MD, MSc, FRCPC
Professor of Medicine
Queen’s University, Kingston General Hospital
Kingston, Ontario
A Randomized Trial of Supplemental Parenteral Nutrition
in Under and Over Weight Critically Ill Patients:
The TOP UP Trial
Hypothesis:
Increased energy and protein delivery to underweight and overweight critically ill
patients (Body Mass Index [BMI] <25 or >35) will result in improved 60 day survival
compared to usual care.
• Multicenter pilot study
• Randomized trial of 160 critically ill adult patients from 8 tertiary care ICU’s in
Canada, United States, and Europe.
• Patients randomized to one of 2 interventions: enteral nutrition (EN) alone or enteral
nutrition plus parenteral nutrition (supplemental PN group).
• Patients stratified on the basis of admission BMI: <25 or >35, medical or surgical
admission diagnosis, by site and if EN was administered between ICU admission and
randomization
Primary outcome: 60 day mortality.
Secondary outcomes: 28 day mortality, hospital mortality, duration of stay (ICU and
hospital), multiple organ dysfunction (SOFA and PODS), duration of mechanical
ventilation, development of ICU acquired infections, functional status at hospital
discharge, and 3 and 6 month survival and health-related quality of life.
Weekly Ultrasounds
• There is emerging evidence that muscle mass and muscle function predict
morbidity in surviving patients and that muscle mass at ICU admission may
predict length of hospital stay.
• We propose to evaluate the effect of differential amounts of protein and
energy provided to study patients on muscle mass and function.
• We can postulate that the beneficial effect of enhanced energy and protein
provision is mediated by the preservation (or attenuated deterioration) of
muscle mass and increased function in these better fed patients, which
would ultimately result in positive outcomes.
• We will evaluate muscle mass in all study patients using non-invasive
bedside ultrasound of the femoral muscle
Testing Ultrasound Reliability before TOP-UP
Before we performed weekly ultrasounds on the study population, we conducted a
trial of the ultrasound protocol to allow us to:
• standardize the training of all Study Investigators performing the ultrasound
assessment
• test the feasibility of the procedures
• determine ‘normal’ values to which we can compare our measures in the study
population
• determine intra-rater (trainer) and the inter-rater (trainee) reliability.
The ultrasound reliability trial involved:
• the 7 participating TOP UP sites
• healthy volunteers
• a standardized protocol:
o ultrasound trainers were to perform ultrasounds twice on each patient
o a trainee repeated the measurement on the same patient
The objective was to evaluate the inter-rater reliability and intra-rater reliability of
bedside ultrasounds of the femoral muscle measuring muscle thickness in healthy
volunteers before using this tool in detailing overall muscle mass in ICU patients.
Reliability Protocol
1) Trainer to complete a 2/3rd and midpoint ultrasound on each leg
2) Trainer to repeat a 2/3rd and midpoint ultrasound on each leg (intra-rater reliability test)
3) Trainee to complete a 2/3rd and midpoint ultrasound on each leg (inter-rater reliability test)
The overall muscle thickness was calculated as the average of the readings measured at the border
between the lower third and upper two-thirds between Anterior Superior Iliac Spine (ASIS) and upper pole
of the patella as well as the reading at the midpoint between the ASIS and the upper pole of the patella
averaged over the right and left legs.
Volunteers’ Demographics
Variable
N
Mean
Std Dev
Minimum
Maximum
Age
64
30.6
8.4
21.0
55.0
Height
78
171.4
11.5
135.0
196.0
Weight
78
71.2
16.4
46.0
136.4
BMI
78
24.1
4.4
16.9
40.7
Intra-rater Reliability Results
Site
Erasme University Hospital,
Brussels, Belgium
Subjects Between Subject Variance Within Subject Variance
10
0.2307
0.01380
NA
ICC
0.94
Grey Nuns Hospital, Edmonton,
Alberta, Canada
0
NA
Royal Alexandra Hospital,
Edmonton, Alberta, Canada
10
0.2425
0.001018 >0.99
Nouvel Hôpital Civil, Strasbourg,
France
4
0.2567
0.000199 >0.99
University of Alberta Hospital
Edmonton, Alberta, Canada
12
0.05866
University of Colorado, Aurora,
Colorado, USA
5
0.2869
0.000613 >0.99
University of Texas Health Science
Center, Houston, Texas, USA
5
0.1774
0.003837
0.98
Pooled
46
0.2648
0.004554
0.98
0.003180
NA
0.95
ICC: Intra-class correlation coefficient, ICC = between subject variance / (between subject variance +within subject variance)
Intra-rater Reliability Results (Continued)
• The paired t-test was used examine the average difference between the first and second
trainer measurements.
• The paired profile plot visualizes the difference in the paired measures.
Mean difference between trainer measurement 1 and trainer measurement 2
Trainer measurement 1
Trainer measurement 2
The mean difference between trainer measurements (95% CI) = 0.037 (0.010 to 0.063) p=0.0077
Inter-rater Reliability Results
Site
Erasme University Hospital,
Brussels, Belgium
Subjects Between Subject Variance Within Subject Variance
10
0.2194
0.02900
ICC
0.88
Grey Nuns Hospital, Edmonton,
Alberta, Canada
13
0.3217
0.000769 >0.99
Royal Alexandra Hospital,
Edmonton, Alberta, Canada
10
0.2305
0.02072
0.92
Nouvel Hôpital Civil, Strasbourg,
France
18
0.2623
0.009972
0.96
University of Alberta Hospital
Edmonton, Alberta, Canada
12
0.03587
0.01360
0.73
University of Colorado, Aurora,
Colorado, USA
5
0.1714
0.02746
0.86
University of Texas Health Science
Center, Houston, Texas, USA
5
0.1704
0.03334
0.84
Pooled
73
0.2584
0.01580
0.94
ICC: Intra-class correlation coefficient, ICC = between subject variance / (between subject variance +within subject variance)
Inter-rater Reliability Results (Continued)
• The paired t-test was used examine the average difference between the first trainer and
the trainee measurement.
• Paired profile plots are provided to visualize the difference in the paired measures.
Mean difference between trainer measurement and trainee measurement
Trainer
Trainee
There was a small but statistically significant difference between the trainer and trainee results:
Mean (95% CI) = -0.061 cm (-0.100 to -0.022), p= 0.0028
Conclusion
• There is excellent inter and intra-rater reliability for ultrasound
measurements of the femoral muscle to determine overall
muscle thickness in healthy volunteers.
• A sample of ‘normal’ values is now available to compare
measures from a study population.
• Further evaluation of this technique must be validated in
critically ill patients.
• Efforts to link the ultrasound measurements to ICU outcomes
should be undertaken.
Questions