Calculations of Reliability We are interested in calculating the ICC

Transcript Calculations of Reliability We are interested in calculating the ICC

Calculations of Reliability

We are interested in calculating the ICC – First step: Conduct a single-factor, within-subjects (repeated measures) ANOVA – This is an inferential test for systematic error – All subsequent equations are derived from the ANOVA table

Trial A1 146 148 170 90 157 156 176 205 156 + 33 Trial A2 140 152 152 99 145 153 167 218 153 + 33  -6 + 4 - 18 + 9 - 12 + 3 - 9 + 13 Trial B1 166 168 160 150 147 146 156 155 156 + 8 Trial B2 160 172 142 159 135 143 147 168 153 + 13  - 6 + 4 - 18 + 9 - 12 - 3 - 9 + 13

Repeated Measures ANOVA

Steps for calculation: 1. Arrange the raw data (X) into tabular form, placing the data for subjects in rows (R), and repeated measures in columns (C).

Sub # 1 2 3 4 5 6 7 8 ΣC Mean ΣX T ΣX 2 Σ(ΣR) 2 N k Trial A1 146 148 170 90 157 156 176 205 (Trial A1) = ΣΣR 2 Trial A2 140 152 152 99 145 153 167 218 (Trial A2) 2 ΣR ΣΣR (ΣR) 2 Σ(ΣR) 2

Repeated Measures ANOVA

Steps for calculation: 2. Square each value = (Trial A1) 2 3. Calculate the row totals ( ΣR) using the original scores 4. Calculate the column totals ( ΣC) using the original scores 5. Calculate the grand total ( ΣX T ) or ( ΣΣR) – same thing.

Sub # 1 2 3 4 5 6 7 8 ΣC Mean ΣX T ΣX 2 Σ(ΣR) 2 N k Trial A1 146 148 170 90 157 156 176 205 1248 (Trial A1) 21316 21904 28900 8100 24649 24336 30976 42025 = ΣΣR 2 Trial A2 140 152 152 99 145 153 167 218 1226 (Trial A2) 2 19600 23104 23104 9801 21025 23409 27889 47524 ΣR 286 300 322 189 302 309 343 423 ΣΣR (ΣR) 2 Σ(ΣR) 2

Repeated Measures ANOVA

Steps for calculation: 6. Sum the row totals = ΣΣR – This is also the “total sum” of all original scores.

– Also = ΣX T 7. Square each row total = ( ΣR) 2 8. Sum the squares of the row totals = Σ(ΣR) 2

Sub # 1 2 3 4 5 6 7 8 ΣC Mean ΣX T ΣX 2 Σ(ΣR) 2 N k Trial A1 146 148 170 90 157 156 176 205 1248 156 2474 (Trial A1) 21316 21904 28900 8100 24649 24336 30976 42025 25275.75

= ΣΣR 2 Trial A2 140 152 152 99 145 153 167 218 1226 153.25

(Trial A2) 2 19600 23104 23104 9801 21025 23409 27889 47524 24432 ΣR 286 300 322 189 302 309 343 423 2474 ΣΣR (ΣR) 2 81796 90000 103684 35721 91204 95481 117649 178929 794464 Σ(ΣR) 2

Repeated Measures ANOVA

Steps for calculation: 9. Compute the mean values for each column.

Sub # 1 2 3 4 5 6 7 8 ΣC Mean ΣX T ΣX 2 Σ(ΣR) 2 N k Trial A1 146 148 170 90 157 156 176 205 1248 156 2474 (Trial A1) 21316 21904 28900 8100 24649 24336 30976 42025 25275.75

= ΣΣR 2 Trial A2 140 152 152 99 145 153 167 218 1226 153.25

(Trial A2) 2 19600 23104 23104 9801 21025 23409 27889 47524 24432 ΣR 286 300 322 189 302 309 343 423 2474 ΣΣR (ΣR) 2 81796 90000 103684 35721 91204 95481 117649 178929 794464 Σ(ΣR) 2

Repeated Measures ANOVA

Steps for calculation: 10. Sum the squares of columns = Σ(Trial A1) 2 11. Sum the sum of squared columns = Σ(Σ(Trial A1) 2 ) – This is also referred to as ΣX 2 . 12.

Σ(ΣR) 2 was calculated in step 8.

13. N = the number of subjects.

14. k = the number of trials.

Sub # 1 2 3 4 5 6 7 8 ΣC Mean ΣX T ΣX 2 Σ(ΣR) 2 N k Trial A1 146 148 170 90 157 156 176 205 1248 156 2474 397662 794464 8 2 (Trial A1) 2 21316 21904 28900 8100 24649 24336 30976 42025 202206 44934.667

= ΣΣR Trial A2 140 152 152 99 145 153 167 218 1226 153.25

(Trial A2) 2 19600 23104 23104 9801 21025 23409 27889 47524 195456 24432 ΣR 286 300 322 189 302 309 343 423 2474 ΣΣR (ΣR) 2 81796 90000 103684 35721 91204 95481 117649 178929 794464 Σ(ΣR) 2

Repeated Measures ANOVA

Steps for calculation: 15. Compute the sum of squares between columns (SS C ), which is the variability due to the repeated-measures treatment effect.

– In this case, SS C is “systematic variability.”

SS C

 ( 

1 ) 2 

( 

2 ) 2  ( 

X T

) 2 (

)(

)

SS C

 ( 1248 ) 2  ( 1226 ) 2 8  ( 2474 ) 2 ( 8 )( 2 )

SS C

 30 .

Repeated Measures ANOVA

Steps for calculation: 16. Compute the sum of squares between rows (SS R ), which is the variability due to differences among subjects.

SS R



k R

) 2  ( 

X T

) 2 (

)(

)

SS R

 794464  2 ( 2474 ) 2 ( 8 )( 2 )

SS R

 14689 .

Repeated Measures ANOVA

Steps for calculation: 17. Calculate the total sum of squares (SS T ), which is the variability due to subjects (rows), treatment (columns), and unexplained residual variability (error).

SS T

 

2  ( 

X T

) 2 (

)(

)

SS T

 397662  382542 .

SS T

 15119 .

Repeated Measures ANOVA

Steps for calculation: 18. Calculate the total sum of squares due to error (SS E ), which is the unexplained variability due to error. This will be used in the denominator for the F ratio.

SS E



SS T



SS C



SS R SS E

 15119 .

75  30 .

25  14689 .

SS E

 399 .

Repeated Measures ANOVA

Steps for calculation: 19. Calculate the degrees of freedom for each source of variance (df C , df R , df E , and df T ).

df C



 1

df C

 2  1

df C

 1

df R



 1

df E

 (

 1 )(

 1 )

df T

 (

)(

)  1

df R

 8  1

df E

 ( 2  1 )( 8  1 )

df T

 ( 8 )( 2 )  1

df R

 7

df E

 7

df T

 15

Repeated Measures ANOVA

Steps for calculation: 20. Construct an ANOVA table: Between Subjects = rows Trials = columns Within Subjects = Trials + Error

MS F p (sig.) Source

Between Subjects Within Subjects Trials Error Total

7 8 1 7 15

14,689.75

430 30.25

399.75

15,119.75

Repeated Measures ANOVA

Steps for calculation: 21. Calculate the mean square for each source of variance (MS C , MS R , and MS E ).

MS C



SS C df C MS R



SS R df R MS E



SS E df E MS MS MS C R E

   30 .

25 2098 .

54 57 .

Source

Between Subjects Within Subjects Trials Error Total

7 8 1 7 15

14,689.75

430 30.25

399.75

15,119.75

2098.54

53.75

30.25

57.11

F p (sig.)

Repeated Measures ANOVA

Steps for calculation: 22. Calculate the F ratio for the treatment effect (columns, F C ).

F W ithin



subjects



MS C MS E F Between



subjects



MS MS E R F W ithin



subjects

 0 .

F Between



subjects

 36 .

Source

Between Subjects Within Subjects Trials Error Total

7 8 1 7 15

14,689.75

430 30.25

399.75

15,119.75

2098.54

53.75

30.25

57.11

36.75

p (sig.)

0.53

Repeated Measures ANOVA

– Determining the Significance of F: Use the F Distribution Critical Values table.

df C df E = df B – columns across the top = df E – rows down the side – If your calculated F ratio is greater than the critical F ratio, then reject the null hypothesis.

There is a significant difference from Trial A1 to Trial A2 There is a significant systematic error – If your calculated F ratio is less than the critical F ratio, then accept the null hypothesis.

There is no difference from Trial A1 to Trial A2 There is no systematic error

Source

Between Subjects Within Subjects Trials Error Total

7 8 1 7 15

14,689.75

430 30.25

399.75

15,119.75

2098.54

53.75

30.25

57.11

36.75

p (sig.)

< 0.05

0.53

> 0.05

Source

Between Subjects Within Subjects Trials Error Total

7 8 1 7 15

14,689.75

430 30.25

399.75

15,119.75

2098.54

53.75

30.25

57.11

36.75

0.53

p (sig.)

0.00005

0.49035

Using ANOVA Table for ICC

2 sources of variability for ICC model 3,1 – Subjects (MS S ) Between-subjects variability (for calculating the ICC) – Error (MS E ) Random error (for calculating the ICC)

ICC

3 , 1 

MS MS S



(



MS E

 1 )

MS E

Equation reported by Weir (2005)

ICC

3 , 1 

MS MS



R R

(

 

1 )

E MS E

Same equation, but modified for our terminology (MS S = MS R ).

MS

or MS

Source

Between Subjects Within Subjects Trials Error Total

7 8 1 7 15

14,689.75

430 30.25

399.75

15,119.75

2098.54

53.75

30.25

57.11

36.75

0.53

p (sig.)

0.00005

0.49035

MS

Using ANOVA Table for ICC

Calculating the ICC 3,1 :

ICC

3 , 1 

MS MS



R R

(

 

1 )

E MS E ICC

3 , 1  2098 .

54  57 .

11 2098 .

54  ( 2  1 ) 57 .

ICC

3 , 1  0 .

947

Interpreting the ICC

If ICC = 0.95

– 95% of the observed score variance is due to true score variance – 5% of the observed score variance is due to error 2 factors for examining the magnitude of the ICC – Which version of the ICC was used?

– Magnitude of the ICC depends on the between subjects variability in the data Because of the relationship between the ICC magnitude and between-subjects variability, standard error of measurement values (SEM) should be included with the ICC

Implications of a Low ICC

Low reliability Real differences – Argument to include SEM values Type I vs. Type II error – Type I error is rejecting H 0 (i.e., H 0 = 0) when there was no effect – Type II error is failing to reject the H 0 effect (i.e., H 0 ≠ 0) when there is an A low ICC means that more subjects will be necessary to overcome the increased percentage of the observed score variance due to error.

Standard Error of Measurement

ICC  relative measure of reliability – No units SEM  absolute index of reliability – Same units as the measurement of interest The SEM is the standard error in estimating observed scores from true scores.

Calculating the SEM

Calculating the SEM 3,1 :

SEM



MS E SEM

 57 .

SEM

 7 .

SEM

We can report SEM values in addition to the ICC values and the results of the ANOVA We can calculate the minimum difference (MD) that can be considered “real” between scores

Minimum Difference

The SEM can be used to determine the minimum difference (MD) to be considered “real” and can be calculated as follows:

MD MD

 

SEM

7 .

 1 .

96 56  1 .

96     2

 20 .

Example Problem

Now use your skills (by hand) to calculate a repeated measures ANOVA, ICC 3,1 , SEM 3,1 , and MD 3,1 for Trials B1 and B2.

– Report your results.

– Compare your results to Trials A1 and A2.

What is the primary difference?

Using the Reliability Worksheet Online

Go to the course website and download the Reliability.xls worksheet.

– Calculate the ANOVA, ICC, SEM, and MD values for both Trials A1 and A2 and Trials B1 and B2 and compare your results.

Calculations of Reliability We are interested in calculating the ICC

Transcript Calculations of Reliability We are interested in calculating the ICC