Transcript Revilla
How to evaluate the cross-cultural equivalence of single items Melanie Revilla, Willem Saris RECSM, UPF Zurich – 15/16 July Cross-cultural equivalence • Usually – Discussed in the frame of cross-national research – Idea: in different countries people can express themselves in different ways – Different cultures can also be defined on other criteria (e.g. language) – Procedure: can be applied in similar way to all kinds of different groups • “Equivalence” measurement equivalence – 2 persons with the same opinion will give the same answer (whatever their group) • Important – because observed differences may result from non equivalent measures and not be real differences • If measurement equivalence does not hold – cannot make comparison across groups!! Important distinction (Northrop, 1947) • Concept by Intuition (CI) – Simple concepts that can be measured directly – Single item – Ex: trust in the parliament • Concept by Postulation (CP) – – – – Complex concepts that cannot be measured directly Also called “construct” Need several CI to measure them Ex: political trust: trust in parliament + legal system + police … • Classic procedure to test for equivalence for CP but not for CI start with a reminder of the procedure for CP When we have multiple indicators CP / complex concepts Basic Confirmatory Factor Analysis model λ11 λ21 CP1 Y1 Y2 λ31 Political trust Y3 intercepts slopes ≈ regression equation [Y i τ1 Answer Trust in the parliament e1 τ2 e2 τ3 Answer Trust in the police e3 error terms = τi + λij CP1 + ei Dependent variable Answer Trust in the legal system Independent variable i = 1,2,3 Multiple Group CFA approach Group 1 Group 2 • Multiple group: – possible to test for equality of the parameters in the different groups – constraints across groups • Can be extended to more groups Different levels of invariance (Meredith, 1993) • Configural Group 1 – Same model holds in all groups • Metric – Configural + Slopes (λij) the same in all groups – Sufficient for comparison of relationships • Scalar – Metric + Intercepts (τi) the same in all groups – Sufficient for comparison of means • More: error terms, etc… Group 2 In practice • Analyses can be done with standard SEM softwares – – – – – LISREL/Mplus based on covariance matrices & means recommended sample size: >200 in each group 3-step procedure: configural, metric, scalar syntax quite easy to get estimates • More tricky but crucial step: testing Testing the model • Assessing global fit – Chi2 test / Fit indices: RMSEA (<.05), CFI (>.9), etc… – Limits: Depends on sample size / Sensitive to deviations from normality • Assessing local fit – Saris & Satorra should test at the parameter level + take into account type II errors (H0 not rejected despite being false) – JRule software (van der Veld, Saris, Satorra) + Jrule for Mplus (Oberski) • See next presentation! • Always check if estimates are really different – Difference may be statistically significant but not substantially meaningful • Partial invariance – What if some indicators are equivalent but not all? – Consistent estimates of the means of the latent variables if at least 2 indicators are scalar invariant (Byrne, Shavelson, Muthén, 1989) If we have single indicator CI / simple concepts Single items Group 1 CI1 λ11 Y1 Group 2 τ1 CI1 e1 Yi = τi + λij CIi + ei • Testing equivalence single items • Testing equivalence for CI λ11 Y1 τ1 e1 Single multiple indicators? • Problem: model just presented not identified • “Single indicator” = single trait in fact • But possible to use multiple methods • So for CI: – Only one trait, but we can always have more than one method – Several indicators = same trait asked using different methods Can apply again MGCFA Group1 Group 2 11 points 6 points Trust in the parliament 4 points • Similar at the previous model (for CP) but now different methods instead of different traits measuring a same concept Same procedure • Different levels of invariance as for CP – – – – Configural Metric Scalar etc • Same procedure to get the estimates and test the model – Multiple group analyses – Test of the model: global / local fit – Partial equivalence Problem for the CI • Fix the scale? – – – – – – As before, necessary to fix the scale of the LV Usually, fix the first loading to 1 Can be done here too Other loadings are relative to the first one But need to be done in all groups If there are differences for the method whose loading is fixed to 1 across groups, may be problematic – Should try to use methods that have been shown to be the most similar across groups: e.g. fixed reference points General model u1 α1 v21 CI1 Y21 v31 c1 Y31 u2 CP1 Y11 1 c2 α2 CI2 1 v22 Y12 Y22 v32 Y32 c3 u3 α3 CI3 1 v23 v33 Y13 Y23 Y33 τ11 e11 τ21 e21 τ31 e31 τ12 e12 τ22 e22 τ32 e32 τ13 e13 τ23 e23 τ33 e33 • Even when working on CP: better to use different methods Final remarks / CCL Equivalence single items • Need to repeat the same item with different methods – 3 or more repetitions – Multi Methods (MM)? • Same persons get the question several times using different methods • Limit: 20 minutes at least to avoid memory effects (Van Meurs & Saris, 1990) – Mix with Split-Ballot (SB) design? • Random assignment of respondents to different versions of the questionnaire • “SB-MM” (CI) or SB-MTMM (CP)? Conclusion • Measurement equivalence can be assessed both for CP and CI using Multiple Group Confirmatory Factor Analysis – For CP, process already well-known and used a lot – For CI, possible do similar analyses But necessary to repeat questions!! specific data • So testing equivalence of simple item can be done using a (SB)-(MT)MM approach – Similar to what exists in the ESS for CP: main + supplementary questionnaires (different versions) – With extension for concepts by intuition In summary • To test single item equivalence – Use multiple methods – Do everything as for multiple items equivalence Thank you for your attention! Questions?