Transcript Slide 1
Survey Research Operations Survey Research Center Institute for Social Research Cross-Cultural Survey Guidelines and Quality Monitoring Beth-Ellen Pennell 2009 International Total Survey Error Workshop (ITSEW 2009) Tällberg, Sweden Unique Challenges – Locating and engaging respondents ITSEW 2009 Beth-Ellen Pennell 2 Literacy A dult literac y rates by developmental s tatus , 2005-2007 100 L iterac y rate 95 90 Total 85 Male F emale 80 75 70 W orld Developed c ountries C ountries in trans ition Developing c ountries A dult literac y rates by reg ion, 2005-2007 UNE S C O reg ions 92.2 93.5 92.8 O c eania 98.9 99.4 99.1 E urope 76.3 A s ia 82.1 87.8 91.9 92.6 92.3 95.6 96.6 96.1 S outh A meric a North A meric a 54.8 A fric a 79.4 W orld 83.9 10 20 30 40 50 60 Male Total 73 63.6 0 F emale 70 80 88.5 90 100 L ite ra c y ra te ITSEW 2009 Beth-Ellen Pennell 3 Gatekeepers/Privacy From the Institute of Social Research’s Population and Ecology Laboratory in Nepal ITSEW 2009 Beth-Ellen Pennell 4 Infrastructure *From the International Telecommunications Union (http://www.itu.int/ITU-D/ict/statistics/ict/index.html) ITSEW 2009 Beth-Ellen Pennell 5 Unique Data Collection Challenges • Other: – Research traditions – Languages – Seasonal – Political – Religious – Geographical ITSEW 2009 Beth-Ellen Pennell 6 Standardization issues • Concentrate on key design aspects • Decide when to be rigid and when to be flexible • Experience is that adherence to standards and regulations must be checked • ISO 20252 What is special about quality in an international setting? • • • • • • • Procedural equivalence is important Concepts must have a uniform meaning Scientific and administrative challenge Risk management differs Financial and methodological resources differ National pride is at stake Conflicts of interest Examples of Quality Assurance • • • • • • • • • • • Central planning and support organization Deep bench of senior experts Up-to-date translation procedure Pretesting of questions and questionnaires Interviewer training Probability sampling design Call scheduling algorithm Formulas for calculating base weights Documentation system User communication channels A set of operational specifications Deviations from specifications Examples from 1995 IALS • Average interviewer workload varied between 6 and 30 • Two countries chose sampling control instead of 100% keystroke validation • One country did not calculate the base weights correctly • One country informed respondents that the survey was just a pretest Evaluations and peer reviews • Recently ESS • Examples of recommendations: – Develop quantitative indicators for all process steps – Standardize contact forms – Set bounds on effective sample size – Improve capacity building – Expand the user base Some thoughts • For some countries it is a challenge to reach minimum standard • Process stability is difficult to obtain in decentralized survey environments • Survey organizations must be aware of the meaning of specifications and the effects of certain methodological choices • Reasons for deviations must be checked • Vigorous monitoring and performance checks necessary ITSEW 2009 Beth-Ellen Pennell 13 Cross-Cultural Survey Guidelines Contributors and Reviewers Contributors: Reviewers: • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • Kirsten Alcser, UM-SRC Ipek Bilgen, UNL Ashley Bowers, UM-SRC Rachel Caspar, RTI Judi Clemens, UM-SRC Peter Granda, UM-ICPSR Sue Ellen Hansen, UM-SRC Janet Harkness, UNL Frost Hubbard, UM-SRC Rachel Levenstein, UM-SRC Christina Lien, UM-SRC Zeina Mneimneh, UM-SRC Rachel Orlowski, UM-SRC Beth-Ellen Pennell, UM-SRC Emilia Peytcheva, UM-SRC Ana Villar, UNL Graphic design assistance • Larry LaFerte, UM-SRC • Ruth Shamraj, UM-ICPSR Formatting and copy-editing • Gail Arnold, UM-SRC • Shaw Hubbard - independent consultant Dorothee Behr, Gesis Bill Blyth, TNS Europe Pam Campanelli, independent consultant Somnath Chatterji, WHO Rory Fitzgerald, European Social Survey Steve Heeringa, UM-SRC Tim Johnson, University of Illinois, Chicago Achim Koch, Gesis Frauke Kreuter, University of Maryland Paul Kussumaul, European Social Survey Kristin Miller, National Center for Health Statistics Peter Mohler, University of Mannheim Meinhard Moschner, Gesis José L. Padilla, University of Granada Alisú Schoua-Glusberg, Research Support Services Eleanor Singer, UM-SRC Tom W. Smith, NORC Jare Struwig, Human Sciences Research Council Rick Valliant, University of Maryland Gordon Willis, National Institutes of Health Christine Wilson, Heriot-Watt University Christof Wolf, Gesis Programming and website maintenance • Tricia Blanchard, UM-SRC ITSEW 2009 Beth-Ellen Pennell 14 Goal • To develop and promote internationally recognized guidelines that highlight best practice for the conduct of comparative survey research across cultures and countries • Initiative of Comparative Survey Design and Implementation (CSDI); 2005 annual meeting ITSEW 2009 Beth-Ellen Pennell 15 Guidelines Initiative Initiative in response to: • Increasing number and scope of cross-cultural surveys over past decade • Desire to increase operational equivalence and survey quality through harmonization – Within and across “units” (e.g. countries) – Across waves of panel study • Lack of published materials on implementation • Balance standardization versus localization ITSEW 2009 Beth-Ellen Pennell 16 Target Audience • Researchers and survey practitioners planning or engaged in cross-cultural or cross-national research – Basic to advanced information – References – Suggested further reading ITSEW 2009 Beth-Ellen Pennell 17 Process • Developed over two and a half years • Weekly meeting of core staff • Each guideline underwent iterative, internal reviews • Sent to selected external reviewers with expertise in topic area • Published last summer • Revised last fall ITSEW 2009 Beth-Ellen Pennell 18 Guideline Topics I. II. III. IV. V. VI. VII. VIII. IX. X. XI. XII. XIII. XIV. XV. Study structure Tenders, bids, and contracts Ethical considerations Sample design Questionnaire design (in development) Translation Adaptation (in development) Survey instrument design Pretesting Interviewer recruitment and training Data collection Harmonization of data Data processing Dissemination Assessing quality ITSEW 2009 Beth-Ellen Pennell 19 Components of Guidelines • • • • • • • • • Introduction to topic area Goal of guideline Guideline Rationale Procedural steps Lessons learned Glossary References Further suggested reading ITSEW 2009 Beth-Ellen Pennell 20 Format of Guidelines • “Drill-down” approach – Increasing level of detail • Links available to – Glossary – References – Other modules – External information ITSEW 2009 Beth-Ellen Pennell 21 22 High Quality Framework Low Cost Burden Design Constraints Professionalism Fitness for Use Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Relevance Accuracy Timeliness Accessibility Interpretability Coherence Comparability ITSEW 2009 Beth-Ellen Pennell 23 Quality Framework Accuracy Total Survey Error Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Cost Burden Design Constraints Professionalism Construct Validity Measurement Error Processing Error Coverage Error Sampling Error Nonresponse Error Adjustment Error ITSEW 2009 Beth-Ellen Pennell 24 Quality Framework Across the Study Lifecycle Quality Monitoring and Control Quality Planning and Assurance Quality Profile Study Structure Tenders, Bids, and Contracts Regulations, standards, guidelines Monitoring quality indicators Process analysis Statistical process control Ethical Considerations Questionnaire Design Translation Adaptation Survey Instrument Design Process improvement plan Process analysis Quality indicators and checklists Recommended corrective actions Quality management plan Recommended preventive actions Performance measurement analysis Pretesting Quality audits Interviewer, Recruitment, & Training Data Collection Harmonization of Data Updates to standards, best practices, and quality management plan Recommendations for quality improvement Documentation of study design, process protocols Final indicators of process quality Final indicators of survey statistic quality, focusing on potential sources of error Data user satisfaction index Lessons learned Recommendations for quality improvement Updates to standards, best practices, and quality management plan Data Processing Dissemination ITSEW 2009 Beth-Ellen Pennell 25 Summary • Website has had~105,214 hits since published in June, 2008 • ~288 hits per day • Evolving and dynamic: feedback and comments welcome • Provides framework for quality control monitoring ITSEW 2009 Beth-Ellen Pennell 26 Thank you. http://ccsg.isr.umich.edu/ ITSEW 2009 Beth-Ellen Pennell 27