Transcript Document
Survey Design and Measurement Jeremy Kees, Ph.D. Some practical issues…. • Qualtrics Research Platform – Free you under VSB’s “site license” – Extremely user friendly, but also very robust – www.qualtrics.com Some practical issues…. • Amazon Mechanical Turk – The most inexpensive way to collect consumer data – Extremely user friendly, but also very robust – www.mturk.com Online Survey (created by you and housed on Qualtrics’ server) Create HIT (Human Intelligence Task) on Mturk • Description of your study and a (Qualtrics) link to it Mturk workers (survey responders) “work” on your HIT (i.e., they take your survey) Data is recorded by Qualtrics. Participants who complete the survey are given a code to input into Mturk. Those that enter a valid code, get paid. Everyone is happy The Mturk data I collected today…. • • • • N=200 Cost = $100 Data collected in less than 1 hour Demographics – – – – Mean age = 36 56% male 76% Caucasian 80% at least some college • 41% are college grads – Median income = $35-50k – Highly engaged! Stages in the Research Process Formulate Problem Determine Research Design Design Data Collection Method and Forms Design Sample and Collect Data Analyze and Interpret the Data Prepare the Research Report Surveys / Questionnaires • The most common measurement instrument when quantitative data is sought – – – – Descriptive research Experiments Modeling Etc…. Developing Surveys • Good, well-specified research objectives lead to good surveys • Research design dictates what types of questions should be used – Exploratory research = unstructured script – Confirmatory research = structured survey Desirable Characteristics • • • • Brief Objective Specific Relevant Survey Methods • Usually should determine administration method prior to developing items – Can dictate what types of questions you should ask • Internet panels have become the most efficient and versatile method to collect data – Phone is still a viable option – Mall intercepts can still be useful – Mail/fax makes little sense anymore Mall Intercepts vs. E-Panels household size average age employed white male college mall tests 2.8 40.5 71% 86% 20% 40% internet tests 2.9 39.2 72% 88% 21% 43% panel members 3.0 37.2 69% 89% 15% 46% Correlation between Responses: purchase intent frequency liking price / value mall vs. internet .86 .94 .85 .90 internet test/retest reliability .94 .97 .91 .99 E-Panels vs. Phone Time survey took to administer Internet 12.5 Phone 19.4 minutes Upon completion, would respondent participate in future studies? 35% yes 26% yes More experienced Internet Users x Used rating scale extreme “endpoints” more frequently Jeff Miller and Alan Hogg “Internet vs. Telephone Data Collection” Burke White Paper series 2 (4) (www.burke.com). Also see Ashok Ranchhod and Fan Zhou “Comparing Respondents of E-Mail and Mail Surveys,” Marketing Intelligence & Planning 19 (2001), 254. x Types of Questions • • • • Screening Variables Independent Variables Dependent Variables Classification Variables – Segmentation – Moderators • Attention Filters Primary Data: Overview • Types of Primary Data – – – – – – – Demographic / Socioeconomic Characteristics Psychological / Lifestyle Characteristics Attitudes / Opinions Awareness / Knowledge Intentions Motivation Behavior • What, how much, where, when, how, who • Purchase behavior vs. use behavior • E.g., --- basic hierarchy of effects models • Example (CWL Study) QUESTION WORDING - General Guidelines • • • • • Use simple words and questions Avoid ambiguous words and questions Avoid leading questions---be objective Avoid implicit alternatives Avoid generalizations and estimates --Be specific • Avoid double-barreled questions 1. What is your income? $10,000 or less………………….1 $10,000 to $25,000……………..2 $25,000 to $50,000……………..3 $50,000 to $75,000………..…….4 $75,000 to $100,000..……..…….5 $100,000 or more…………..……6 What is the problem and how would you revise the question? 4. Is the speed and efficiency of the drive-in teller services at your regular bank…..(READ CATEGORIES) Very Satisfactory………………………4 Somewhat satisfactory…………………3 Somewhat unsatisfactory………………2 Very unsatisfactory……………….……1 Question Wording • It is good practice to use scales whenever possible – Likert or semantic differential – Multi-item Itemized Rating Scales • The respondents are provided with a scale that has a number or brief description associated with each category. • The categories are ordered in terms of scale position, and the respondents are required to select the specified category that best describes the object being rated. • The commonly used itemized rating scales are the Likert and semantic differential Types of Scales • Nominal scales: those that use only labels • Ordinal scales: those with which the researcher can rank-order the respondents or responses • Interval scales: those in which the distance between each descriptor is equal • Ratio scales: ones in which a true zero exists Examples… Nominal Ordinal Which of the soft drinks in the following list do you like? (Check ALL that apply): ___Coke ___Dr. Pepper ___Mountain Dew ___Pepsi ___Seven Up ___Sprite Rank the soft drinks according to how much you like each (most preferred drink = 1, and least preferred drink = 6): ___Coke ___Dr. Pepper ___Mountain Dew ___Pepsi ___Seven Up ___Sprite Interval Ratio Please indicate how much you like each soft drink by checking the appropriate position on the scale: dislike like a lot dislike like a lot Coke ____ ____ ____ ___ Dr. Pepper ____ ____ ____ ___ Mountain Dew ____ ____ ____ ___ Pepsi ____ ____ ____ ___ Seven Up ____ ____ ____ ___ Sprite ____ ____ ____ ___ Please divide 100 points among these soft drinks To represent how much you like each: ___Coke ___Dr. Pepper ___Mountain Dew ___Pepsi ___Seven Up ___Sprite 100 21 Itemized Rating Scales • Likert Scales – requires the respondents to indicate a degree of agreement or disagreement with each of a series of statements about the stimulus objects Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 1. Wal-Mart sells high quality merchandise. 1 2X 3 4 5 2. Wal-Mart has poor in-store service. 1 2X 3 4 5 3. I like to shop at Wal-Mart . 1 2 3X 4 5 4. Wal-Mart has low prices . 1 2 3X 4 5 Itemized Rating Scales • Semantic Differential Scales – End points associated with bipolar labels that have semantic meaning SEARS IS: Powerful --:--:--:--:-X-:--:--: Weak Unreliable --:--:--:--:--:-X-:--: Reliable Modern --:--:--:--:--:--:-X-: Oldfashioned Decisions for Itemized Scales • Number of scale items • More is better, but there is a diminishing return around 11 points (Nunnally 1978) • 7-point scales are customary • Enough to discriminate • Allows for a scale midpoint • Manageable • Odd/even number of categories • Forced vs. non-forced Why Multi-Item Scales?? Construct Abstract Concept “Unobservable” “Latent” “Psychological” **Single items are typically not sufficient to assess unobservable constructs Multi-Item Scales are More “Reliable” • True Score Test Theory – All measures have 1. 2. • • “True” Score “Error” (Random and Systematic) Good measures minimize the systematic error component of the score Types of Reliability – – – Inter-Rater Test-Retest Internal Consistency (Cronbach’s Alpha) Developing Sound Measures Step 1: Specify Domain of the Construct Step 2: Generate Sample of Items Step 3: Collect Data Step 4: Purify Measure Step 5: Assess Validity Question Sequencing • After you have developed your measures, think about the order in which they should be asked QUESTION SEQUENCING General Guidelines • Use (more) simple, interesting opening questions • Use the funnel approach, asking broad questions first, and follow with more specific questions • Carefully design branching questions – Skip/display logic – Ask for classification information last • Place more difficult or sensitive questions near the end QUESTION SEQUENCING General Guidelines Question ordering #1 1 – EVALUATION OF FAT LEVEL OF A PRODUCT 2 – EVALUATION OF OVERALL PROD. NUTRITIOUSNESS 3 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE Question ordering #2 1 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE 2 – EVALUATION OF LEVEL OF PROD. NUTRITIOUSNESS 3 – EVALUATION OF FAT LEVEL OF PRODUCT FOP Labeling Study • We were interested in consumer evaluations of: 1. Facts Up Front 2. All On-Package Labeling 3. Front-of-Package Nutrition Info Why was question sequencing critical?? Tips for Maximizing Participation • • • • • Offer an incentive ($$$) Importance/relevance of the research project and its purpose Completing the questionnaire will take only a short time Answers are anonymous or confidential Reminder 2-3 days after the initial ask Attention Filters • Always include an attention filter to ensure that you are getting “quality” respondents – Eliminate “click throughs” Attention Filters (Case Study) • Advertising Experiment – Very stringent screening criteria • Total # that started the study = 15,458 • Number that qualified = 870 – Incidence Rate (IR) = 5.6% • Number that qualified and passed the attention screener = 451 – 48% failed the attention filter!!! • NOT GOOD, criticalmix! “Easy” Attention Filter “Difficult” Attention Filter And finally, remember the golden rule…. Do unto your respondents as you would have them do unto you!! Team Assignment #2 • Refine your research questions – Need to be clear, concise, and “testable” • Based on your research questions 1. Design 2 potential studies that could address your research questions 1. 2. Explain the benefits and weaknesses of each approach Pick the “best” design and explain your decision (Note: Don’t worry about measurement or sampling too much---you’ll have your chance to do that later) Team Assignment #3 ** Don’t start on this assignment until you’ve read Fowler (CH 6-7) • Based on your research design 1. Write a paragraph about what your measurement instrument is supposed to accomplish 2. Make a list of what should be measured to accomplish the goals of the study 3. Develop your measurement instrument Team Assignment #3 • Deliverables include: 1. A very clean, polished version that you could use to actually collect data • This means you will need to carefully think through all of the issues we covered tonight (e.g., set-up, ordering, length, multi-item scales, etc.) 2. Intro paragraph and variable list (see previous slide) (Note: Don’t worry about defining your sample--you’ll have your chance to do that next week)