Surveys, questions and other cheap ways of getting data

How to do stuff 1/14/02 Michael Ramscar

Plan

• Surveys • What is correlational research?

• Sampling a population • Ways of collecting survey data • How to formulate “Good” questions • Scales • Soothing strategies

Correlational Research

• The nature of correlation – A numerical relationship between two variables – Predictions: • Will score A covary with score B?

• Can be either negative or positive • Presence of correlation allows prediction – Prediction can be tested on samples

Correlational Research

• Does drinking beer make you appear cooler, funnier and sexier?

– How do you test this as a hypothesis?

Correlational Research

• 2 ways: – Experimentally • Randomly assign people to groups • Get one to drink no beer; other drink 3 beers/day • Measure the change in their coolness in relation to pre-test – Conduct a survey • Ask people to estimate how much beer they drink • Ask people to rate them for coolness, sexiness and funniness • Find the relationship between these two variables

Surveys: Pros and Cons

• Advantages of survey research – Can sample a large number of individuals – Do not have to worry that the situation is artificial – Surveys are often easy to administer

Surveys: Pros and Cons

• Disadvantages of survey research – Danger of biased surveys • Sample may not be representative • Questions may bias responses – People may not answer truthfully/competently – Asking questions may not be the right methodology – Cannot make statements about causality

Correlation and Causality

• Correlation – A numerical relationship between two variables – One value need not cause the other • The

third variable problem - “spurious relationships”

– Does drinking beer make you cool?

• Perhaps uncool people are just too intimidated to go into bars • A Mantra: “Correlation does not equal causation!”

Sampling

The hard part of conducting a survey is getting a representative sample Population Sampling Frame Sample

Potential sampling problems

• Flawed surveys often reflect poor samples – Sampling bias can enter in two places • At selection of sampling frame – Use of phone directories – List of subscribers to a magazine • At selection of a sample – Use of an email survey – Emails made to students on Friday and Saturday nights

Potential sampling problems

• Suppose you were interested in student attitudes to beer on campus – Sampling frame might be the student email directory – You could randomly sample names from this • What is good and bad about this frame?

Types of Samples

• Nonprobability samples – No guarantee that elements of pop can be included – Will (almost never) be reflective of the population • May be helpful for studying ideal cases • May be helpful for studying specialized populations – Accidental samples • Whoever happens to be around...

– Surveys in the street – Newspaper call-in polls • Problems with accidental samples – Built in bias – Generalizability?

More non-probability samples

• Purposive sampling – Studying a particular population – May seek out special individuals • Survey of types of employee • May provide insight into what makes good vs. bad • Generally assume that non-probability samples are biased – True for many mail-in surveys

Subject Selection Effects

• Any study in which subjects self-select themselves for participation are potentially problematic – Some types of selection are worse than others • Those willing to participate for money • Those interested in a particular topic • Those with a particular viewpoint • Those with a particular skill • Problem with passive net surveys?

What to do?

• Probability samples – In a probability sample, there is a better chance that the sample will reflect the underlying population • Types of probability samples • Simple random sampling – Select N individuals from sampling frame • Stratified random sampling – Useful if there are key subgroups in data – Suppose your students were half male and half female • Divide sampling frame into males and females – Select N/2 males and N/2 females • Ensures that this aspect of the sample matches the population

Survey research designs

Cross-sectional designs •Try to get a view of a population at one time

Example

• Suppose we were interested in the number of left handers in the population – We could conduct a survey – What if we found that left-handers make up about 10% of the population up to age 60, and then there is a continuous decrease in percentage • Might conclude that left-handers die earlier • Other factors?

Successive Independent Samples

• Can be used to assess changes in attitude over time • Must be sure that samples are comparable • Want results to be due to changes in time, not changes in population

Longitudinal Studies

• Follow a single group of individuals over time – Requires a lot of time and effort – Subject mortality a problem • Drop out -- sometimes literal mortality • Most often people drop out for some reason • Must ensure that there is not a systematic reason for the mortality – Left handers again?

Summary So Far...

• Research is only as good as the sample you draw • Must ensure the most representative sample possible • Survey and correlational research cannot determine causal relationships • Next - how do we go about it?

Conducting Surveys

• There are many ways to conduct surveys – Through the mail – In person via an interview – Over the phone – Via the internet • some of all of the above can apply to this • Each of these methods has advantages and disadvantages

Mail Surveys

• Mail surveys are used frequently – Minimal effort needed to collect data – Good for personal / embarrassing topics • Respondent’s replies are anonymous • May cut down on the amount of socially desirable responding • Problems with mail surveys – Respondents cannot ask for clarification • A big problem is questions are often poorly worded – No follow-up questions can be asked – No control over the order in which a survey is completed – Big potential for response bias

Self-selection in mail surveys

• Respondents in mail surveys always self-select – Respondent must decide to do survey and mail it • Controlling response bias in mail surveys – Try to ensure a high return rate • Over 50% is pretty good • Response bias is less severe with return rates over 50% – Make survey attractive • Include a pre-paid reply envelope • Personal introductions • Raffle a prize – But…

Personal Interviews

• Survey can be conducted in person – Advantages • High response rate • Can control order of responding • Respondent can ask for clarification • Interviewer can ask follow-up questions

Personal Interviews

• Survey can be conducted in person – Disadvantages • There is a potential for interviewer bias – May suggest the desired response (nonverbally) • People may give socially desirable responses – They may not want to express their true beliefs to the interviewer • Difficult to ask about embarrassing topics – No anonymity for the respondent

Telephone Surveys

• Doing surveys over the phone has some advantages – More anonymity than for personal interviews – Can control order of responding – Respondent can ask for clarification – Interviewer can ask follow-up questions

Disadvantages of telephone surveys

• People may not want to be bothered over the phone – Particularly now that telemarketers often use surveys as a sales pitch (think also email spam) • No control over surroundings – Respondent may be doing a number of things at once • Still the potential for interviewer bias • People may provide socially acceptable responses – They may not want to express their true beliefs to the interviewer

Constructing Surveys

• Kinds of questions • Open-ended – Range of allowable responses not determined in advance – Respondents decide how much information to give – Open-ended question can be difficult to score

Constructing Surveys

• Kinds of questions • Closed questions – A fixed set of responses is provided – Respondent must pick one – Easy to score – May miss responses that respondent wants to give – Limited in the scope of the answer that can be given

More on questions

• Mixed questions – A fixed set of responses is provided – An open-ended choice is given to cover any alternatives that may have been missed – Still has a limited scope • What meanings can “bank” have: – 1. A place to keep money – 2. The edge of a river – 3. A large snail – 4. To lean an aeroplane into a turn … etc...

– 7 Other. _________________

Response Scales

• For closed-ended questions, a list of responses is needed – For some questions, the list is a set of options • What meanings can “bank” have: – 1. A place to keep money – 2. The edge of a river – 3. A large snail … etc.

– Many questions ask for a degree of preference • Yes/No questions – Easy response – No information about strength of preference

Response Scales

• Likert Scales – Labeled points on scale • Strongly agree, Agree, Disagree, Strongly disagree • Variations on these scales – Scale from 1-5 with ends labeled • How many values should be put on a scale?

– People are reasonably good with 7±2 options – With more options, people sub-divide the space • Ratings are not

necessarily

they are on a 1-9 scale – False feeling of accuracy more accurate on a 1-100 scale than

How many points?

– How many points • The number of points you use depends on how much your subjects will be able to discriminate, and the size of the effect you want to measure.

• If the effect you're trying to measure is very small, then you'll need more points on the scale • If you don't think people can meaningfully discriminate more than 3 levels of a variable, then you should only have 3 points - having more points would just add noise to your data.

Even or odd number of points?

– This makes a difference when you're measuring an effect in the middle of the scale – An even number of points is good when you want to force people to make a decision that commits them to one side of a scale or another – An odd number of points is good when you're interested in when people will take a middle of the road stance • When will they say “don’t know”

When might you not use a 7 point scale?

• 1. Measuring a small effect in the middle of the scale: How good at maths are women generally?

1------2------3------4------5------6------7------8------9------10 not at all.............................................................................…extremely

When might you not use a 7 point scale?

• 2. Getting more info out of your subjects than they think they know Is this picture old or new?

1------2------3------4 I'm sure it's old.........................I'm sure it's new

When might you not use a 7 point scale?

• 3. Finding out that people aren't sure Is this picture old or new?

1------2------3 I'm sure it's old.................I'm sure it's new

When might you not use a 7 point scale?

• 4. Another example of forcing a binary decision Please rate whether each item below is masculine or feminine: violin ..........M or F hammer.......M or F chair............M or F salt..............M or F

Writing Good Questions

• A survey (or test) lives and dies with the questions – Unclear questions can confuse respondents – Biased questions can skew results • Vocabulary in questions should be clear and simple – Never use two syllables where one will do

Writing Good Questions

• Questions should be clear and specific – Is something wrong with these sentences?

– How do you feel about the grammar of these sentences?

– Please indicate if one of the following sentences is grammatical or not, and indicate why.

More on questions

• Edit questions for readability • Use short questions – Some words are ambiguous meaning that people attribute multiple meanings to them. Given this, do you think or not that more than one meaning could be given to the following words?

More on questions

• Do not assume that participants share your enthusiasm • Get to the point • Did I mention use short questions – Help us understand memory - do this….

• Q-day!

Asking questions

• Participants may not always help you even when they want to – Show me how you stop your children from doing naughty things

Asking questions

• Participants may not always help you even when they want to • How to get round this?

Asking questions - stratagems

• Participants may not always help you even when they want to – I bet you couldn’t stop little Jimmy from doing something he really wanted to do…

Asking questions - stratagems

• Participants may not always help you even when they want to • Children especially may be too eager to please / to nervous to perform

Asking questions - stratagems

• Participants may not always help you even when they want to • Children especially may be too eager to please / to nervous to perform • Deflect the task – Could you help this really dumb cookie monster to…

Asking questions - stratagems

• Participants may not always help you even when they want to • Do fat people eat more when they are stressed?

Asking questions - stratagems

• Participants may not always help you even when they want to • What happens when people have folk psychological theories?

– Or when theories are famous?

Things to avoid

• Leading questions – Questions that suggest the right answer • People often recognise ambiguities -- or multiple meanings - in sentences. Re-write the following sentences in a way that uses different words, but preserves their meanings?

Things to avoid

• Loaded questions – Questions that are emotionally charged • Competent readers can often spot ambiguities when reading. Does this sentence have more than one possible reading?

Things to avoid

• Double-barreled questions – Packing too much into one question • Say which aspects of the the following sentence are ambiguous and why you think they are.

Other things

• Include conditional information before the key idea in the question alter responses?

– If you were reading the following sentence, would you consider alternative meanings for it?

– Would you consider alternative meanings if you were reading the following sentence?

Other things

• Consider varying the polarity of the questions – Some questions should be phrased negatively – Other questions should be phrased positively – Controls for a ‘positivity’ or ‘negativity’ bias • Some people just like to say No (or Yes).