Chapter 6

Tools for Interrogating Frequency Claims: Surveys and Polls © 2012 by W. W. Norton & Company

Detailed Learning Objectives: Question Wording and Observations 1. Give examples of how question wording can change the results of a survey or poll. 2. Describe the different ways questions can be worded: open-ended, forced choice, and using rating scales. 3. Explain how to increase the construct validity of questions by wording them carefully and by avoiding leading questions, double-barreled questions, and double negatives. 4. Explain how question order can change the meaning (and validity) of a question.

5. Explain ways to increase the construct validity of questions by preventing respondent shortcuts (such as nay-saying), biases (such as trying to look good), or simple inability to report. 6. Describe how observational techniques for measurement are different from survey techniques. 7. Explain ways to improve the construct validity of observations by reducing observer bias and target reactivity.

Detailed Learning Objectives: Sampling 8. Indicate the population that a given sample might represent. 9. Define three sampling problems that lead to biased samples. 10. Understand why a random sample is more likely to be a representative sample and why representative samples have external validity to a particular population. 11. Explain five techniques for random sampling: simple random, multistage, cluster, stratified random sampling, and oversampling. 12. Describe three techniques of nonrandom sampling: purposive, convenience, and snowball sampling. 13. Explain why representative samples may be especially important for many frequency claims. 14. Consider times when an unrepresentative sample may be appropriate for a frequency claim. 15. Explain why it is more important to ask how a sample was collected rather than how large the sample is.

Question Wording

Construct Validity of Surveys and Polls

• • • Choosing question formats Writing well-worded questions Encouraging accurate responses

Choosing Question Formats

Open-ended questions Forced-choice format Likert scale Semantic differential format (example shown)

Writing Well-Worded Questions

How can you write items that avoid these problems? • • • • Leading questions Double-barreled questions Double negatives (vs. negatively worded items) Question order

Encouraging Accurate Responses

What are some solutions to these problems? • • • • Response sets (three types) – – Yea-saying/ nay-saying Fence sitting Faking good or bad Saying more than we can know Measuring subjectivity vs. objectivity

Behavioral Observations

Construct Validity of Behavioral Observations • • • Examples of frequency claims based on observational data Are observations better than self-reports?

Good observations are both reliable and valid

Some Examples of Frequency Claims Based on Observational Data

Are Observations Better than Self-Reports?

What are some solutions to these problems? Observers may see what they expect to see Observers can affect what they see Masked research designs Observer effects Unobtrusive observations Wait it out Measure results of a behavior Observe ethically Figure 6.6

Unobtrusive

observations This one-way mirror allows researchers to unobtrusively record the behaviors of children in a preschool classroom.

Good Observations Are Both Reliable and Valid

Good Observations Are Both Reliable and Valid Figure 6.8

Table 1 from Campos and colleagues’ (2009) study of family interactions.

The table depicts the degree of interrater reliability for each of the behaviors coded.

Sampling

Generalizing to Others: Sampling Participants • • • • • Populations and samples How to get a representative sample When does external validity matter?

When a representative sample is not the top priority Are bigger samples better samples?

Key Questions

a. Read the first three paragraphs carefully. What do you think of the sampling techniques used in this poll? To what population can we generalize the results from this poll? b. Your friend comments, “wow! 375,000 people responded to this survey! We can definitely generalize this to the American population.” What do you say?

Populations and Samples

Population is ?

Sample is ?

Representative Samples

Probability sampling Simple random sampling Cluster sampling Multistage sampling Stratified random sampling Oversampling Systematic sampling

Unrepresentative Sampling

Self-selected Purposive Convenience

Working with your own examples

Homework

• • What do you think of the construct validity of the polls your group brought in? What do you think about the external validity of the polls your group brought in?

More Practice

Imagine that you are planning to estimate the price of the average book at the UD college bookstore. The bookstore carries 13,000 titles, but you plan to sample only 200 books. You will select a sample of 200 books, record the price of each book, and use the average of the 200 books to estimate the average price of the 13,000 books in the bookstore.

Based on this information, answer the following questions: a. What is the sample in this study, and what is the population?

b. How might you collect a simple random sample of books?

c. How might you collect a stratified random sample?

d. How might you collect a convenience sample?

e. How might you collect a systematic random sample?

f. How might you collect a cluster sample of 200 books?