Transcript INTERPRETING THE QUESTION

QUALITATIVE APPROACHES TO
QUESTIONNAIRE DESIGN:
METHODS AND APPLICATIONS
RICHARD B. WARNECKE, Ph.D.
UNIVERSITY OF ILLINOIS AT CHICAGO
MINORITY HEALTH PROJECT
UNIVERSITY OF NORTH CAROLINA AT
CHAPEL HILL
JUNE 15, 2000
Copyright  2000, Richard B. Warnecke
ANSWERING QUESTIONS
 INTERPRETATION
 RECALL
 JUDGMENT
FORMATION
 RESPONSE
FORMATTING
 EDITING
Copyright  2000, Richard B. Warnecke
LABORATORY METHODS
FOCUSED
INTERVIEWS
THINK-ALOUDS
PARAPHRASING
CONFIDENCE RATINGS
RESPONSE LATENCY
SORTS
Copyright  2000, Richard B. Warnecke
INTERPRETING THE
QUESTION
 LITERAL
MEANING
 INTENDED
MEANING
 RESPONSE ALTERNATIVES
Copyright  2000, Richard B. Warnecke
INTERPRETATION PROBES
 PARAPHRASE
QUESTION.
 DEFINE A WORD OR PHRASE.
 WHAT DOES A RESPONSE MEAN TO
YOU?
 BOUNDARIES (WHICH OF THE
FOLLOWING WOULD ALSO BE …..?)
 LABEL PREFERENCE
 INTERPRETATION DIFFICULTY
Copyright  2000, Richard B. Warnecke
WHAT TYPES OF PHYSICAL
ACTIVITY OR EXERCISE DID
YOU PERFORM DURING THE
PAST MONTH?
________________________________
________________
Copyright  2000, Richard B. Warnecke
WHAT DID YOU THINK WE
MEANT WHEN WE SAID
“PHYSICAL ACTIVITY?
Copyright  2000, Richard B. Warnecke
WHICH, IF ANY, OF THE FOLLOWING
WOULD YOU (ALSO) CONSIDER TO BE
PHYSICAL ACTIVITY? WHAT ABOUT . . .
WALKING,
HOUSEWORK, SUCH AS CLEANING THE
BATHROOM OR KITCHEN,
WORK-RELATED ACTIVITY, SUCH AS
CONSTRUCTION
YARD WORK, SUCH AS MOWING THE
LAWN OR RAKING LEAVES?
Copyright  2000, Richard B. Warnecke
LOGISTIC REGRESSION ANALYSIS OF PHYSICAL
ACTIVITY QUESTION MEANING PROBES
(UNSTANDARDIZED COEFFICENTS REPORTED)
Behavioral frequency
Male (1=Yes)
0.22*
0.02
0.07
0.12
Male
(1 = Yes)
-0.91***
-0.36**
0.41*
-0.43**
African American
(1 = Yes)
0.07
0.042**
-0.01
0.59**
Mexican American
(1 = Yes)
0.48
0.25
0.36
0.43
Copyright  2000, Richard B. Warnecke
LOGISTIC REGRESSION ANALYSIS OF PHYSICAL
ACTIVITY QUESTION MEANING PROBES
(UNSTANDARDIZED COEFFICENTS REPORTED) cont’d.
Puerto Rican
(1 = Yes)
0.26
0.38*
-0.09
0.34
Age
0.08**
0.01
-0.00
0.00
Education
0.12
0.10
-0.12
-0.03
Model chi-square
30.11***
18.42**
8.21
18.00**
Sample Size
365
372
371
371
Copyright  2000, Richard B. Warnecke
RECALLING AND
RETRIEVING
 RECALL
PRIOR INFORMATION
 COMPUTE
 ESTIMATE
 USE A SCHEMA (A PATTERN OF
ASSOCIATIONS AROUND WHICH
THINGS ARE RECALLED, I.E. PAP
SMEAR AS PART OF A PHYSICAL
EXAM).
Copyright  2000, Richard B. Warnecke
RETRIEVAL PROBES
 RECALL
STRATEGY
 RECALL INTERVAL
 SEARCH STRATEGY (PROXIMAL,
DISTAL, ANYWHERE)
 LONG TERM RECALL--LINK TO
EVENTS TO HELP REMEMBER
 RECALL FRAME OF REFERENCE-WHAT KINDS OF THINGS HELPED
YOU REMEMBER?
Copyright  2000, Richard B. Warnecke
HOW MANY TIMES PER WEEK
OR PER MONTH DID YOU TAKE
PART IN (FIRST ACTIVITY
MENTIONED)?
_______________ TIMES
PER WEEK
____________________ TIMES PER MONTH
Copyright  2000, Richard B. Warnecke
HOW DID YOU REMEMBER THE
NUMBER OF TIMES YOU DID
THIS? (PERFORM PHYSICAL
ACTIVITY)
Copyright  2000, Richard B. Warnecke
IN A TYPICAL WEEK, HOW
MANY TIMES DO YOU TALK ON
THE TELEPHONE WITH FAMILY,
FRIENDS, OR NEIGHBORS?
______ TIMES PER DAY
______ TIMES PER WEEK
Copyright  2000, Richard B. Warnecke
IN RECALLING THIS INFORMATION
ABOUT HOW MANY TIMES YOU TALK ON
THE TELEPHONE, DID YOU SPECIFICALLY
THINK ABOUT THE MOST RECENT WEEK
OR DID YOU AVERAGE TOGETHER
EXPERIENCES YOU’VE HAD OVER A
NUMBER OF WEEKS, OR DID YOU THINK
ABOUT IT IN SOME OTHER WAY?
THE MOST RECENT WEEK
 AVERAGE WEEK
 OTHER (SPECIFY) __________________
 DON’T KNOW

Copyright  2000, Richard B. Warnecke
JUDGMENT FORMATION
 FIT A REFERENCE
PERIOD
 FIT A DEFINITION
 RELATE
TO QUESTIONNAIRE
CONTEXT-- ORDER EFFECTS
Copyright  2000, Richard B. Warnecke
JUDGMENT PROBES

JUDGMENT EXPLANATION
– HOW ARRIVED AT ANSWER
 JUDGMENT
CERTAINTY
– SCALE OF CERTAINTY (1-7)
 ANCHORING
STRATEGIES
– COMPARATIVE
 ATTITUDE
CRYSTALLIZATION
– THOUGHT ABOUT BEFORE
– FREQUENCY OF PREVIOUS THOUGHT
– DISCUSS WITH OTHERS
Copyright  2000, Richard B. Warnecke
HOW LIKELY DO YOU THINK IT IS
THAT YOU WILL HAVE SERIOUS
HEALTH PROBLEMS FROM
SMOKING? DO YOU THINK IT
IS…..
VERY LIKELY
SOMEWHAT LIKELY
SOMEWHAT UNLIKELY
VERY UNLIKELY
DEFINITELY NOT POSSIBLE?
DON’T KNOW
Copyright  2000, Richard B. Warnecke
BEFORE WE ASKED YOU THE
QUESTION ABOUT SERIOUS
HEALTH PROBLEMS, HAD YOU
THOUGHT ABOUT IT BEFORE?
YES
NO
Copyright  2000, Richard B. Warnecke
ON A SCALE FROM 1 TO 7,
WHERE 1 IS ONLY RARELY AND 7
IS VERY OFTEN, HOW OFTEN
HAVE YOU THOUGHT ABOUT IT
BEFORE?
1
2
3
4
5
ONLY
RARELY
Copyright  2000, Richard B. Warnecke
6
7
VERY
OFTEN
ORDER EFFECTS-I
 MINIMIZING
ORDER EFFECTS
– GENERAL BEFORE SPECIFIC (FUNNEL)
– SPECIFIC BEFORE GENERAL
(INVERTED FUNNEL)
– USE ORDER TO REDUCE EFFECTS OR
ITEM NON RESPONSE
Copyright  2000, Richard B. Warnecke
ORDER EFFECTS -2
 DEVELOP A LOGICAL
FLOW TO HELP
RESPONDENT
– SEGMENT BY TOPIC
– ASK ABOUT RELATED TOPICS
TOGETHER
– ASK RECALL FROM MOST RECENT TO
LEAST RECENT
– USE TRANSITIONS TO CHANGE
TOPICS
Copyright  2000, Richard B. Warnecke
ORDER EFFECTS-III
 ESTABLISH
RAPPORT
– START WITH EASY, NON-THREATENING
QUESTIONS
– DON’T START WITH DEMOGRAPHICS
END WITH THEM
– DON’T START WITH KNOWLEDGE AND
AWARENESS
Copyright  2000, Richard B. Warnecke
EXAMPLE OF USE OF ORDER
TO RESOLVE CONFUSION
 ORIGINAL
QUESTION:
HOW SATISFIED ARE YOU WITH
YOUR RELATIONSHIP WITH YOUR
SPOUSE/SIGNIFICANT OTHER?
 PROBE:
WHAT
PARTS OF THE RELATIONSHIP DID
YOU THINK ABOUT?
Copyright  2000, Richard B. Warnecke
THE PROBLEM
 QUESTION
WORKED WELL IN
ENGLISH.
 HOWEVER IN SPANISH THE WORD
FOR RELATIONS IS THE WORD FOR
SEXUAL RELATIONS
Copyright  2000, Richard B. Warnecke
SOLUTION
 REVERSED
ORDER OF QUESTIONS.
ASKED ABOUT SATISFACTION WITH
SEXUAL RELATIONS FIRST.
 THEN ASKED ABOUT OTHER
ASPECTS OF MARITAL
RELATIONSHIP.
Copyright  2000, Richard B. Warnecke
FORMATTING A RESPONSE
 RESPONSE
 RATING
CATEGORIES
SCALES
 RESPONSE
SET
– EXTREME CHOICES
– ACQUIESCENCE
Copyright  2000, Richard B. Warnecke
Mean Group Responses to Measure of
Extreme Response Style
African
American
(N=102)
Mexican
American
(N=106)
Puerto
Rican
(N=84)
White
nonHispanic
(N=101)
Mean**
7.78
8.09
8.69
6.7
Standard
deviation
4.65
3.99
4.20
4.42
**F-ratio p< .01.
Note: Means with different superscripts differ significantly
(p<.05) on the basis of multiple range test.
Copyright  2000, Richard B. Warnecke
Mean Group Responses to Acquiescence Scale
African
American
(N=102
Mexican
American
(N=106)
Puerto
Rican
(N=84)
White
nonHispanic
(N=101)
Mean**
10.00
9.96
9.43
8.76
Standard
deviation
2.84
2.91
3.12
2.71
**F-ratio p< .01.
Note: Means with different superscripts differ significantly (p<
basis of multiple range test.
Copyright  2000, Richard B. Warnecke
.05) on the
Mean Group Responses to the Marlowe-Crowne
Social Desirability Scale
African
Mexican
American American
(N=110)
(N=110)
Puerto
Rican
(N=87)
White
nonHispanic
(N=108)
Mean**
5.57
5.92
5.47
4.85
Standard
deviation
2.47
2.30
2.16
2.29
**F-ratio p<.01.
Note: Means with different superscripts differ significantly
.05)
(p<on the basis of multiple range test.
Copyright  2000, Richard B. Warnecke
SCALE OF SATISFACTION WITH
QUALITY OF LIFE
 QUALITY
OF LIFE SCALE WITH
RANGE OF VALUES.
–
–
–
–
–
–
–
–
–
VERY SATISFIED
SATISFIED
SOMEWHAT SATISFIED
A LITTLE SATISFIED
NEITHER SATISFIED NOR DISSATISFIED
A LITTLE DISSATISFIED
SOMEWHAT DISSATISFIED
DISSATISFIED
VERY DISSATISFIED
Copyright  2000, Richard B. Warnecke
ASSESSMENT
SATISFACTION RANGE
DISSATISFIED
RANGE

VERY
100-60

VERY
0-80

SATISFIED
100-50

DISSATISFIED
0-80

SOMEWHAT
90-45

SOMEWHAT
10-54

A LITTLE
70-30

A LITTLE
10-53
Copyright  2000, Richard B. Warnecke
SOLUTION
 USED A BAR
GRAPH WITH SINGLE
SCALE WHERE BLANK EQUALS
“NOT AT ALL SATISFIED” AND 6
EQUALS “VERY SATISFIED.”
 SIMPLIFIED RESPONSE ON ONE
DIMENSION
 AVOIDED LABELS THAT WERE
CONFUSING.
Copyright  2000, Richard B. Warnecke
EDITING A RESPONSE
 SOCIAL
DESIRABILITY
 SELF PRESENTATION
 EFFECT OF INTERVIEWER
– GENDER
– AGE
– CULTURE/ ETHNICITY
Copyright  2000, Richard B. Warnecke
RESPONSE EDITING PROBES
 PERCEIVED
SOCIAL DESIRABILITY
 PERCEIVED EMBARRASSMENT
 PERCEIVED HONESTY
 PERCEIVED INTERVIEWER EFFECTS
– ETHNICITY
– AGE
– GENDER
Copyright  2000, Richard B. Warnecke
INTERVIEWER EFFECTS: AGE
ORIGINAL QUESTION:
HAVE YOU EVER SMOKED
MARIJUANA, HASH OR HASH OIL OR
TAKEN THEM IN SOME OTHER WAY?
 PROBE:
DO YOU FEEL THIS IS A QUESTION THAT
A PERSON YOUR AGE WOULD BE
COMFORTABLE OR UNCOMFORTABLE
TALKING ABOUT TO A SURVEY
INTERVIEWER OF THE SAME AGE OR
WHO WAS MUCH YOUNGER OR OLDER?

Copyright  2000, Richard B. Warnecke
Projected Comfort Level Discussing Marijuana Use with Hypothetical
Interviewer of Same Age/Different Age
Respondent age
Interviewer age
Younger
(18-29)
Older
(30-50)
Same age
77.5
78.6
Younger or
Older
30.3
54.6
Copyright  2000, Richard B. Warnecke
INTERVIEWER EFFECTS:
GENDER
ORIGINAL QUESTION:
DURING THE PAST WEEK, HOW
OFTEN HAVE YOU HAD CRYING SPELLS?
 PROBE:
DO YOU FEEL THIS IS A QUESTION
THAT (RESPONDENT’S GENDER) WOULD
BE COMFORTABLE TALKING ABOUT TO A
SURVEY INTERVIEWER OF THE SAME
SEX? OF THE OPPOSITE SEX?

Copyright  2000, Richard B. Warnecke
Projected Comfort Level Discussing Crying Spells with Hypothetical
Interviewer of Same Sex/Opposite Sex
Interviewer
gender
Male
Female
Same gender
34.1
93.3
Opposite gender
28.1
41.3
Copyright  2000, Richard B. Warnecke
INTERVIEWER EFFECTS:
ETHNICITY
 ORIGINAL
QUESTION:
DO YOU THINK OF YOURSELF AS
HETEROSEXUAL, HOMOSEXUAL OR
BISEXUAL?
 PROBE:
DO YOU FEEL THIS IS A QUESTION
THAT (RESPONDENT’S ETHNIC GROUP)
WOULD BE COMFORTABLE DISCUSSING
WITH AN INTERVIEWER OF THE SAME
(ETHNIC GROUP)? A SURVEY
INTERVIEWER OF A DIFFERENT ETHNIC
GROUP?
Copyright  2000, Richard B. Warnecke
Projected Comfort Level of Respondents by Topic, By Culture
of Hypothetical Interviewer, And By Culture of Respondent
(Percent Comfortable)
Comfort Level
Topic: Sexual
orientation
Interviewer
race/ethnicity:
Same
race/ethnicity
Different
race/ethnicity
African
Mexican
American American
(N=101) (N=100)
Puerto
Rican
(N=82)
White
NonHispanic
(N=99)
72.3
64.0
65.4
82.8
61.0
51.5
54.9
71.4
Copyright  2000, Richard B. Warnecke