Transcript Assessing the Validity of Generalized Trust Questions

Field experiments for assessing
question validity
Patrick Sturgis, Department of
Sociology, University of Surrey, UK
Paper presented at conference on ‘Survey Measurement: Assessing the
Reliability and Validity of Contemporary Questionnaire Items’ The Royal
Statistical Society,10 April 2008
1
Plan of Talk
• Standard validity assessment for survey
questions
• Field experiments
• Example 1 – Political knowledge
• Example 2 – Social trust
• Concluding remarks
2
Standard validity assessment
•
•
•
•
•
•
•
•
Nothing
Face/process validity
Correlation with criterion variables
Multi-trait-multi-method (MTMM)
Expert panels
Behaviour coding
Interviewer debrief
Thinkaloud protocols/cognitive interview
3
Limitations
• Small n/purposive selection – do
inferences generalize?
• Do different techniques/researchers
identify same ‘problems’
• Do modifications increase validity?
• Paradoxical limitations for survey
research!
4
Field Experiments
• Large n with randomization of alternate
forms
• Clean and powerful inference
• Lack of criterion reference can be
problematic
• But theory can help!
5
Example 1
(with Nick Allum, Patten Smith)
Measuring Political Knowledge:
Guessing and partial knowledge
6
Standard approach
•
•
MCQ format:
“The Number of MPs in Parliament is about 100”
a. True
b. False
c. DK
•
•
DKs ‘encouraged’
Two key problems (Mondak 2001; 2002):
– Some say DK when they can answer correctly at p > 0.5
(partial knowledge)
– Some provide a substantive answer when they cannot
answer correctly at p >0.5 (guessing)
7
Personality Variance
• Variation in knowledge scores reflects
more than just knowledge
• Men more likely to guess in absence of
knowledge
• Women more likely to say DK with partial
knowledge
• Thus, men ‘appear’ to know more about
politics than women
8
The Solution?
• Force all respondents to provide an
answer even if they genuinely DK
(Mondak 2001)
• Randomly allocate residual DKs across
substantive categories
• Removes personality variance by omitting
option of guessing
• And of saying DK in presence of partial
knowledge
9
Study 1 - Partial Knowledge
•
•
•
•
BMRB CATI omnibus (quota sample)
Interviewing 17-19 December 2004
N = 1006
Three true/false knowledge items:
–
–
–
Britain's electoral system is based on proportional
representation
MPs from different parties are on parliamentary
committees
The Conservatives are opposed to the ratification of
a constitution for the European Union
10
Design
• “For the next few questions, I am going to read
out some statements, and for each one, please
tell me if it is true or false. If you don't know, just
say so and we will skip to the next one”
• If respondent answers DK:
• “You said earlier that you don't know whether
the number of MPs is about 100. Could you
please just give me your best guess?”
• Partial knowledge in initial DK responses if %
correct after probe > .5
11
Probed DK Responses
incorrect
correct
Total
Item 1
91 (49.5%)
93 (50.5%)
184
Item 2
33 (33.7%)
65 (66.3%)
98
Item 3
69 (47.6%)
76 (52.4%)
145
12
Results
Binary logit predicting correct answer (0,1)
answer category
initial answer
probed guess
constant (randomly allocated)
logit
0.90
0.15
-0.01
Odds Ratio
2.46
1.17
0.98
p (95%)
0.000
0.363
0.950
n
2638
427
223
Model predicted probabilities of correct answers =
•
71% for those giving an initial response
•
53% for probed DKs
•
50% for random allocation and
No gender difference
13
Study 2 - Guessing
•
•
Ask standard format knowledge questions but
where answer options are all wrong
Respondents choosing any substantive
alternative are ‘guessing’:
1. Who is the Secretary of State for Trade and
Industry? Is it,
a. Geoff Hoon
b. Peter Hain or
c. Do you not know? (correct=Alan Johnson)
•
BMRB omnibus n=2011, 4-6 November 2005
and 9-11 December 2005
14
Who is the Secretary of State for Trade and Industry? Is it,
a. Geoff Hoon
b. Alan Johnson, or
c. Do you not know?
50
47
45
41
40
37
35
30
32
26
Alan Johnson
25
Alistair Darling
DK
20
16
15
10
5
0
male
female
15
Who is the Secretary of State for Trade and Industry? Is it,
a. Geoff Hoon
b. Peter Hain, or
c. Do you not know?
80
72
70
60
57
50
42
correct
40
guess
27
30
DK
20
10
1
1
0
male
female
16
Binary Logit Model
Dependent variable guess=1, dk=0
Predictor
sex (male=1)
political knowledge
self-confidence
social grade
age
Constant
logit
0.42*
0.60**
0.14
-0.15**
0.06
-1.73
s.e.
0.15
0.09
0.08
0.05
0.05
0.39
Odds ratio
1.52
1.82
1.15
0.86
1.07
0.18
17
Conclusions
• No evidence that DKs in survey knowledge items
conceal partial knowledge
• Guessing, however, is common and differential
(favouring men)
• Guessing also related to political knowledge
• Recommendation: use ‘standard’ format items
• For marginal comparisons, randomly allocate DKs
to substantive categories
• For associational relationships use number right
scoring (treat DK and incorrect as equivalent)
18
Example 2
(with Patten Smith)
Investigating Social Trust
Using thinkalouds
19
Conceptions of Trust
• Trust is a ‘good thing’
• Trusting citizens are good citizens (voting,
volunteering, civic engagement)
• Trusting societies are good societies
(more democratic, egalitarian, > economic
performance)
• Trust ‘lubricates’ social and economic
transactions
• Reduces ‘monitoring costs’
20
‘Thick’ Trust
•
•
•
•
Also ‘particularized’ or ‘strategic’ trust
Between people who know one another
Based on personal experience
Encapsulated interests; ‘your interests are
my interests’ (Hardin)
• I trust x to do y
21
‘Thin’ trust
• Also ‘social’ or ‘generalized’ trust
• Trust between people not personally
known to one another
• More akin to a core social value or attitude
• “an evaluation of the moral standards of
the society in which we live” (Newton)
• A ‘default position’ in transactions with
unknown others
22
Does this matter?
• Primary social and individual returns are to
thin/social trust
• Thick and thin trust may even be
negatively correlated
• The less we trust people in general, the
more we retreat to the safety of those we
know
• So, empirically distinct measures are
clearly essential
23
The standard trust question
• Generally speaking, would you say that
most people can be trusted, or that you
can't be too careful in dealing with people?
– Most people can be trusted
– Can’t be too careful
• Usually credited to Rosenberg (1959), the
‘Rosenberg Generalized Trust’ (RGT) item
24
The Local Area Trust item
• How much do you trust people in your
local area?
– a lot
– a fair amount
– not very much
– not at all
• Reflects Putnam’s emphasis on trust being
a property of local areas
25
Trust by Question type
• These items are both used more or less
interchangeably as measures of generalized trust
• Yet, they yield very different estimates of trust at
the national level. e.g.:
– Social Capital Community Benchmark survey: 47%
most people can be trusted; 83% trust people in local
area ‘some’ or ‘a lot’
– UK Taking Part survey: 44% most people can be
trusted; 74% trust ‘many’ or ‘some’ of the people in
their local area
• Why such a large discrepancy in generalized trust
(trust in strangers)?
26
Research Design
• Ipsos-MORI general population omnibus survey
• Random selection of small areas, quota
controlled selection of individuals
• n=989 (fieldwork, November 2007)
• Respondents randomly assigned to RGT or TLA
item
• In answering the last question, who came to
mind when you were thinking about ‘most
people’/ ‘people in your local area’?
27
Distributions for trust questions
TLA item (n=481)
RGT item (n=508)
Most people can be trusted
48% (229)
A lot
20% (100)
Can’t be too careful
52% (252)
A fair amount
60% (302)
Not very much
17% (88)
Not at all
3% (17)
28
Primary Codes
1. colleagues/ex-colleagues
2. family/family member
3. friends
4. most people I know/meet
5. neighbours
6. people from my church
7. anyone/all people
8. everyone/everybody
9. foreigners/ethnic minorities
10. general public/people in general
11. children/young people
12. no-one in particular
13. strangers
14. people in this town/village
15. doctors
16.officials/authority
figures/professionals
17. police
18. politicians/political parties
19. salesmen/sales people
20. tradesmen
21. don't know these days
22. identity theft
23. you have to place trust in people
24. people interested in themselves
25. people mostly trustworthy
26. trust people until they upset me
27. trusting is naïve
28. other answers
29. don't know/not stated
Higher Order Codes
% mentioned
Known others
42%
Unknown others
22%
Local community
5%
Named job/profession
10%
Other (not relevant)
13%
Don’t know/no answer
22%
29
Who comes to mind by RGT
80%
70%
60%
most people can be trusted
can't be too careful
% mentioned
50%
40%
30%
20%
10%
0%
known others
unknown others
named
job/profession
people in local
area
code
other
don't know/not
stated
30
Who comes to mind by TLA
80%
70%
a lot
a fair amount
not at all/not very much
60%
% mentioned
50%
40%
30%
20%
10%
0%
known others
unknown others
named
job/profession
people in local
area
code
other
don't know/not
stated
31
Who came to mind – both questions
60%
50%
RGT
TLA
% mentioned
40%
30%
20%
10%
0%
known others
unknown others
named
job/profession
people in local area
code
other
don't know/not
stated
32
Explanatory Models 1
Covariates
Age (years)
Sex (male=1)
social class (ABC1=1)
longstanding illness (yes = 1)
Highest qualification (ref=no qualifications)
Degree
GSCE or above
Marital status (ref = single, never married)
Divorced
Married
Widow
Who came to mind? (ref=2. unknown others)
1. known others
3. people in local area
4. named job/profession
5. other (not relevant)
6. non-one/don't know/not stated
Constant
RGT Item – Binary Logit Model
Model 1a
Model 2a
O.R
Logit (S.E.)
.
Logit (S.E.)
0.028 (0.036)
1.03
0.013 (0.038)
0.057 (0.197)
1.06
0.091 (0.207)
0.817 (0.213)*** 2.26
0.949 (0.227)***
0.355 (0.335)
1.43
0.462 (0.349)
O.R.
1.01
1.09
2.58
1.59
0.944 (0.337)**
0.108 (0.261)
2.60
1.11
1.029 (0.354)**
0.142 (0.276)
2.80
1.15
0.236 (0.454)
0.176 (0.274)
-0.124 (0.516)
1.27
1.19
0.88
0.508 (0.476)
0.413 (0.291)
0.272 (0.540)
1.66
1.51
1.31
-1.178 (0.345)
0.31
1.535 (0.267)***
1.885 (0.763)**
-0.255 (0.373)
0.257 (0.328)
1.043 (0.280)***
-2.161 (0.410)
4.64
6.60
0.78
1.29
2.84
33
0.12
Explanatory Models 2
Covariates
Age (years)
Sex (male=1)
social class (ABC1=1)
longstanding illness (yes = 1)
Highest qualification (ref=no qualifications)
Degree
GSCE or above
Marital status (ref = single, never married)
Divorced
Married
Widow
Who came to mind? (ref=2. unknown others)
1. known others
3. people in local area
4. named job/profession
5. other (not relevant)
6. non-one/don't know/not stated
Constant
TLA Item – Ordered Logit Model
Model 1b
Model 2b
O.R
Logit (S.E.)
.
Logit (S.E.)
0.097 (0.034)**
0.076 (0.034)*
1.10
-0.393 (0.186)**
-0.255 (0.190)
0.68
0.751 (0.204)*** 2.12
0.771 (0.207)***
0.230 (0.293)
0.297 (0.297)
1.26
0.605 (0.312)*
0.218 (0.255)
-0.247 (0.409)
0.323 (0.249)
0.516 (0.440)
1.83
1.24
0.425 (0.320)
0.075 (0.258)
1.53
1.08
0.78
1.38
1.68
-0.206 (0.418)
0.275 (0.253)
0.447 (0.448)
0.81
1.32
1.56
1.559 (0.305)***
0.953 (0.408)*
0.087 (0.305)
0.383 (0.356)
0.579 (0.346)
-
4.75
2.59
1.09
1.47
1.7834
-
-
O.R.
1.08
0.77
2.16
1.35
-
Concluding Remarks
• Large-scale field experiments are a useful
way of assessing validity of questions
• Random sample + random manipulation
yields strong inferential power
• Under-utilized due to cost considerations
• But are they really so costly?
• A complement to rather than replacement
for small n approaches
35
Papers
• Sturgis, P. Allum, N. & Smith, P. (2008) The Measurement
of Political Knowledge in Surveys Public Opinion Quarterly
72,90-102.
• Sturgis, P. and Smith, P. (2008) Assessing the Validity of
Generalized Trust Questions: What kind of trust are we
measuring? Paper presented at the ‘Conference on
Composite Scores’ ESADE, Barcelona, 14-15 February
2008.
• Sturgis, P. and Smith, P. (2007) Fictitious Issues Revisited:
political knowledge, interest, and the generation of
nonattitudes. (under review).
• Sturgis, P., Choo, M. & Smith, P. (2007) Response Order,
Party Choice, and Evaluations of the National Economy: A
Survey Experiment. Survey Research Methods (in press).
36