Transcript Slide 1

AIR 51st Annual Forum
May, 2011
Estimating Behavior Frequency:
Do Vague and Enumerated Estimation Strategies Yield Similar Results?
Presented by:
James Cole, Ph.D.
Assistant Scientist
Ali Korkmaz, Ph.D.
Assistant Scientist
Indiana University
Center for Postsecondary Research
Estimating Behavior Frequency
Probably most everyone in this room has collected self-reported data
on behavioral frequency. Questions like . . . .
How often did you meet with your advisor?
How many hours per week do you study?
Estimating Behavior Frequency
Probably most everyone in this room has collected self-reported data
on behavioral frequency. Questions like . . . .
How often did you meet with your advisor?
How many hours per week do you study?
You then examine your data. . .
34% said they met with their advisor “very often”
16% said they studied more than 15 hours per week
Estimating Behavior Frequency
Probably most everyone in this room has collected self-reported data
on behavioral frequency. Questions like . . . .
How often did you meet with your advisor?
How many hours per week do you study?
You then examine your data. . .
34% said they met with their advisor “very often”
16% said they studied more 15 hours per week
You wonder . . .
“What does ‘very often’ mean?”
“How did the student estimate 15 hours per week?”
Estimating Behavior Frequency
The purpose of this presentation is to provide additional
information regarding the use of vague and enumerated
quantifiers in survey research. Specifically, do vague and
enumerated responses produce consistent results?
Estimating Behavior Frequency
Surveys of student behavior commonly collect data regarding estimations
of behavior frequency in two ways:
Vague quantifiers
Beginning Postsecondary Students Longitudinal Study (NCES) includes items
such as “how often respondent wrote essay answers as part of exams
during the 2003-2004 academic year” with response categories “never,”
“sometimes,” and “often.”
Estimating Behavior Frequency
Surveys of student behavior commonly collect data regarding estimations
of behavior frequency in two ways:
Vague quantifiers
Beginning Postsecondary Students Longitudinal Study (NCES) includes items
such as “how often respondent wrote essay answers as part of exams
during the 2003-2004 academic year” with response categories “never,”
“sometimes,” and “often.”
Enumerated (or tallied) quantifiers
Beginning College Survey of Student Engagement (CPR), respondents are
asked “During your last year of high school, about how many hours did you
spend in a typical 7-day week doing each of the following?” and then
provided with a list of activities and numeric response categories (0, 1-5, 610, 11-15, etc).
Estimating Behavior Frequency
Referencing Versus Enumerating When Estimating Behavior Frequency
The process of estimating behaviors using vague quantifiers is distinct
from the process used to estimate behavior by trying to tally or count
occurrences.
Vague response sets invoke a process of comparison to estimate the
behavior frequency (Pace & Friedlander, 1982; Schaeffer, 1991; Wanke,
2002). People compare their behavior to:
similar behaviors they also engage in, or
behavior of others engaged in same behavior.
Estimating Behavior Frequency
Referencing Versus Enumerating When Estimating Behavior Frequency
Enumerated or tallied responses involves one or two steps depending
on the time frame, behavior frequency, and schedule (fixed or variable)
(Brown, 2002; Schaeffer & Presser, 2003).
1. Simple enumeration for low frequency behaviors regardless of the
time frame or schedule
2. As the frequency of the behavior increases and time lengthens
estimation becomes a two-step process: enumeration and
extrapolation
Estimating Behavior Frequency
Though enumerated responses have the appearance of accuracy, it is
important to note that enumerated responses are not necessarily
accurate (Brown, 2002; Schaeffer, 1991).
•
Satisficing
•
Accuracy of stored and recalled information (Ashcraft &
Radvansky, 2009)
•
Good faith effort to accurately recall information (Brown &
Sinclair, 1999).
Estimating Behavior Frequency
Given the many difficulties with enumeration, some researchers have
suggested that relative, vague judgments of behavior are preferred
(Bradburn & Danis, 1984).
“Since behavioral frequency reports are error-prone anyway, why bother
asking respondents for reports that suggest more precision than they
can provide?” (Sudman, Bradburn and Schwarz, 1996 , p. 226).
Estimating Behavior Frequency
Given the many difficulties with enumeration, some researchers have
suggested that relative, vague judgments of behavior are preferred
(Bradburn & Danis, 1984).
“Since behavioral frequency reports are error-prone anyway, why bother
asking respondents for reports that suggest more precision than they
can provide?” (Sudman, Bradburn and Schwarz, 1996 , p. 226).
However, Sudman, Bradburn and Schwarz (1996) go on to say:
“vague frequency expressions carry their own load of problems. . .
different respondents use the same term to mean difference objective
frequencies of the same behavior” (p. 226).
Thus, it is not surprising that researchers sometimes find group
differences for meanings assigned to vague quantifiers (e.g., Wanke,
2002).
Estimating Behavior Frequency
Group Differences in Estimating Behavior
Due to the comparative process involved when a respondent
interprets vague response sets students may interpret vague
quantifiers differently based on their background, social groups, or
past experiences.
“When the meaning of vague quantifiers differ by group, relative and
absolute responses may lead to different conclusions”
(Schaeffer, 1991 p. 397).
This norming effect can become particularly problematic when
examining groups differences using responses to vague quantifiers
(Wright et al., 1994).
Estimating Behavior Frequency
This study extends this line of research by investigating the following
questions:
• Is there a relationship between the frequency of the behavior and the
time frame selected to estimate the frequency of the behavior?
• For each behavior is there an increasing enumerated estimation with
each increasing level of vague quantifier?
• Are there group differences between vague and enumerated estimates
of the same behaviors based on gender, type of high school
(public/private), or completion of calculus (an indicator of academic
achievement level)?
Estimating Behavior Frequency
Data Source
2010 administration of BCSSE with 6 experimental items added
Estimating Behavior Frequency
Sample
Sample includes 30,964 entering first-year students enrolled at 81
institutions across the US.
Institution characteristics include:
15% doctoral
30% baccalaureate
48% masters
7% other
Student characteristics include:
59% female
8% African American
8% Asian/Asian American
9% Latino/Hispanic
67% White
8% Other
17% attended private high school
36% reported passing high school calculus
Estimating Behavior Frequency
Data Analysis
Student responses to the enumerated items were recoded so that
all responses were on a per week basis. Students could report
frequency based on five options (per day, week, month, academic
term, and academic year).
Week was taken as the baseline and responses with other time
frames were adjusted by appropriate multipliers (day = 5, month =
.25, academic term = .111, and academic year = .02778). There are
180 days in a typical high school academic year, 4 terms of 9
weeks.
Results
Is there a relationship between the frequency of the behavior and the
time frame selected to estimate the frequency of the behavior?
Time Frame Used to Estimate Behavior
Frequency
Rank
order
Behavior
Day
Week
Month
Term
Year
1
Ask questions
67.9%
27.9%
2.6%
0.5%
1.1%
2
Discussed ideas from readings with others
21.9%
44.1%
20.9%
5.3%
7.9%
3
Discuss grades/assignments w/teachers
9.9%
40.1%
31.0%
12.3%
6.8%
4
Discussed ideas from readings with teacher
11.2%
32.8%
27.0%
10.1%
18.9%
5
Worked with classmates outside class
5.3%
28.3%
35.3%
15.9%
15.2%
6
Class presentations
3.3%
26.6%
49.9%
12.0%
8.1%
Results
For each behavior is there an increasing enumerated estimation with each
increasing level of vague quantifier?
Ask
Class Pres
Faculty outside
class
Discuss grades
Students
Outside Class
Others
Outside Class
Never
Sometimes
Often
Very Often
Never
Sometimes
Often
Very Often
Never
Sometimes
Often
Very Often
Never
Sometimes
Often
Very Often
Never
Sometimes
Often
Very Often
Never
Sometimes
Often
Weekly
Mean
2.32
7.36
15.31
22.86
0.80
0.49
0.98
1.30
0.16
1.09
2.67
4.01
0.17
0.89
2.14
3.19
0.11
0.71
1.66
2.26
0.30
1.63
4.07
N
329
5913
10338
11709
518
11459
11930
4725
7042
13456
5570
2313
1259
12181
10528
4667
3115
14954
7943
2670
2259
12198
9420
SD
5.3
8.5
12.4
14.9
0.3
0.5
0.8
0.9
0.7
1.5
2.3
2.9
0.7
1.3
2.1
2.6
0.5
0.9
1.4
1.6
1.3
2.1
3.3
df
3
F
2142.4
sig
.001
Omega sq
.185
3
1997.8
.001
.173
3
4222.2
.001
.309
3
2272.5
.001
.192
3
3073.3
.001
.243
3
4016.4
.001
.300
Results
Games-Howell post-hoc test results
DAY
WEEK
MONTH
TERM
YEAR
Never
a
8.9
2.1
0.5
0.4b,c,d
0.0
Sometimes
b
14.9
3.5
1.0
0.6a,c,d
0.3c,d
Often
c
20.4
5.4
2.2d
1.0a,b,d
0.4b,d
Very Often
d
26.0
7.6
3.7c
2.5a,b,c
0.7b,c
Never
a
2.1
0.9
0.2
0.1
0.0
Sometimes
b
10.4c,d
1.8d
0.5
0.3
0.1
Often
c
12.5b,d
1.9c
0.7
0.5d
0.2
Very Often
d
12.3b,c
2.3
0.8
0.6c
0.3
Faculty
Never
a
0.8
1.2
0.4
0.2
0.0
grades
Sometimes
b
6.5c,d
1.9
0.5
0.3
0.1
Often
c
6.9b,d
2.5
0.8
0.4d
0.2d
Very Often
d
7.0b,c
3.1
1.0
0.5c
0.3c
Ask
Questions
Class
Presen
Note - Superscript indicates non-significant differences between indicated level(s). No superscript indicates significant
difference with other level(s).
Results
Games-Howell post-hoc test results
DAY
WEEK
MONTH
TERM
YEAR
Students
Never
outside
Sometimes
Often
Very Often
a
b
c
d
0.2
4.7c,d
5.0b,d
5.0b,c
1.3
2.0
2.4
2.9
0.3
0.5
0.8
1.0
0.1
0.3
0.4
0.6
0.0
0.1
0.2d
0.3c
Discuss
Ideas
w/fac
Never
Sometimes
Often
Very Often
a
b
c
d
0.5
6.5
7.0d
7.4c
1.5
2.0
2.7
3.4
0.3
0.5
0.9
1.1
0.1
0.3
0.5d
0.7c
0.0
0.1c
0.2b,d
0.3c
Discuss
Never
Ideas
Sometimes
w/others
Often
Very Often
a
b
c
d
1.3
7.2
8.1
9.1
1.7
2.3
3.3
4.6
0.4
0.6
1.1
1.6
0.1
0.3d
0.6d
0.7b,c
0.0
0.1
0.3d
0.4c
Note - Superscript indicates non-significant differences between indicated level(s). No superscript indicates significant
difference with other level(s).
Results
Are there group differences between vague and enumerated estimates of the same behaviors based on
gender?
Gender
Male
Female
sig
d
Ask questions
Vague
3.09
3.16
.001
-.088
Enumerated
16.81
16.47
.043
.024
Class presentations
Vague
Enumerated
2.65
0.82
2.75
0.81
.001
.253
-.128
.014
Discuss grades
w/faculty
Vague
Enumerated
2.66
1.78
2.69
1.63
.001
.001
-.037
.073
Discuss w/studs
outside class
Vague
Enumerated
2.29
1.07
2.37
1.04
.001
.026
-.097
.027
Discuss w/faculty
outside class
Vague
Enumerated
2.13
1.50
2.12
1.34
.032
.001
.015
.076
Discuss w/others
outside class
Vague
Enumerated
2.45
2.87
2.62
3.21
.001
.001
-.194
-.097
Results
Are there group differences between vague and enumerated estimates of the same behaviors based on
completion of type of high school?
High School Type
Public
Private
sig
d
Ask questions
Vague
3.13
3.19
.001
-.080
Enumerated
16.16
19.46
.001
-.235
Class presentations
Vague
Enumerated
2.71
0.82
2.69
0.81
.026
.121
.022
.024
Discuss grades
w/faculty
Vague
Enumerated
2.69
1.67
2.66
1.80
.004
.001
.028
-.064
Discuss w/studs
outside class
Vague
Enumerated
2.31
1.02
2.41
1.20
.001
.001
-.126
-.141
Discuss w/faculty
outside class
Vague
Enumerated
2.10
1.35
2.25
1.67
.001
.001
-.175
-.156
Discuss w/others
outside class
Vague
Enumerated
2.53
3.01
2.63
3.35
.001
.001
-.124
-.100
Results
Are there group differences between vague and enumerated estimates of the same behaviors based on
completion of calculus?
Calculus
Ask questions
Vague
Enumerated
No
3.10
15.86
Yes
3.20
17.97
sig
.001
.001
d
-.116
-.149
Class presentations
Vague
Enumerated
2.70
0.82
2.68
0.81
.037
.118
.017
.019
Discuss grades
w/faculty
Vague
Enumerated
2.71
1.74
2.62
1.59
.001
.001
.110
.074
Discuss w/studs
outside class
Vague
Enumerated
2.28
1.01
2.40
1.12
.001
.001
-.153
-.082
Discuss w/faculty
outside class
Vague
Enumerated
2.11
1.41
2.14
1.37
.001
.110
-.043
.020
Discuss w/others
outside class
Vague
Enumerated
2.51
2.97
2.62
3.26
.001
.001
-.124
-.083
Discussion
Overall this study found that:
The more frequent the behavior, the shorter the time frame the
respondent uses when estimating the behavior. The implication for
researchers is that time frame is an important factor when asking
respondents to enumerate estimates of past behaviors.
Discussion
Overall this study found that:
The more frequent the behavior, the shorter the time frame the
respondent uses when estimating the behavior. The implication for
researchers is that time frame is an important factor when asking
respondents to enumerate estimates of past behaviors.
Vague quantifiers are associated with increasing enumerated
responses for the same behavior. This provides additional reliability
evidence for the use of vague quantifiers.
Discussion
Overall this study found that:
The more frequent the behavior, the shorter the time frame the
respondent uses when estimating the behavior. The implication for
researchers is that time frame is an important factor when asking
respondents to enumerate estimates of past behaviors.
Vague quantifiers are associated with increasing enumerated
responses for the same behavior. This provides additional reliability
evidence for the use of vague quantifiers.
There were minimal group differences regarding these estimates. The
implication of these results is that for these types of behaviors there are
likely no meaningful differences between these groups of students and
how they interpret vague and enumerated estimates of behavior.
AIR 51st Annual Forum
May, 2011
Any discussion, questions, or comments?
Thank you!
Jim Cole
[email protected]
Ali Korkmaz
[email protected]