Transcript PowerPoint Presentation - Indiana University Bloomington

Using Cognitive Interviews to
Improve Survey Instruments
Presented at the Association for Institutional
Research Forum
June 2-6, 2012
Heather Haeger, Indiana University
Amber Lambert, Indiana University-Bloomington
Jillian Kinzie, Indiana University-Bloomington
James Gieser, Indiana University
Agenda
• Introduction
• Conceptual Framework
• Methods
• Findings
• Conclusion and Discussion
Current Context
• Pressure to assess educational practices
remains high
• Crucial to ensure that instruments accurately
measure educational practices and
experiences
o Questionnaires must measure what they intend;
respondents must understand & correctly interpret
items
• NSSE’s widespread use (1,500 institutions)
makes it particularly important to rigorously
subject items to cognitive research testing
Context for NSSE’s Cognitive
Interviews
• Cognitive Interviews (CI) part of
NSSE survey design from outset
• Focus in 2005 to test survey
among historically underrepresented students
• Planned NSSE update for 2013
provided occasion for multiple
rounds of CIs
Purpose
• Purpose of cognitive interviews
o Identify and analyze sources of
response error
• Focus
o Cognitive processes
o Access respondents
interpretation & meaning
of items
Purpose (cont.)
• In brief, cognitive interviews are
meant to identify…
o Whether subjects understand the
question…
o In a way consistent across subjects…
o And in the way intended by
researchers.
Background
• Four actions of the cognitive
process:
o Comprehend the question
o Retrieve information
o Make a judgment about relevance and
accuracy
o Formulate and provide a response
Methods
• “Think-aloud”
o Explicit activity in which the subject verbalizes, his/her
thought processes as s/he answers survey questions.
• Interviewer reads the question, then observes and records as the
subject responds.
• Interviewer is mainly passive in process, aside from providing
encouragement to “tell me what you’re thinking” if s/he hesitates or
pauses.
o Advantages: Freedom from bias imposed by frequent
interviewer interjections; minimal interviewer training
requirements; open-ended design.
o Disadvantages: Subject usually requires training in
method, or may resist technique; possibility for subject
to stray from the topic at hand; subject may bias
his/her description of his/her decision processing.
Methods (cont.)
• Verbal probing: concurrent and
retrospective
o Concurrent: after interviewer asks a question and
subject answers, interviewer asks more specific
questions designed to elicit further information about
the response. Probes can be scripted or
spontaneous.
o Retrospective: at the end of interview, subject is
asked to verbalize their thoughts about questions
answered earlier when taking the questionnaire.
o Advantages: Interviewer maintains control of the
interview; relative ease of training the subject.
o Disadvantages: Artificiality – criticism that this
technique is not reflective of a real survey interview,
in which interviewer simply asks questions and
respondent answers them; potential for bias through
poor selection of probes.
Analysis
3 stages and related sub-stages that
respondents faced during cognitive interviews:
1) Understanding the survey question and response options
a) Comprehending the survey question
b) Comprehending the response options
2) Performing the primary survey tasks
a) Retrieving information
b) Deduction; making conclusions about information
c) Mental arithmetic computation
3) Formatting responses
a) Mapping data yielded by primary task processes to
an explicit response option
b) Response option is not available/offered
Types of Problems
Coding within these stages can
address any of the following
problems:
o Language problems
o Inclusion/exclusion problems
o Temporal problems
o Logical problems
o Computational problems
Example 1
Question: In your experience at your institution
during the current school year, about how
often have you had serious conversations with
people who differ from you in the following
ways? (Never, Sometimes, Often, Very Often)
a) Political views,
b) Economic and social background,
c) Religious beliefs or philosophy of life,
d) Race, ethnic background, or country of
origin,
e) Sexual orientation
Example 1 (cont.)
• Language Problem
o Too wordy, too complex
o Diversity of interpretation
• Talked with people who are different
• Talked with people who are different
but only about that topic
• Talked about that topic with anyone
• Logic problem
o Serious conversations + people who are
different from you + in the following ways
Example 2
Question: Indicate the quality of
your interactions with the
following people at
your institution:
a) Student Affairs Professional
Example 2 (cont.)
• Language problem
o Not knowing what a Student Affairs Professional was
o Wording was changed to “Student services staff (campus
activities, housing, career services, etc.)”
• Inclusion/exclusion problem
o Students were including everyone
• Added parenthetical to help narrow focus
o This raises a question for researchers: are we ok with
students thinking broadly about this question (e.g. including
dining hall staff and campus security staff)?
• Formatting problem
o Response options ranged from Poor=1 to Excellent=7
o Need to add an NA option
Example 3
Question: During the current
school year, in about how
many of your courses did you
do the following?
Participated in a community-based
project as part of a regular course
(i.e., service-learning)
Example 3 (cont.)
• Language problem
o This problem was site specific
• Inclusion/Exclusion problem
o Students wondered what activities to
include
• What counts as a “community-based”
activity, and what did we mean by
“community” (campus only? Surrounding
neighbor/town too? Elsewhere in
US/overseas?)
• Some students thought of volunteering on
their own, not in connection to a
particular course
Example 4
Question: Which of the
following have you done or do
you plan to do before you
graduate from your institution?
Participate in a formal program where
groups of students take two or more
classes together (sometimes called a
learning community)
Example 4 (cont.)
• Language problem
o As with service learning, students’ understanding
of this question was also site specific
• Formatting problem
o Location on survey made students less likely to
complete the item if they didn’t understand it
o Solution: We restructured where the item
appeared in the instrument
• We moved the item lower on the page (it
originally was the first question at the top of
the page)
• We moved the item lower within the question
itself (it originally was the first item in series of 6
sub-questions
Example 5
Question: In a typical week this
year, about how many total
pages have you read for all of
your courses?
Example 5 (cont.)
• Computation problem
o Difficult for students to compute number of
pages, especially if sources didn’t have page
numbers (e.g. web pages, e-readers)
• Inclusion Exclusion problem
o Many students didn’t count readings in nontraditional book format (e.g. readings online,
lab reports)
o Some students counted what they actually
had read, but some said that they counted
all the pages assigned in the course, even
those that they had not actually read.
Applications to Your Campus
• How might these methods help triangulate NSSE
results on your campus?
• What questions are you most concerned about in
terms of what your students mean by their
responses?
• Are there item terms that may have less face
validity with your student populations?
• How might it help to know more about students’
interpretations of response option in terms of what
to do with findings?
• How might you initiate this activity on your
campus? Who might be interested in this work?
Who should conduct the interviews?
Conclusion
• Cognitive interviews and focus
groups can provide students an
opportunity to reflect on
their behaviors in college
• Institutions gain concrete
information about THEIR students
experiences, perceptions
o Can then be more explicit about opportunities
and academic services of which students can
take advantage
Contact Information
• Heather Haeger – email:
[email protected]
• Amber D. Lambert – email:
[email protected]
• Jillian Kinzie– email:
[email protected]
• James Gieser – email:
[email protected]
Introducing Updated
NSSE!
• Retains NSSE’s focus on diagnostic
& actionable information
• New Engagement Indicators
o
o
o
o
o
o
o
•
•
•
•
Academic challenge
Deep approaches to learning
Collaborative learning
Quantitative reasoning
Experiences with faculty
Campus environment
Interactions with diversity
Modules
New & Updated Items
Comparisons to Prior-Year Results
FSSE & BCSSE Updates
Register Now for
NSSE 2013
(deadline Sept. 25)