Transcript Presentation
Writing Survey Questions
“Items”
Overview
Characteristics of good survey items Two basic kinds of questions Some Common item writing problems Scale selection Evaluating questions after you write them
Its harder than you think
Despite your best efforts, you will probably write questions that raise more questions than they answer There are no real hard and fast rules, but rather a set of guidelines Common sense is the real rule CAVEAT of all survey research - - - surveys record second hand information, communicated perceptions, opinions, beliefs. Direct observation is always better (but not always possible)
What will you measure
Always begin by asking yourself what you intend to measure or evaluate. This serves as the guide to what items you need to make a decision.
Ask yourself what covariates you will need (blocking variables) Common ones include, age, gender, SES, race, ethnicity, but you may identify some that are relevant to your study (Fowler, 1995) A guiding question should be “on what might the answers depend” (Yovanoff, 2005) More is usually better than less – you can always throw them away if you don’t need them, but you usually cant go back and get more info
5 Basic Characteristics
There are some basic characteristics of questions and answers that are fundamental to good measurement (Fowler, 1995) 1.
Question need to be consistently understood – the same for each person 2.
Questions need to be consistently administered – sometimes an issue with in person interviews, less so with online surveys 3.
Respondents need to be capable of understanding the question, and understanding how to answer 4.
Respondents need to be willing to answer In general – we want to maximize the degree to which an question produces answers that measure something
Two Basic Kinds of Questions
Those that aim to gather facts (objective) e.g., In what month were you born?
e.g., What is your approximate annual income in thousands of CDN dollars?
Those that aim to gather opinion (subjective) e.g., Indicate on the following scale how much you liked last night’s math homework.
e.g., What is the top reason you think an SDSU prof works for UofO is in BC?
Questions to Gather Facts
(e.g., month born, annual salary, etc.) Begin by clearly defining what you want to measure – overall and with each item. This definition will guide you in item development.
Decide how you want the response to look – on what scale (more on that later)
Questions to Gather Facts
Think of a common question you might see on a survey designed to gather a ‘fact’
Questions to Gather Opinion
(e.g. liking homework, why SDSU for UO in BC, etc.) When you gather opinion, you are gathering subjective data There is no right or wrong answer Sometimes these look like factual questions, but if people will respond in different ways, they are not e.g., How friendly is your teacher?
Open Ended Questions
They are needed when … you cannot put all possible responses on a scale, e.g., what are your work duties?
You want to understand thinking, e.g., how would you solve this problem?
When you really don’t understand what the answer might look like, e.g., how did you become homeless?
Unlike with scales or MC questions, open ended questions allow for more respondent flexibility – but it is is coupled with difficult analysis later People are not constrained to an area of response, so there is a lot of variability in the data Leads to subjectivity in coding / imposition of scale
Common Item Writing Problems (and how to deal with them)
Multi-dimensionality Ambiguous Stems Response Restriction Sensitive Items Distorted Responses
Item Writing - Dimensionality
A single dimension in a question is usually best – both in the stem and possible response e.g., How would you rate your health?
In this case “health” is ill defined, and may mean different things to different people. Is it level of fitness? Absence of disease? Weight? This construct has multiple dimensions e.g., How would you rate your teacher?
a) b) smart and not confident c) smart and confident no smart and confident d) not smart and not confident
Item Writing - Dimensionality
Sometimes this is referred to as double barreled In questions watch out for coordinating conjunctions, because they are designed to join clauses "and," "but," "or," "nor," "for," "so, “ , "yet “ e.g., Although the system of education in BC is of good quality, it really should not be mirrored in other provinces. Agree or Disagree?
You may agree the BC system is good or not good You may think it should be mirrored, or not.
Item Writing - Ambiguous
Avoid ambiguous words Be concrete, define key terms in questions Do you spend a lot of time studying in the Master ’ s Program?
On average, how many hours do you study each day ?
Do you usually prepare before class?
On average, how many hours do you spend preparing for class each week ?
Item Writing – Restricted Response
During the semester, on average, how many hours do you spend preparing for this class each week?
Less than 1 hour Between 1 and 2 hours Between 2 and 4 hours More than 4 hours VS During the semester, on average, how many hours do you spend preparing for this class each week?
Enter response _____
Item Writing - Leading
Watch out for statements of supposed fact e.g., Overall, would you agree with most people that our federal government is corrupt?
e.g., better – Overall, is our federal government corrupt?
e.g., This year, wild dogs killed more cats than in any other year, ever. Do you agree that dogs ownership should be better regulated?
e.g., better – Should dog ownership be regulated?
Item Writing - Sensitivity
Simple fact - some sensitive questions will not elicit truthful responses Many respondents will answer positively to avoid the question e.g., Do you love your children?
Better to triangulate with… How much time do you spend with your children?
Do you play games with your children?
How much do you cuddle your children?
Have you ever cried because of your child ’ s behavior?
Have you ever hit your child with a wooden bat?
Item Writing - Sensitivity
Take note – sensitivity is in the eye of the beholder – it is not so much the question, but the way a person will answer e.g. Have you been hospitalized in the last year? a.
b.
c.
For those who have not, it isnt sensitive – truthful response For those who have been to hospital for the flu, it probably isnt – probably truthful response For those who have been to hosptial for teenage pregnancy or an STD – it probably is – probably untruthful response
Item Writing - Sensitivity
Some studies seem to indicate that respondents tend to answer questions in a way that might make them look better, to the surveyor or the public (Locander, Sudman, Bradburn, 1976) Tend to under-report disease - when questioned about health (Cannell, Fisher, Bakker, 1965) Tend to report status quo - when questioned about voting (Madow, 1967) Tend to highly underreport - when questioned about masturbating (Sudman & Bradburn, 1982)
Item Writing - Sensitivity
To reduce sensitivity Assure confidentiality, use blind data analysis Better yet, don’t ask for names /personally identifying info (or do so at the end of the survey) Communicate the importance of accurate responses Reduce the role of the interviewer (if face to face) Reduce the amount of detail you ask for, so it is less obtrusive
Item Writing– Distortions
There are two primary reasons we might not get the answer we want outside of social desirability – “distortions in answers” The respondent may have forgotten The respondent may not have the information Be aware of respondents and how these factors may play into creating measurement error. Try to design around them. e.g., lead into a question with a brief description imagine yourself at home when you are eating dinner as you answer this question Keep in mind your responses are completely anonymous as you answer the following questions
Scales
With your question stem, you also need to design a response format. It might be a selection of choices, might be open ended, but often is on a scale.
Scale Selection
Your selection of response format (the scaling) is determined, in part, by the kind of analysis you might want to use Nominal – words/names – no order, no comparative meaning eg., red, blue, green, Mark, Mary, Chris Good for descriptive data analysis only Ordinal – ordered, less to more eg., no experience, a little experience, some experience, a lot Good for descriptive data, too, lends itself to proportions analysis like % or chi-square Interval (no zero) / ration (zero) – equal interval scales eg., month experience Lends itself to parametric statistics -t-tests, correlations, regression, ANOVA, etc.
Scale Selection
No matter what scale you choose, that scale is open to subjectivity. This is a constant source of measurement error, and is difficult to quantify.
e.g., if I ask you to rate this class on a scale from 1 to 10, one person’s 6 might be pretty good, where another’s 6 might mean just palatable.
Often, we use numbers, words, and pictures to let people express their feelings and opinions, each may be more or less appropriate in different settings
Response Scale Examples
Numbers eg., on a scale of 1 to 10… Words eg., a lot, some, only a little, not at all Pictures
Scales – the details
How many categories? Fewer categories leads to less variance (less data to analyze) But, research points to <10 (Andrews, 1984) being adequate In truth, 5 or 7 is sufficient (10 point scale just seems to be as much as people can handle) An even number will force one way or the other An odd number permits sitting on the fence
Evaluating Questions
1.
After creating questions, there are several methods we can use to evaluate their efficacy/quality Focus group discussions did you like this question? Why or why not?
2.
Interviews that focus on how a person answers a question (a cognitive approach) what were you thinking when you answered this question 3.
Field test under realistic conditions (pilot testing) e.g., give to 10 middle school age students first, before giving to the whole middle school
Evaluating Questions
1.
We can also evaluate the degree to which our questions work by looking at data after they are administered Predictive relationships of responses known in theory (e.g., how much do you like your job? What is your level of job stress?) 2.
3.
4.
Comparison of data from questions that ask almost the same thing (but are worded differently) Comparison of answers to records (when data available) Consistency of answers from a respondent at two time points
Activity
With a partner or small group, design a good question
Activity
With a partner or small group, design a good question Now, make it bad in just one of the ways I covered
Activity
With a partner or small group, design a good question Now, make it bad in just one of the ways I covered Come type on my computer when ready
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Some Bad Questions
Xxx Xxx Xxx Xxx Xxx Xxx Xxx Xxx Xxx xxx
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Activity – fix the questions
Xxx Xxx Xxx Xxx Xxx Xxx Xxx Xxx Xxx xxx
Another Activity
(pick an area to evaluate) Write 3-5 questions that fit your topic In the same groups, review the items to make sure they are good.
Consider covariates (usually demographic variables) you might want – these are also called blocking variables