Surveys and Questionnaires Research Methods & Data College of Advancing Studies Brendan Rapple.
Download ReportTranscript Surveys and Questionnaires Research Methods & Data College of Advancing Studies Brendan Rapple.
Surveys and Questionnaires Research Methods & Data College of Advancing Studies Brendan Rapple Surveys Provide Important Knowledge Economists, psychologists, health professionals, political scientists, and sociologists conduct surveys to study such topics as: Income and expenditure patterns among households; Voting behavior; Effects on family life of women working outside the home, etc. Auto manufacturers use surveys to find out how satisfied people are with their cars. And a multitude of other topics This presentation owes much to the American Statistical Association brochure series on survey research: http://www.amstat.org/sections/srms/whatsurvey.html Specific Purpose Essential Objectives of a survey should be as Specific Clear-cut Unambiguous as possible "Men's Health Practices" is a very nebulous topic. Better: How often do African-American males aged 40-49 visit the dentist? Or A survey of 50-60 year old male professors at BC about their weekly exercise habits Steps in Conducting A Survey Define precise purpose Specify population Specify appropriate sample How to administer survey? Draft of survey instrument Pretest it Revise it Administer survey Analyze, write it up, and communicate the results Use results meaningfully Decide on Mode of Data Collection Mailed Questionnaire Telephone In Person Interview Computer Questionnaire Advantages of In-Person Interview Interviewer can clear up any ambiguity in the questions. The interviewer can gather more complex and “sophisticated” answers/input. The interviewee does not have to be literate. Of course, in-person interviews can be very costly as well as time-consuming. Use Mailed/Computer Questionnaires or Not? Perhaps “better” responses by phone -- but phone interviewing is very time-consuming. On other hand, people tend to be more truthful with anonymous questionnaires. Generally cheaper than one-on-one interviews. Perhaps a mix of questionnaire and interviewing? For All Surveys it is Essential to explain purpose of survey very clearly and precisely. explain any potential use the results will have for the respondents themselves. stress voluntary nature - respondents are doing a favor to the researcher. explicitly promise confidentiality. mention a clear expression of thanks. Whatever Format Used . . . . . . Important to specify a deadline for response. Follow-up letter (e-mail) often advisable for a mailed/computer questionnaire Always essential for PILOT STUDY or PRE-TEST Main problems revolve about: Question content, e.g. confusion with overall meaning of question as well as misinterpretation of individual terms or concepts Population → Sample Usually the population to be surveyed is too large. Accordingly, one must select a smaller, representative sample. “This sample is usually just a fraction of the population being studied.” Samples The quality of the sample – whether it is up-to-date and complete – is probably the dominant feature for ensuring adequate coverage of the desired population to be surveyed. Must be representative of population. Are the distributions of attributes, opinions, and beliefs in the sample the same as in the population? You want to be able to make inferences about the population as a whole based on what you find to be true of the sample. Variability Variability is large, then sample often should be large Converse also true 2 Barrels of Apples Barrel A (low variability) -- all apples about 5 ins. in diameter (range 5.1 to 4.8 ins.) Barrel B (high variability) -- apples range from 2 to 6 ins. in diameter Picking 3 apples from Barrel B might give result well below (above) average. Still, Size of Sample Isn't Everything Large numbers do not, in and of themselves, increase the representativeness of a sample. Most professional survey conductors hold that a moderate sample size is enough statistically and operationally. Representative Sample Survey: Success of unwed teenage mothers in a specific community in raising children? To be representative, sample must contain same proportion of unwed teenage mothers at --each age level --each educational level --each socio-economic status in the community Population -- Sample It is essential that you select sample in such a way that every name on the population list has an equal chance of being included in the sample. Random Sample Example: 500 part-time students in Advancing Studies Sample of 20% is required Assign each student a number from 1 to 500 Randomly select 100 numbers Systematic Random Sampling Example 1. 2,000 in population and you want a sample of 200, then you might select every 10th name Example 2. 500 part-time students in Advancing Studies Sample of 20% is required --Randomly Select a Number from 1 to 5 --Select Every 5th Person --002, 007, 012. 017, 022, and up to 497. Possible Problem Staff in govt. agency may be listed unit by unit Each unit has 9 line-level workers and 1 supervisor. The supervisor is the 10th person on the list. It’s a survey of 20% -- every 5th person is selected. If first no. selected is 1, 2, 3, or 4 then no supervisor will be selected, though they comprise 10% of population. If first number selected is 5, then supervisors will be greatly overrepresented. Thus, possibility of bias due to periodicity or patterns. Stratified Sampling Population: 2,000 (800 females; 1,200 males) Sample required: 200 If gender is an important variable in your survey, then both females and males should be included in appropriate numbers, that is, in proportions that correspond to their presence in the population. Strategy: Treat both sexes as separate populations and take 10% sample from each. OR Make sure that all females are listed first and then take every tenth name. Either way you will end up with 80 females and 120 males Convenience Sampling Could ruin an otherwise well-conceived survey. It’s simple and cheap to select a sample of names from a phonedirectory to find out which candidate people intend to vote for. However, this sampling procedure could give incorrect results since persons without telephones or with unlisted numbers would have no chance to be reflected in the sample. Also, more and more people have got rid of their landlines. Their voting preferences might be quite different from persons who have listed telephones. Confidentiality Confidentiality of data supplied by respondents is of prime concern to all reputable survey organizations. Important that individual respondents are not identified in reporting survey findings. All of the survey’s results should be presented in totally anonymous summaries, such as statistical tables and charts. Problems with Volunteers Example: TV programs asking viewers to vote. people call who are most committed to issue. “stuffing of ballots” by multiple calls. Time of day is important – who’s available? Margin of Error Error margin of 1,000 randomly chosen individuals is generally said to be about 3.1%. Thus, if a random sample of 1,000 indicates that 59% will vote for Obama, the actual number could range from 55.9% to 62.1%. Questions in a Questionnaire Should Be Woven together Flow smoothly Avoid confusion A good questionnaire forms an integrated whole. Different Understandings Everyone should see/understand the exact same question – no ambiguity. But people from different backgrounds, with diverse frames of reference, may have different perceptions of the same question. KISS Principle Keep It Simple, Statistician Questions should be, as far as possible: Simple Clear, Easy to answer Personally relevant to them Often recommended that questionnaires be written at the 5th grade reading level Do You Need Permission for a Survey? Administering a survey may be a sensitive issue. Perhaps you require permission from the organization where you work? Perhaps you require approval from your college’s/university/s Institutional Review Board (IRB)? Remember these Four Points: It’s essential to ask the correct question that the respondents understand the question that the question is one that the respondents know that the respondents are both willing and capable of answering the question Some Examples of Questions Avoid Ambiguity Do you favor governmental involvement in health care? “What is your income?” __________________ do you mean: weekly monthly annual pretax after tax from salary or from all sources Don’t be Vague Bad What major kitchen appliances did you purchase for your home during the past 12 months? List them. Better Check off which major kitchen appliances you purchased for your home during the past 12 months: -- Washing machine -- Dishwasher -- Refrigerator -- Kitchen oven Another Example: (A blouse manufacturer wants to ascertain what type of sleeves teenage females prefer in their blouses) 1. 2. 3. Do you like short sleeve blouses? YES__ NO__ Do you like long sleeve blouses? YES__ NO__ Do you like sleeveless blouses? YES__ NO__ Another Example “Do you jog regularly?” Problem, of course, lies with REGULARLY Avoid jargon, slang, abbreviations Plumbers talk about “snakes” Psychologists about “oedipus complex” Lawyers about “mens rea” “NATO” usually means North Atlantic Treaty Organization But some respondents might take it to mean: --National Auto Tourist Organization --Native Alaskan Trade Orbit --North African Tea Office Types of Questions Open-Ended Questions: What is your age? What is the total turnover in your company? Which of the four seasons do you prefer? How would you spend a a $1,000,000 lottery win? Disadvantages of Open-ended Questions: Variation in answers make coding/scoring difficult Advantages of Open-ended Questions Do not impose researcher's opinion on respondent Can lead to a very precise answer Forced-Choice Questions How many books do you read each year? Please check as appropriate below: ___ ___ ___ ___ ___ none 1 to 5 6 to 10 11 to 20 more than 20 What do you recall about the frequency of snowfall in Boston during the winter of 2009? ___ ___ ___ ___ It snowed almost every day It snowed about once a week It snowed about once every two weeks It snowed about once a month Importance of Wording Take a very simple question: How many drinks do you have each day? (Check one of the following) ____ 5 or more ____ 4 ____ 3 ____ 2 ____ 1 ____ none Better to Ask Are your daily drinking habits reasonably consistent --i.e. do you take about the same number of alcoholic drinks each day? _____ YES _____ NO (if you mark "NO," skip the following question). Checking Respondents' Consistency Q. 4. Check one of the following: _x_ I believe that manufacturing should be increasingly computerized, even if layoffs ensue. ___ Preserving the jobs of workers is more important than computerizing manufacturing. Later on, the Questionnaire might ask: Q. 30. Check one of the following: ___ I support G.M.'s increasing use of robotics, despite the resulting massive layoffs. _x_ The government should force G.M. to curtail computerization to ensure a reduction in layoffs. Avoid Emotional Language “What do you think about a policy to pay murderous terrorists who threaten to steal the freedoms of peace-loving people?” Problematic Words: murderous freedoms steal peace Avoid Loaded Questions: "Should the mayor spend even more tax money trying to keep the streets in top shape?" "Should the mayor fix the pot-holed and dangerous streets in our city?" Arrangement Is Important What is your present marital status? 1 never married 2 married 3 divorced 4 separated 5 widowed A Better Arrangement Would Be: What is your present marital status? (circle number) 1. 2. 3. 4. 5. never married married divorced separated widowed Danger of Overlapping Check How Many Children are in Your Family: a. 0-1 children b. 1-2 children c. 2 or more children Avoid Prestige Bias “Most doctors say that cigarette smoke causes lung disease for those near a smoker”. Do you agree?” “Do you support the president’s policy regarding Zanozui?” Avoid Double-Barreled Questions: "Does your company have pension and health insurance benefits?" Be Careful with Sensitive Questions Sensitive Question: What is your annual income? $______ Better to Phrase the Question as . . . Which of the following categories does your annual income best fit: ___ Less than $10,000 ___ $10,001-$20,000 ___ $20,001-$30,000 ___ $30,001-$40,000 ___ $40,001-$50,000 ___ $50,001-$60,000 ___ over $60,001 Avoid Double Negatives Poor Question: Do you not think that it is an unsatisfactory policy for the U.S. to continue sending financial aid to Egypt? Ranking Answers (Likert Scale Format): Women should automatically receive three months maternity leave in your company strongly agree ___ agree partly agree disagree ___ ___ ___ strongly disagree ___ How often do you attend church? Never ____ Yearly ___ Monthly ___ Weekly ___ Daily ___ Visual Analog Scales BIPOLAR scales I see myself as: Extremely interested in politics ______:______:______:______:______ Extremely uninterested in politics Comments Adding a “Comments” box is often very useful.