Transcript TUTUT - Tampereen Teknillinen Korkeakoulu

IHTE-1800 Research methods:
Surveys and interviews
Sari Kujala, Spring 07
http://www.cs.tut.fi/ihte
Contents
• Survey
- Types
- Process
• Interviews
- Types
• Literature
http://www.cs.tut.fi/ihte
What is a survey? (Pfleeger& Kitchenham, 2001)
• A comprehensive system for collecting
information to describe, compare or
explain knowledge, attitudes and behavior
http://www.cs.tut.fi/ihte
Types of surveys
• Supervised
- Telephone interviews
- Group surveys
• Unsupervised
- Mailed questionnaire
- Electronic questionnaire
http://www.cs.tut.fi/ihte
Survey process (Pfleeger &Kitchenham, 2001)
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Setting measurable objectives
Planning and scheduling the survey
Ensuring that appropriate resources are available
Designing the survey
Preparing the data collection instrument
Validating the instrument (piloting)
Selecting participants
Administrating and scoring the instrument a
Analyzing the data
Reporting the results
http://www.cs.tut.fi/ihte
Setting measurable objectives
• Preferably research questions or
hypotheses
• Statements of the survey’s expected
outcomes
- What information will be identified
- Target population
• Definitions of all potentially ambiguous
terms
http://www.cs.tut.fi/ihte
Designing the survey
• The goal is to provide the most effective
means of obtaining the information
- No bias
- Apparopriate (makes sense in the context of
-
the population)
Cost-effective
http://www.cs.tut.fi/ihte
Descriptive designs
• Cross sectional
- Information at one fixed point in time
• Cohort
- Information about changes in a specific
population
• Case control
- Retrospective information about previous
circumstances to help explain a current
phenomenon
http://www.cs.tut.fi/ihte
Experimental designs
• Concurrent control studies - participants are
•
•
•
•
randomly assigned to groups
- E.g. is training changing attitudes?
Concurrent control studies – participants are
not randomly assigned to groups
Self-control studies
- Pre- and post-treatment measures
Historical control studies
- E.g. comparison with previous surveys
Combination of techniques
http://www.cs.tut.fi/ihte
Sample size
• Sample size should be big enough
- If different groups are compared, each group
should have app. 30 data points in order to
make statistical analyses
• Start sampling by defining a target
population
- May be a subset of a larger population,
inclusion or exclusion criteria may be used
http://www.cs.tut.fi/ihte
Sampling methods
• Probabilistic
- Random
- Stratified random sample (subgroups)
- Systematic sampling (every nth member)
- Cluster-based sampling (belonging to a
defined group)
• Non-probabilistic
- Convenience (who is available), snowball
http://www.cs.tut.fi/ihte
Response rate
• A low response rate can destroy a good
survey (range is from 10 % up to 90 %)
• The reasons for non-response should be
known
http://www.cs.tut.fi/ihte
Improving response rate
• Over-sampling, reminders, rewards
• Ensuring that people are able, willing and
motivated to answer the questions
• Respondents should see some clear
benefit to answering the questions
http://www.cs.tut.fi/ihte
Data collection instrument
• Search the relevant literature
- Existing instruments?
• Construct an instrument
• Evaluate/pilot the instrument
• Document the instrument
http://www.cs.tut.fi/ihte
Constructing an instrument: what to ask?
• Use survey objectives
• Consider the respondents
- Questions should be easy and accurate to
-
answer, events not happened long in the past
Respondents have sufficient knowledge and
position to answer
• Remember background or demographic
questions to identify the respondent
http://www.cs.tut.fi/ihte
Question types
• Open questions
• Standardized response format
- Multiple-choice question
- Likert scale statements
http://www.cs.tut.fi/ihte
How to ask
• Don’t ask too many questions – too time
demanding to answer –low response rate
- Think every time: how you can use the answers, what
is your hypothesis
• Don’t ask too many open questions
- Laborious to answer
- Difficult to classify, time consuming analyze
• Standardize responses when appropriate
- offer possibility to answer “other”
• Give the respondent enough instructions
http://www.cs.tut.fi/ihte
Information and instructions for respondents
• Include a cover letter providing a contact name
•
•
•
•
•
and information
Explain the purpose and relevance of the study
Describe who is sponsoring the study and how
confidentiality will be preserved
Explain how the respondents were chosen and
why
Explain how to return the questionnaire
Provide a realistic estimate of the time to
complete the questionnaire
http://www.cs.tut.fi/ihte
A good question is:
• Purposeful from the respondents’ point of
•
•
•
•
•
•
view
Clear
Neutral, not leading
Concrete
Concentrating on essentials
Asks only one issue
All in all easy to answer
http://www.cs.tut.fi/ihte
Example: Requirements quality questionnaire
Requirements are completely
defined.
The requirements describe a
system that meets user needs.
Disagree Agree don’t
1 2 3 4
know
1 2 3 4 []
1
2
3
4
[]
The requirements are based on
1
the information gained from users
and customers.
In all likelihood, there are
1
moderately few errors in the
requirements.
2
3
4
[]
2
3
4
[]
http://www.cs.tut.fi/ihte
Piloting
• Is the most important way of improving the
quality of the questionnaire
- Use real people belonging to target group
- How is your questions and the used words
-
understood?
Is all options available?
Iterate
http://www.cs.tut.fi/ihte
Types of reliability
• Test-retest reliability
• Internal consistency
- Is a group of items forming a single scale?
- Statistical measures
• Inter-observer reliability when observers
are completing a survey instrument
- Statistical measures
http://www.cs.tut.fi/ihte
Types of validity
• Face validity
• Content validity
- A group of reviewers with knowledge of the subject
matter and target population members review the
survey contents
• Criterion validity
- A measure of how well one instrument compares with
another instrument or predictor
• Construct validity
- The extent to which different data collection
approaches produce similar results
http://www.cs.tut.fi/ihte
Analyzing the data
•
•
•
•
Check the incomplete questionnaires
Partition the responses according to subgroups
Make figures of the results
Statistical analysis depending on the scale type
of replies
- Frequency, mean, variance etc.
- Chi-squared test to measure associations among
-
nominal scale variables
Variance analysis, correlations
http://www.cs.tut.fi/ihte
Interviewing types
• Structured interviews
- Questionnaire used
• Semi-structured or thematic interviews
- Pre-defined themes used
- Additional questions in order to understand the
answers and find new interesting issues
• Open interviews
- More like free discussions
• Individual or group interviews, focus groups
http://www.cs.tut.fi/ihte
Literature
• Bourque, L. and Fielder, E. (1995) How to conduct self•
•
•
•
•
administrated and mail surveys, Sage Publications Inc.
Fink, A. (1995). The Survey Handbook. Sage Publications.
Pfleeger and Kitchenham (2001). Principles of survey research.
Parts 1-5. Software Engineering Notes.
Straub, D.W. (1989) Validating Instruments in MIS Research, MIS
Quarterly, 13, 2, 147-169.
Andrews, D., Nonnecke, B., Preece. J. (2003) Electronic survey
methodology: A case study in reaching hard-to-involve internet
users. International journal of human-computer interaction, 16, 2,
185-210.
Ropponen, J. and Lyytinen, K. (2000) “Components of Software
Development Risk: How to Address Them? A project manager
survey”, IEEE Transactions on Software Engineering, 26, 2, 2000,
98-112. (good example)