Transcript Response rate
Research methodology in
DBA IMI / GOLESTAN
Research Methodology (1)
• What is research?
• Ways of knowing
• Characteristics of research
• What is “good” research?
• Approaches to research
• Research process overview
What is research?
• The systematic, controlled, empirical, and
critical examination of hypothetical propositions
about a phenomenon in order to enhance
Ways of knowing
Characteristics of common sense
• Unless we decide not to, we usually observe
• We usually generalize from only a few cases
• We observe selectively to see what we’re
• We make things up to fill in the gaps
• We believe in luck and fate
• We get personally and emotionally involved
Characteristics of research
• We consciously decide how to observe
• We explicitly sample for generalizing
• We consciously decide what to observe
• We base conclusions only on the evidence
• We believe in probability
• We have to respect scientific norms regardless
Research is more conscious and more careful
than knowing through authority, intuition or
What is “good” research?
• Carefully planed
• Theoretically grounded
• Carefully conducted
• Theory generates…
• Hypotheses, which are then subjected to…
• Observation and possibly lead to…
• Observation generates…
• Patterns leading to…
• Tentative hypothesis, that is next integrated
• The world does exist
• It is possible to study it objectively
• The world does not exist
• It is impossible to study it objectively
• Critical realism
The research process (1)
• A. Plan the research
– Define the research question(s)
– Define the population and sampling method
– Determine variables of interest and measures
– Determine model /set of hypothesis
– Select research design and statistical tests
– Write proposal
The research process (2)
• B. Conduct the research
– Identify sample
– Conduct experiment/ collect data
– Analyze data
– Test hypotheses / research questions
• C. Report the results
– Write research report
– Write academic paper
Research Methodology (2)
Response rate is…
# that answered
# you contacted
The proportion of people who return the survey
It is calculated by dividing the number of returned
surveys by the total number of surveys distributed.
Example: If you distribute 250 questionnaires and
you get 85 questionnaires back, your response rate
Low response to our surveys
We often send out surveys and find that few are
returned. Response rates of 30% and lower are
common in Extension. Often, the number of
returned surveys is too small to aggregate in a
meaningful way or make any comparisons.
Best practice says that a response rate under 70%
should be a warning.
Why is response rate important?
It’s the only way to know if your survey results
High response rate promotes confidence in
Low response rate increases the probability of
a higher response rate is preferable because
the missing data is not random
Comparing Early, Late, and NonRespondents
• Identify subjects who responded to the first mailing within the
deadline date, and label them as early. Similarly, identify all other
subjects who responded to subsequent mailings, and label them as
late. After the data collection is complete, identify and label the
non-respondents. According to Miller and Smith (1983), nonrespondents tend to be similar to late respondents in responding to
surveys. Therefore, compare the early and late respondent groups
on key variables (Figure 1). If you find no significant differences
between early and late respondents, you can statistically conclude
that non-respondents are perhaps similar to late respondents and
thus generalize the findings to the population. The other accepted
procedure is to follow-up with a telephone call to 15-20% of the
non-respondents, and collect data from them on key variables.
Then do a comparison between early and late, early and nonrespondents, and late and non-respondents.
Logic of Comparing Early, Late, and Non-Respondents
• If the comparison indicates no differences between
these three groups of respondents, then you can
generalize the findings to the population. On the other
hand, if you find significant differences, you cannot
generalize the findings to the population. Therefore,
use your judgment as to whether or not to include the
differed variables for final analysis. Normally, the
differed variables are eliminated from further analysis.
Explain why the subjects differed on the key variables.
Otherwise, provide justification for including the
differed variables in the final analysis.
• Use independent t-test to compare early and late
respondents; early and non-respondents; and late and
non-respondents. Use ANOVA if you want to compare
all three (early, late, and non respondents) response
types and conduct a post-hoc analysis to determine
group (early, late, and non respondents) differences.
Response rate (RR)
•How to achieve a high response rate?
•Getting a high response rate (>80%) from a small,
random sample is considered preferable to a low
response rate from a large sample.
There is no standard response rate
“The higher, the better.”
While it is not actually the % that matters but
WHO responds, a higher response rate means
that you can be more sure that the answers
reflect the population.
So, we want to remove barriers and
motivate as many people as possible to complete
and return the questionnaire.
How can we do that?
Most important things that
influence response rate:
Importance of the topic – interest in the topic of
Personalized request and communications
Multiple follow-up contacts
Sponsor of the survey is respected, trusted
Questionnaire is brief and easy to complete
• Early vs. late Respondents
• Respondents vs. non- Respondents
10 Ways to increase response rate
1. Generate positive publicity for your survey.
2. Appeal to people’s helping tendencies – ask
them to help by providing their input.
3. Make the survey topic salient – important
Ensure that respondents see the value of the
survey and their response.
Point out their personal connection to the topic
4. Tailor, personalize communications
5. Make the questionnaire attractive and easy to
complete AND easy to return
10 ways to increase response rate,
6. Provide incentives (token of appreciation)
7. Show positive regard; Say thank you
8. For mail survey, provide 1st class
9. Make (multiple) follow-up contacts
– by mail, email, telephone, in person…
10. Use a combination of survey modes –
telephone plus mail; internet plus mail.
Research Methodology (3)
Temperature in Celsius degrees (from 10°C to 20°C) ?
Age (from 0 to 99 years) ?
Date (from 1457 BC to AD 2013) ?
beautiful vs. ugly?
male vs. female?
The term "level of analysis" points to the location, size, or scale of a research target
True Score Theory
True Score Theory
is a theory about measurement. Like all theories, you
need to recognize that it is not proven it is postulated
as a model of how the world operates. Like many
very powerful model, the true score theory is a very
simple one. Essentially, true score theory maintains
that every measurement is an additive composite of
two components :true ability (or the true level) of
the respondent on that measure; and random error.
True Score Theory
• We observe the measurement -- the score on the test,
the total for a self-esteem instrument, the scale value
for a person's weight. We don't observe what's on the
right side of the equation (only God knows what those
values are!), we assume that there are two
components to the right side.
• The simple equation of X = T + eX has a parallel
equation at the level of the variance or variability of a
measure. That is, across a set of scores, we assume
Var (X) = var (T) + var (eX)
• In survey sampling, respondent error refers to any error
introduced into the survey results due to respondents
providing untrue or incorrect information. It is a type of
systemic bias .
• Several factors can lead to respondent error :
• Misunderstanding (Language and educational )
• Recall bias can lead to misinformation (misrecalling the facts
• Social desirability bias(he or she thinks is correct or better or
less embarrassing, rather than providing true and honest
• Improper administration or execution of a survey results
in administrative errors. Such errors can be caused by
carelessness, confusion, neglect, omission or another
blunder. There are four types of administrative errors.
1- Data-processing error
A category of administrative error that occurs in data
processing because of incorrect data entry, incorrect computer
programming or other error during data analysis.
2- Interviewer error
This type of administrative error is caused by failure of an
interviewer to correctly pose questions or record responses.
Interviewer error generally leads to biased results, and perhaps
to an increase in variability
3- Interviewer cheating
The practice of filling in fake answer or falsifying questionnaire while
working as an interviewer.
4- Sample selection error
Selection bias : an administrative error caused by improper
selection of a sample during a survey , resulting in accidental
bias in the results.
Same time : consistency
Different time : stability/consistency 77
What is construct?
Construct is a trait or attribute
You cant measure it.
But you can evaluate it by theories bakground
Intelligence , creativity , love
• Construct validity seeks agreement between
a theoretical concept and a specific
measuring device or procedure.
Construct validity can be broken down into
1- Convergent validity
2- Discriminate validity
• Convergent validity
• Discriminate validity
Issues with unstructured format
Can generate unpredictable responses
Dependent on number of respondent
Requires content analysis
Places a greater load on the respondent –
important issues not occur to them