Transcript Quantitative Research Methods

QUANTITATIVE RESEARCH
METHODS
Irina Shklovski
Quantitative Research Methods


Include a wide variety of laboratory and nonlaboratory procedures
Involve measurement…
Quantitative Research Methods

Measurement
 Populations
and Sampling
 Random Assignment
 Generalizability
Quantitative Research Methods

Measurement
Populations and Sampling
 Random Assignment
 Generalizability


Time
Cross-sectional studies & single experiments
 Longitudinal studies & repeated measures

Quantitative Research Methods

Method
 Experiments
& Quasi-experiments
 Behavioral Measures
 Questionnaires & Surveys
 Social Network Analysis
 Archival and Meta-Analysis
What we will talk about today

Measurement
 Population
& Sampling
 Random Assignment
 Generalizability

Method
 Experiments
& Quasi-experiments
 Questionnaires & Surveys
Measurement – Sampling

Specify your population of concern

Sampling
 Selecting
respondents from population of concern
 Random sampling
 Systematic selection
 Stratified sampling
 Convenience sampling
 Snowball sampling
Sampling Biases

Non-response bias
 Be
persistent
 Offer incentives and rewards
 Make it look important

Volunteer bias
 Some
people volunteer reliably more than others for
a variety of tasks
Random assignment

Different from random sampling

Mostly used for experiments or quazi-experiments

Protects against unsuspected sources of bias


Does NOT guarantee to balance out the differences
between participants
Chance is LUMPY
Generalizability



How do you know that what you found in your
research study is, in fact, a general trend?
Does A really, always cause B?
If A happens, is B really as likely to happen as you
claim? Always? Under certain conditions?
Association vs. Causality
Thanks to
Sara Kiesler
for these graphs!
Experiments & Quasi-experiments

ex·per·i·ment
Pronunciation: \ik-ˈsper-ə-mənt also -ˈspir-\
 Function: noun
 Etymology: Middle English, from Anglo-French esperiment,
from Latin experimentum, from experiri
 Date: 14th century


An operation or procedure carried out under controlled
conditions to discover an unknown effect or law, to test
or establish a hypothesis, or to illustrate a known law
Experiments & Quasi-experiments

Key feature common to all experiments:
 To
deliberately vary something in order to discover
what happens to something else later
 To seek the effects of presumed causes
An Experiment is

A controlled empirical test of a hypothesis.

Hypotheses include:
A causes B
 A is bigger, faster, better than B
 A changes more than B when we do X


Two requirements:
Independent variable that can be manipulated
 Dependent variable that can be measured

Experiments in Research





Comparing one design or process to another
Deciding on the importance of a particular feature
in a user interface
Evaluating a technology or a social intervention in a
controlled environment
Finding out what really causes an effect
Finding out if an effect really exists
Remember


Experiments explore the effects of things that can
be MANIPULATED
(but there is a caveat)
Types of Experiments



Randomized – units/participants assigned to receive
treatment or alternative condition randomly
Quazi – no random assignment
Natural – contrasting a naturally occurring event
(i.e. disaster) with a comparison condition
If your study involves experiments

Experimental design:
Shadish W.R., Cook T.D. & Campbell P.T. (2002) Experimental
and Quasi-Experimental Design for Generalized Causal
Inference. Boston, Mass: Houghton Mifflin

Experimental data analysis:
Bruning, J. L. & Kintz, B. L. (1997). Computational handbook of
statistics (4th ed.). New York: Longman.
Questionnaires & Surveys

Self-report measures
 Questionnaires
 Interviews
 Diaries

Types
 Structured
 Open-ended
& surveys
Questionnaires & Surveys

Advantages
 Sample
large populations (cheap on materials & effort)
 Efficiently ask a lot of questions

Disadvantages
 Self-report
is fallible
 Response biases are unavoidable
Response biases

Relying on people’s memory of events & behaviors
 Emotional
states can “prime” memory
 Recency effects
 Routines are deceiving

Social desirability
 Solution:

none that are simple
Yea-saying
 Solution:
vary the direction of response alternatives
General Survey Biases




Sampling – are respondents representative of
population of interest? How were they selected?
Coverage – do all persons in the population have
an equal change of getting selected?
Measurement – question wording & ordering can
obstruct interpretation
Non-response – people who respond differ from
those that do not
Design is KEY




Format – booklet, printed vertical, one-sided
Question ordering – earlier questions can prime
answers to later questions
Page layout – group similar items & use consistent
fonts and response categories
Pre-testing – conduct think-alouds with volunteers
demographically similar to expected participants
Common Problems

Avoid complicated & double-barrel questions
 Complexity


increases errors & non-response
Navigation is paramount – make sure the survey is
EASY to follow
Open-ended questions
 The
size of the field allotted will determine the number
of words

Incentive is key
 BUT
amount differences have little impact
If your study involves surveys

Designing surveys:
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet,
mail, and mixed-mode surveys : the tailored design method (3rd
ed.). Hoboken, N.J.: Wiley & Sons.
Fowler, F. J. (1995). Improving survey questions : design and
evaluation. Thousand Oaks: Sage Publications.

Analyzing data:
Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied
multiple regression/correlation analysis for the behavioral
sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum
Associates.
So… what?



Difference between quantitative methods is in the
questions they can answer
There are a LOT of methods and even more
statistical techniques
Regardless of the method, if it’s not an experiment,
you CAN NOT prove causation
Things we did NOT talk about




Reliability assessments
Validity assessments
Statistical analysis of data
Interpretation of results