Transcript First Steps in Designing Quantitative Research: Asking the Right

Susan R. Hutchinson, Ph.D.
Department of Applied Statistics & Research Methods
CETL Workshop, July 10, 2012
First Steps in Designing
Quantitative Research:
Asking the Right
Questions
1
• Overview of purposes for conducting
quantitative research
• Role(s) served by research questions
• Attributes of useful RQs
• Group activity on evaluating RQs
• Exercise on constructing RQs
CETL Workshop, July 10, 2012
Workshop Outline
2
CETL Workshop, July 10, 2012
Pretest
3
False
CETL Workshop, July 10, 2012
All quantitative studies include
one or more research
questions.
4
False
CETL Workshop, July 10, 2012
Research questions and
research hypotheses are the
same thing.
5
True
CETL Workshop, July 10, 2012
Research questions should
“foreshadow” the data
analyses.
6
False
CETL Workshop, July 10, 2012
Research questions should be
posed after data are collected.
7
False
CETL Workshop, July 10, 2012
Research questions should be
written using non-specific
language to allow flexibility in
designing the study.
8
True
CETL Workshop, July 10, 2012
All of the following are types of
variables that might be
included in research questions:
dependent, moderator,
mediator, extraneous,
independent
9
True

CETL Workshop, July 10, 2012
Writing quantitative research
questions requires knowledge
of statistics.
10
CETL Workshop, July 10, 2012
What are the major purposes
for conducting quantitative
research?
11
Quantitative research is
intended to examine:
• Differences between groups
• Change over time
• Direction, magnitude, and between-group differences
• Relationships between 2 or more variables
• Direction, magnitude, type/form, between-group
differences
CETL Workshop, July 10, 2012
• Statistical significance, magnitude, and direction of
differences
• Descriptions of populations and phenomena
12
CETL Workshop, July 10, 2012
As you’re constructing your
research questions, keep in
mind the overriding purpose of
quantitative research is to
understand differences
13
CETL Workshop, July 10, 2012
What role do research
questions play in designing
quantitative research?
14
• Indicate the target population
• Identify the variables/constructs and what we
want to know about them
• Foreshadow the type of data analysis (and
sometimes the research design)
CETL Workshop, July 10, 2012
What information should
research questions include?
15
What characterizes a “good”
research question?
• Clearly conveys the variables/constructs and the nature of
their relationships
• Must be answerable based on data from the study
• RQs should not be rhetorical or require a value judgment to
answer
• RQs should match the research design
• Should be worth investigating
• A good RQ does not address a trivial issue; “so what?”
• Answers to RQs should further the researcher’s understanding of
the phenomenon
CETL Workshop, July 10, 2012
• Includes independent, dependent, moderator, and mediator
variables (where applicable)
16
CETL Workshop, July 10, 2012
“It is better to have an
approximate answer to the
right question than an exact
answer to the wrong one.”
John Tukey
17
• Dependent
• Independent
• Extraneous
• Moderator
• Mediator
CETL Workshop, July 10, 2012
Types of variables that might
appear in research questions
18
Dependent variable(s)
• The dependent variable is a phenomenon one is
attempting to explain or predict
• In a survey study, often this is the variable or
construct of primary interest to the researcher
• It is something which the researcher wishes to
understand more fully
• For many researchers conducting nonexperimental
research this is the starting point in planning a study
SRM 627 Survey Research
Methods, Hutchinson
• Also referred to as the outcome or criterion variable
19
• In an experimental or quasi-experimental design,
an independent variable would be the
manipulated or treatment variable in your study
• In a survey or other type of nonexperimental
study, where no variables are manipulated,
independent variables are the factors used to
explain or predict the DV
SRM 627 Survey Research
Methods, Hutchinson
Independent variable(s)
20
• Extraneous variables are also called nuisance
variables
• These are variables which are not usually of
primary interest but are believed to be related
to the independent and/or dependent variables
• Their effects need to be controlled in order to
obtain meaningful results
• Extraneous variables are of two broad types:
• dispositional and situational
SRM 627 Survey Research
Methods, Hutchinson
Extraneous variables
21
salary
job
satisfaction
A researcher is interested in examining the relationship
between salary and job satisfaction. However, the
researcher suspects that factors other than salary (such
as job autonomy) are also responsible for an
employee’s level of job satisfaction.
SRM 627 Survey Research
Methods, Hutchinson
Illustration of controlling for
extraneous variables
22
job
autonomy
Relationship with “job autonomy" taken
into account or controlled
SRM 627 Survey Research
Methods, Hutchinson
salary
job
satisfaction
23
Moderator variable(s)
• A moderator variable influences the strength of
the relationship between the primary
independent variable and the dependent variable
• A moderator variable would be the variable that
interacts with the independent variable
SRM 627 Survey Research
Methods, Hutchinson
• Usually a moderator variable is of secondary interest
24
SRM 627 Survey Research
Methods, Hutchinson
Illustration of moderator effect
Computer skills/comfort
25
• Also referred to as an intervening variable
• A mediator (or mediating) variable is some
construct through which the independent
variable indirectly affects the dependent variable
SRM 627 Survey Research
Methods, Hutchinson
Mediator variable
26
Parents’
educational
level
Achievement
motivation
Performance
on standardized tests
SRM 627 Survey Research
Methods, Hutchinson
Illustration of mediator
variable
27
• What are psychologists’ perceptions of professional
development?
• Are university faculty depressed?
• What is the relationship between self-esteem and happiness?
• How many undergraduate students dropped out of college in
2011?
• What factors predict job satisfaction?
• How are exercise and blood pressure related?
• Is there a difference between gender and depression?
• What is the impact of test anxiety on test performance?
• Is religion good for society?
CETL Workshop, July 10, 2012
Examples of less than ideal RQs
28
• What is the researcher hoping to learn?
• What are the variables/constructs in the RQ?
• What is the target population?
• How might this RQ be improved?
• Which aspects of professional development do
psychologists rate most versus least positively?
CETL Workshop, July 10, 2012
What are psychologists’ perceptions of
professional development?
29
CETL Workshop, July 10, 2012
Or, instead of posing a
descriptive question about a
particular construct, consider
rewording so that you can
explain differences on the
construct.
30
CETL Workshop, July 10, 2012
• To what extent are private practice psychologists’
perceptions of the need for professional
development explained by their years of
experience, area of specialization, and
professional degree?
31
• What are the independent and dependent
variables?
• What assumption about university faculty is
implied by the research question?
• What are possible revisions?
• Does level of depression among university
faculty differ based on their gender,
departmental affiliation, and tenure status?
CETL Workshop, July 10, 2012
Are university faculty depressed?
32
• What are the independent and dependent
variables?
• What assumption about university faculty is
implied by the research question?
• What are possible revisions?
• Does level of depression among university
faculty differ based on their gender,
departmental affiliation, and tenure status, after
controlling for marital status and life
satisfaction?
CETL Workshop, July 10, 2012
Are university faculty depressed?
33
CETL Workshop, July 10, 2012
Hypothetical depression scores
34
CETL Workshop, July 10, 2012
For complex constructs that
are likely to be multiply
determined, you should
attempt to control for
potentially extraneous
variables.
35
• What do you think the researcher wants to
know?
• How might the question be revised to more
clearly convey the type of information sought?
• What is the strength of relationship between
self-esteem and happiness?
• Is self-esteem linearly related to happiness
among college-educated adults?
CETL Workshop, July 10, 2012
What is the relationship between selfesteem and happiness?
36
How many undergraduate students
dropped out of college in 2011?
• Which of the following variables predicted
undergraduate college student dropout in 2011?
(grade point average, perceived social support,
….)
CETL Workshop, July 10, 2012
• What is wrong with this question?
• How might it be improved in order to provide
more useful information?
37
• What are the independent and dependent
variables?
• What is the target population?
• How might you reword this RQ so that it
provides a clearer focus the researcher?
• To what extent do salary, autonomy, ….
CETL Workshop, July 10, 2012
What factors predict job satisfaction?
predict job satisfaction among full-time employees?
38
• What is the independent and dependent
variable?
• Target population?
• How would you reword this RQ if you had reason
to believe gender could be a moderator
variable?
• Does the strength of relation between amount of
daily aerobic exercise and blood pressure differ
for male and female adults?
CETL Workshop, July 10, 2012
How are exercise and blood pressure
related?
39
• What flaw do you notice in this RQ?
• What do you think the researcher actually wants
to know?
• What is a possible revision?
CETL Workshop, July 10, 2012
Is there a difference between gender and
depression?
40
• Assuming the researcher plans to collect data via
survey, what part of this RQ needs to be
reworded and why?
• How might you revise this item if you wanted to
determine if study strategies was a mediator
variable?
• Do study strategies mediate the relationship
between test anxiety and test performance
among doctoral level social science majors?
CETL Workshop, July 10, 2012
What is the impact of test anxiety on test
performance?
41
Is religion good for society?
• Are rates of violent crimes lower in communities
with higher attendance at formal religious
services?
CETL Workshop, July 10, 2012
• Why is this not a suitable research question?
• How could you revise the question so that it is
suitable?
42
Other considerations when
writing RQs
• Be careful about your choice of words
• Think about how you plan to answer your
research questions as you write them
• Consider the research design, measurement, and
data analysis
CETL Workshop, July 10, 2012
• Verbs such as improve, impact, affect, and predict
imply certain types of research designs
43
Research question templates
• Do male and female undergraduate students differ
in their level of statistical test anxiety?
CETL Workshop, July 10, 2012
• Do [groups of interest] differ on [name of
construct]?
44
Templates – cont’d
• Does level of participation in Greeley Parks &
Recreation Center classes differ by age,
educational level, gender, or ethnic group?
CETL Workshop, July 10, 2012
• Does [name of construct] differ by [variable]?
45
Templates – cont’d
• To what extent do salary, working conditions,
and types of job tasks predict job satisfaction?
CETL Workshop, July 10, 2012
• To what extent does a set of variables explain (or
predict) a given outcome variable?
46
Templates – cont’d
• Is there a relationship between teachers’ job
satisfaction and perceived level of administrative
support?
CETL Workshop, July 10, 2012
• Is there a relationship between [variable] and
[variable]?
47
Templates – cont’d
• Is the relationship between teachers’ job
satisfaction and perceived level of
administrative support the same for novice
versus experienced teachers?
CETL Workshop, July 10, 2012
• Is the relationship between [variable] and
[variable] the same for [names of groups]?
48
Templates – cont’d
• Does child’s achievement motivation explain
performance on statewide standardized tests,
after controlling for parents’ SES and academic
self-concept?
CETL Workshop, July 10, 2012
• Does a given explanatory variable (or set of
variables) explain a criterion, after taking into
account one or more extraneous variables?
49
Templates – cont’d
• Does reading level improve during a twomonth summer intensive reading program for
elementary school English language learners?
CETL Workshop, July 10, 2012
• Does [dependent variable] change [or improve
or decline] across time among [population of
interest]?
50
Templates – cont’d
• Is the rate of improvement in reading level the
same for males and females?
CETL Workshop, July 10, 2012
• Is the pattern of change [or improvement or
decline] on [name of dependent variable] the
same for [names of groups]?
51
• Consider a quantitative study you might conduct
• Think about the overall purpose and how the
research questions will help support the purpose
• You are to compose at least two research
questions, each of which should include a
dependent variable and one or more
independent variables.
• One of your RQs will include one or more
extraneous variables and the other will include a
moderator variable.
CETL Workshop, July 10, 2012
Question Writing Exercise
52