Transcript Research Methods - Southeast Missouri State University

Research Methods
Techniques
Strengths
Problems
1
Experiments
Experiments involve manipulation and
control. They use:
 IVs and DVs.
 Experimental and control groups
 Pretesting and posttesting
 They seek to understand causation.
The IV is thought to lead to, affect, the
DV

2
•Logic of experiments
DV
(IV)
Experimental group
DV
DV
Control group
DV
Ideally, random assignment of subjects to
experimental and control groups
3
Logic of experiments (continued)
If random assignment (randomization) is
used, the groups are likely to be
equivalent. The P is that the groups will
be the same, differences canceled out.
 Logic or rationale: If the control and
experimental group are the same on the
DV at the start of the study, and the only
difference is that the experimental

4
Logic (continued)
group receives the IV and the control
group does not, then
 If there are differences between the
groups on the DV at the end of the
study, then they must be due to the IV.

5
Equivalent groups
Importance of equivalent groups
 Randomization best, but not always
feasible
 Might match on important variables (sex
race, SES, intelligence, etc..)
 Problem with matching: might not
match on some important variable, and
the groups will not be equivalent

6
Some designs
Classic experiment
 One group pre-post test
 Static-group comparisons

7
Threats to validity (extraneous)
Internal validity: variables which might
affect the DV, aside from the IV
 History: events which occur during the
course of the study
 Maturation: changes that take place
because of the aging process
 Testing: changes that occur because of
pretesting

8
Threats (continued)
Selection: nonequivalent groups
 Regression: problematic when extreme
groups are used. People with very low
or very high scores tend to score closer
to the mean on retesting. (holds true for
other phenomenon, such as inheritance
to traits)
 Mortality: dropping out of a study

9
External validity (generalization)
One study shows that the IV affects the
DV. Will this be true in other settings
with other subjects?
 Interaction of testing and IV
 Experimenter effects, reactive
arrangements
 Hawthorne study

10
Advantages of experiments
We can more directly observe the
effects of one variable on the other,
holding other variables constant
 Sometimes can be done with little time
and money
 Experiments can be more easily
replicated than other methods

11
Disadvantages of experiments
Artificiality
 Example: TV and aggression
 Example: Zimbardo study

12
Surveys
Ask people what they think, feel, did
 Initial steps in surveys
 Selection of population and sample
 Development of the questions into a
questionnaire
 Pretesting and review

13
Survey methods
Group administered
 Mailed
 Face to face interviews
 Telephone interviews
 Each has its advantages and
disadvantages

14
Group administered
Assemble subjects together, have them
fill out the questions. A proctor is
available to answer questions
 Advantages: economical, quick,
problems can be handled immediately
 Problem: it may not be possible to
assemble subjects. Works best with
groups who will be together anyway

15
Mailed surveys
Mail out: (1) questionnaire; (2) cover
letter explaining the purpose of the
study, who is doing it, and what the
results will be used for; and (3) a selfaddressed “stamped” envelope.
 bulk rate
 on the return envelope: stamps,
metering, business reply permits

16
Mailed (continued)
business reply permits : more
expensive to purchase, but you only pay
for those surveys returned.
 Follow-ups--reminder postcards vs. new
questionnaire, cover letter and envelope

17
Advantages of mailed surveys
Relatively inexpensive
 Fairly fast results
 Geographic accessibility--can reach
almost anyone

18
Disadvantages
Low response rates, with the potential
for biased results
 No control over the subjects
 No control over misinterpretations of the
questions
 Will subjects be truthful?
 More impersonal than other methods

19
Face to face interviews
Training of interviewers
 Identification
 How to dress (depends on subjects)
 Approaching people (appointment vs.
showing up)
 How to administer questions and record
responses

20
Face to face (continued)
Practice sessions
 Anticipating as many situations as
possible and handling problems
 Need to monitor interviewers

21
Advantages of face to face
High response rates
 Control over subjects
 Can “probe” and ask more complicated
questions
 Deals better with less educated or more
illiterate populations
 Better for establishing rapport and
getting more information

22
Disadvantages of face to face
Time-consuming
 Expensive
 Interviewer bias
 Some people may lie to face to face
interviewers

23
Telephone interviews
Same procedures as face to face, but
over the telephone
 Use of computer systems
 Random digit dialing
 Coding data as questions are answered

24
Advantages of telephone
Fast
 Less expensive than face to face
 Can still ask more complicated
questions and probe

25
Disadvantages
More impersonal
 Can terminate the interview more easily
 response rate not as high as face to
face, but higher than mailed
 Cannot be very long

26
Observational Research
Problem with surveys: what people say
and what they actually do are not
always the same.
 Additionally, there may be important
information that no one thought to ask
 Therefore it is sometimes useful to
observe behavior as it occurs, not ask
about it or do experiments

27
Distinctions in observations
Laboratory vs. field
 One subject vs. a group
 Unstructured vs. structured
 Unstructured: observe a variety of
aspects of environment and behavior
 Structured: interested in a particular
behavior

28
Distinctions (continued)
Unstructured: Margaret Meade’s
studies, prison subculture studies
 Structured: police brutality, police
discretion at the time of arrest
 Participant vs. non-participant observer
 Participant: joins in the activities in the
environment. May be a genuine
member (Becker as a jazz musician)

29
Distinctions (continued)
Might be pretend to be a member of the
group. Problem: is it ethical to
deceive? (Rosenhan study, “end of the
world” study)
 With participant, the observer might
become too involved to be objective,
and might influence the group’s
behavior

30
Distinctions (continued)
Nonparticipant researcher
 Might conceal him/herself. Problem:
people’s right to privacy (Not a problem
if observing in a public place, where
people do not have an expectation of
privacy)
 Might identify self as researcher.

31
Distinctions
Problem: Might be rejected. Also,
people might act differently if they know
they are being observed.
 The pros and cons must be considered
in such observational studies

32
What do observers study?
Practices (behaviors)
 Episodes and encounters
 Roles
 Relationships
 Groups and organizations

33
Steps in observational research
Literature
 Informants
 “Getting in” and rapport, explanation of
research
 Sampling: often nonprobability-convenience, quota, or snowball.
 Observations

34
Steps in observation (continued)
Unstructured: field notes
 Structured: record events in categories
as they occur
 Unstructured interviews--probe and try
to determine what is going on
 Categorizing observations
 Examples: history of group, biographies

35
Steps in observation (continued)
religion, rule-breaking, roles.
 Finding patterns, deviations in the
patterns
 Writing up results
 More difficult than other methods

36
Advantages of observation
Depth of understanding gained
 Flexibility of the method
 Rich source of hypotheses for other
types of studies

37
Disadvantages of observation
Might misinterpret
 Difficulties of interpretation
 Small number of subjects
 Sampling problems--will the
observations be representative
 Time-consuming
 Expensive

38
Unobtrusive research
Looking at traces left behind
 Example: looking at well-worn floors in
museums to determine which exhibits
are most popular
 Our only way of being able to study
prehistoric man
 Problem: dangers of misinterpretation

39
Record Research
With the advent of the written word, we
can study behavior. Our only method
for studying the past.
 For example, has America always had a
crime problem, particularly violent
crime, or has this always been part of
our culture?
 Have child rearing practices changed?

40
Content analysis
Use of documents to study a behavior
such as crime: books, magazines,
songs, newspapers, letters, diaries
 Example: fear of crime in Switzerland
studied by examining newspapers and
Parliamentary debates for concern
about crime (In contrast to the U.S., the
topic was not raised often)

41
Content analysis (continued)
Themes put into categories
 Need to distinguish manifest (actual)
and latent (hidden meanings) content
 Use of computers to search for words in
documents (advantages and
disadvantages)

42
Record (archival) research
Use of existing statistics and records
 government documents
 computer tapes of agencies
 agency files
 Examples: Do judges sentence females
differently than males? What
percentage of cases are plea
bargained? etc..

43
Advantages of study of records
Inexpensive
 Possible to study processes occurring
over long periods of time
 No effect on those being studied;
records are being kept anyway
 Can be used for historical and
comparative uses
 Large numbers of subjects

44
Disadvantages of record research
Records may have gaps, be incomplete
 Ecological fallacy: patterns at a group
level may be different than those at the
individual. Example: Protestant
countries have higher suicide rates than
Catholic countries. Are Protestants
committing suicides--could be Catholic
minority

45
Disadvantages--record research
Problems with statistics
 Official statistics--unreported crime
 Crimes depend on laws, which vary
 Numbers depend on discretion of
officials and their norms, at arrest, plea
bargaining and convictions, sentences,
probation/parole revocation
 Necessary to look behind the numbers

46
Evaluation Research
Determining the effects of programs,
agencies, policies and laws
 Four components
 Needs assessment
 Monitoring
 Outcomes
 Cost/benefit and cost/effectiveness

47
Needs assessment
Is the service/change/intervention
needed?
 Who is the target population? i.e.., who
do we want to reach the most? What
are their characteristics?
 What problems can be anticipated as
the change is made? i.e.., halfway
houses expect neighborhood resistance

48
Needs (continued)
Can problems be addressed
successfully?
 Ways of assessing: records, surveys,
key informants and gatekeepers
 Without needs assessment,
inappropriate interventions might be
made

49
Monitoring
Looks at the processes taking place as
the intervention is made
 Are the services being provided? In the
case of a law, is it being utilized?
 Is the intervention reaching the target
population? i.e., if your intervention is
school based and your targets are
truants, it will not work

50
Monitoring (continued)
Are there unanticipated side effects?
 Is the intervention working/happening?
 Assessing: records, interviews with
people carrying out the interventions,
interviews with people affected by the
intervention

51
Outcomes research
What are the goals and outcomes
expected? Usually multiple goals
 Is the intervention achieving those
outcomes?
 Assessing: experiments or quasiexperiments (nonequivalent control
groups, before-after), time series
designs

52
Cost benefit & cost effectiveness
Cost-benefit: monetary costs of the
intervention are compared to the
monetary savings as the result of the
program
 Cost effectiveness: costs are compared
to the outcomes
 Example: bail program

53
Cost benefit/cost effectiveness
Example: Halfway house program
 Costs must be estimated
 Effectiveness a more difficult issue
 How successful does a program have to
be to be “worth it?” a 5% reduction in
recidivism? 10% ? etc.. compared to
what costs?

54
Strengths of evaluation
Determining what works and what does
not, rather than just trying intervention
after intervention
 Determining what the unanticipated
effects are (ex: net-widening)
 Determining why programs do not work,
which can be valuable (example: laws
and prosecutors)

55
Problems in evaluation
Problems of uncontrolled experiments
 Measurement problems
 Lack of control over agencies
 Results ignored, or might be unpopular
(for example, pornography studies)

56