Transcript IS 4800 Empirical Research Methods for Information Science Class Notes April 6, 2012

IS 4800 Empirical Research Methods
for Information Science
Class Notes April 6, 2012
Steps in the research process
• Identify a phenomenon of interest
• Iterate:
– Investigate current state of knowledge (lit. review ?)
– Narrow down your interest to a research question or
hypothesis
• Identify research method to employ (survey,
experiment, ethnography, case study)
Steps in the research process (cont.)
• Operationalize research question or hypothesis
– Define the source of your data
• Sample population and recruitment method (if relevant)
• And/or the location/activities to be observed
– Define variables and/or data collection methods and
instruments
– Identify how the analysis will be carried out
----YOU NOW HAVE A RESEARCH PROPOSAL ------
Steps in the research process (cont.)
• Carry out your observations
• Analyze your data
• Draw conclusions, write up the results
Qualitative Data and Collection
Methods
• Direct observation
– Participant observation
• In-depth interviews
– Focus groups
• “Artifacts” – usually text or Databases
Direct Observation
– May be in person or use audio or videotape,
observe through a 1-way mirror
– Unlike participant observation, often focused on
specific events (how many, how often, by whom,
observe patterns – for example, interruptions at a
meeting)
What to observe
• Spatial relations
• Activities
• Communication
– Verbal
– Other
• Tasks
– How work is allocated
How to be an effective observer
• Preparation
• Stay in the background
• Be factual and objective in your notes
– (interpretation comes later)
• Taking notes:
– Hand written usually
– Type in to computer later
• EXPANDING NOTES (ASAP)
“(Participant) observation”: in
natural setting
• “Participant” observation occurs when you
interact casually and/or form relationships with
informants
• How much you actually “participate” depends
on the goals of the study.
Participant observation
• Advantages:
– Offers insights into complex behavior
– Identify the “right questions” for further study
– Verify/correct self-reports
• Disadvantage:
– Time consuming
– Data collection is difficult
– Problem of subjectivity
How to operationalize
• Field notes
– Text
– Diagrams, maps
– Can result in numerical data
• Interviews (interviewer more clueful)
• Focus groups (facilitator more clueful)
What to observe
• Spatial relations
• Activities
• Communication
– Verbal
– Other
• Tasks
– How work is allocated
– See Table 3 in reading
Ethics
• Do not disrupt the activity your are observing
versus
• Do not mislead
• No formal rules about disclosing your role as a
researcher when engaging in casual
conversation – article suggests a point where
you want to ask specific question
• Disclosure includes: right of refusal,
confidentiality
Protecting confidentiality when
data is unique
• Separate identify info from field notes entered
into the computer
• People, organizations/companies, should be
given fictitious names
How to be an effective observer
• Preparation
• Stay in the background
• Be factual and objective in your notes
– (interpretation comes later)
• Taking notes:
– Hand written usually
– Type in to computer later
• EXPANDING NOTES
Tips
•
•
•
•
Leave space
Take notes strategically
Use abbreviations
Cover a range of observations:
Body language, etc.
Tips
•
•
•
•
Leave space
Take notes strategically
Use abbreviations
Cover a range of observations:
Body language, etc.
Participant observation: in natural
setting
• “Participant” observation occurs when you
interact and/or form relationships with
informants
• Demanding and time-consuming
• How much you actually “participate” depends
on the goals of the study.
• Subjects may “forget” you are a researcher
How to operationalize
direct/participant observation
• Field notes
– Text
– Diagrams, maps
– Can result in numerical data
• Interviews (interviewer more clueful in P.O.)
• Focus groups (facilitator more clueful in P.O.)
“Water cooler” effect
Participant observation
• Advantages:
– Offers insights into complex behavior
– Identify the “right questions” for further study
– Verify/correct self-reports
• Disadvantage:
– Time consuming
– Data collection is difficult
– Problem of subjectivity
Ethics of direct observation/
participant observation
• Do not disrupt the activity your are observing
versus
• Do not mislead
• No formal rules about disclosing your role as a
researcher when engaging in casual
conversation – some authors suggest a point
where you want to ask specific question
• Disclosure includes: right of refusal,
confidentiality
Protecting confidentiality when
data is unique
• Separate identify info from field notes entered
into the computer
• People, organizations/companies, should be
given fictitious names
In-depth interview/focus group
• Probes the interviewee(s) views of the
phenomenon of interest
• Interviewer/facilitator should be neutral
• Data collected: transcript, audio/video
recording, notes
In-depth interview/focus group
Interviewer should:
• Start with some open-ended questions
• Follow up by asking “how” and “why”
• Keep the discussion on track
Documents
•
•
•
•
•
Memos and meeting notes
Transcripts of conversations or speeches
Manuals and policy handbooks
Newspapers and magazines
Internet-based research
– Email
– Web sites
– Blogs
• Especially important in case studies
III. Artifacts: Content Analysis
• Used to analyze a written or spoken record for
occurrence of specific behaviors or events
• Archival sources often used as sources for data
• Response categories must be clearly defined
• A method for quantifying behavior must be defined
26
Example Study
• The CEO of Global Enterprises, Inc. is very
worried about the low morale in the company, as
evidenced by the amount of flame email she
receives. She considers sending every office on a
“ropes” course, but to do this would cost the
company $10M. She asks you to do a study to tell
how well her scheme might actually work in
reducing her flame mail.
27
Analytic Induction
Nonexperimental, Qualitative analogue to scientific method
1. Phenomenon tentatively defined
2. Hypothesis is developed
3. A single instance is considered to determine if
hypothesis is confirmed
4. If hypothesis fails, then phenomenon or hypothesis
is redefined
5. Additional cases are examined and, if the new
hypothesis is repeatedly confirmed, some degree of
certainty results
6. Each negative case requires that the hypothesis be
reformulated until there are no exceptions
28
Typical Use of Analytic Induction
• Say you’re interested in employee’s impressions of
WizziWord.
• You interview 3 people, transcribe your notes, and
categorize all important statements into themes
– e.g. “It’s too slow.”, “It looks cool.”, etc.
• You interview 3 more people, categorize their
comments.
• Repeat until no new (significant) categories/themes
emerge.
29
Meta-Analyses
• Compare/Integrate “all” studies that have
investigated a given phenomena
– E.g., use of a particular medication for a
particular disease
• Common in the literature (esp. medical)
• Very methodical
– Search for articles
– Eligibility criteria
– Statistical analyses
30
Case Study Research
Steps: design, conduct (data collection), analysis, write-up
Exploratory research and the role of prior theory
Impacts case selection, data collection
Scientific method – observations should have the
potential to disconfirm
Case study research questions
Example: Telemedicine paper
31
Case study research
Design steps
Define unit of analysis
Case selection
Creation of a protocol (plan of work)
Data Collection
Analysis and reporting
Case Study Research
I. Design steps:
Select the unit(s) of analysis
(temporal, organizational, technological)
Case selection (“sampling”??) – one or several
critical case
theory based (confirming or disconfirming)
extreme v. typical
intense
criterion (e.g., budget > $X)
convenience
33
Case Study Research – Design steps (cont.)
Use of a protocol: (the “plan of work”, should be required)
1. Overview of study, including overview of data collection
strategy.
2. Details of data collection (sources, procedures)
3. Interview guidelines and instruments
4. Outline of the expected project report
Issues to be addressed in (2):
access to the organization
resources sufficient to collect the data in the field
scheduling of data collection activities
providing for unanticipated events
34
Case Study Research – Data Collection
The more different methods employed, the fuller the picture
of the phenomena being studied.
-- Documents (meeting minutes, project reports, newsletters,
manuals)
-- Archival documents (service records, system usage data)
may provide quantitative information
-- Interviews
Typical: 95 interviews over 6 months
-- Field observations (when a visit is conducted):
usually meetings. (also can observe user training, etc.)
-- Artifacts (problem reports – why not archival docs??)
35
Case Study Research – Data Collection
Selecting interviewees:
a. maximum variation (preferred method)
b. Homogeneous
c. Snowball or chain
d. Purposeful v. opportunistic
Benefits of semi-structured interviews
Unstructured when questions not known in advance
What is “triangulation”? (paper mentions construct
validity)
When to STOP collecting data
36
Case Study Research – Data Analysis
Data analysis very different from analysis of experiment
and survey data. Why?
Stages of analysis:
Preliminary analysis (early steps)
Within-case analysis
Cross-case analysis
Qualitative analysis most difficult and least standardized
part of empirical research.
37
Case Study Research – Data Analysis
Goals of qualitative data analysis:
Identify themes
Develop categories
Explore similarities and differences
Describe patterns that explain why
(Propose models that predict)
38
Case Study Research – Data Analysis
Techniques for qualitative data analysis: Preliminary Stage
1. Coding
2. Database
What is a code? a word or short phrase attached to each
segment (e.g., paragraph, answer to interview question) of the
collected data, indicating the “presence” of that concept.
Codes can be arranged in (or derived from) a taxonomy.
A good taxonomy will yield codes that reveal patterns in the
data.
39
Case Study Research – Data Analysis
Where do codes come from:
Prior work or theory
Study of initial data (defined “inductively”)
Iterative nature of coding
Use of independent raters to validate codes
The code book or code manual: desirable attributes
Detailed description of each code
Inclusion and exclusion criteria
Examples of collected data to illustrate each code
Development of higher-level “pattern codes” identify
themes or relationships that are relevant to the study’s
40
research questions
Case Study Research – Data Analysis
Case study database:
•uninterpreted data
•complete data (answers criticism of “selective quoting”)
•analogous to raw data collected in experiment or survey
Contents of case study database
Field notes (interviews, observations)
Documents (including transcripts)
Quantitative data (including questionnaire data if any)
Contemporaneous notes (reflective remarks)
41
Case Study Research – Data Analysis
Techniques for qualitative data analysis: within-case stage
Goal: identify larger themes, relationships and propositions
Looking for larger themes and patterns:
Pattern-matching
compare expected elements with actual data
perform cross-checking of interview transcripts
and other data collected
desire two or more sources for each proposition
Explanation building:
challenge  tactics  results approach
Use of charts, table, graphs, timelines to aid understanding
42
Case Study Research – Data Analysis
Techniques for qualitative data analysis: cross-case stage
Depends on availability of several cases
Two approaches:
Analyze similarities and differences among cases
(e.g., factors, behaviors, results)
If goal is theory-building, develop a theory using one
case and systematically compare its
propositions to other cases
43
Case Study Research – Write-up
Weakest part of the article
Goals:
•Includes the goals of all professional writing, e.g., clarity,
shows relationship to earlier work, data support conclusions
•For positivist case research, shows applicability (general
relevance) to other examples with similar circumstances
•Constructive – propositions translate into “lessons learned”
that offer guidance on how to make use of the results (do’s
and don’ts)
44
Qualitative Data Analysis
by John V. Seidel
• Description of how to go about analyzing transcripts
of interviews, documents, and/or field notes.
• Focus on “coding”
–
–
–
–
First identify “events”
Assign terms that represent concepts of interest
Organizing codes into a scheme
Building qualitative models using the coding scheme as the
model vocabulary
• Focus on iterative nature of QDA
Two perspectives on coding
• Objectivist perspective
– Condensed representation of facts
– Can be subjected to hypothesis testing
– Strong burden of consistency/completeness
• Heuristic perspective
– Signposts pointing to things you care about
– Foundation for further analysis
Elements/types of qualitative models
• Examples from Rogers’ theory of innovation
diffusion
–
–
–
–
–
VCR’s
Cell phones
Metric system
Seat belts in cars
Dvorak keyboard
Three analogies to explain this
• Jigsaw puzzle analogy
• A little data and a lot of right brain
• Multi-threaded DNA (patterns among the
patterns)