Transcript e-Research. Methods, Strategies and Issues

e-Research: Methods, Strategies
and Issues
Open University
May, 2003
Terry Anderson Ph.D.
Canada Research Chair
in Distance Education
Athabasca University
[email protected]
Context – Athabasca University,
Alberta, Canada
Largest Open University in Canada
Fastest growing University in Canada
50,000 enrolments
Individualized and Open instruction
*
Athabasca University
Largest MBA and MDE programs
in Canada
Context – Athabasca University,
Alberta, Canada
Canada is a great country, much too
cold for common sense, inhabited by
compassionate and intelligent people
with bad haircuts”
*
Athabasca University
Yann Martel, 2002
Agenda:
• How the Net changes Research
• Rationale for E-Research
• The E-research Process
–
–
–
–
Problem setting
Literature review
Ethics approval
Gathering data
• Qualitative
• Quantitative
– Dissemination
• Future of e-Research
Why eResearch?
"Our Age of Anxiety is, in great part,
the result of trying to do today's jobs
with yesterday's tools."
Marshall McLuhan
“the Net changes everything”
Oracle Corporation
Affordances of the Net
• “Affordances are both objective and persisting and, at
the same time, subjective, because they relate to the
species or individual for whom something is afforded.”
(Gibson, 1982, p. 237).
• Access to huge amounts of information
– (est. 5 billion pages, 2000)
• Unparalleled communication capacity
– voice, text, video
– Synchronous, asynchronous
– very inexpensive
– Place of business, play and education
• Emerging Intelligence capacity
– emergence of the Semantic Web, will allow
autonomous agents to work for us 24 hours a
e-Research in Education
– Research in educational environments
supported by the web
•
•
•
•
Interactions
Virtual and remote labs
Learning communities
Simulations, games and evaluation
– Data collection by devices on the web
– Dissemination and feedback on the web
– The scholarship of teaching
Activity On
the Net –
‘Cyberspace’
Research
Activity On
the Net
Research
Activity NOT
On the Net
Activity NOT
On the Net
‘Real world’
E-Research
Internet Skills
Effica cy
Resea rch Skills
Problem Sta tem ent
M enta l M odels
Litera ture Review
Access
E- RESEARCH
Da ta Collection
Term inology
Da ta Ana lysis
Experience
Trouble Shooting
Dissem ina tion
Types of E-Research
• Online surveys.
• Open ended or structured text-based interviews
conducted via email or computer-mediated
conferencing.
• Focus groups using real time Net-based video, audio
conferencing or asynchronous computer
conferencing.
• Analysis of Web logs and other tracking tools for
measurement and synthesis of online activities.
• Net-based telephone interviews.
• Analysis of text transcripts of learning or social
activities.
• Analysis of social behavior in virtual reality
environments.
• Online assessment and/or evaluation of performance or
knowledge, web cams
Challenges of e-research
• Data collection can be more challenging
than F2F
• More hidden contextual variables
• Greater heterogeneity of population
– Kember 1989
• Variations in access to and proficiency with
technology
• Anonymity and privacy issues
The Online Population
• How can we use online sampling when not
everyone is online?
– Quickly growing majority of users are online
• (USA 61% have used the net during the previous 4 weeks,
Sept. 2002)
– Many sub-populations, including students, have much
higher rates of participation, some approaching 100%
– “profile of the US Internet population is very similar to
the United States as a whole” (Sheehan, 2002)
The Literature Review Process in eResearch
‘A few months spent in field research
can frequently save a few hours in the
library.’
• Verifying authenticity
• Finding that missing reference
• New tools linking scholarly ideas such as
Buckingham-Shum’s ScholOnto
http://kmi.open.ac.uk/projects/scholonto/
Bibliographic Databases
the “killer app” of scholarly research
• Stores references to works you have read
– Sort, select, keywords, URLs
– Can also store quotations or notes from references
• Inserts references into word processing documents
– Integrates with Word and WP
• Creates reference lists in multiple formats (600
types)
• Searches and retrieves references from databases
and catalogues on the Net
Informal support and reference
•
•
•
•
•
Finding a discipline email group
What’s the use of the NewsGroups?
Virtual Conferences
Direct email
Filtering information for your professional
group or friends
• Knowledge management systems
Ethics
• Cyberspace “has a geography, a physics, a nature
and a rule, of human law” (Benedikt,1991),
• Can the same ethical constraints be placed upon
researchers when anonymity, trust and privacy
issues are so different
• Logistic difficulties of obtaining consent are so
much greater?
• Should teachers be able to publish the results of
“action research projects” using the net without
asking student permission as long as they don’t
identify individuals in publications?
• Are the Ethics Police out of control???
Collaboration Research Online
Much productive research is now done in
distributed or multidisciplinary teams
Communication tools –
Information and knowledge sharing tools –
Application sharing tools –
Project management tools Examples:
Microsoft Sharepoint
Community Zero
Groove
Time Track
www. communityzero.com
www.groove.net
Research Interviewing
• Email Interviews
–
–
–
–
Asynchronous allowing for reflection and time shifting
More easily ignored than face-to-face or phone
Keep short 2-4 questions
Develop relationship with respondents
• Real time, net-based audio or video conference
interviews offer low cost, multimedia
communications but bandwidth and hardware
issues problematic
Like Lyceum at Open Univ.
E-Luminate for
Interviews of Focus
Groups
audio
application sharing
no long distance
chat
whiteboard
Over Internet
Net-Based Consensus Research
• Research to gain insight from opinion(s) of a
group of experts
–
–
–
–
high quality opinion
authority
allows for knowledge creation through disagreement
efficient
On the web:
accessible, time and place shifted, equitable
Net-Based Consensus Research
Types of consensus research
• Delphi
• Nominal Group (real time)
• WIKI and BLOGs
See WIKI site http://openwiki.com/ or
http://www.personaltelco.net/index.cgi/CategoryC
ommunity
What role do BLOG play in undertsanding human
behaviour?
Kennedy, 2002
Quantitative Data collection
The (overhyped?) promise of Data Mining
what are my online students doing?
what resources are they viewing? For how long?
which students are online?
how do these statistics relate to student
achievement? Gender? Access?
Most tools designed for e-commerce market
What are the ethical issues of such study?
Email and Web Surveys
•
•
•
•
Much faster results
More convenient for subjects
Much lower cost than telephone or mail surveys
Integrated data processing can reduce errors and
allow for instantaneous results
• Return rates dropping over the past 10 years
– 37% 1986-2000 – 29% 1997-2000 (Sheehan, 2001)
– Major spam response
– Major concerns with sample bias in general population
surveys
– Need to investigate incentives
Response time and mode –
Email attachment survey (return rate 26%)
Days to
Respond
Same day
1
2
3
4
5
6
8 –14
# of Email
Surveys
22
14
4
17
10
5
6
6
#Mailed or
faxed
0
0
0
0
4
0
1
3
Totten, 2002
E-Mail
Limited error checking
and respondent feedback
May require re-keying to
analysis program
Web Based
Error checking, quality
control and instant
analysis
Anonymous capability
Easy to compose
Allows sophisticated
multimedia
Linear, with awkward
branching
Allows multiple paths
Pushes to respondents
desk
Must pull respondent to
the web site
Much brighter future predicted for web based (Totten, 2002)
Commercial Survey Services
• Factors that increase response rate –
similar to paper based
– follow-up contacts with non-respondents,
– personalized contacts, and
– contacting sampled people prior to
sending out the survey
– Cook, Heath & Thompson (2000)
Content Analysis of transcripts
• See our Community of Inquiry site at
www.atl.ualberta.ca/cmc
Purpose of Transcript Analysis
• Objectify and quantify the intuitive
impressions that instructors form while
participating in online conferences.
• Reveal additional insights that are not
obvious from superficial reading or
participation.
• Allow educational researchers to compare,
replicate results, increase understanding.
Qualitative vs.
Quantitative Analysis
• Qualitative
– Uses cultural sensitivity of the researcher
– In CMC often uses grounded theory to read
transcripts, identify themes, and build theory
from these themes
• Quantitative
– Strives to eliminate researcher bias and achieve
transferability
– Focus on reliability and validity
Procedure
• Identify variable, construct to be studied – often
generated by theory
• Sample texts or gather complete set
• Identify Unit of Analysis
• Construct a coding frame – that fits both theory
and text include
– categories, indicators, heuristics and rules.
• Assess reliability of coding process
• Analyze data and use for statistical display and
analysis
• Triangulate or otherwise confirm validity
Reliability
• Would the reader have categorized the data similarly
to the author of the report?
– Percent Agreement Measures
• total # of agreements / total # of decisions
– Chance Agreement Measures
• 2 categories = 50% of agreements could be due to chance
• Cohen’s kappa accounts for chance agreement in reliability
calculation
• See
http://ourworld.compuserve.com/homepages/jsuebersax/agr
ee.htm#basics.
Nature of Content
(Potter and Levine –Donnerstein 1999)
• Manifest
– concrete, obvious, meaning resides on surface
• spelling errors, use of emoticons, first names, threading
• Latent Pattern
– several manifest cues, in an appropriate pattern
• I.E. style of dress formal or casual (tie, belt, hat, jewelry)
• Latent Projective
– meaning resides in coders personal schema
• I.E. attractiveness, humor, critical thinking
Unit of Analysis
• What segment of the transcript will coders
categorize?
– Whole posting, sentence, paragraph, phrase,
thematic unit?
• CMC communication is idiosyncratic
• variables often do not organize themselves into
syntactic packages—paragraphs, postings too long,
sentences, phrases too short.
– We’ve tried all of the above – which is Best?
Units of Analysis
• Message?
– manageable data set,
– objectively identifiable by coders
– useful when it encompasses the variable
• Thematic Unit?
– a single item of information in its natural form
– most commonly used unit of analysis
– unreliable, coders are not alerted to the need for a
decision.
• Speech Turn?
Unitizing conundrum – how many units
in the message below?
Hi folks
I really hate the way we have to answer ALL
THESE QUESTIONS before getting the data
right- how about you? I have very little time this
week for this assignment. The last time I tried
this I got bogged down with chapter 3 do we
really have to know about semiotics, I want to
get onto the project analysis first then decide if
we need all this theory how about I’ll do the first
question and you guys do the rest.
Your-confused-comrade-in-arms
Software to Aide Analysis
• Qualitative analysis software
– NUD*IST, Hyperqual, Atlas-ti
• facilitates coding process both autocode and
manually
• Simple statistics and export
• Quantitative analysis software
– SPSS
• facilitates assessment of interrater agreement
• Presentation of descriptive statistics
Theoretical Model: Model of Critical Inquiry in CMC Context –
Garrison, Anderson and Archer (2,000)
Category
Facilitating
discussion
Indicator
Identifying areas of
agreement/disagreement
Seeking to reach
consensus/understanding
Encouraging, acknowledging,
reinforcing contributions
Setting climate for learning
Drawing in participants,
prompting discussion
Assess the efficacy of the
process
Teaching Presence results:
90
80
70
60
50
40
30
20
10
0
Instructional
Design
Facilitating
Discourse
Direct
Instruction
Education
Course
Health Course
Percentage of instructor messages
Applications of Teaching Presence to Peer
Moderating
(Rourke and Anderson, 2002) JIME
• Peer teams (3-4 persons) moderated conferences
last half of graduate course
• Higher levels of all three indicators of teacher
presence than instructor!
• Many more moderator postings by peers
• Shows value of sharing and delegating teaching
presence
• Interviews found insufficient probing by peer
facilitators – too much share and compare
Dissemination
Paper versus online publication
• Costs
• Visibility and retrievability
• Marketing
• Editing
• Time
• Audience
• Standards
• Prestige
• Quality of review – see Journal of Interactive
Media in Education JIME
Virtual Conferences
Buy the Book!!
Future of E-research
www.e-research
Activity: You buy the book!!!
Future of E-research
• Easier data collection and monitoring via research
agents of the Semantic Web
• Structured data of the Semantic Web will allow
much improved classification of e-research,
search, retrieval and information sharing
• Increasing access will further expand population
that can be studied using e-research techniques
• Mobile computing will allow e-research to study
more informal learning
• Need to develop and assess participant incentives
Your Comments and Questions??
Terry Anderson
[email protected]