Transcript Terry Anderson

Research Methods in Distance Education:
Design Based Research
Terry Anderson
PhD Seminar
Leicester UK
Feb. 2012
My Focus: The context of Distance
Education Implementation
• Disruptive innovation (Christensen, 2008)
simpler, not wanted by main stream
customers
• Rapid gains in functionality
• Cheaper
• Adaptive
• Moving from peripheral to mainstream
(blended and online for full time students)
Research Paradigms
Research Paradigms
Research Paradigms
Quantitative ~ discovery of the laws that
govern behavior
Qualitative ~ understandings from an
insider perspective
Critical ~ Investigate and expose the
power relationships
Design-based ~ interventions,
interactions and their effect in multiple
contexts
Paradigm 1
Quantitative Research
• employs a scientific discourse derived
from the epistemologies of positivism and
realism.
• “those who are seeking the strict
way of truth should not trouble
themselves about any object
concerning which they cannot have
a certainty equal to arithmetic or
geometrical demonstration”
– (Rene Descartes)
• Inordinate support and faith in
randomized controlled studies
Quantitative Example 1 –
CMC Content Analysis
• Anderson, Garrison, Rourke 1997-2003
– http://communitiesofinquiry.com - 9 papers reviewing results
focusing on reliable , quantitative analysis
– Identified ways to measure teaching, social and cognitive
‘presence’
– Most reliable methods are beyond current time constraints of
busy teachers
– Questions of validity
– Serves as basic research as grounding for AI methods and major
survey work of the future
– Serves as qualitative heuristic for teachers and course designers
Quantitative – Meta-Analysis
• Aggregates many effect sizes creating large N’s more
powerful results.
• Ungerleider and Burns (2003)
• Systematic review of effectiveness and efficiency of
Online education versus Face to face
• The type of interventions studied were
extraordinary diverse –only criteria was a
comparison group
• “Only 10 of the 25 studies included in the indepth review were not seriously flawed, a
sobering statistic given the constraints that went
into selecting them for the review.”
• 1928-2008
• distance delivery modes
from correspondence
schools, radio, television,
video, and now e-learning
• when the course
materials and teaching
methodology are held
constant, there are no
significant differences
(NSD) in learner
outcomes
http://www.nosignificantdifference.org/
Slide from Tom Reeves
Ungerleider, C., & Burns, T. (2003). A systematic review of the effectiveness
and efficiency of networked ICT in education. Ottawa: Industry Canada.
Is DE Better than Classroom Instruction?
Project 1: 2000 – 2004
• Question: How does distance education compare
to classroom instruction? (inclusive dates 19852002)
• Total number of effect sizes: k = 232
• Measures: Achievement, Attitudes and Retention
(opposite of drop-out)
• Divided into Asynchronous and Synchronous DE
Bernard,
R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A., Wozney, L.,
Wallet, P.A., Fiset, M., & Huang, B. (2004). How does distance education
compare to classroom instruction? A meta-analysis of the empirical literature.
Review of Educational Research, 74(3), 379-439.
12
Quantitative Research Summary
• Can be useful especially when fine tuning well
established practice
• Provides incremental gains in knowledge, not
revolutionary ones
• The need to “control” context often makes results of
little value to practicing professionals
• In times of rapid change too early quantitative
testing may mask beneficial positive capacity
• Will we ever be able to afford blind reviewed,
random assignment studies?
Paradigm 2
Qualitative Paradigm
• Many different varieties
• Generally answer the question ‘why’ rather
then ‘what’, ‘when’ or ‘how much’?
• Presents special challenges in distributed
contexts due to distance between participants
and researchers
• Currently most common type of DE research
(Rourke & Szabo, 2002)
Qualitative Example
–Dearnley (2003) Student support in
open learning: Sustaining the Process
–Practicing Nurses, weekly F2F tutorial
sessions
–Phenomenological study using
grounded theory discourse
Core category to emerge was “Finding the
professional voice”
Dearnley and Matthew (2003 and 2004)
Qualitative example 2
• Mann, S. (2003) A personal inquiry into an experience of
adult learning on-line. Instructional Science 31
• Conclusions:
– The need to facilitate the presentation of learner and teacher
identities in such a way that takes account of the loss of the normal
channel
– The need to make explicit the development of operating norms and
conventions
– reduced communicative media there is the potential for greater
misunderstanding
– The need to consider ways in which the developing learning
community can be open to the other of uncertainty, ambiguity and
difference
3rd Paradigm
Critical Research
• Asks who gains in power?
• David Noble’s critique of ‘digital diploma Mills’
most prominent Canadian example
• Are profits generated from user generated
content or OERs exploitative?
• Confronting the “net changes everything”
mantra of many social software proponents.
• Who is being excluded from online world?
See Norm Friesen’s
Friesen, N. (2009) Re-thinking e-learning
research: foundations, methods, and
practices. Peter Lang Publishers
• Why does Facebook own all the content that
we supply?
• Does the power of the net further marginalize
the non-connected?
• Why did the One Laptop Per Child project fail?
Or did it?
Do Either Qualitative or Quantitative
Methods Meet Real Needs of Practicing
Distance Educators?
But what type of research has most
effect on practice?
– Kennedy (1999) - teachers rate relevance and
value of results from each of major
paradigms.
– No consistent results – teachers are not a
homogeneous group of consumers but they
do find research of value
– “The studies that teachers found to be most
persuasive, most relevant, and most
influential to their thinking were all studies
that addressed the relationship between
teaching and learning.”
But what type of research has most
effect on Practice?
– “The findings from this study cast doubt on
virtually every argument for the superiority
of any particular research genre, whether the
criterion for superiority is persuasiveness,
relevance, or ability to influence practitioners’
thinking.” Kennedy, (1999)
Design-Based Research
• Developed from frustration of the lack of
impact of educational research on educational
systems.
4th Paradigm
Design-Based Research
• Related to engineering and architectural
research
• Focuses on the design, construction,
implementation and adoption of a learning
initiative in an authentic context
• Related to ‘Development Research’
• Closest educators have to a “home grown”
research methodology
Design-Based Research Model
Phase 2
Phase 1
Analysis of
Practical
Problems by
Research and
Practitioners in
Collaboration
Development of
Solutions
Informed by
Existin Design
Principles and
Technological
Innovations
Phase 3
Iterative
Cycles of
Testing and
Refinement
of Solutions
in Practice
Phase 4
Reflection to
Produce “Design
Principles” and
Enhance
Solutions
Implementation
Refinement of Problems, Solutions, Methods, and Design Principles
(Reeves, 2006, p. 59)
Reeves, T. C. (2006). Design research from the technology
perspective. In J. V. Akker, K. Gravemeijer, S. McKenney, & N.
Nieveen (Eds.), Educational design research (pp. 86-109). London:
Routledge.
Design-Based Research Studies
– iterative,
– process focused,
– interventionist,
– collaborative,
– multileveled,
– utility oriented,
– theory driven and generative
• (Shavelson et al, 2003)
Critical characteristics of
design experiments
• According to Reeves (2000:8), Ann Brown
(1992) and Alan Collins (1992):
– addressing complex problems in real contexts in
collaboration with practitioners,
– integrating known and hypothetical design
principles with technological affordances to
render plausible solutions to these complex
problems, and
– conducting rigorous and reflective inquiry to test
and refine innovative learning environments as
well as to define new design-principles.
Design-based research
• Methodology developed by educators for
educators
• Developed from American pragmatism – Dewey
(Anderson, 2005)
• Recent Theme Issues:
–
–
–
–
The Journal of the Instructional Sciences, (13, 1, 2004),
Educational Researcher (32, 1, 2003) and
Educational Psychologist (39, 4, 2004)
See bibliography at
http://cider.athabascau.ca/CIDERSIGs/DesignBased
SIG/
• My article at www.cjlt.ca/abstracts.html
Number of DB Scholarly
Articles Published
400
350
300
250
200
150
100
50
0
2000
01
02
03
04
05
06
07
08
09
10
Anderson, T., & Shattuck, J. (2012). Design-Based Research: A Decade of Progress in
Education Research? Educational Researcher, 41(Jan/Feb.), 16-25. Retrieved from
http://edr.sagepub.com/content/41/1/7.full.pdf+html.
Anderson, T., & Shattuck, J. (2012). Design-Based Research: A Decade of Progress in
Education Research? Educational Researcher, 41(Jan/Feb.), 16-25. Retrieved from
http://edr.sagepub.com/content/41/1/7.full.pdf+html.
Design Tradition
• “Learning and productivity are the
results of the designs (the
structures) of complex systems of
people, environments, technology,
beliefs and texts” New London
Group 2000
• DBR opens the door for teachers,
researchers and learners to become
designers, not merely
consumers, bosses or observers .
Integrative Learning Design
(Bannan-Ritland, 2003)
• “design-based research enables the creation
and study of learning conditions that are
presumed productive but are not well
understood in practice, and the generation of
findings often overlooked or obscured when
focusing exclusively on the summative effects
of an intervention” Wang & Hannafin, 2003
• Iterative because
• ‘Innovation is not restricted to the prior design
of an artifact, but continues as artifacts are
implemented and used”
• Implementations are “inevitably unfinished”
(Stewart and Williams (2005)
• intertwined goals of (1) designing learning
environments and (2) developing theories of
learning (DBRC, 2003)
Design Based research and the Science of
Complexity
• Complexity theory studies the emergence of
order in multifaceted, changing and previously
unordered contexts
• This emerging order becomes the focus of
iterate interventions and evaluations
• Order emerges at the “edge of chaos” in
response to rapid change, and failure of
previous organization models
DBR Examples
Call Centres At Athabasca:
•
•
Answer 80% of student inquiries
Savings of over $100,000 /year
Anderson, T. (2005). Design-based research and its application to a call center
innovation in distance education. Canadian Journal of Learning and Technology,
31(2), 69-84
Design Based research in Action
• Phase 1 Exploration – surveys, talking to
faculty and tutors, investigating open source
tools, setting research questions
• Phase 2. Building the intervention – Elgg
through two versions and 85 plugins (on
going)
Design Based research in Practice
• Athabasca Landing
– Elgg based
– Started in 2008
– 3500 users
– Unpaced
– Paced Courses
– Informal Learning
– Staff and alumni networking
– Problem of critical mass
• Phase 3 Evaluation – Before and after survey’s
see:
– Anderson, T., Poelhuber, B., & McKerlich, R. (2010). Self
Paced Learners Meet Social Software. Online Journal of
Distance Education Administration
– Dr students – Use of past student archives
– Ongoing iterations and development of tools
• Phase 4 – Design Principles
– Development of design principles/patterns Nets and Sets
Dron, J., & Anderson, T. (2009). How the crowd can teach.
In S. Hatzipanagos & S. Warburton (Eds.), Handbook of
Research on Social Software and Developing Community
Ontologies.
Network Tool Set (example)
Text
Text
42
Stepanyan, Mather & Payne, 2007
Access Controls in Elgg
• Need to study usability, scalability and
innovation adoption within bureaucratic
systems
• Allow knowledge tools to evolve in natural
context through supportive nourishment of
staff
Conclusion
• Education research is grossly under-resourced to
meet the magnitude of opportunity and demand
• Paradigm wars are unproductive
• Design-based research offers a promising new
research design model
• It can be used for Doctoral dissertations see
• Herrington, J., McKenney, S., Reeves, T., & Oliver, R.
(2007). Design-based research and doctoral students:
Guidelines for preparing a dissertation proposal.
Your comments and questions most
welcomed!
Terry Anderson [email protected]
Blog: terrya.edublogs.org
Twitter: terguy