Transcript Quantitative and Qualitative Inquiry in Educational

VI KONFERENCJA EWALUACYJNA
WARSZAWA, 13-14.12.2010
PhD, Professor
KATRIN NIGLAS
TALLINN UNIVERSITY
Podejścia mieszane – przyczynek do modelu
„piątej generacji” badań ewaluacyjnych
Some Visions for Fifth Generation Evaluation
Some background information
•
My interest in research methodology goes back to the beginning of 1990s.
•
Since 1994 I have been a lecturer in data analysis;
since 2004 also lecturer in research methods.
•
My (5y+2y) Master’s thesis, defended in 1996, was about teaching statistics in
the framework of research methods courses.
•
Got interested in mixed methods research in 1998 during my one-year study in
Cambridge University
•
In the doctoral dissertation published in 2004 I focused on the combined use of
quantitative and qualitative approaches in educational research
•
Belonging to the editorial board of Journal of Mixed Methods Research and
International Journal of Multiple Research Approaches
Further reading related to the topic of speech
•
Niglas, K. (2010). The multidimensional model of research methodology.
An integrated set of continua. Tashakkori, A., Teddlie, C. (Eds). Handbook of Mixed Metods
Research. 2nd Ed. Sage Publications Ltd
•
Niglas, Katrin (2008). How the novice researcher can make sense of mixed methods designs.
International Journal of Multiple Research Approaches, 1, 13 - 33.
•
Niglas, K.; Kaipainen, M.; Kippar, J. (2008). Multi-perspective exploration as a tool for mixed
methods research. . M.M. Bergman (Ed). Advances in Mixed Methods Research: Theories and
Applications. Sage Publications Ltd
•
Niglas, Katrin (2007). Spreadsheet Software Can Facilitate Mixed Methods Research –
Using Old Tools in a New Context! Journal of Mixed Methods Research, 1(3), 297 – 29
•
Niglas, K. (2004) The Combined Use of Qualitative and Quantitative Methods in Educational
Research. Tallinn Pedagogical University. Dissertations on Social Sciences.
Short version: http://www.tlulib.ee/files/arts/95/nigla32417030233e06e8e5d471ec0aaa32e9.pdf
PS! Published version of the dissertation (book) is available by request (e-mail: [email protected])!
•
Several papers available at Education Line http://www.leeds.ac.uk/educol/
Change in the aims and forms of evaluation
Guba and Lincoln (1989) describe 4 generations in
evaluation of social and political programs:
1. Evaluation ≈ measurement, testing
2. Evaluation ≈ description, mapping
3. Evaluation ≈ substantive evaluation, judgement, advise
4. Evaluation ≈ collaboration, negotiation, initiation and
guidance of change
From positivism to constructivism …
Critique of the first three generations by G & L (1989)
1. A tendency towards managerialism
managers who organise evaluation as well as evaluators are outsiders and
their relationship with/to stakeholders is disempowering
2. A failure to accommodate value pluralism
scientific mode comes with the claim of the value-freedom, but as evaluation
is essentially about valuing, the value-free in practice means imposing the
values of one group and not accommodating value differences
3. An overcommitment to the scientific paradigm of inquiry
context stripping; overdependence on quantitative measurement; belief in
scientific truth leaves no room for negotiation or alternative explanations and
frees evaluators from moral responsibility
From positivism to constructivism …
Main principles for 4th generation evaluation by G&L (1989)
4th Generation evaluation
1. is organised by the claims, concerns, and issues of
stakeholding audiences
2. utilises the methodology of the constructionist paradigm
3. is a process whereby evaluators and stakeholders jointly and
collaboratively create (or move towards) a consensual valuing
construction of some evaluand
This process is: local, sociopolitical, emergent, continuous, recursive, and
divergent, involves teaching/learning of all parties and sets them into
hermeneutic dialectic relationship
From positivism to constructivism …
Where are we 20 years after manifestation of 4th Generation?
1. Are we there?
(read: has constructivist paradigm won the paradigm wars and we all
work within this framework?)
2. Do we want to be there?
(read: are we convinced that constructivism as The paradigm is a
desired end for our journey on conceptualising evaluation and it’s
methodology?)
3. Do we have an alternative?
(read: is our choice constrained by positivism and constructivism or …?)
From positivism to constructivism …
My answers to the three questions set on the last slide:
1. Are we there?
(read: has constructivist paradigm won the paradigm wars and we all
work within this framework?)
No!
– a quick glance at the evaluation practice tells us that wealth of
evaluation studies are not conceptualised and carried out in this
framework
From positivism to constructivism …
My answers to the three questions set on the last slide:
2. Do we want to be there?
(read: are we convinced that constructivism as The paradigm is a
desired end for our journey on conceptualising evaluation and it’s
methodology?)
No, not all of us!
– Even though there is a lot to learn from constructivist tradition many
of us resist the forced choice between positivism and constructivism
and do not accept all the assumptions that constructivism brings with it
(i.e relativist ontology, strong belief in and either or stance on paradigms,
overemphasized belief in impossibility of representativeness and generalizations
as well as that change cannot be engineered, etc.)
From positivism to constructivism …
My answers to the three questions set on the last slide:
3. Do we have an alternative?
(read: is our choice constrained by positivism and constructivism or …?)
Yes!
We can:
– replace the bi-polar paradigmatic view by the conceptualization of research
methodology as a multidimensional integrated set of continua
– reason that the research/evaluation practice is inevitably lead by sophisticated
and changing personal worldviews or mental models of the parties rather than
well defined and incommensurable paradigms
– set facilitating change as the sovereign aim and the staple framework for the
evaluation without having to declare either the stakeholders’ or the
managers/experts’ position, needs, views, etc. being always paramount
The idea of incommensurable paradigms
 The Conventional and Constructivist Belief Systems
(Adapted from Guba and Lincoln 1989)
Fundamental
questions
Conventional
beliefs
What is there that can be
REALISM
known? –
ONTOLOGY
What is relationship of
the knower to the known OBJECTIVIST
(or knowable)? –
EPISTEMOLOGY
What are the ways of
finding out knowledge? – INTERVENTIONIST
METHODOLOGY
Constructivist
beliefs
RELATIVISM
SUBJECTIVIST
HERMENEUTIC
Common dichotomies in methodological literature
qualitative
quantitative
* bad
descriptive
empiricism
atheoretical
good
predictive
rationalism
theoretical
subjective
inductive
participant observation
anthropology
naturalism
art
hermeneutics
aristotelian
teleological
finalistic
understanding
Verstehen
phenomenological
objective
deductive
survey techniques
sociology
anti-naturalism
science
positivism
galilean
causal
mechanistic
explanation
Erklaren
logical positivism
Arguments to support rejection of the idea of
paradigms and moving towards the idea of continuum
• view which divides social research into limited number of
paradigms, whether based on philosophical categories or
methodological distinction, is misleading
• research practice is much more variegated and complicated
than that proposed by a paradigmatic view
• misguided attribution of two distinct and often opposite
sets of qualities to the two large families of methods
Arguments to support rejection of the idea of
paradigms and moving towards the idea of continuum
• misrepresentation of complex philosophical ideas
• major differences in philosophical as well as methodological
preferences within the camp of qualitative researchers as
well as within the ranks of quantitative researchers
• it is rather a complex and multifaceted individual mental
model, which is formed by many factors like education and
training, personal values, disciplinary perspectives,
philosophy of science, etc., than formalized paradigm that
guides the work of social inquirers and evaluators
On the way towards the idea of continuum
• Extension of the paradigm model to 5 paradigms
by Guba and Lincoln (2001/2005/2008)
• The idea of mixed methods as an interactive continuum
by Ridenour & Newman (2008)
• QUAL-MM-QUAN continuum
• Multidimensional continuum of research projects
by Teddlie & Tashakkori (2009)
On the way towards the idea of continuum
Integrated Multidimensional Continuum of Research Methodology
(adapted from Niglas 2001)
• not static or “all-inclusive”
• functions as an illustration supporting an argument against
validity of paradigmatic approach to methodology
• helps to organize our thinking about methodology and thereby
serves educative purposes
Original figure available at: www.tlu.ee/~katrin/mmrm/
Experiences from the field ...
“[...] experience in conducting research to bring about social
change and influence social policy indicates that the most
persuasive policy research includes both elements: numbers
that define the scope and patterns of the problem, and a story
that shows how the problem works in daily life and provides
for emphatetic understanding. These two elements stem from
quantitative and qualitative research. Integrated approaches
are effective for [...] analysis of programs, policies, or actions
that helped to overcome persistent inequalities.”
Spalter-Roth, 2000
The choice of methodology and methods for a study
depends on the purposes and research questions
epistemological,
different schools of thought;
different research traditions
philosophical level:
Needs of life and practice;
goals of funding bodies; …
RESEARCH AIMS &
PROBLEM(S)
or
methodological
level:
quantitative
approach
or
technical
level:
quantitative
methods
or
data:
quantitative
data
(Methodological) skills &
commitments; resources; …
or
qualitative
approach
or
qualitative
methods
or
qualitative
data
Best fit between aims/problems and designs/strategies
(adapted from Creswell & Plano Clark 2007)
Need to ...
Usually best suited research
design/strategy
see if treatment is effective
Experimental design
identify trends/attitudes/... in a population
Survey design
learn about and describe a culture shared
by a group
Ethnography design
generate a theory of a process
Grounded theory design
imporve the subsistence and/or empower
certain group
Action research design
develop or imporve some policies/artifacts
Design research approach
Facilitating change as sovereign aim for evaluation practice and change oriented research approaches
 Design research
 Action research
Cyclical nature of the research process:
 Diagnose / Analyze
 Plan action / design & develop
 Act / test / implement
 Monitor
 Evaluate / Learn
Need for variegated empirical data –> best results will be
often achieved by utilizing mixed methods approach
Traditional model of methodological aspects is
here incorporated into more general framework!
•
Research problem
(question, hypothesis, purpose, …)
•
Strategy
(case study, survey, experiment, grounded theory, action research …)
•
Sampling
(random sample, one case, purposefully chosen cases, … )
•
Data collection
(structured questionnaire, unstructured/open interview, …)
•
Data analysis
(statistical methods, open coding, discourse analysis,… )
•
Interpretation and conclusions
(descriptions, empirical generalisations, …)
What is design research?
• Design research investigates the process of designing with
the overall aim to develop an accessible, robust body of
knowledge that enhances our understanding of design
processes, applications, methods and contexts. Often, this
knowledge helps to define best practice and workable
methods in dealing with design and design related problems.
• Alternative approaches/terms used:
design-based research, design-science research,
design study, design experiment, developmental research,
development study, formative research, ...
What is design research?
• Design research is distinguished from design by the
production of interesting (to a community) new knowledge
• Research driven
• Systematic documentation
• Formative evaluation
• Generalization
• DR strives to combine the creativity of design communities
with appropriate adherence to standards of traditional
research
Characteristics of good design research
•
Development and research takes place through continuous cycles of
design, enactment, analysis, and redesign
•
The central goals of designing artifacts or environments and
developing theories about design process but also about the related
phenomena are intertwined
•
DR must lead to sharable theories that help communicate relevant
implications to practitioners and other designers
•
Research must account for how design functions in authentic settings
•
The development relies on methods that can document and connect
processes of enactment to outcomes of interest
Design research – the methodological cycle
Indicating the need for development and related knowledge ->
-> initial plan of the study
• Analysis of the problem
(needs, goals, pre-existing knowledge, ... )
•
Designing / monitoring / documenting
– planning of the design&development process
(work allocation, schedule, applicable methods, ... )
– monitoring the d&d process
(memos, team discussions, …)
– describing the result of d&d process
(sketches, alternatives, resulting design and/or artifact, …)
•
Evaluation (incl initial implementation)
(testing the design result, evaluation according to standards, feedback
from users and/or experts, ... )
•
Generalisations
(suggestions, requirements, standards, concepts, models, ontologies, theories,...)
Our design research project:
Induction year for novice teachers in Estonia
4 Evaluating the
model and
theoretical
standpoints
3. Applying the
model (being
reflective)
Novice
teacher
5. Shortcomings of
the model and
Generalizations
0. Need to design a
support system
2.Design process,
constructing/
improving the
model
1. Analysing problem,
collecting and
syntezising theoretical
information
Induction year for novice teachers in Estonia
Partners and implementation activities:
•
Schools (mentors, novie teachers, school leaders)
•
Universities
•
Ministry
creating propitious environment for professional development
mentoring beginning teachers
(re-)developing theoretical model for mentoring
mentor training
support programme for beginning teachers
developing policy
ensuring resources
 Joint expert group
(re-)designing implementation model
facilitating networking
quality assurance
Induction year for novice teachers in Estonia
Design research cycles
Design cycles of our project can be divided into two general phases:
•
Cycles I and II - emphasis on planning, designing and re-designing
implementation model for induction programme
Analysis of theoretical and secondary materials
Focus group interviews with experts, novice, mentors and school leadres
Survey for beginning teachers
Semi-structured questionnaire for mentors
Open interviews for the school leaders
Monitoring and self-evaluation (diaries, peer visits, peer groups, ...)
•
Cycles III, IV, etc – emphasis on implementation of the induction
programme
- Constant evaluation and improvement of the model => monitoring and
research activities continued
Experiences from our design research project
Well established cyclical process of design research where QUAL and
QUAN data is systematically collected and analysed in order to inform
design & development process, empovered by strong partnership links
provide good basis for evidence based educational restructuring
ensuring
•
Continuity of the process
- ongoing evaluation and development
- ongoing partnership
•
Sustainability of the project
- acceptance by the practitioners
- wide-scale implementation
Towards Fifth Generation Evaluation …
some conclusions (read: visions)
The 5th Generation Evaluation can be characterised by:
Conceptualising evaluation as a tool for facilitating development
and therefore utilizing cyclical methodological designs which place
evaluation into the process of continuous pursuit for improvement
(evaluation not as a purpose of the project per se but implemented
within a project to serve wider purposes)
Valuing initiation from stakeholders as well as from experts to
start and lead systematic process of evidence-based development
(utilizing designs from action research to design research with various
hybrids in between)
Towards Fifth Generation Evaluation …
some conclusions (read: visions)
Open and inclusive stance towards various worldview positions
and methodological approaches where paradigmatic confrontation
is commuted for continuum model accommodating overlaps between
and being enriched by the differences there are between traditions
(dignifying personal mental model instead of paradigm, following the
principle of informed creativity, choosing methods and techniques in
concord with specific aims and questions considering QUAN, QUAL as
well as MM approaches)
Acceptance that evaluation can generate results that apply to the
unique local settings as well as results that serve more general
shared needs for development
(accepting and utilizing different forms of generalizations)
Thank you !
Katrin Niglas
PhD, Professor
Institute of Informatics
[email protected]; www.tlu.ee/~katrin/