Transcript Lect 11_Ian 17_Mixed Methods_Evaluation research_on line

GSSR
Research Methodology and Methods of Social Inquiry
socialinquiry.wordpress.com
January 17, 2012
I. Mixed Methods Research
II. Evaluation Research
MULITPLE-METHODS APPROACH
Triangulation:
applying 2 or more dissimilar measures and/or methods
(research strategies) to investigating a certain problem.
Why do it:
increased confidence in findings
Key:
what you want to study (i.e. the nature of the research
question, the phenomena considered) should determine
your research strategy/methods!
- the relative strengths & weaknesses of alternative approaches
should be weighted in deciding which methods to select,
and how best to combine them when possible.
Table 12.2 in Singleton and Straights, p. 399
Multiple methods can also be used within a single approach:
- allows exploiting the strengths & weaknesses of
complementary methods.
Ex:
One approach (survey method), but mail questionnaire to
probability sample, & face-to-face interviews on smaller
sample of non-respondents, to estimate non-response bias.
Vignette experimental designs in survey research;
Use of archival records to identify groups for field research …
II. EVALUATION RESEARCH
www.socialresearchmethods.net/kb/intreval.php
http://ec.europa.eu/regional_policy/sources/docgener/evaluation/evalsed/sourc
ebooks/method_techniques/index_en.htm
Application of social research methods for:
(a) assessing social intervention programs & policies instituted to
solve social problems;
(b) in the private sector: assess policy, personnel, products.
Major goal of evaluation:
Influence decision-making/policy formulation through providing
empirically-driven feedback.
Evaluation takes place within a political & organizational context,
where researchers face multiple stakeholders.
Stakeholders:
- individuals/ groups/ or organizations that have a significant
interest in how well a program/product functions/performs.
Ex:
Program sponsor (actor who initiates & funds the
program/product)
Evaluation sponsor (who mandates & funds the evaluation)
Policymaker/decision maker who determines the fate of the
program/product, …
Outcome of evaluation:
Detailed technical report that describes the research design,
methods and results.
Plus: executive summaries, memos, oral reports geared to the
needs of specific stakeholders.
Evaluation Strategies
Scientific-experimental models (see
socialresearchmethods.net/kb/intreval.php)
Take values & methods from the social sciences;
- prioritize on the desirability of impartiality, accuracy, objectivity
& the validity of the information generated.
Ex:
- experimental & quasi-experimental designs;
- objectives-based research that comes from education;
- econometrically-oriented perspectives including costeffectiveness and cost-benefit analysis;
- theory-driven evaluation.
Management-oriented systems models
- emphasize comprehensiveness in evaluation, placing
evaluation within a larger framework of organizational
activities.
The Program Evaluation and Review Technique (PERT)
The Critical Path Method (CPM).
The Logical Framework -- "Logframe" model developed at U.S.
Agency for International Development Units Treatments
Observing Observations Settings (UTOS);
Context Input Process Product (CIPP)
Qualitative/anthropological models
Emphasize:
- the importance of observation;
- the need to retain the phenomenological quality of the
evaluation context
- the value of subjective human interpretation in the evaluation
process.
Ex: naturalistic or 'Fourth Generation' evaluation; the various
qualitative schools; critical theory & art criticism approaches;
and, the 'grounded theory' approach of Glaser and Strauss
among others.
Participant-oriented models
Emphasize the central importance of the evaluation participants,
especially clients & users of the program or technology.
Ex: Client-centered and stakeholder approaches; consumeroriented evaluation systems.
Types of Evaluation
Formative Evaluation (Product):
Needs assessment: who needs the program? How great the is
the need? & What might work to meet the need?
Evaluability assessment: is an evaluation feasible & how can
stakeholders help shape its usefulness?
Structured conceptualization: helps stakeholders define the
program/ technology, the target population, & the possible
outcomes
Implementation evaluation: monitors the fidelity of the program
or technology delivery
Process evaluation investigates the process of delivering the
program or technology, including alternative delivery
procedures
Summative evaluation (Effects/Outcome):
Outcome evaluations: did the program/technology produce
demonstrable effects on specifically defined target outcomes?
(effect assessment)
Impact evaluation: broader; assesses overall/ net effects -intended or unintended -- of the program/ technology as a
whole
Cost-effectiveness & cost-benefit analysis address questions of
efficiency by standardizing outcomes in terms of their dollar
costs & values
Secondary analysis: reexamines existing data to address new
questions or use methods not previously employed
Meta-analysis integrates the outcome estimates from multiple
studies to arrive at an overall/ summary judgement on an
evaluation question
Methodological Issues in Evaluation Research
Effect Assessment: did the program/technology caused
demonstrable effects?
The ‘black box’ paradigm
We can observe:
- what goes into the ‘black box’ – the inputs (here, the
program/product/intervention)
and
- what comes out of the box – the output (certain effects).
Theory as guide to research
Research Design & Internal Validity
Ideal strategy for effect assessment: experiment, with units of
analysis randomly assigned to at least 2 conditions (one with
intervention present, one without).
Measurement Validity
- need good conceptualization
- reliable and valid measures of cause (treatment program) and
effect (expected outcome).
Issues with creating valid indicators of program outcomes.
Timing of outcome measurement:
- lagged effect of the program; continuous/gradual effects, vs.
instantaneous effects.
To increase measurement validity: multiple measurement
(independent measures) & different points in time.
External Validity
- Random sample, or ‘true’ experiments are most often not
feasible  non-probability sample
Selection biases:
- self-selection into the treatment group;
- selection of program participants because they are likely to
generate positive results / are available;
Social context of evaluation may threaten external validity