Transcript the Participant-Oriented Evaluation

PARTICIPANT-ORIENTED
EVALUATION APPROACHES
EDF5461
Summer 2011
Alison Moore
Jenna Oster
Aaron Paquette
DEFINITION
 Participant-Oriented Evaluation Approaches
"currently include many different models, but
their commonality is that they all use
stakeholders- people with an interest or "stake"
in the program- to assist in conducting the
evaluation." (Fitzpatrick 189)
KEY CONCEPTS
 Involves participants in the program, managers,
staff, and other key stakeholders (anyone
involved with planning and implementation of
the evaluation) as a key principle to good
evaluation
 Involve stakeholders to teach evaluators about
program
 Founded in Constructivism (humans gain
knowledge and meaning from interaction and
experience)
DATA COLLECTION METHODS
 In Situ Observation
 Observing stakeholders in their interaction with
the program
 Interviews
 Talking to staff, managers and stakeholders to
learn more about the program
 Documents
Using program documents to help with the
evaluation
TIMELINE
Pre-1967
1967
Mid-1970s
Present
No desire to
include the
human
elements
and/or
qualities within
the evaluation
process
A push to bring
human
elements
and/or
qualities into
the evaluation
process: make
people/
stakeholders a
part of the
evaluation
process
Observations
and identifying
concerns and
issues of the
stakeholders
were examined
and became
enwound with
the P-PE
evaluation
Jean King:
“…overarching
term that involves
program staff
actively in
decision making”
1975:
Responsive
evaluation
Cousins and Earl:
“Applied social
research…
partnership
between evaluator
and stakeholders”
New methods of
participatory
evaluation
continue to
emerge
MAJOR DEVELOPERS
 Robert Stake: One of the first advocates for participantorientated evaluation
 The Countenance of Educational Evaluation (1967)
 Allows for a wide range of data to be collected
 A congruence between the intention and what is observed
 This model looks for continuity between what happens and what is
intended
 Countenance Framework (1967)
 Countenance Matrix
 Responsive Evaluation
 Provides qualitative evidence of a
programs effectiveness
 Includes stakeholders in the
implementation of policies and goals
MAJOR DEVELOPERS
(CONTINUED)
 Egon Gruba and Yvonna Lincoln
 Naturalistic Inquir y (1985)
 Collects qualitative data to interpret and understand unique situations where there is a level of
uncertainty
 Reliability and Validity: “Trustworthiness”
Credibility
Transferable
Dependable
Conformability
 Naturalistic Evaluation
 Responsive Evaluation + Naturalistic Methodologies = Naturalistic Evaluation
 Evaluators participate in the evaluation and try to uncover the needs of the all members
 Three phases
1. Familiarizing Phase
2. Action Phase
3. Synthesis Phase
 Four th-Generation Evaluation (1989)
 Uses constructionist paradigm to gather, analyze, and organize evaluative material based on
the needs of stakeholders
 Never-ending process
APPROACHES
Participatory evaluation: “An overarching term for any
evaluation approach that involves program staf f or participants
actively in decision making and other activities related to
planning and implementation of evaluation studies.” (p. 199)
Two main types of participatory evaluation:
P-PE
Practical Participant Evaluation
is used for practical reasons,
limiting the evaluation to a
particular organization.
T-PE
Transformative Participant
Evaluation means just that, to
transform and empower the
stakeholders (social change).
CATEGORIES OF PARTICIPATORY
APPROACHES
 There are 3 dimensions in which participatory
approaches differ, according to Cousins,
Donohue, and Bloom (Fitzpatrick 200):
 1) Control over the evaluation or technical decisionmaking process
 2) Stakeholder selection
 3) Depth of Participation
APPROACHES
(CONTINUED)
M AT R I X ( P. 2 0 4 )
Approach
Practical Participatory Evaluation
(P-PE)
Key Elements
Context
• Explicit focus on evaluation
decisions
• Evaluators work closely with
primary stakeholders
• Formative decisions
• Stakeholder control
• Many stakeholders
• Emphasis on program
participants
• Much involvement
• Participants are oppressed
• Social justice is a concern
• Participant empowerment is
the goal
Empowerment Evaluation
• Stakeholder control
• Much involvement
• Need for empowerment to build
staff
• Need for building internal
• Self-monitoring and program
improvement
Democratically Oriented
Evaluation
• Evaluator control
• Many stakeholders
• Limited involvement
• Need for a dialogue among
stakeholders
Transformative Participatory
Evaluation (T-PE)
STRENGTHS OF PARTICIPANT-ORIENTED
EVALUATION APPROACHES
 When stakeholders are involved in evaluations, they . . .
 . . . provide data sources, distribute surveys, and set up focus groups
 . . . are familiar with program context and environment
 . . . establish validity of the study because they possess/incorporate
knowledge to inform decisions
 . . . can communicate program situation/details to facilitate
evaluator’s understanding
 Including stakeholders in evaluations . . .
 . . . gains their trust in the evaluator and the evaluation process
 . . . leads to better chances of stakeholder use/adoption
 . . . improves stakeholders’ understanding of program/organization
LIMITATIONS OF PARTICIPANT-ORIENTED
EVALUATION APPROACHES
 Competence/feasibility
 Is the evaluator truly capable of acting as facilitator, coordinator, and
consultant during the evaluation process?
 Are stakeholders able to perform tasks required of them ?
 Credibility
 Because stakeholders are deeply invested in the program, will they
act to change the program based on the evaluation results?
 Political
 Due to involvement of people (primary stakeholders, managers, staff)
 Time-consuming and expensive
REVIEW
 Evaluators include stakeholders in evaluation planning and
implementation
 Stake and Gruba and Lincoln responded in the 1970s and
1980s to established methods of evaluation
 Wanted approaches offering consideration of program stakeholders
 Two main branches of participant -oriented approaches
 Practical participatory evaluation (P -PE): encourage
involvement/ownership, thus increasing use of results by
stakeholders
 Transformative participatory evaluation (T-PE): empower stakeholders
by direct involvement in evaluation process
 Three primary dimensions of participatory approaches
 Evaluator or stakeholders possess control
 Individual stakeholder selection
 Extent of stakeholder involvement
REVIEW
(CONTINUED)
 Practical participatory (P -PE) approaches often rely on
 Stakeholder involvement
 Practical participatory evaluation
 Developmental evaluation
 Transformative approaches often rely on
 Empowerment evaluation
 Deliberative democratic evaluation
 Many definitions for participative evaluation exist
 All involve stakeholders, but no additional details are standard