Transcript EVAL 6000: Foundations of Evaluation

EVAL 6000: Foundations
of Evaluation
Dr. Chris L. S. Coryn
Kristin A. Hobson
Fall 2011
Agenda
• Stage One theories
– General characteristics
– Michael Scriven
• Activity
• Questions and discussion
• Encyclopedia of Evaluation entries
General Characteristics of
Stage One Theories
• Stage One theories can be
characterized as a period of social
problem solving
• Emphasis was on scientific rigor and
valid causal knowledge of the effects
of social programs
• Reflects the tenor of the 1960s, a
period of rapid development in the
social sciences
Stage One Theory of Social
Programming
• Programs largely viewed as
immediately implementable solutions to
social problems
• Rested on assumptions that
A. Implementation and evaluation would be
unambiguous
B. “Successes” would be adopted
C. A and B combined, would lead to
ameliorating social problems
• Concerned primarily with
innovations/pilots/demonstrations
Stage One Theory of
Knowledge Construction
• Priority was given to “truth”
• The dominant view was that an
external reality probably exists,
though it may never be perfectly
known
• Valid knowledge could be constructed
(more or less)
• Emphasis on sources of bias and the
best methods for reducing them
Stage One Theory of Valuing
• Theorists at this stage agree that
valuable social programs solve
important social problems
• They disagree about how value
claims are made and whose values
should be given priority
– Prescriptive versus descriptive valuing
Stage One Theory of
Knowledge Use
• Dominant assumption was “naïve
instrumentalism”
• Effective programs would be
maintained or scaled-up and
ineffective ones would be eliminated
(i.e., a rationalist view)
• Use would occur almost naturally
– Evaluation findings, being scientifically
rigorous, would inform decision making
Stage One Theory of
Evaluation Practice
• Emphasis on assessing a program’s
effectiveness at solving important
social problems
• Primary task is summative claims or
judgments
– Methods of arriving at summative
statements were given priority
• Distance is crucial to scientific rigor
and integrity
“Bad is bad and good
is good and it is the
job of evaluators to
decide which is
which”
— Michael Scriven
Biographical Sketch
• Born in 1928, in Bealieu, Hampshire,
England
• Ph.D. in Philosophy, Oxford University
• M.A. in Mathematics and Philosophy,
Melbourne University
• Former President of AERA and AEA
• Author or more than 300 publications
• Recipient of numerous awards and
prizes
Scriven’s View of Evaluation
• Scriven argues that a singular
statement of value is the sine qua
non of evaluation and is the primary
task of evaluators
• Evaluation is an autonomous
discipline, and a vital component of
all other disciplines
Scriven’s Influence
• Scriven is widely viewed as the first
major evaluation theorist
• Whether one agrees or disagrees
with his views, Scriven has directly
and indirectly influenced nearly every
other theorist and theory
“Not to know Scriven
is to be illiterate in
evaluation”
— Thomas D. Cook
Scriven’s Major Contributions
•
•
•
•
•
•
•
•
•
Definition of evaluation
Valuing theory
Formative and summative distinction
Goal-free evaluation
Causal logic and cause-probing
methods
Logic of evaluation
Theory-free evaluation
Metaevaluation
Key Evaluation Checklist
Goal-Free Evaluation
• Premised on examining what effects
a program is actually having rather
than intended effects
• Aimed at bias reduction through
avoiding knowledge of program goals
• Relies on needs assessment to
determine the extent to which a
program meet needs
Causal Logic and Reasoning
• Modus operandi method (MOM) and
general elimination method (GEM)
for causal analysis
• Causal claims ‘beyond a reasonable
doubt’ should be the standard within
and outside of science, as opposed to
those supported by ‘the balance of
the evidence’
• Overdetemination problem
Theory-Free Evaluation
• The role of evaluators is to determine
only whether programs work, not to
explain how they work
• To attempt explanations is beyond
the capability of most evaluators
• Requires the correct (not just the
believed) logic or theory of the
program
Metaevaluation
• An evaluation of an evaluation
• Internal and external quality control
• Identify strengths, weaknesses, and
uses
• Appropriate criteria
– Validity
– Credibility
– Utility
– Cost-effectiveness
– Ethicality
Key Evaluation Checklist
• Organized into four parts with 15
checkpoints
– Part
– Part
– Part
– Part
A: Preliminaries
B: Foundations
C: Subevaluations
D: Conclusions & Implications
Scriven’s Theory of Social
Programming
• Social programs exist to meet
important societal needs
– Consumer needs should be given the
same primacy in an evaluation
• Good programs should be expanded
and poor programs should be
abandoned
“If I had asked people
what they wanted,
they would have said
faster horses”
— Henry Ford
Scriven’s Theory of Knowledge
Construction
• Realist philosophy and ideology
• Valid knowledge is possible, but
nearly always incomplete
• Practical inference rather than the
development of scientific theories
about social programs
• Socio-organizational controls to
reduce bias in knowledge claims
Scriven’s Theory of Valuing
• Prescriptive theory premised on
some values having higher priority
than others
• Valuing is no less scientific in its
validity than “value-free” science
• Relative valuing, rather than absolute
valuing, is preferable
Scriven’s Theory of Knowledge
Use
• Use is “self-evident”
• “Naïve instrumentalism”
• Primarily concerned with summative
statements emphasizing short-term
instrumental use
“Evaluation is what
it is, the
determination of
merit and worth,
and what it is used
for is another
matter”
— Michael Scriven
Scriven’s Theory of Evaluation
Practice
• Methodologically pluralistic
• Audience of evaluation is
predominately “consumers”
• Evaluations should serve the “public
good”
• Emphasizes bias minimization
through socio-organizational controls
Encyclopedia Entries
•
•
•
•
•
•
•
•
•
•
•
Bias
Causation
Checklists
Chelimsky, Eleanor
Conflict of Interest
Countenance Model of
Evaluation
Critical Theory
Evaluation
Effectiveness
Efficiency
Empiricism
Independence
• Evaluability
Assessment
• Evaluation Use
• Fournier, Deborah
• Positivism
• Relativism
• Responsive evaluation
• Stake, Robert
• Thick Description
• Utilization of
Evaluation
• Weiss, Carol
• Wholey, Joseph