Transcript Name of presentation - Western Michigan University

EVAL 6000: Foundations of Evaluation
Dr. Chris L. S. Coryn & Carl D. Westine
September 23, 2010
Word of the Day
hu·mil·i·ty (noun) \hyü-ˈmi-lə-tē, yü-\
: The quality or state of being humble
Synonyms: demureness, down-toearthness, humbleness, lowliness,
meekness, modesty
Antonyms: arrogance, assumption,
bumptiousness, conceit, egoism, egotism,
haughtiness, hauteur, huffiness,
imperiousness, loftiness, lordliness,
peremptoriness, pomposity, pompousness,
presumptuousness, pretense (or pretence),
pretension, pretentiousness, pride,
pridefulness, superciliousness, superiority,
toploftiness
Evaluation ‘Lens’
– In 30-40 minutes
• Using the assigned perspective identify
a set of criteria for evaluating this
room
• On each criterion identified, rate the
room’s quality using standards of
your choice
• Determine how to create an overall
rating
– In 10-15 minutes
• Present your reference lens, criteria,
and ratings to the class
• How did your group arrive at its
selected criteria, standards,
observations (measurement), and
synthesis?
Evaluation Theory and Logic
Like statistics, evaluation is a subject of
amazingly many uses and yet few
effective practitioners
—Coryn (2009)
Evaluation Theory and Logic
Theory (Social Science) Evaluation Theory
• A set of interrelated
constructs, definitions,
and propositions that
present a systematic view
of phenomena by
specifying relations
among variables, with the
purpose of explaining and
predicting phenomena
• (Normative) Evaluation theories
describe and prescribe what
evaluators do or should do
when conducting evaluations
(and their anticipated
consequences)
• They specify such things as
evaluation purposes, users and
uses, who participates in the
• Not to be confused with
evaluation process and to what
natural or biological
extent, general activities or
theories, phenomena,
strategies, methods choices,
predictions, explanations,
and roles and responsibilities of
principles, or laws, among
the evaluator, among others
others
Evaluation Theory Taxonomies
• Four systems are useful in understanding, describing, and
classifying various types of evaluation theories
1. Shadish, Cook, and Levition’s (1991) five principles
that undergird evaluation
• This system is directed toward theories of program
evaluation
2. Stufflebeam’s (2001) taxonomy
• Classifies evaluation approaches into distinct
categories based on primary orientation
3. Alkin and Christie’s (2004) evaluation theory tree
• Describes major theorists’ orientations
4. Fournier’s (1995) more general ‘logic of evaluation’
(largely derived from Scriven’s earlier works)
• This system is more generalizable and useful for
describing nearly all forms of evaluation (e.g.,
personnel, product, program) and approaches (e.g.,
goal-based, participatory, empowerment)
Shadish et al. Five Principles
• Theories of evaluation can be (somewhat)
described by five dimensions (we’ll come
back to this next week)
1. Social programming: the ways that social
programs and policies develop, improve, and
change , especially in regard to social
problems
2. Knowledge construction: the ways
researchers/evaluators construct knowledge
claims about social programs
3. Valuing: the ways values can be attached to
programs
4. Knowledge use: the ways social science
information is used to modify programs and
policies
5. Evaluation practice: the tactics and
strategies evaluators follow in their
professional work, especially given the
constraints they face
Stufflebeam Taxonomy
• General classification scheme
• Pseudoevaluations
• Questions- and methods-oriented
• Improvement- and accountabilityoriented
• Social agenda and advocacy
• Eclectic
Stufflebeam Taxonomy
• Pseudoevaluations
• Shaded, selectively released,
overgeneralized, or even falsified findings
• Falsely characterize constructive efforts—
such as providing evaluation training or
developing an organization’s evaluation
capability
• Serving a hidden, corrupt purpose
• Lacking true knowledge of evaluation
planning, procedures, and standards
• Feigning evaluation expertise while
producing and reporting false outcomes
Stufflebeam Taxonomy
• Question- and method-oriented
• Address specific questions (often
employing a wide range of methods)—
questions-oriented
• Typically use a particular method
(methods-oriented)
• Whether the questions or methods are
appropriate for assessing merit and worth
is a secondary consideration
• Both are narrow in scope and often
deliver less than a full assessment of
merit and worth
Stufflebeam Taxonomy
• Improvement- and accountabilityoriented
• Fully assess an evaluand’s merit and
worth
• Expansive and seek comprehensiveness
• Consider the full range of questions and
criteria to assess and evaluand
• Often employ needs assessment as the
source of foundational criteria
• Look for all relevant outcomes, not just
those keyed to objectives
Stufflebeam Taxonomy
• Social agenda- and advocacy-oriented
• Aimed at increasing social justice through
evaluation
• Seek to ensure that all segments of
society have equal access to
opportunities and services
• Advocate affirmative action to give the
disadvantaged preferential treatment
• Favor constructivist orientation and
qualitative methods
Stufflebeam Taxonomy
• Eclectic
• No connection to any particular
evaluation philosophy, methodological
approach, or social mission
• Advanced pragmatic approaches that
draw selectively from a wide range of
other evaluation approaches
• Designed to accommodate needs and
preferences of a wide range of clients and
evaluation assignments
• Unconstrained by a single model or
approach
The Evaluation Theory Tree
Use
Valuing
Methods
Alkin & Christie (2004)
Accountability &
Fiscal Control
Social Inquiry
The Evaluation Theory Tree
• The trunk is built on a dual foundation of
accountability and social inquiry
• These two areas have supported
development of the field in different ways
• The need and desire for accountability
presents a need for evaluation
• Accountability is broad in scope
• It is not a limiting activity, but rather
is designed to improve and better
programs (and other things), society,
and the human condition
The Evaluation Theory Tree
• The social inquiry root of the tree
eminates from a concern for employing a
systematic and justifiable set of methods
for determining accountability
• While accountability provides the rational,
it is primarily from social inquiry that
evaluation models (i.e., theories,
approaches) have been derived
• The main branch of the tree is the
continuation of the social inquiry trunk
• This is the evaluation as research, or
evaluation guided by research methods,
branch (designated METHODS)
The Evaluation Theory Tree
• Initially inspired by Michel Scriven, the
VALUING branch firmly establishes the
vital nature of the evaluator in valuing
• Those on this branch maintain that
placing value on objects is the central
task of evaluation
• Subsequent theorists extend the
evaluator’s role to include facilitating the
placing of value by others (e.g., Guba &
Lincoln)
The Evaluation Theory Tree
• The third major branch is USE, which
originated with the work of Daniel
Stufflebeam and Joesph Wholey, where
evaluation was focused on decision
making
• Work done by theorists on this branch
express a concern for how evaluation will
be used and by whom
• Michael Patton, more than any other
theorist, has developed the most
comprehensive, extensive theory of use
Evaluation Logic
• Four steps of the general (working) logic
1. Establishing criteria
• On what dimensions must the
evaluand do well?
2. Constructing standards
• How well should the evaluand perform?
3. Measuring performance and
comparing with standards
• How well did the evaluand perform?
4. Synthesizing and integrating
information/data into a judgment of
merit or worth
• What is the merit or worth of the
evaluand?
Evaluation Logic
• This logic requires two general types of
premises
1. Factual premises
• The nature, performance, or impact of
an evaluand or evaluee
• Roughly equivalent to description
(“what’s so?”)
2. Value premises
• The properties or characteristics (i.e.,
criteria and standards) which typify a
good, valuable, or important evaluand
or evaluee of a particular class or type
in a particular context
Evaluation Logic
• The value premise can be further broken
down into
1. General values
• The merit-defining criteria by which an
evaluand or evaluee is evaluated; the
properties or characteristics which define
a ‘good’ or ‘valuable’ evaluand or evaluee
2. Specific values
• The standards (i.e., levels of performance;
usually an ordered set of categories)
which are applied and by which
performance is upheld, in order to
determine if that performance is or is not
meritous, valuable, or significant
• The ‘sum’ (i.e., synthesis) of these two
answer the “so what?” question
Evaluation Logic: General
Evaluation Logic: General & Working
Evaluation Logic: Working
Evaluation Logic: General & Working
Evaluation Logic: Working
Evaluation Logic
• Several methods of synthesis (step 4)
• Primary method is fact-value synthesis, which
is comparing performance to standards on a
single criterion
• Typically using decision trees/rules and
rubrics
• Sometimes quantitatively (e.g., allocation
or distributional methods including
differational weighting)
• Synthesis across multiple criteria or
dimensions normally uses one of two
methods
• Numeric Weight and Sum (NWS)
• Qualitative Weight and Sum (QWS)
• No matter what the method, this type of
synthesis is not a precise/exact
procedure (yet) and subject to numerous
sources and types of error
Encyclopedia Entries for Last Week
•
•
•
•
•
•
•
•
•
•
•
Assessment
Accountability
Auditing
Campbell, Donald
T.
Cook, Thomas D.
Criteria
Evaluand
Evaluation
Evaluation Theory
External
Evaluation
Formative
Evaluation
• History of
Evaluation
• Independence
• Logic of Evaluation
• Objectivity
• Scriven, Michael
• Shadish, William R.
• Standard Setting
• Standards
• Summative
evaluation
• Value-free inquiry
• Value judgment
• Values
Encyclopedia Entries for This Week
•
•
•
•
•
•
•
•
•
•
Bias
Causation
Checklists
Chelimsky,
Eleanor
Conflict of interest
Countenance
model of
evaluation
Critical theory
evaluation
Effectiveness
Efficiency
Empiricism
• Independence
• Evaluability
assessment
• Evaluation use
• Fournier, Deborah
• Positivism
• Relativism
• Responsive
evaluation
• Stake, Robert
• Thick Description
• Utilization of
evaluation
• Weiss, Carol
• Wholey, Joseph