Chapter 6: Program-Oriented Approaches
Transcript Chapter 6: Program-Oriented Approaches
Presentation by Jay Kerstetter, Amanda Brown,
Jody Yoos, & Jane Lightner
#1 What are the key concepts of the
objectives-oriented evaluation approach?
#2 How has this approach influenced
What is it?
Objectives-oriented evaluation approach helps
determine whether some or all of the program
objectives are achieved and, if so, how well they
Evaluators may work with stakeholders to
establish if program objectives are met.
Information from this approach can assist with
deciding to maintain, terminate, or change
approaches within the program.
1. Ralph Tyler is credited with initiating this approach in the 1930’s
2. Tyler began to formulate his views on education and evaluation.
3. His approach included the following steps:
Establish goals or objectives
Classify the goals or objectives
Define objectives in behavioral terms
Find situations in which achievement of objectives can be shown
Develop or select measurement techniques
Collect performance data
Compare performance data with behaviorally stated objectives
4. This approach was readily adoptable by evaluators and had great
influence on evaluation theorists.
Tylerian Evaluation Approach
Discrepancy Evaluation Model:
Developed by Malcolm Provus; viewed evaluation as a
continuous information management process.
Provus stemmed key characteristics of his proposal from Tyler.
Provus viewed evaluation as a process.
This process was called DEM; Discrepancy Evaluation Model,
which are broken into four developmental stages.
◦ Cost-benefit analysis (optional)
The DEM was designed to facilitate the development of programs
in large public school systems and later applied to state
evaluations by federal bureau.
The DEM was one of the earliest approaches to evaluation and
elements of it can still be found today in many evaluation.
Orienting Question #3:
How is the objective-oriented
evaluation approach used today?
Accountability in education
Performance monitoring systems
used in many
refinements to the system since the
Ralph W. Tyler:
Tylerian Evaluation Approach
Influenced the Elementary and Secondary
Education Act (ESEA) of 1965, first act to
require evaluation of educational
Started the National Assessment of
Educational Progress (NAEP), only way to
examine educational achievement of al 50
Approach evaluated in Pittsburgh public
Viewed evaluation as a continuous
information- management process
designed to serve as “the watch-dog of
program-management” and the
“handmaiden of administration in the
management of program development
through sound decision making”
Malcolm Provus: Provus’s
Discrepancy Evaluation Model
Orienting Question #4a: How are
Logic Models used in evaluation?
an extension of
Designed to fill
in those steps
program and its
Today Logic Models are used in program
Help program staff articulate /discuss how
program might achieve goals
What elements are important to evaluate at
any given time
Build internal evaluation capacity or think in
an evaluative way
United way of America
WK Kellogg Foundation
Annie E Casey Foundation
Is used by evaluators to gain a better
understanding of the program
Better define the evaluation questions the study
To aid their choices of what concepts to measure
and when to measure them
To improve their interpretation of results and
their feedback to stakeholders to enhance use.
Orienting Question #4b:
How are Program Theories used in
Steps in Theory Based Evaluation
1) Engage Relevant stakeholder
2) Develop a first draft of program
evaluator or evaluation team
3) Present draft to stakeholders for further
discussion, reaction, and input
4) Conduct a plausibility check
5) Communicate findings to key
6) Probe arrows for model specificity
7) Finalize program impact theory
Theory Driven Evaluation
Work with stakeholders to identify key questions to be answered
in the evaluation and the appropriate designs and methods for
answering those questions
Emphasis is on testing the program model
Provide guidance as to what to measure and when to measure it
The selection of the evaluation questions to be addressed depend
on the stage of the program and what stakeholders hope to learn
Provides evaluator with critical information that can be used
throughout the evaluation
Orienting Question #5: How do
theory-based evaluation and
objectives approaches differ?
The objectives are
identified by group
looking for the
Purpose of activity
specified and then those
purposes are being
Individual creditedRalph W. Tyler
The evaluator discusses
goals, basics, and
objectives of program
The evaluator decides
how the program should
work and then sees if it
works that way.
Huey Chen and Leonard
Bickman helped to
develop approaches to
Science based and
Question #6: What are the strengths and
limitations of program oriented
Objective oriented – the simplicity of the
concept and program makes it easy to
understand, follow and possibly
◦ Face validity – the evaluator is being held
accountable for what is being asked to be
evaluated. They want to see if what is currently
happening or working based on their own goals
and objectives for the program.
Theory oriented – the chance for the
evaluator to partake in dialogue with the
stakeholders is a strength of the program
because it helps them to expand their
knowledge on the topic.
◦ This then leads to the evaluator to have a clear
understanding of the program so they know
how to properly evaluate the program.
Objective oriented – the evaluator can have
a single minded focus on the objectives
which will cause them to overlook the
complications, elements and factors
contributing to that programs success or
a. This approach does not ask for the evaluator to
gain an understanding of the context in which
the program operates this could then be what is
affecting the program’s success or failure.
b. Evaluator could ignore the actual value of the
objective. Since the evaluator is told what the
objectives are they are not asked to evaluate
whether that objective even fits the program.
Theory oriented – this like the objective
oriented can cause the evaluator to ignore
certain aspects of the program that are
important because the evaluator is so
focused on the theory of how it runs and not
how it is actually running.
a. Evaluators may ignore the needs or values of
stakeholders involved with the program.
b. This approach may also oversimplify the
complexity of the program making it feel easier
to evaluate than it really is because not all of the
surrounding factors are accounted for in the
Orienting Question # 7: What is
Rationale: “Goals should not be taken as
Developed by Scriven (1972)---believes the
most important function of a goal-free
evaluation is to reduce the bias that occurs
from knowing program goals
Thus…increase objectivity in judging the
program as a whole.
Orienting Question #8: What does
it teach us about conducting an
Goals can act as “blinders” causing us to
possibly miss the most important outcomes
not related to the goals.
Goal-free evaluation was proposed to
primarily indentify the unanticipated side
effects that an objectives-oriented
evaluation might miss
Major Characteristics of a GoalFree Evaluation
becoming aware of
goals are not
permitted o narrow
the focus of the
4. The goal-free
minimal contact with
manager and staff.
on actual outcomes
rather than intended
the likelihood that
effects will be noted.
Internal Goal Evaluator vs.
How well is program
What does the
Looking at ALL the
intended or not
Goal Directed + Goal-Free=
can work together
Information taken from
Alternate Approaches and