Transcript Chapter 3 Identifying Issues and Formulating Questions

Chapter 3 Identifying Issues and
Formulating Questions –
Mary Ellen Good



Identification and formulation of questions
is a critical phase of the evaluation process
The essence of evaluation is generating
credible answers to questions about the
performance if a social program.
Questions must be answerable with the
resources available
Evaluation revolves around the
hub of carefully crafted
evaluation questions





Questions need to be meaningful for
stakeholders and program decision makers
Who will use the the evaluation results?
What info do they need?
How do they expect to use it?
The evaluator’s own analysis of the
program is also important
Communication with decision
makers and stakeholders



Questions are formulated through discourse
and negotiation
Engaging these groups increases their
understanding and appreciation, and makes
effective use of the findings when available
Evaluators can’t depend solely on their
perspectives – they are not experts on
evaluation
Role of the Evaluator




Framing the questions – the evaluator knows how
to analyze a program and focus an evaluation.
Raise issues that might be overlooked
Identify aspects of program’s operation and
outcomes that warrant inquiry
Draw out concerns from stakeholders that can be
translated into questions that can be answered by
evaluation research
Written summary of questions
Helps evaluator guide the evaluation design.
This is a useful reference for design and
selecting research procedures.
Can be shared with stakeholders to ensure
their concerns are addressed.
Safeguards against later misunderstanding of
what the evaluation was suppose to
accomplish.
The 2 most important topics
related to evaluation questions:

1. How to formulate evaluation question
that can be addressed using research
procedures available to the evaluator

2. How to determine specific questions
that the evaluation should focus.
What makes a good evaluation
question?



Focus the evaluation on the areas of
program performance at issue for key
decision makers and stakeholders
Facilitate design of data collection to
provide meaningful info about performance
Identify a distinct dimension of program
performance that can be credibly assessed
What it means to evaluate
something




Establishing criteria of merit – on what
dimension must the evaluand do well?
Constructing standards – how well should should
the evaluand perform?
Measuring performance and comparing with
standards – How well did the evaluand perform?
Synthesizing and integrating data into a judgment
of merit or worth. What is the merit or worth of
the evaluand
……a thought

To evaluate anything means to assess the
merit or worth of something against criteria
and standards.
Dimensions of Program
Performance


Good evaluation questions must first be
reasonable and appropriate
The questions must be answerable –
specific, concrete, practical, and
measurable
Reasonable and Appropriate
Questions



Work with stakeholders to scale down and
focus the questions –
Lit Review – social science and social
service literature
Get acquainted with the program – its
structure, activities, roles and tasks of
personnel, nature of participants, and
assumptions.
Questions must be Answerable



Easy to formulate unanswerable questions
without realizing it….Does this program enhance
family values?
Must be possible to identify some evidence or
“observables” that can be obtained and will be
credible as the basis for an answer.
Have measurable performance dimensions stated
in unambiguous terms with non-controversial
definitions.
Criteria for Program Performance



What distinguishes evaluation questions is they
have to do with performance and are associated
with criteria by which performance can be
judged.
Performance Criterion (or standard) that apply, as
well as the performance dimension at issue.
An evaluation that only describes performance,
but doesn’t assess it, is not truly evaluation.
Standards by which program
performance may be evaluated:






Needs/wants of target population
Program goals and objectives
Professional standards
Customary practice, norms
Legal requirements
Ethical/moral values: social justice, equity
Standards…continued






Past performance; historical data
Targets set by program managers
Expert opinion
Pre-intervention baseline levels for target
population
Conditions expected in the absence of the
program
Cost or relative cost
Places to look for performance
criteria:




Professional standards – esp in medical and
health programs
Prior experience
Evaluation and program literature
Judgement ratings from stakeholders to
establish criterion levels or ranges (low to
high performance)
Typical evaluation Questions
1.
2.
3.
4.
5.
Need for program services (needs assessment)
Program’s conceptulization or design
(assessment of program theory)
Program operations and service delivery
(assessment of program process)
Program outcomes (impact assessment)
Program cost and efficiency (efficiency
assessment)
Order of the five categories or
questions



The questions draw meaning from the answers to
the prior questions.
Implementation failure – poor outcomes due to
program activities needed to bring about desired
improvements did not actually occur. Nutritional
status of homeless people did not improve
because soup kitchen was rarely open.
Theory failure- Program conceptualization and
design can’t generate desired outcomes no matter
how well implemented – soup kitchen was far
away.
Evaluation Hierarchy –
Assessment of:





1. Need for the Program
2. Program Design and Theory
3. Program Process and Implementation
4. Program Outcome?Impact
5. Assessment of Program Cost and
Efficiency
When developing questions




Start at the bottom of the evaluation
hierarchy and consider what is known and
what needs to be known about the issues
Then move up the ladder
Logical interdependencies between the
levels
Premature attention to higher-order
evaluation questions can be avoided
Specific Questions Evaluators
Should Ask


Don’t rely only on input from sponsors or
stakeholders – they are familiar with the
program and can overlook critical, routine
aspects or performance.
Evaluator should make an independent
analysis of program performance pertinent
for investigation
continued



Multiple stakeholders = multiple views
Set priorities with them and integrate as
many relevant concerns as possible
If they have constraints, the evaluator must
decide whether or not to conduct the
evaluation anyway or negotiate to broaden
additional perspectives
Essential

Strive to ensure stakeholders understand
and accept the nature of the eval process,
the type of info it will produce, what it
might mean if results come out one way or
another, and what ambiguities or
unanswered questions remain.
Snowball approach


Identify and contact a person or group, they
then nominate others, who in turn nominate
others.
The process ends when no new
nominations are given.
Topics to discuss with
Stakeholders



Why the evaluation is needed
The program goals and objectives
What are the most important questions for
the evaluation to answer?
Backward mapping
Start with a specification of the desired
endpoint and work backward to determine
what must be done to get there.
Analysis of Program
Assumptions and Theory



Construct a conceptual model of how the
program is expected to work and the connections
presumed between activities and functions and
social benefits it is intended to produce.
Program theory is the set of assumptions about
relationships between strategy and tactics and the
social benefits it expects to produce.
Program theory is also called program
conceptulaization, program plan, blueprint or
design.
Logic Model

Lays out expected sequence of steps going
from program services to client outcomes.
Collating Eval Questions and
Setting Priorities



Questions tend to cluster around program
functions – ex services or outcomes
Also around issues – ex need, design,
impact
When the evaluator prioritizes the
questions for a selected program, the next
phase is design.
Summary





Assess the need for a program
Program theory or plan for addressing the need
Implementation of the program and associated
process
Impact or outcome of the program
implementation on the social need
Efficiency with which the program attains its
outcomes