Transcript PowerPoint 演示文稿 - East China Normal University

INTRODUCTION TO
POLICY ANALYSIS
Text:
Patton and Sawicki, Basic Methods of Policy
Analysis and Planning, 2nd edition.
1. THE NATURE OF PUBLIC PROBLEMS
In
•
•
•
the public sector, problems:
are fuzzy and ill-defined;
have political as well as purely technical aspects;
often lack a good cause-effect knowledge base;
• may be solved only by producing new problems;
• often involve tradeoffs between cost and
effectiveness;
• may be hard to measure adequacy of results;
• may be hard to measure fairness of results.
CLASSICAL RATIONAL PROBLEM -SOLVING
MODEL
In theory, problems can be approached using a rational,
comprehensive problem solving model. The demands of
this model are:
1) Define the problem
2) Determine important social values
3) Identify all alternatives
4) Assess all alternatives
5) Select optimal alternative
6) implement optimal alternative
LIMITATIONS IN THE PUBLIC SECTOR
Theoretical Model
1) Define the problem
2) Determine important
social values
3) Identify all alternatives
4) Assess all alternatives
on all values
5) Select optimal
alternative
6) implement optimal
alternative
Public Sector Limitation
1) Problems are interlined
2) No agreement on social
values
3) Limited time, knowledge
4) Limited resources, lack
of predictability
5) Pressure to select the
first good solution
6) Short time horizon to
produce results
-Traditional research is concerned with broad,
theoretical, complex questions.
-It uses explicit scientific steps and invariant
procedures.
- Policy analysis, on the other hand, is practical,
situational and flexible.
- It addresses local problems and focuses on making
decisions.
- It is more craft or art than science.
Traditional
Research
• Seeks "truth"
• Explicit steps and
procedures
• Addresses broad
questions
• Focus on
complexity
• Science
Policy Analysis
• Is practical
• Flexible,
situational
• Addresses local
problems
• Focus on
decision-making
• Craft
CASE STUDIES IN POLICY ANALYSIS
- Problems in the public sector are multi-faceted and
difficult to pin down.
- As if that was not bad enough, the knowledge domain
of public policy is ill-structured.
- This means that there is no "one best way" to solve all
problems.
- Giving policy analysis only one methodology is like
giving a home owner only a hammer to solve all
household problems.
- A new approach is needed to learning in this area.
This approach is offered by case studies.
- Case studies link problems to a reality;
- They offer the opportunity for the application of policy
analysis techniques in a concrete context.
The way information is remembered and use is linked to
the way it is learned.
- Case studies provide cues to the types of techniques
that are needed to approach a solution to the problem.
- These cues help policy analysts learn multiple
approaches to learning and to problem solving.
Use of case studies will help to:
1) Recognize situations where analysis is appropriate and
productive;
2) Become competent in the application of different
approaches and methods;
3) Learn how to communicate the results of policy analysis.
TIPS FOR PRACTICAL POLICY ANALYSIS
1) Quickly identify the central decision criterion of the
problem
(What is the most important factor in buying a car?
Taking a new job?)
2) Identify what types of public sector actions can be
taken
(Taxing, spending, sanctions, incentives, moral suasion,
education?)
3) Avoid the "one best way" approach
(Have many tools in the tool box, not just one)
4) Learn how to deal with uncertainty
(Admit it, estimate its possible effects)
5) Say it with numbers
(Charts, graphs, tables, maps, etc.)
6) Make the analysis simple and transparent
(Provide details in a technical appendix)
7) Check and re-check the facts
(Use multiple sources of facts, triangulation)
8) Learn to anticipate the objections of opponents
(Improves the ultimate product)
9) Give analysis, not decisions
(Distinguish between analysis and advocacy)
10) Push the boundaries of the envelope
(Expand the problem definition; introduce novel solutions)
11) Policy analysis is never 100% complete, rational, and
correct
(How much time, money, and personnel is available to do
the job?)
2. ALTERNATIVE POLICY ANALYSIS MODELS
1) Quade
a. Policy formulation
b. Search for alternatives
c. Forecast the future
d. Model the impacts of the alternative
e. Evaluate, compare, and rank the alternatives
2) MacRae and Wilde
a. Define the problem
b. Determine criteria
c. Generate alternatives
d. Choose course of action
e. Evaluate policy after implementation
3) Stokey and Zeckhauser
a. Determine the underlying problem
b. Determine the objectives
c. Generate alternatives
d. Predict consequences of each alternative
e. Determine criteria for measuring achievements
f. Choose course of action
4) Urban Institute
a. Define the problem
b. Identify objectives
c. Select criteria
d. Specify the client
e. Calculate the cost of each alternative
f. Assess the effectiveness of each alternative
g. Present the findings
5) Weiner and Vining
a. Problem analysis
a.1. Understand the problem
a.2. Choose goals and constraints
a.3. Choose method of solution
b. Solution analysis
b.1. Choose evaluation criteria
b.2. Specify alternatives
b.3. Assess alternatives
b.4. Recommend solution
6) Hill
a. Define problem
b. Identify alternatives
c. Quantify alternatives
d. Apply decision aids
e. Choose alternative
f. Implement solution
7) Patton and Sawicki
a. Verify, define and detail the problem
b. Establish evaluation criteria
c. Identify alternative policies
d. Assess alternative policies
e. Display and distinguish among alternatives
f. Implement, monitor, and evaluate the policy
SIX STEP POLICY ANALYSIS
1)
2)
3)
4)
5)
6)
Verify, define and detail the problem
Establish evaluation criteria
Identify alternative policies
Assess alternative policies
Display and distinguish among alternatives
Implement, monitor, and evaluate the policy
1) VERIFY, DEFINE AND DETAIL THE PROBLEM
State the problem meaningfully:
Determine the magnitude and extent of the problem
Continually redefine the problem in light of what is possible
Eliminate irrelevant material
Question the accepted thinking about the problem
Question initial formulations of the problem
Say it with data
Locate similar policy analyses
Locate relevant sources of data
Eliminate ambiguity
Clarify objectives
Resolve conflicting goals
Focus on the central, critical factors
Is it important?
Is it unusual?
Can it be solved?
Identify who is concerned, and why?
What power do concerned parties have?
Make a quick estimate of resources required to deal with
the problem
2) ESTABLISH EVALUATION CRITERIA
What are the important policy goals, and how will they be
measured?
Identify criteria central to the problem and relevant to the
stakeholders
Clarify goals, values and objectives
Identify desirable and undesirable outcomes
Is there a rank order of importance among the criteria?
What will be the rules for comparing alternatives?
Administrative Ease
Costs and benefits
Effectiveness
Equity
Legality
Political acceptability
3) IDENTIFY ALTERNATIVE POLICIES
Consider a wide range of options
Consider the status quo, or no-action alternative
Consult with experts
Brainstorming, Delphi, Scenario writing
Redefine the problem if necessary
4) ASSESS ALTERNATIVE POLICIES
Select appropriate methods and apply them correctly
Estimate expected outcomes, effects, and impacts of
each policy alternative
Do the predicted outcomes meet the desired goals?
Can some alternatives be quickly discarded?
Continue in-depth analysis of alternatives that make the
first cut
5) DISPLAY AND DISTINGUISH AMONG
ALTERNATIVES
Choose a format for display
Show strengths and weaknesses of each alternative
Describe the best and worst case scenario for each
alternative
Use matrices, reports, lists, charts, scenarios, arguments
6) IMPLEMENT, MONITOR, AND EVALUATE THE
POLICY
Draw up a plan for implementation
Design monitoring system
Suggest design for policy evaluation
Was the policy properly implemented?
Did the policy have the intended effect(s)?
ROLE OF THE POLICY ANALYST
Policy analysis is a systematic evaluation of the technical
and political implications of alternatives proposed to solve
public problems. Policy analysis refers to both the process
of assessing policies or programs, and the product of that
analysis.
A policy analyst:
• uses qualitative and quantitative data;
• uses a variety of approaches to the problem;
• applies appropriate methods correctly.
Who does policy analysis?
Is public policy analysis a calling?
A vocation?
A service?
A guild?
A cult?
The role of the policy analyst is to:
• Produce arguments for debates about public policy
• Produce evidence for decisions about public policy
• Act as internal organizational consultants
• Act as external policy consultants
• Handle both technical and people aspects of policy
analysis
All policy represents the distribution of power and
resources. These policies are an expression of values.
Values and beliefs are often used as short-cuts to
decision-making.
• What code of ethics should the policy analyst adopt?
• What about the professional values of obligation,
responsibility, discretion, and citizenship?
• What about published professional codes of ethics, such
as ASPA, ICMA, AICP, NASW, NSPE, etc.?
The policy analyst has responsibilities, to the client, the
customer, the self, the profession, the public interest,
fairness, equity, law, justice, efficiency, effectiveness, and
the practice itself.
Who is to define what is good?
Whose values or goals should be pursued?
What is the right thing to do?
Who or what is ultimately to be served?
Should the analyst try first and foremost to do good, or
to do no harm?
Should the analyst give neutral advice, or normative
advocacy?
Should the analyst be supportive or adversarial?
Bias is inevitable in policy analysis. To mitigate the effects
of bias, the analyst can:
• identify all underlying assumptions
• keep accurate records
• use multiple sources of information
• use replicable methods and models
• identify the client's goals and values
• identify the formal and informal actors and institutions
• address relevant professional and ethical considerations
3. CROSS-CUTTING METHODS
SELECTING TECHNIQUES
Selecting the appropriate techniques to use in policy
analysis depends on a variety of factors:
• what the client wants to know
• the time available
• knowledge of the decision criteria
• complexity of the issue
• available data
Some techniques commonly used in various stages of
policy analysis include:
1. Verifying, Defining, and Detailing the Problem
• Back-of-the-envelope calculations
• Quick decision analysis
• Creation of valid operational definitions
• Political analysis
• Issue paper/first cut analysis
2. Establishing Evaluation Criteria
Technical feasibility studies
Economic and financial feasibility studies
Political viability studies
Administrative operability studies
3. Identifying Alternatives
Researched analysis
No-action analysis
Quick surveys
Literature reviews
Comparison of real-world experiences
Passive collection and classification
•
•
•
•
•
•
Development of typologies
Analogy, metaphor, and synectics
Brainstorming
Comparison with an ideal
Feasible manipulations
Modifying existing solutions
4. Assessing Alternative Policies
• Extrapolation
• Theoretical forecasting
• Intuitive forecasting
• Discounting
• Cost/Benefit analysis
• Sensitivity analysis
• Allocation formulas
• Quick decision analysis
• Political feasibility analysis
• Implementation analysis
• Scenario writing
5. Displaying Alternatives and Distinguishing Among Them
• Paired comparisons
• Satisficing
• Lexicographic ordering
• Non-dominated alternatives method
• Equivalent alternatives method
• Standard-alternative method
• Matrix display systems
• Scenario writing
6. Implementing, Monitoring, and Evaluating Policies
Before-and-after comparisons
With-and-without comparisons
Actual-versus-planned performance
Experimental models
Quasi-experimental models
Cost-oriented approaches
7. Cross-Cutting Methods
• Identifying and gathering data
• Library search methods
• Interviewing for policy data
• Quick surveys
• Basic data analysis
• Assessing information quality
• Communicating the analysis
CROSS-CUTTING METHODS
Cross-cutting methods are techniques of policy analysis
that can be used at nearly any stage in the analysis.
They are useful tools for the policy analyst to know how
to use. They include:
• Identifying and gathering data
• Library search methods
• Interviewing for policy data
• Quick surveys
• Assessing information quality
• Basic data analysis
• Communicating the analysis
IDENTIFYING AND GATHERING DATA
Policy analysts need to know how to search for existing
information, such as:
• academic journal articles
• archives
• census records
• hearings
• legislative history
• news media reports
• past policy analyses
• public agency reports
• public records
People are also good sources of information, including
• advocacy groups
• experts
• issue networks
• personal contacts
• professional colleagues
Even personal observation can be a source of data.
Personal observation can furnish data on usage patterns,
compliance patterns, insights into the problem, anecdotes,
and innovative suggestions. However, observation is time
consuming and may suffer from problems with accuracy,
bias, limited samples, and difficult to quantify data.
Observational methods include "sidewalk surveys,"
mechanical counting devices, measures of erosion,
satellite images, etc.
Other sources of information include:
• federal agencies
• libraries
• local agencies
• non-profit agencies
• private organizations
• research institutes
• state agencies
• think tanks
• universities
Policy analysts should seek information from multiple
sources ("triangulation"), especially on controversial
data. Problems with sources of data include:
• outdated statistics
• irrelevant data
• misleading data
• poor quality data
• biased data
Looking for documents that may be helpful in doing the
policy analysis is important. But three questions that
must be asked are:
1) do such documents exist?
2) can they be obtained in a reasonable time?
3) when is additional searching no longer worthwhile?
LIBRARY SEARCH METHODS
Libraries are excellent sources of policy-related
information. To make the most of library resources, follow
these strategies:
1) look up basic policy-related terms and definitions in
encyclopedias, dictionaries, or a subject-related thesaurus;
each policy issue area has its own terms and jargon
2) develop a list of search terms for searching
computerized bibliographic data bases, electronic guides
to library holdings, and Internet access;
3) identify key journals in the field and skim their table
of contents for the past 1-2 years;
4) check guides to current periodicals, newspapers,
news magazines, trade journals, and guides to the
literature
5) check annual reviews in the policy subject area;
conference proceedings on the subject; government
hearings on the subject, etc.
The federal government offers a wide variety of sources:
• Congressional Directory
• Government Yearbooks
• Guide to Federal Statistics
• International Statistics
• Population Reports
• Statistical Abstract of the U.S.
• U.S. Census
• U.S. Government Printing Office catalogues
• U.S. Government Manual
Many sources of legal information have bearing on policy
issues:
• Adjudication and case law
• Agency regulations
• Code of Federal Regulations
• Federal and State statutes
• Federal Register
• Legal Periodicals
• Municipal ordinances
• Supreme Court decisions
INTERVIEWING FOR POLICY DATA
Interviewing is typically conducted with either mass, elite,
intensive, or focus group methodologies. Interviewing is
typically used:
• to gather historical background, context, and evolution
of the policy
• to gather basic facts about the problem
• to assess political attitudes and resources of major
players
• to gather ideas about the future, trends, and forecasts
• to generate additional contacts and materials (snowball
technique)
Elite (specialist) interviewing is most typically used when:
• it is a short-term policy project
• it is on a new topic
• there is a lack of existing literature
• informants are reluctant to put information into writing
• no quantitative data are available
• it is not feasible to use hired interviewers
To set up interviews, the policy analyst usually:
• arranges appointments in advance
• makes formal or informal requests (letterhead,
telephone)
• sends a reminder letter and follows up with a phone
call
• gives the name of a mutual friend or influential person
as a reference
• collects background information prior to the interview
• will conduct a telephone interview if a face-to-face
interview is not possible
When conducting the actual interview, it is usually
accepted behavior to:
• ask before using any recording device
• promise anonymity and/or confidentiality of information
• take notes during the interview
• keep to the allotted time
• thank the person for the interview
• send a follow-up letter
QUICK SURVEYS
Surveys can be conducted by mail, in person, or by
telephone. Survey methodology is described in many
standard research texts. Cross-sectional interviews are
conducted at one point in time across a wide sample of
the population. Longitudinal interviews are conducted
repeatedly over many time intervals (months, years,
decades) with the same individuals. A comparison of the
advantages and disadvantages of the most typical
surveys is displayed below.
Type of
Survey
Response
Rate
Sample Concerns
Staffing
Other concerns
Mail
15%
May not be representative
Least staff time required
Response
with gifts
Telepho
ne
50%
Limited to
telephones
Moderate
required
InPerson
75%
May be needed for lesseducated
those
with
rate
improves
staff
time
Short and simple questions
Most intensive;
supervisors
most
Can cover complex issues
ASSESSING INFORMATION QUALITY
When collecting information and data for policy
analysis, the analyst must assess the quality of the
information and data collected.
Document Analysis:
• When was the document generated?
• What was the original purpose of the document?
• Is there an obvious bias in the document?
• What is the pattern of word usage?
• Does the document omit important information?
Are there errors in the document?
Assessing Interviews:
• Was the information plausible?
• Was the information consistent?
• Does the information diverge from accepted facts?
• Did the respondent report direct experience?
• Did the respondent have ulterior motives?
• Did the respondent operate under some constraints?
• Was the respondent candid?
• Did the respondent acknowledge areas of ignorance?
• Was the respondent self-critical?
Data quality:
• Are multiple sources of information consistent?
• Were data collected independently, from separate
sources?
• Is the data original or re-organized?
• Do the data pertain to a particular geographic locale?
• Were the data collection methods systematic?
• For what purpose were the data originally collected?
• How old are the data? Were they affected by timing?
• Was there bias or special motivation in the collection of
the data?
BASIC DATA ANALYSIS
Data are not generally useful in their raw form. Instead,
they must be analyzed. Data are most often analyzed
using descriptive and/or inferential statistics. Descriptive
statistics search for patterns in the data and look for
relationships to gain insight into the problem. Inferential
statistics attempt to estimate a characteristic of a
population from data gathered from a sample.
Descriptive univariate statistics look for patterns in the
data. They are best presented in graphical form, using
frequency distributions, cumulative distributions, bar
charts, histograms, pie charts, and frequency polygons.
Statistics include the mean, median, and mode, as well as
the range, standard deviation, and variance.
Descriptive bivariate statistics look for relationships in the
data. They are best presented in tables, plots,
scattergrams, and time series graphs. Measures of
association include Lambda, Gamma, and Pearson's r.
Inferential statistics make probabilistic statements (or
inferences) about a whole population based on the results
obtained from a partial sample. Measures of statistical
significance are used to estimate whether two groups
differ from one another, or whether there is a chance that
a relationship observed in a sample also exists in a
population. These measures include Chi Square, Z-scores,
t-tests, and F-tests.
COMMUNICATING THE ANALYSIS
Written Communication:
• Work from an outline--keep separate folders for each
section of the analysis
• Work from goals and deadlines--generate a complete
draft and then fill in the holes
• Get help--with editing of rough drafts; revise for clarity;
incorporate new ideas
• Include a Table of Contents--sections include Executive
Summary; Problem Definition; Decision Criteria;
• Alternatives; Comparison of Alternatives; Conclusions
and Recommendations;
• Use graphics--charts, graphs, flow charts, tables, maps,
pictures, diagrams, drawings, etc.
• Use geographic information systems (GIS)--to generate
maps of data distributions
• Simplicity--use the active voice for verbs
• Accuracy--verify facts; triangulate; check all calculations
• Documentation--note all formulas used and assumptions
made
• Fairness--use references and give credit to your sources
of data
• Neatness--use good grammar, spelling, punctuation,
syntax, etc.
Oral Presentations
• Know your audience
• Keep it short and simple
• Use visual aids and handouts
• Allow time for questions, comments and criticisms
4. VERIFYING, DEFINING, AND DETAILING THE
PROBLEM
PROBLEM DEFINITION
The first thing the policy analyst must do is to ask:
1) Does a problem exist?
2) Can anything be done about it?
3) Does the client have the power?
If the answers are no, then there is no point in doing a
policy analysis.
Pitfalls in public policy problem definition:
1)
2)
3)
4)
5)
accepting the client's definition of the problem
looking only for the simple and obvious
thinking that any and all problems need a public solution
confusing the need for short- versus long-term solutions
confusing the values of individuals versus collectivities
Don't Need Public Policies
Do Need Public Policies
Individual problems
Social problems
Widespread problems
Serious problems
Relative problems
Absolute problems
DEVELOPING PROBLEM STATEMENTS
In
1)
2)
3)
4)
5)
6)
7)
developing problem statements:
think about the problem
delineate the boundaries of the problem
develop a fact base
list goals and objectives for policy solutions
identify the policy envelope (key players)
develop preliminary costs and benefits
review the problem statement
BACK-OF-THE-ENVELOPE CALCULATIONS
One of the first things a policy analyst will do is to try to
get a handle on the possible dimensions of the problem
and potential solutions. The analyst may ask,
1) How many people are we talking about?
2) What is the likely cost per unit of service?
3) How much of the target population can we serve?
4) How much do we have available to spend?
5) Will more staff be needed?
6) Will this impact the budget/tax rate?
7) What are the trends in this area?
8) What will happen if we do nothing?
For example, try to estimate these parameters if half the
children in the state are not receiving the required
immunizations before beginning school. Start with the
number of children in the state up to age 5.
• Which immunizations are required?
• How much does each one cost?
• How many children could realistically be reached?
• How much do we have available to spend?
• Could we get more from the Federal government?
• Will more state staff be needed, or can this be handled
by the private/non-profit sector?
• Will this impact the budget/tax rate?
• What are the trends in this area--is the problem
increasing or decreasing over time?
• What will happen if we do nothing?
The information for doing back-of-the-envelope
calculations can come from
1) reference works
2) experts
3) past studies or quick research
4) informed guesses, extrapolation, rules of thumb,
estimation, parallel reasoning, triangulation, etc.
QUICK DECISION ANALYSIS
Quick decision analysis is a variation on the technique of
making decision trees. Decision trees are ways of
diagramming a problem, when the problem has more than
one solution. It is a tool to help policy analysts see the
logical alternatives to a problem.
POLITICAL ANALYSIS
Policy analysts recognize that politics is important at all
stages of the policy process, including policy analysis.
There are a number of ways to communicate about
potential political influences or factors that may impinge
on the policy analysis. These techniques attempt to allow
political factors to be treated like any other important
considerations in policy analysis.
The analyst may draw up a list of issues involved in
defining the problem, and identify a number of potential
political actors who have taken positions on those issues. A
table can display the likely support or opposition of each
group to each issue.
For example, what are the issues involved in raising the
age at which teens can get a driver's licence to 18? Which
groups are likely to support (+) or oppose (-) problem
definitions that focus on these issues?
Lower teen auto accident, death
& injury rates
Hardship for teens who work or
commute to school
Lower insurance rates for
family cars
M.A.D.D.
+
?
?
Parents/Vo
ters
+
-
+
Insurance
Lobby
+
-
-
Groups
FIRST CUT POLICY ANALYSIS
An issue paper is a study that is conducted in
preparation of making a decision on whether or not to
do a policy analysis. It describes the problem, the
attendant issues, the political groups involved, and
concludes whether or not a policy analysis will be
feasible.
A first cut policy analysis concentrates on
identifying preliminary recommendations. It is a
mini-policy analysis, conducted in a short period of
time, using simple techniques. It forms the basis
for a much more in-depth, complex, and thorough
full-fledged policy analysis.
Researched analysis refers to a more traditional research
project, perhaps conducting a pilot study of several policy
alternatives to generate concrete data on which to base
recommendations.
However, policy analysts rarely have the luxury of the
time and resources needed, nor do they often work for
someone who is far enough removed from the problem to
resist pressures for a quick solution.
5. ESTABLISHING ANALYSIS CRITERIA
WHAT ARE CRITERIA
Every time a policy problem is identified, some
statement of goals is adopted. The goals are what the
adopted policy alternative should accomplish. Goals are
broad, formal, long-term problem-solving achievements
that are desired. An example might be to make sure that
all rivers are safe, clean, and usable.
Goals are translated into objectives. Objectives are more
concrete statements about desired end states, with time
tables, target populations, and resource limits. An
objective might be to make the Colorado River safe for
swimming and fishing.
Criteria are the measurable dimensions of objectives.
Criteria are used to compare how close different proposed
policy alternatives will come to meeting the goals of
solving the problem. Criteria set the rules to follow in
analyzing and comparing different proposed policy
alternatives (solutions).
Sample criteria for improving river water quality might
be:
• effectiveness--how much of an improvement in water
quality will this alternative produce?
• cost--how much will it cost to improve the quality of
the river using this alternative?
• technical--do we have the equipment and know-how to
use this alternative?
• political--is this alternative politically acceptable?
Measures are the actual measurements that will be taken
of each proposed policy alternative. For example,
measures such as the following might be employed:
• effectiveness--how many milligrams of pollutants per
liter of water will this alternative clean up?
• cost--how many dollars will be required to implement
this alternative?
• technical--is the necessary equipment for this
alternative available and are people trained to use it?
• political--what percentage of the voting-age population
will favor this alternative in a statewide poll?
One difficulty in specifying criteria and measures is that
many problem statements have vague, fuzzy, or even
conflicting goals.
This is often necessary in order to get consensus on
taking some action about the problem.
But this complicates the selection of criteria.
If dirty rivers are a problem, and the goal is to have clean
rivers, what is the most important considerations in
choosing between different ways of cleaning up the rivers?
• Is it cost?
• Is it effectiveness?
• Is it equity?
• What do we mean by "clean"?
It is impossible to get rivers 100% clean.
• Do we use Federal, State, or local standards on
admissible levels of toxicity?
• How will we measure the level of cleanliness that
different policy alternatives are likely to produce?
RELIABILITY AND VALIDITY
The criteria and their measures must be unambiguous.
They should be relatively straightforward and simple to
measure. Their application should produce uniform results,
no matter who does the measuring of different
alternatives. And repeated measurements of the same
alternative should produce the same results, again, no
matter who does the measuring.
Criteria and measures should be appropriate to the unit of
analysis.
That is, if the goal of a proposed policy alternative is to
change the investment strategies of cities, the unit of
measurement is cities, not individuals.
Be sure to specify whether the unit of measurement is
households or families, census tracts or neighborhoods,
school children or school districts, etc.
ECONOMIC CRITERIA
Most policy analysis involves at least one economic
criterion. These include impacts on the economy, on
expected public sector revenues, on government spending,
etc.
The most common economic criteria are costs. These may
include:
• borrowing costs--the costs of borrowing funds
• decreases in net worth--decreases in assets and/or
liabilities
• direct costs--directly attributable to the policy alternative
• indirect costs--additional impacts not included in the
goals
• intangible costs--costs that cannot be counted or
quantified
• monetarizable costs--can be expressed in dollars
• one-time fixed costs--new capital expenditures,
equipment, training, etc.
• operations and maintenance costs--ongoing costs of the
alternative
• opportunity costs--other things that could have been
done with the same resources instead
• tangible costs--can be counted and quantified
Costs need to be counted.
One cannot assume that the money was going to be
spent anyway.
Costs should be identified as completely as possible,
eliminating unpleasant surprises down the road.
Another type of cost criterion that is often employed is
marginal cost.
That is, if some good or service is already being produced,
how much more will it cost to produce one additional unit
of output?
The types of costs that are considered in marginal
analysis are:
• fixed costs--these do not vary in the short run, no
matter how many units are produced
• variable costs--these vary directly with the volume of
output of goods or services
• average costs--the total of units of output divided by
the total costs of output
• marginal costs--the costs of producing one additional
unit of output
• sunk costs--these are costs that can be ignored as they
have already been spent in the past
Another type of economic criterion is benefits.
Benefits are the opposite of costs.
Benefits are ways in which the policy actors will be better
off.
Benefits can be measured in many of the same ways as
costs, including:
• direct benefits--directly attributable to the policy
alternative
• increases in net worth--increases in assets and/or
liabilities
• Indirect benefits--additional benefits not included in the
goals
• interest earned--interest that will accrue or be paid
• intangible benefits--benefits that cannot be counted or
quantified
• monetarizable benefits--can be expressed in dollars
• one-time benefits--one-time reduction in the problem
• ongoing benefits--continuing decreases in the problem
• tangible benefits--can be counted and quantified
Benefits are often more difficult to quantify than costs.
One alternative is to use "shadow prices," or the value of
the benefits in a perfectly competitive market, for
example, free recreation facilities, wilderness areas, parks,
etc.
EQUITY CRITERIA
Efficiency and effectiveness are technical and economic
questions, but equity is a public question.
Equity asks about the social allocation of burdens and
benefits.
Equity asks the questions of "who pays?" and "who
benefits?"
A proposed policy alternative may impact equity if it will
change the distribution of burdens and benefits in society.
There is no universally approved optimal or right answer
for how benefits and burdens should be distributed in
society.
That is a continuing area of contention, and essentially a
political decision.
Horizontal equity asks whether burdens and benefits
are being shifted among groups in society which are
relatively equal.
Vertical equity asks whether burdens and benefits are
being shifted among groups in society which are relatively
unequal.
Inter-generational equity asks whether burdens or
benefits are being shifted from one time period to another,
whether younger generations will have to pay more and
receive less than older ones, or vice versa.
Groups are often identified on the basis of:
• residence
• income
• citizenship
• race or ethnicity
• sex
• age
• family status
• home ownership
• educational status
• veteran status
• criminal record
• substance abuse
• health
Problems in assessing equity include:
• how should the population be sub-divided?
• how should groups be defined?
• should historical criteria, the status quo, or desired
states be used?
• what is a burden?
• what is a benefit?
• what is a degree of need?
• what is an ability to pay?
TECHNICAL CRITERIA
Effectiveness is often used as a criterion by which to
judge policy proposals.
Effectiveness is the extent to which the proposed policy
will attain the goals set forth in the problem statement.
For example, if the goal is to decrease the current teenage
driver accident rate,
how much will each policy alternative decrease the rate
below current levels?
Another technical criterion is technical feasibility.
This asks whether the technology exists or is readily
available to implement a proposed alternative.
For example, one proposed policy alternative may be to
install in all cars a breath analyzing device that would not
let a car start if the driver has been drinking.
However, this technology is not widely or cheaply available.
Other technical criteria may question whether the
measurement of criteria can be conducted at the desired
level of reliability and validity.
For example, are there tests that can adequately measure
whether students in bilingual education programs have
the same level of literacy as students in non-bilingual
education programs?
POLITICAL CRITERIA
Many times the client for the policy analysis will hold a
political office. In that case, the policy analyst must often
include political criteria in the assessment of proposed
policy alternatives.
Political viability asks whether or to what extent a
proposed policy alternative will be acceptable to relevant
powerful groups, decision makers, legislators,
administrators, citizens, neighborhoods, unions, or others.
Other ways of assessing political viability include:
• acceptability--is the proposed alternative acceptable to
policy makers, policy targets, the general public, voters,
etc.?
• appropriateness--is the proposed alternative
appropriate to the values of the community, society, the
legislature, etc.?
• Legal--is the proposed alternative legal under current
law, or will statutes have to be amended or enacted?
• responsive--will the proposed alternative meet the real
or perceived needs of the target group, the public, etc.?
ADMINISTRATIVE CRITERIA
Many public policies are implemented by public agencies.
Therefore, administrative operability or administrative ease
are often used as criteria for judging proposed public
policies.
Questions that may be addressed include:
• authority--does the agency have the authority to
implement the proposed policy?
• commitment--does the proposed policy have the
commitment of top managers, field staff, and support staff?
• capacity--does the agency have the resources to
implement the proposed policy, in terms of staff, skills,
money, training, expertise, etc.?
• support--are the facilities, equipment, and other support
available for the proposed policy?
6. IDENTIFYING ALTERNATIVES
GENERATING ALTERNATIVES
Before alternatives can be generated,
1) the problem must be correctly identified, and
2) relevant criteria for judging alternatives must be
specified
At first, the policy analyst can generate a large number of
alternatives, but later reduce them to a manageable size
(between four and seven). Consider alternatives like the
status quo, but also radically different. Consider what
may be possible under different circumstances.
Some criteria that are often used in judging the suitability
of alternatives include:
1) cost--can we afford it; will it be cost-effective?
2) reliability--does it have proven success, or is it subject
to failures?
3) stability--will it still work if conditions change?
4) invulnerability--will it work if one of its component
parts fails?
5) flexibility--can it accomplish more than one thing?
6) riskiness--is there a high chance of all or nothing?
7) communicability--is it easy to understand?
8) merit--does it address the problem?
9) simplicity--is it easy to implement?
10) compatibility--is it congruent with existing norms and
procedures?
11) reversibility--can we return to our prior state if it fails?
12) robustness--can it succeed in different future states?
SOURCES OF ALTERNATIVES
1) The status quo or no action alternative
This means that current efforts will continue at the same
level. It is important to consider how effective any
different alternative will be at changing the status quo.
A baseline analysis:
• identifies clear trade-offs with the present;
• clarifies project objectives;
• underlines whether there is a need for action or not;
• provides linkages to existing efforts;
• identifies problems likely to emerge;
• confirms that no optimum solution exists.
2) experiences of others with similar problems, from
reported research findings, experts, laws, public opinion
polls, new technology, etc.
3) re-define the problem from others' points of view,
including opponents of any change
4) consider the ideal, then apply political, economic, and
other constraints
5) start from generic, to modified, to custom-made
alternatives
6) Quick Surveys by telephone, fax, or e-mail, of peers,
old MPA classmates, people in the policy issue network,
public meetings or hearings, content analysis of editorials,
letters to the editor, etc.
7) Literature review of professional and academic journals,
government reports, collected proceedings from
conferences, on-line services (lexis-nexus, first search,
article first etc.).
8) Case studies of real world experiences: why was the
alternative adopted, what were the circumstances, what
other alternatives were considered and discarded, how did
it eventually work out, what modifications were made after
implementation.
9) Passive collection and classification: keep a folder for
collecting interesting policy solutions on a regular basis,
even if no problem exists at the moment, from clients,
superiors, advocates, media, interest groups, etc. Then
refer to the folder in emergencies.
10) Develop Typologies: identify all the types of persons
likely to be affected by any policy alternative, and what
the probable reaction of each group would be to each
type of alternative suggested; then develop alternatives
that can overcome the objections of most of the groups.
11) Use analogies: 'new' problems are really just like other
'old' problems.
• Personal Analogy: pretend to be someone affected by
this problem, identify with the problem to see what types
of policy alternatives suggest themselves;
• Direct Analogy--look at solutions to other problems to
see if they can be applied to this one;
• Symbolic Analogy--imagine the most aesthetically
satisfying solutions rather than merely technologically
sound ones;
• Fantasy Analogy--image the ideal solution, and try to
preserve as much of it as possible when working
backwards through real world constraints.
12) Brainstorming--can be oral, written, or electronic.
Brainstorming has two phases, first a pure ideageneration phase, where no judgments are made about
any ideas; and second, an evaluation and ranking phase,
to help arrive at concrete solutions.
13) Feasible Manipulation--takes existing policy activities
and develops alternatives based on limited, moderate, or
wide manipulation of the range of possible activities.
14) Modify existing solutions:
• Magnify--do more, more often, larger, longer, exaggerate,
add new components, new resources
• Minify--do less, less often, smaller, shorter, omit, remove,
split apart, under use, fewer resources
• Substitute--switch components, apply in different order,
use different materials, try a different location or different
sponsor
• Combine--blend approaches, combine units, combine
purposes, combine sponsors
• Re-arrange--reverse, invert, change sequence, speed up,
slow down, randomize
• Location--use single or multiple locations, node versus
scattered, temporary versus permanent
• Timing--accelerate, lag, stagger, run concurrently,
shorter span, longer span, time sharing
• Finance--provide, purchase, tax, user fee, subsidy, copay, deductible, voucher, contract out
• Organization--centralized versus decentralized,
mandated versus voluntary, regulated, prohibited,
enforced, inform, implore
• Decision Sites--individual, unit, organization, elected,
appointed, advisory, binding
• Influence Points--users, providers, intermediaries,
beneficiaries, payers
• Risk Management--guarantees, insurance, remedial
correction
PITFALLS
1) Too much reliance on past experience
2) Failure to capture ideas and insights (listen, write them
down, record them)
3) Too early closure on problem definition
4) Sets a policy preference too soon before all the
alternatives are known
5) Criticizing new ideas as they are offered
6) Some alternatives are ruled out too early on
7) Failure to re-consider discarded alternatives if conditions
change
7. ASSESSING POLICY ALTERNATIVES
Which policy alternative should be adopted? In this step
in the policy analysis process, the policy analyst takes
each of the proposed policy alternatives and, one by one,
applies each of the decision criteria to each alternative.
For example, say we have specified that we will be using
the criteria of efficiency, cost, political acceptability, and
equity to make our decision. We have defined what we
mean by each of these and how they will be measured.
For example, efficiency is defined as the amount of
reduction in the teenage driving accident rate, and is
measured by the Department of Motor Vehicles as the
number of accident involving teen drivers divided by the
total number of teens in the state.
We must then look at each proposed policy alternative,
one at a time, and ask,
what would be the efficiency of this alternative?
What would be the cost of this alternative?
What would be the political acceptability of this alternative?
And how will this alternative affect equity?
We then repeat this process for every alternative,
including the no-action alternative.
How do we know what the efficiency of each alternative
will be?
That involves forecasting.
FORECASTING
The criteria that will be important in assessing proposed
policy alternatives determine what needs to be forecast.
For example, if the goal of a proposed policy alternative
is to lower the teenage driving fatality rate, then what
needs to be forecast is the teen driving fatality rate, first
under the assumption that no action is taken, and the
under the assumption that the policy alternative being
considered is implemented.
There are a variety of methods used to make forecasts.
Forecasting methods range from simple stereotyping to
complex statistical formulas.
Intuition may use techniques such as Delphi, scenario
writing, or feasibility assessment. However, it requires
that the participants be quite knowledgeable, and it
needs to be checked for logical consistency.
Theoretical models identify important variables and specify
the nature of the linkages among them. Then the model is
used to predict outcomes when one or more of the
variables are changed. Models are built from information,
experience, expert advice, etc.
Constructing a model helps to get to the key elements of
the situation, and focus on the most important concerns.
It identifies the key factors and the relationships among
them which will likely be impacted by any proposed policy
alternative. It demonstrates the likely consequences of
either the no action alternative, or any other rival
alternative.
Models may be expressed in words, in physical dimensions
(e.g., architectural models), or in numerical form.
Extrapolation uses the past to predict the future, assuming
there are stable patterns. For example, if the population of
Arizona has been growing at 50% every ten years, then a
graph showing past growth can be extended into coming
years to predict future growth.
Extrapolation is useful for conducting a baseline analysis,
showing what is expected if the status quo or no action
alternative is adopted. It is relatively simple and cheap
and can be accurate in many circumstances. Data can
be either raw numbers or a computed rate of change.
Extrapolation requires precise definitions of criteria and
measures, and accurate measurement. It is most often
used when there are linear patterns in the data.
Extrapolation is less useful in the case of new problems,
new issues, or new policy areas, where there is little or
no past data.
The most commonly used form of regression is linear
regression, and the most common type of linear
regression is called ordinary least squares regression.
Linear regression uses the values from an existing data
set consisting of measurements of the values of two
variables, X and Y, to develop a model that is useful for
predicting the value of the dependent variable, Y for
given values of X.
Elements of a Regression Equation
The regression equation is written as Y = a+bX + e
Y= Dependent variable (Y), what is being predicted or
explained
a=Alpha, a constant; equals the value of Y when the
value of X=0
b=Beta, the coefficient of X; the slope of the regression
line; how much Y changes for each one-unit change in X.
X= Independent variable (X), what is predicting or
explaining the value of Y
e=The error term; the error in predicting the value of Y,
given the value of X (it is not displayed in most regression
equations).
For example, say we know what the population has
been in past years for a given area. We can compute
the values of the components of the regression
equation, and, and use them to predict what the area's
population will be in future years.
Year
Actual Population
1910
15000000
1920
18000000
1930
21000000
1940
24000000
1950
27000000
1960
30000000
1970
33000000
1980
36000000
1990
3900000
Predicted Population
2000
42000000
2010
45000000
If the data are not linear, that is, if on a graph the line
that best shows the relationship between the two
variables is not a straight line, then simple linear
regression cannot be used to extrapolate into the future.
Instead, the data must be converted, for example, to
logarithms, or a different sort of regression must be
used.
ECONOMIC ANALYSIS
One of the most widely used economic analysis tools is to
look at the long term costs and the long term benefits of
a proposed policy alternative. From there, the policy
analyst can calculate either the Net Present Value (NPV),
the Cost-Benefit Ratio, or the Internal Rate of Return (IRR)
of each alternative.
To calculate the long term costs and long term benefits of
a proposed policy alternative, the policy analyst must
assemble estimates of the initial or implementation year
costs and benefits of the alternative, and the subsequent
costs and values for each additional year the project will
be in effect.
Say that the city wants to upgrade its data processing
function. It asks for bids showing costs to be submitted
by different vendors. It also estimates the savings that
will occur from each bid (for example, from a reduction in
personnel).
For the first bid, the policy analyst estimates the following
YEAR:
0
1
2
3
4
5
Costs
$15,000
0
0
$1,223
0
0
0
$4,000
$4,000
$4,000
$4,000
$4,000
Benefits
Discounting
The next step is for the policy analyst to decide on a discount
rate. The discount rate assumes that money spent in the future
will not cost as much as money spent today.
Similarly, money gained in the future will not be worth as much
as money gained today. This is based on the human preference
for wanting to put off costs (or payments) as long as possible,
and wanting to receive benefits (or pay) as soon as possible.
The discount rate is usually obtained from economists,
from agency policy, or from the nature of the project
being considered (i.e., whether a large infrastructure
project, a revenue-bond based project, or a general
obligation bond based project). Another source is the
discount rate charged by the Federal Bank, or the interest
rate paid on government bonds.
At times, the choice of which discount rate to use has
been highly politicized. Because many government projects
have high initial costs but a long stream of benefits, a low
discount rate will make a project look more favorable, and
a high discount rate will make a project look less favorable.
The same discount rate is generally applied to both the
project costs and the project benefits. If inflation is going
to be factored in, it should be applied to both the costs
and benefits separately, before the discount factor is
applied.
To calculate the discounted costs, multiply each year's
costs by that year's discount factor (the discount rate
factor be obtained from a table of discount rates):
YEAR:
0
1
2
3
4
5
Total
Costs
$15,000
0
0
$1,223
0
0
$16,223
Discount rate
4%
4%
4%
4%
4%
4%
Discount factor
1.0
.9615
.9246
.8890
.8548
.8219
$15,000
0
0
$1087
0
0
Discounted
costs
$16,087
To calculate the discounted benefits, multiply each year's
benefits by that year's discount factor (the discount rate
factor be obtained from a table of discount rates):
YEAR:
0
1
2
3
4
5
Total
Benefits
0
$4,000
$4,000
$4,000
$4,000
$4,000
$20,000
Discount rate
4%
4%
4%
4%
4%
4%
Discount factor
1.0
.9615
.9246
.8890
.8548
.8219
0
$3846
$3698
$3556
$3419
$3288
Discounted
benefits
$17,807
Net Present Value
The Net Present Value is the value of the project if all the
costs were paid today and all the benefits were gained
today. To find NPV, subtract discounted costs from
discounted benefits: Discounted Benefits $17,807 Discounted Costs $16,087 = $1,720
The Net Present Value of each policy alternative must be
calculated separately, and then it can be compared to the
NPV of each other policy alternative, to find the one with
the highest NPV.
Cost-Benefit Ratios
The costs and the benefits of any policy alternative
can be compared in a number of ways. Cost-benefit
ratios are obtained by dividing discounted benefits by
discounted costs:
Discounted benefits =$17,807
Discounted costs =$16,087
Benefit/Cost ratio =1.1
Note that the highest benefit-cost ratio may not have the
highest NPV. These are two different types of analysis. The
most efficient projects have the highest benefits-to-costs
ratio, but many policy analysts prefer to maximize NPV. In
any case, NPV should be a positive number, and the
benefit-cost ratio should be greater than 1.0
Internal Rate of Return
The internal rate of return is an expression of the discount
rate at which discounted benefits would equal discounted
costs. For the example above, at an 8% discount rate,
discounted benefits would equal $15,971 and discounted
costs would also equal $15,971.
If the calculated IRR is greater than the discount rate being used
for the project, then that is an indication that the project should be
carried out. Generally, IRR is not comparable to either NPV or
the benefits-to-costs ratio. The IRR from one project, however,
can be directly compared to the IRR from an alternative project.
Sensitivity Analysis
Often there is no clearly superior potential policy alternative,
but several that seem equally acceptable. One alternative
may be better on the criterion of efficiency, while another is
better on costs, and a third on political acceptability.
A policy analyst will usually try to see how sensitive the
analysis is to changes in assumptions. Things that the policy
analyst will test include:
1) the length of the project (how long will benefits continue)
2) the discount rate
3) the value placed on various quantities (costs, benefits,
probabilities, etc.)
For example, a city wants to replace old garbage trucks
with newer models, and it assumes the new trucks will
last 20 years. What if the benefits only last 10 years? Or
if the annual maintenance costs are 50% higher than
what was budgeted?
Or does a project still have a positive NPV if the discount
rate is raised from 4% to 6%? Is the IRR still greater than
the discount rate? Is the benefit-to-cost ratio still greater
than 1.0 ?
In another example, the city assumes that
building a new parking garage will raise an
additional $2,000 per parking space per year in
sales taxes, as well as the revenues from parking.
What if only $1,000 is raised?
Or say a university wants to get more students to
park on campus instead of on nearby
neighborhood streets. It thinks that if it reduces
the parking permit fee by $10, then 25% more
students will buy one and park on campus. What
if only 5% more students buy one?
A city has vacant land that it can sell, lease, or keep. The
city wants to sell. It assumes that someone will buy the
land and develop it, increasing the city's property tax
revenues. But what if the office building remains mostly
vacant? Would this change the city's decision on selling?
What are the probabilities of the different outcomes?
What if the probabilities for the favored outcome
decrease?
Another type of sensitivity analysis is to identify the
break-even point. This can vary according to:
the length of the project (how many years are needed
to break even?)
the discount rate (how low before benefits equal costs?)
the value of other quantities (e.g., amount of extra
parking permits sold?)
Contingency analysis identifies what will happen if one of
the basic assumptions about the project is altered. For
example, what if there are large cost over-runs? What if
people do not behave as predicted (e.g., buy more
parking permits?)
A fortiori analysis examines the likelihood that any one
factor will take on a value that makes the project
infeasible. For example, what if the project takes two
years to complete instead of one? What if interest rates
rise dramatically? What if new regulations are adopted
that make the project technically impossible?
To perform sensitivity analysis,
1) list all relevant considerations;
2) establish the range of values that each variable can
take, from low to high;
3) holding all other values constant, vary the value of one
variable at a time;
4) test sensitive values to find the break-even,
contingency, and a fortiori points.
RISK ANALYSIS
Some decision-makers are risk averse. They want to
minimize any possible losses, rather than to pursue the
(riskier) maximum possible gains. They will want to go for
the sure thing (the alternative with the highest probability-in this case, do nothing--especially if limits their possible
losses (for this alternative, the worst case scenario is to
break even at 0).
Another way to begin the appreciate the different
possible outcomes of different policy alternatives is to
use quick decision analysis. This is a way to visually
represent a small number of alternatives and their
consequences.
Quick decision analysis identifies key issues, and helps
the policy analyst to decide what information is necessary
to assess each possible alternative. It helps to structure
thinking about the probability or likelihood that certain
outcomes will occur.
It also helps the policy analysts or decision-makers to
reveal their attitudes about risk and uncertainty. And it
alerts the policy analyst to the possible political
ramifications of predicted outcomes.
The steps in constructing a quick decision analysis are:
1) identify the dimensions of the analysis (problem,
alternatives, outcomes)
2) construct a diagram
3) forecast the likely outcome for each alternative
4) assess how likely each outcome is in terms of probability
5) calculate the expected value of each alternative
For example, say a city wants to know if it should offer
tax abatement to encourage economic development. The
two alternatives are, simply, to do nothing (not offer the
abasement), and to do something (offer the abatement).
Each possible policy alternative has two possible
outcomes: new economic development occurs, or new
economic development does not occur.
Do Nothing
Likely Outcomes
Offer Abatement
Developmen
t
Occurs
No new
development
Developmen
t
Occurs
No new
development
+$100 m
0
+$900 m
0
Cost of offering the abatement
0
0
-$600 m
-$200 m
Cost of additional city services
-$25 m
0
-$100 m
0
Net
+$75 m
0
+$200 m
-$200 m
p=0.3
p=0.7
p=0.6
p=0.4
Change in
Revenues
Property
Tax
Probability of this outcome
Expected Value of this alternative
0.3 x $75 m =
$22.5 m
0.7 x $0 m =
$0 m
+$22.5 m
0.6 x $200 m =
$120 m
0.4 x-$200 m =
-$80 m
+$40 m
However, it is important to question quick decision analysis.
-What studies were used to estimate outcomes and
probabilities?
-Were discount rate applied?
-What time frame was considered?
-What were the opportunity costs (how could the money be
spent elsewhere?)
-How sensitive are these figures to changes in the economy?
-At what probability would the expected value of the two
alternatives be equal?
If there is a great deal of uncertainty about the analysis,
there are a number of strategies:
1) delay until more is known
2) map out all uncertainties and the information that is
needed
3) collect more data to reduce uncertainty
4) estimate a wide range of possible values for those
which are uncertain
5) develop alternatives under a wide range of possible
conditions
6) build in more flexibility
7) build in more backup
8) compromise to an acceptable alternative, even if it is
not the optimal one
9) choose a strategy that minimizes the maximum
possible losses
10) conduct in-depth research to provide the information
needed
POLITICAL ANALYSIS
Often one criteria for assessing proposed policy
alternatives is political acceptability to the client. A political
feasibility analysis can help the policy analyst identify the
important elements to be considered for each proposed
policy.
1) Actors--people, groups, and organizations
2) Beliefs and motivations--which are negotiable, and
which are non-negotiable?
3) Resources--power, influence, money, staff, public
opinion, etc.
4) Effectiveness--leadership, ability to use resources
effectively
5) Sites--agendas, windows of opportunity, sequencing of
decisions, etc.
A political feasibility analysis takes each proposed policy
alternative and examines how well it will hold up in the
current political reality. Which actors will favor or oppose it,
and why (beliefs and motivations)? What resources do
they have, and how effective will they be at supporting or
opposing the policy? Where is the debate on the policy to
occur, and which actors or groups will be most powerful
there? Does any group have veto power?
IMPLEMENTATION ANALYSIS
Even after a policy is adopted, there still may be
resistance to its implementation. In conducting an
implementation analysis, the policy analyst looks at factors
that will make the alternative easier or more difficult to
implement, such as:
1) are there few or many actors required to implement
this alternative?
2) will there be one or multiple implementation settings?
3) will there be a single or multiple sets of instructions?
4) what is the degree of consensus around this
alternative?
5) what magnitude of change will be required?
6) how much of the political conflict from the adoption
stage will be displaced into the implementation stage?
7) can game theory be used to model the possible
outcomes?
8) are the necessary resources present, such as
administrative will, competence, budget, skills, authority,
personnel, etc.
8. DECISION RULES and DISPLAYING
ALTERNATIVES
DECISION RULES
Policy assessment techniques do not determine which
policy should be adopted. Policy analysis presents the
benefits and drawbacks of each alternative, but in
addition one or more decision rules are needed in order
to determine which policy is the "best."
There are many problems in trying to determine which
policy to adopt.
1) Many problems in the public sector have multiple facets.
Policies are designed with multiple goals or objectives.
There may be no dominant objective, or several objectives
may be in conflict.
2) there are multiple criteria to take into account--technical,
economic, political, and administrative--but who decides
which is the most important?
3) not all important considerations can be converted into
comparable units, such as dollar values.
4) which is the proper criterion to use, greatest net present
value? greatest internal rate of return? largest benefit-cost
ratio?
5) there is often a lack of agreement beforehand on
decision rules, or which rules to apply
6) even if each decision criterion is optimized separately,
there may still be a sub-optimal choice at the end (a
camel is a horse designed by a committee).
The policy analyst is often faced with trying to present
multiple policy alternatives which have been assessed in
terms of multiple decision criteria. There are various
methods which can be used to display this information in
a way that facilitates decision-making.
PAIRED COMPARISONS
If there are 5 different ways of collecting the city's
garbage, then the analyst can compare method 1 with
method 2 and determine which is superior. The better of
the two is then compared with method 3, and again one of
them is determined to be superior. The winner of each
contest is then compared with another of the remaining
alternatives until all have been evaluated and the winner
of the last contest is the overall winner.
This method presents a simple, step-by-step
comparison that is relatively easy to follow, if somewhat
tedious. The problem with this method is that the
outcome may be influenced by the order in which the
alternatives were considered.
SATISFICING
The analyst presents all the alternatives that meet the
minimum threshold levels on all criteria. The minimum
threshold is then increased on each criterion, and those
alternatives which do not meet the new levels are
dropped. This process continues until only one alternative
is left.
For example, say there are 6 alternatives to solve the
parking problems on campus and three criteria: net
revenues, number of permits sold, and student
satisfaction. The minimum levels are $50,000 in
revenues, 10% increase in permits sold, and at least
50% students satisfaction. Two alternatives are dropped
because they do not meet one or more of these
minimum levels.
The levels are then raised to $75,000 in revenues, 20%
increase in parking, and 65% student satisfaction. Another
two alternatives drop out. Finally, raising the minimums to
$100,000 in revenues, 25% increase in parking, and 75%
student satisfaction eliminates one alternative and the only
remaining alternative that meets all these minimums is the
winner.
This method assures that the minimum "needs" for
the policy will be met, and, in addition, offers the
prospect of meeting higher than minimum needs
("desires"). The problem is that it changes the definition
of acceptable after the analysis has been completed.
GRADING METHOD
The consequences of each alternative on each
criterion are considered. A grade of "Pass" or "Fail" is
assigned to each alternative on each criterion. Only
those alternatives which "Pass" on all criteria are
retained; those which have any "Fails" are rejected. The
retained criteria are then compared further. This is
comparable to the satisficing method discussed above.
LEXICOGRAPHIC ORDERING
The analyst lists all those alternatives that ranked
most highly on the one most important criterion.
These alternatives are all considered to be roughly
equal. The alternatives are then compared on the
second most important criterion. Those alternatives
which rank most highly are retained, and they are
then compared on the third most important criterion.
This process is repeated until the alternative that is
most highly ranked on all the criteria is found.
For example, say there are 5 alternatives for economic
development for the downtown area, and there are
three criteria: increased revenues, increased jobs, and
citizen satisfaction. Three alternatives rank highly on
the most important criterion of increased jobs. These
three are then compared on the second criterion,
citizen satisfaction. Only two are highly ranked. These
last two are compared on the third criterion of
increased revenues, and the most highly ranked is the
winner.
This is a rather straightforward process of comparison.
However, it assumes there is agreement on which is the
first most important criterion, the second most important,
and so on. For situations with multiple objectives (criteria)
or multiple decision makers, this may be difficult.
NON-DOMINATED ALTERNATIVES
All alternatives are measured on all criteria,
and their rank order on each criterion is
displayed. Every alternative that is ranked the
most highly on any criterion is retained; any
alternative that is not ranked #1 on at least one
criterion is discarded. The result is only "nondominated" alternatives, that is, alternatives
that are clearly superior on at least one criterion.
These alternatives can then be compared
further by another method.
Criterion 1
Criterion 2
Criterion 3
Alternative A
Rank #4
Rank #5
Rank #6
Alternative B
Rank #2
Rank #1
Rank #3
Alternative C
Rank #3
Rank #4
Rank #1
Alternative D
Rank #1
Rank #3
Rank #2
Alternative E
Rank #6
Rank #6
Rank #5
Alternative F
Rank #5
Rank #2
Rank #4
In this example, only Alternatives B, C, and D
were ranked #1 in at least one category.
Alternative B is not dominated by any other
alternative on Criterion 2; Alternative C is not
dominated by any other alternative on Criterion 3;
and Alternative D is not dominated by any other
alternative on Criterion 1. Alternatives B, C, and D
must now be considered further.
Say there are five designs for a new community
building, and the designs are ranked in terms of
their suitability for athletics, as well as their
suitability for arts and crafts. Only one design
will be ranked #1 in each category. These two
designs are the only non-dominated alternatives.
Since they are each ranked #1 on one category,
and they are each ranked #2 in the other
category, they are considered equal.
A third criterion may be added as a "tiebreaker." For example, the two alternatives
might be ranked in terms of their suitability for
meetings. The higher ranking of the two will be
the winner.
This method is straightforward and relatively
easy to follow. However, it brings in additional
criteria (tie-breakers) after the analysis has
been completed and some alternatives have
been eliminated.
EQUIVALENT ALTERNATIVES
When at least one of the criteria can be
measured in quantitative units, for example,
dollars, this method can be used to compare
two alternatives. It involves converting other
units of measurement to dollars as well and
then comparing the two alternatives again.
For example, you have been offered two
possible jobs. You assess these jobs in terms of
five criteria: salary (measured in dollars), climate
(measured in days of sunshine per year),
commute time (measured in minutes), nature of
job (interesting or uninteresting), and potential
for advancement (measured as good or poor).
Salary
Climate
Commute
Nature
Advance
Job A
$36,000
240
20
I
G
Job B
$42,000
200
30
U
P
Since the two jobs are both measured on at least
one criterion--salary--in a quantitative form
(dollars), the next step is to convert the value of
the other criteria to dollars as well.
For example, how much of the $42,000 salary
would you be willing to give up to have more
days of sunshine per year? Say that the extra 40
days of sunshine per year are worth $1,600 (you
would be willing to take Job B at $1,600 less in
pay if it had 40 more days of sunshine). Show
these calculations in a revised chart.
Salary
Climate
Commute
Nature
Advance
Job A
$36,000
240
30
I
G
Job B
$40,400
240
20
U
P
Now, how much of the remaining salary of Job B
would you be willing to give up to have a shorter
commute? Say that cutting commute time by 10
minutes per day is worth $1,000 (you would be
willing to take Job B at $1,000 less in pay if it had
10 minutes less of commuting). Show these
calculations in a revised chart.
Salary
Climate
Commute
Nature
Advance
Job A
$36,000
240
20
I
G
Job B
$39,400
240
20
U
P
Now, how much of the remaining salary of Job B
would you be willing to give up to have a more
interesting job? Say that it is worth $1,400 (you
would be willing to take Job B at $1,400 less in
pay if it was more interesting). Show these
calculations in a revised chart.
Salary
Climate
Commute
Nature
Advance
Job A
$36,000
240
20
I
G
Job B
$38,000
240
20
I
P
Now, how much of the remaining salary of Job
B would you be willing to give up to have
greater potential for advancement? Say that it
is worth $2,500 (you would be willing to take
Job B at $2,500 less if there was greater
advancement potential). Show these
calculations in a revised chart.
Salary
Climate
Commute
Nature
Advance
Job A
$36,000
240
20
I
G
Job B
$35,500
240
20
I
G
Now the two jobs are "equalized" on all criteria
except the quantitative on (salary). Since the two
jobs are equal except for salary, then the job with
the higher salary (Job A) is the winner.
This method is rather complex and has many
steps. It is also very subjective and best suited to
situations where there are only individual decision
makers rather than groups. It does help the
decision maker to clarify their personal
preferences, however, and may provide useful
insight into the most important facets of the
problem.
WEIGHTED DECISION CRITERIA
When there are multiple decision criteria, a
ranking or weighting system can be developed to
reflect the relative importance of each criterion in
the decision making process. These are sometimes
referred to as "importance weights."
Say that the state wants to adopt a plan to give
grants to the elderly to defray their utility expenses.
The decision criteria are number of elderly reached,
speed at delivering the grants, and controls on
fraud. Each alternative is measured on each
criterion and a raw score is assigned.
Elderly Reached: 5 = 80% - 100%; 4 = 60% - 80%;
3 = 40% - 60%; 2 = 20% - 40%; 1 = 0% - 20%
Speed of Delivery: 5 = 0-5 days; 4 = 6-10 days;
3 = 11-15 days; 2 = 16-20 days; 1 = 21+ days
Curbs on Fraud: 5 = Very strong; 4 = Strong;
3 = Moderate; 2 = Weak; 1 = Very Weak
RAW SCORES
Elderly Reached
Speed of Delivery
Fraud Curbs
Alternative A
5
4
3
Alternative B
4
4
4
Alternative C
4
3
3
Alternative D
2
3
5
The decision criteria have been weighted by the
decision makers, so that the number of elderly
reached is worth 50% of the decision, the speed
at delivering grants is worth 30%, and controls on
fraud is worth 20%.
The raw score for each alternative on each
criterion is then multiplied by the importance
weight for each criterion. The result is a weighted
score. The weighted scores for each alternative on
all the criteria are then added for the total
weighted score.
WEIGHTED SCORES
Elderly Reached
(50%)
Speed of Delivery
(30%)
Fraud Curbs
(20%)
Total
Score
Alternative A
5 x .5 = 2.5
4 x .3 = 1.2
3 x .2 = 0.6
4.3
Alternative B
4 x .5 = 2.0
4 x .3 = 1.2
4 x .2 = 0.8
4.0
Alternative C
4 x .5 = 2.0
3 x .3 = 0.9
3 x .2 = 0.6
3.5
Alternative D
2 x .5 = 1.0
3 x .3 = 0.9
5 x .2 = 1.0
2.9
The alternative which scores the highest on the
number of elderly reached (Alternative A) is the
alternative with the highest total score. An
alternative which has very good controls on
fraud, but does not do as well in the areas of
reaching the elderly or speed, has the lowest
total score.
The advantages of this method are that arriving
at the superior alternative is straightforward and
relatively easy. Rather than considering
alternatives one at a time, or considering criteria
one at a time, this method considers all
alternatives on all criteria simultaneously. The
disadvantages are that many criteria are not
measurable in quantitative form and there may
be disagreement over the importance weights of
the criteria.
GROLLER SCORECARD
The matrix (or "scorecard") is a way of
displaying the impacts of each alternative in
terms of each criterion in "natural" units of
measure, such as reduction in fatality rate,
political acceptability, administrative ease,
effectiveness, impact on need for additional
landfills, etc. The scorecard indicates the extent to
which each alternative attains the objectives
specified by each criterion.
This approach the viewer(s) to grasp a complex
analysis by summarizing it in a chart. It can help a
group to come to a decision without the need to
impose quantitative measurement or assign
importance weights to criteria.
REDUCE
PERMIT
PRICE
STRONGER PARKING
ENFORCEMENT
EDUCATIONAL
CAMPAIGN
More students buy permits; no net
loss or gain
Higher costs in enforcement
Moderate costs for materials
SOLVES
PARKING
PROBLEM
Cuts illegal parking by 25%
Cuts illegal parking by 40%
Cuts illegal parking by 10%
ACCEPTABILITY
TO STUDENTS
Highly acceptable
Highly unacceptable
Moderately acceptable
ACCEPTABILITY
TO COMMUNITY
Moderately acceptable
Highly acceptable
Unacceptable
COST
TO
UNIVERSITY
THE
Best option
Worst option
9. IMPLEMENTATION, MONITORING, &
EVALUATION
IMPLEMENTATION ANALYSIS
The full policy process is often described by the
following steps:
1) problem definition
2) alternative generation
3) analysis of alternatives
4) policy adoption
5) policy implementation
6) policy evaluation
While this course has focused on the first three
steps, the last three steps are equally important.
A thorough policy analysis will include some
consideration of policy implementation,
monitoring, and evaluation.
The policy analyst can sketch out an
implementation plan for the most highly ranked
alternative(s) that considers:
1) relevant actors and their interests
2) required resources and who might provide
them
3) facilitators and barriers likely to be
encountered
4) reasonable time frame
Implementation analysis might involve writing a
"best-case" scenario and a "worst-case" scenario
for each policy alternative, as well as the "most
likely" outcome. The idea is to think
systematically through the implementation
process, identify potential problems, and develop
actions that can be taken to either avert
catastrophes or reduce losses.
POLICY MONITORING
Policy maintenance refers to keeping the policy or
program going after it is adopted. Policy monitoring
refers to the process of detecting how the policy is
doing.
To monitor a policy, some data about the policy
must be obtained. A good implementation plan
will suggest some ways in which ongoing data
about the policy can be generated in the regular
course of policy maintenance, for example, from
records, documents, feedback from program
clients, diary entries of staff, ratings by peers,
tests, observation, and physical evidence.
POLICY EVALUATION
Policy evaluation is the last step in the policy
process. It may ask deep and wide-ranging
questions, such as:
1) was the problem correctly identified, or was
the correct problem identified?
2) were any important aspects overlooked?
3) were any important data left out of the analysis?
did this influence the analysis?
4) were recommendations properly implemented?
5) is the policy having the desired effect?
6) are there any needs for modification, change,
or re-design? what should be done differently
next time?
When policies fail to have the intended effect, it is
usually due to one of two types of failure: theory
failure, or program failure.
A theory failure occurs when the policy was
implemented as intended, but failed to have the
desired effect. This may occur when, for example,
a school adopts school uniforms to curb violence
in the school, but the violence remains at the
same level.
The policy was implemented (uniforms were
adopted) but the expected change did not occur.
The theory that violence occurs due to style of
dress is wrong. There must be some other cause of
school violence, which would require a different
policy to address.
An implementation failure occurs when the policy
is not implemented as intended. For example, the
school may adopt a uniform policy, but the
majority of the students ignore it. The level of
violence in the school does not change. We still do
not know whether adopting school uniforms would
lower the level of violence in the schools; we only
know that uniforms were not adopted.
FORMATIVE EVALUATION
If adequate monitoring processes are in effect, it
should be fairly easy to detect whether a policy
has been implemented as intended. This type of
policy monitoring has been referred to as
formative evaluation. Formative evaluation
documents and analyzes how a policy is
implemented, with the objective of making
improvements as the implementation process
unfolds.
SUMMATIVE EVALUATION
Summative evaluation is conducted after a
program has been fully implemented. It looks at
whether the program is meeting its objectives, and
why or why not.
Evaluations may be unpopular for many reasons:
1) the program is controversial;
2) there are strong political interests in seeing it
succeed or fail;
3) there are difficulties in measuring program
accomplishments;
4) those involved may be uncooperative;
5) program effects may be influenced by outside
developments.
To decide whether an evaluation will be helpful,
the answer to the following questions should be
"yes":
1) will the evaluation be accepted by politicians,
administrators, and/or participants?
2) has an evaluator been involved from the
beginning?
3) are there measurable objectives?
4) are data available?
5) are multiple evaluation methods plausible?
6) has the program remained stable over time?
7) can program staff become involved in the
evaluation?
8) will the findings be made widely available?
EVALUATION DESIGN
Policy evaluation applies accepted social science
research methods to public programs. The same
research designs used in laboratory experiments
are not always practicable in the field, but the
same principles can guide the planning and
execution of policy evaluation.
Before-and-After Evaluation: a policy is evaluated
for the changes it has produced since its
implementation; the situation is controlled to
exclude other possible influences on the outcome.
With-and-Without Evaluation: a policy is evaluated
for producing changes in the target population,
compared to another population without the policy.
After-Only Evaluation: the extent to which the
policy goals were achieved, compared to the state
of affairs before the policy was implemented; but
the situation is not controlled to exclude other
possible influences on the outcome.
Time-Series Evaluation: the changes produced by
the policy, tracked over a long time period.