Transcript Barriers to a More Even-Handed Treatment of Uncertainties

BARRIERS TO A MORE EVEN-HANDED TREATMENT OF
UNCERTAINTIES IN PROJECTED ECONOMIC COSTS AND
HEALTH BENEFITS OF ENVIRONMENTAL HEALTH
REGULATIONS
DALE HATTIS
CLARK UNIVERSITY
Approach to the Problem
• Adam has introduced the issues and prior
observations of imbalance in the quantification of
uncertainties in costs vs benefits.
• I chose to first build additional observations from
the most recent available EPA RIAs.
• This gave me a basis to interview participants in
creating these RIAs and discuss ideas about why
they were crafted as the were.
• Today I will first build some historical context for
“impact analysis” before moving to the
observations from recent RIAs.
Historical Background of “Impact Analysis”-• Which “Facts” need to be assembled for value choices
to be made?
• Corps of Engineers tradition of traditional cost benefit
analysis (promoting societal aggregate net benefit as
the goal)
• Environmental movement of Late 60s-early 70s aimed
to right distributional wrongs
• NEPA (1969)
– Required analysis of environmental impacts not previously
assessed.
– Allowed enforcement by citizen suits.
– Did not change the underlying decision-making mandate
of affected agencies.
Political Opponents of
Environmentalists Borrow a Page
• 1974—Ford “Whip Inflation Now (WIN)” program institutes
requirement for “Inflation Impact Statements”
• Requirement is based on an Executive Order, not a
congressional law, so enforcement is not via the courts but
solely by administrative appeal to the Office of
Management and Budget
• For both kinds of impact statements, advocates were not
powerful enough to change underlying legislative
mandates, but sought impact statements as a way of less
formally changing the mix of considerations influencing
administrative choices.
• Statement initially required to say whether the costs are
“justified” by the benefits, not the stronger statement
(later used under Regan) that benefits “outweigh” costs.
Development of Mandates for
Uncertainty Analysis
• Risk analysis emerges as a discipline beginning in the midlate 1970s and early 1980s.
• Agencies develop “conservative” procedures for estimating
risks and associated risk reduction benefits—simplifying
assumptions thought likely to overstate risks and therefore
lead to protective standards—placing the burden of
uncertainty on protected parties.
• Such “conservative” estimation procedures, however did
not correspond to the “risk neutral” arithmetic mean
“expected value” estimates called for under traditional cost
benefit criteria.
• (They may also not have been as health-protective as the
participants may have imagined or sold to managers).
Take-away from this background
• This discussion sets the policy context for
understanding the role of impact analyses in decision
making. The impact analysis advocates did not have
the power to change the basic legislative mandates,
but did have enough influence to at least get their
considerations analyzed and disclosed. This helps to
understand why the mandated analyses were never
placed at the very center of the decision-making
processes and made highly rigorous, but tended to be
conducted more with a goal to satisfice (be sufficient
to appease) potential opponents after important
policy choices had already been made
Bush 1991 “Regulatory Program of the
United States”
• Unfortunately, risk-assessment practices continue to
rely on conservative models and assumptions that
effectively intermingle important policy judgments
within the scientific assessment of risk. Policymakers
must make decisions based on risk assessments in
which scientific findings cannot be readily
differentiated from embedded policy judgments. This
policy environment makes it difficult to discern serious
hazards from trivial ones, and distorts the ordering of
the Government’s regulatory priorities. In some cases,
the distortion of priorities may actually increase health
and safety risks.
OMB Circular A-4 (2003)
• “When benefit and cost estimates are uncertain
(for more on this see "Treatment of Uncertainty"
below), you should report benefit and cost
estimates (including benefits of risk reductions)
that reflect the full probability distribution of
potential consequences. Where possible, present
probability distributions of benefits and costs and
include the upper and lower bound estimates as
complements to central tendency and other
estimates.”
Obama (2011)
• “The goal of the agency’s uncertainty analysis
is to present both a central “best estimate,”
which reflects the expected value of the
benefits and costs of the rule, as well as a
description of the ranges of plausible values
for benefits, costs, and net benefits, which
informs decision-makers and the public of the
degree of uncertainty associated with the
regulatory decision.”
New Observations
Example 2
Example 3
Example 4
Rule
Date
Office
Goal Statement
Uncertainties in
Benefits
Analyzed?
Uncertainties in Costs
Analyzed?
USEPA.
National
emission
standards for
hazardous air
pollutants:
Mercury
emissions from
mercury cell
chlor-alkali
plants--signed
draft federal
register notice,
2011(45)
2011
Air
National emission
standards for
hazardous air
pollutants
(NESHAP) for
mercury emissions
from mercury cell
chlor-alkali plants in
response to a petition
for reconsideration
filed by the Natural
Resources Defense
Council (NRDC).
This action proposes
two options. The
first option would
require the
elimination of
mercury emissions
and thus encourage
the conversion to
non-mercury
technology. The
second option would
require the measures
proposed in 2008 but
would still allow
chlor-alkali plants to
operate..
Yes—derived from
particle related
mortality reduction cobenefits with
monetization. The
primary mercuryrelated benefits are not
estimated “due to data,
time, and resource
limitations”. “We
estimate the monetized
energy co-benefits of
the non-mercury
technology option to
be $22 million to $43
million (2007$, 3
percent discount rate)
in the implementation
year (2013). The
monetized co-benefits
of the regulatory action
at a 7 percent discount
rate are $14 million to
$33 million (2007$).
Higher or lower cobenefits estimates are
plausible using other
assumptions.
Cost benefit summary table
presents only a single point
estimate of costs. However there
is appreciable uncertainty
analysis in costs traceable to
challenges to the cost analysis
from both environmental groups
petitioning for the change and
industry. “In this revised
analysis, we recognize that there
are significant uncertainties in
estimating these costs, and
consider ranges of the potential
costs (and savings) associated
with each cost element. For each
element, we do select a “best
estimate” to allow the estimation
of capital and annual costs of
conversion for each
facility... .Using more
conservative assumptions, our
best estimate is that the average
annual costs would be between
$800,000 and $7 million per year
per plant.
Summary Results from New Observations
• Overall 9 of the 14 new cases had clearly greater depth of analysis
of uncertainties in benefits than in costs—in accord with the prior
findings.
• Of the 4 cases where we judge the treatment of uncertainties in
costs and benefits to be reasonably comparable, three represent
cases where there was little or no quantitative treatment of
uncertainties at all—impact estimates, when given, were single
points, often derived using “conservative” thought likely to
overstate health/biological benefits.
• In the remaining case of comparable uncertainty treatment, the
treatment of cost uncertainties seems to reflect effects of
controversy arising from challenges by both environmentalists and
industry to a baseline estimate originally derived by/for the agency.
• A final case of arguably greater analysis of cost uncertainty was the
adoption of general drinking water standards for use in air planes—
where there was some analysis of uncertainty in costs but
quantitative estimation of benefits (and their uncertainty) was not
attempted
Results of Interviews
• Perceptions of a key EPA analyst
– engineering cost estimates don’t come naturally
attached to uncertainty statement, unlike the
epidemiologically-based confidence limits for the
mortality estimates and other effects associated with
fine particle exposures.
– overall the most important source of uncertainty is
probably the changing baseline where effects both
positive and negative occur far into the future (e.g.
2020 for the NAAQS particulate standard).
Another Interview
• Perceptions of 2001-6 OMB/OIRA head John
Graham:
– EPA economists, while willing to use statistical
confidence ranges as measures of uncertainty, were
very reluctant to go beyond this to potentially more
extensive quantitative characterizations of the
uncertainties in analyses of costs and benefits.
– EPA economists were comfortable with sensitivity
analyses, and happy to illustrate results of different
scenarios, but did not recognize the appropriateness
of non-statistical analyses and necessarily subjective
weightings of relative likelihood of different scenarios.
– Economists’ position on this was reinforced by the
general unwillingness of EPA lawyers to sanction the
explicit incorporation of numerical uncertainties in
official documents such as RIAs.
Speculation on Lawyers’ Reasons
• Adam’s perception: I think laypeople (lawyers, judges) think
they understand economics, but are more willing to admit
they aren't scientists-- and hence they see an error bar for
science and see it as reasonable, whereas an error bar for a
cost figure connotes to them "go back and come up with
the right answer; you are being lazy to stop with the
range."
• Additional possibility: lawyers want to preserve the
traditional legal doctrine that courts should show
“deference” to the judgments of expert executive agencies
within the agencies’ field of expertise. Stating uncertainties
in scientifically-quantified health benefits may be judged to
disturb this tradition less than more accessible conclusions
with respect to costs.
Interpretation
• It is clear that the EPA analysts are very aware and cite
in their documents the record of ex-ante vs ex post
predictions of costs, and the empirical data on learning
curve effects from the innovation/business literature.
Learning curve effects resulting in reduced cost
estimates have been incorporated into regulatory
impact analyses for some mobile source emission
rules. However the EPA analysis team seems to have
been generally reluctant to draw on non-case-specific
information to quantify Bayesian “prior” expectations
for the bias and uncertainty in the engineering
estimates of compliance costs that they receive.
Positivism vs Critical Realism—Role of
Theory/”Metaphysics”
• Positivism—Theory is to be avoided wherever
possible in favor of simple
analysis/summarization of the data. (Role of
statistical analysis is to describe/summarize
data.)
• Critical Realism—building a picture of
underlying causal relationships—some of
which may not be directly observable--is the
essence of the goal for scientific research.
Frequentist vs Bayesian Concepts of Probability
• Frequentist—Probability statements can only be
made by sampling from some data set; there is
no information relevant to an analysis beyond the
data.
• Bayesian—Probability is a degree of belief or
perception of the likelihood of an event or
circumstance, considering all available
information.
• Contrast—implications of finding an alligator in
the Charles River (near Boston) vs a cockroach in
Aunt Suzie’s kitchen.
Conclusion
• The most likely principal interpretation for the
economists’ behavior is a frequentist orientation to
statistical analysis—case specific information is
required for quantitative uncertainty analysis.
• The record of ex-ante vs ex-post comparisons of cost
estimates could provide a reasonable starting point for
understanding the direction and magnitude of likely
errors in cost estimation. Moreover, distributional
uncertainty estimates specific to particular cases could
be constructed by quantitatively analyzing the past
record for the effects of known likely causal
determinants of systematic and random errors.
Likely Gains From Improved Uncertainty Analysis
of Both Costs and Benefits
• Use distributions to replace the single point
“uncertainty” factors now used to adjust estimates of
“acceptable daily intakes” and “reference doses” from
primary observations for both cancer and non-cancer
effects. However this requires setting explicit
quantitative goals for RfDs in terms of residual
incidence and severity of effects with a particular
degree of confidence/uncertainty
• Provides a long term incentive for improvement of the
estimates and accountability for the estimators.