Transcript National Action Plan for Energy Efficiency www.epa.gov/ eeactionplan Model Energy Efficiency Program Impact Evaluation Guide - Basics of EM&V Steve Schiller. Schiller Consulting [email protected] www.schiller.com NARUC Workshop February 2008

National
Action
Plan for
Energy
Efficiency
www.epa.gov/
eeactionplan
Model Energy
Efficiency Program
Impact Evaluation
Guide - Basics of
EM&V
Steve Schiller. Schiller Consulting
[email protected]
www.schiller.com
NARUC Workshop
February 2008
Topics

Evaluation objectives

Categories of programs and evaluations

Some more on the Action Plan Guide

Calculating energy savings and avoided emissions

Issues to think about when designing evaluation
efforts and issues for discussion today

Resources - references
www.schiller.com
2
Why Evaluate Energy Efficiency
Programs?

Accountability. Document and measure the effects of a
program in order to determine how well it has met its
intended outcomes (goals).

Improvement. Understand why those effects occurred and
identify ways to improve current and future programs
www.schiller.com
3
Efficiency Program Categories by their primary objectives




Resource acquisition. Directly achieve energy savings,
and possibly avoid emissions, through specific actions.
Market transformation. Change the way in which energy
efficiency markets operate and achieve either naturally
occurring efficiency or incremental codes/standards. Can
also include support for emerging technologies.
Codes and standards. primary objective is to define and
enforce mandated levels of efficiency in buildings and
products.
Education and training. primary objective is to inform
consumers and providers about energy efficiency and
encourage them to act on that information.
www.schiller.com
4
Efficiency Evaluation Categories

Impact evaluations. Determine the impacts (usually energy
savings) and co-benefits that directly result from a program.

Process evaluations. Assess how efficiently a program was
or is being implemented.

Market effects evaluations. Estimate a program’s influence
on encouraging future energy efficiency projects because of
changes in the marketplace.

Cost-effectiveness analyses. Typically seen as an extension
of impact evaluations, but may also take into account market
evaluation results considering market penetration over the
expected lifetime of the measures.
www.schiller.com
5
Back to the NAPEE Guide Why a Program Guide?

Programs are different from projects

There are widely recognized protocols for the
measurement and verification (M&V) of
energy savings from single projects
 e.g., International Performance
Measurement and Verification Protocol
(IPMVP at www.evo-world.org)

Policy neutral guides for determining savings
from programs must be interpreted from
existing specific program protocols
www.schiller.com
6
Guide Scope: Programs Addressed

Primary focus (i.e., includes detailed guidance):
 Resource acquisition, downstream energy efficiency
programs - directly achieve energy and/or demand savings,
and possibly avoid emissions, through specific actions

Secondary focus (i.e., addressed, but no detailed guidance):
 Other demand-side programs: Market transformation,
codes and standards, demand response, and upstream
efficiency programs
 Supply-side programs: renewable energy and combined
heat and power (CHP) program
www.schiller.com
7
Guide Scope: Evaluation Focus

Primary focus:
 Impact evaluation, including: kWh, kW, therm savings and
avoided emissions
• Three basic gross savings analysis approaches
• Four basic net savings analysis approaches
• Two basic avoided emission analysis approaches

“Very” Secondary focus:
 Process and market evaluations
 Potential studies
 Cost-effectiveness evaluation
www.schiller.com
8
Guide Contents
Also includes about 40 “sidebars” of examples/clarifications and 25 figures and tables
Part 1
Executive Summary
Part 2
Chapter 1: Introduction
Chapter 2: Energy Efficiency Program Evaluation
Chapter 3: Impact Evaluation Basics
Part 3
Chapter 4: Calculating Gross Energy and Demand Savings
Chapter 5: Calculating Net Energy and Demand Savings
Chapter 6: Calculating Avoided Air Emissions
Part 4
Chapter 7: Planning An Impact Evaluation
Part 5
Appendix A: Leadership Group List
Appendix B: Glossary
Appendix C: Other Evaluation Types
Appendix D: Uncertainty
Appendix E: Resources and References
Appendix F: Renewables and Combined Heat and Power Program Evaluation
Appendix G: References
www.schiller.com
9
Guide Audience

Policy-makers:


Information about the basic principles of efficiency evaluation
Program evaluation managers:


Resources and background on evaluation and statistical analyses
Guidance, or a roadmap, on process and key issues relating to documenting
energy/demand savings and avoided emissions

Program designers:


Understanding of how their programs will be evaluated
Energy efficiency community:




Common terminology definitions
A central reference document
References for publicly available best practices resources
Understanding of efficiency’s potential value as an emissions control strategy
www.schiller.com
10
Impact Evaluation Results Reported

Estimates of gross savings. Gross energy (or demand) savings are
the change in energy consumption and/or demand that results directly
from program-promoted actions taken by program participants regardless
of the extent or nature of program influence on their actions.

Estimates of net savings. Net energy savings refer to the portion of
gross savings that is attributable to the program. This involves separating
out the impacts that are a result of other influences, such as consumer selfmotivation. Given the range of influences on consumers’ energy
consumption, attributing changes to one cause (i.e., a particular program)
or another can be quite complex.

Estimates of co-benefits. A co-benefit commonly documented and
reported is avoided air emissions: the air pollution or greenhouse gases
that would have been emitted if more energy had been consumed in the
absence of the energy efficiency program.
www.schiller.com
11
Approaches for Determining Gross
Energy Savings

One or more measurement and verification (M&V)
methods from the IPMVP are used to determine the savings
from a sample of projects. These savings are then applied to
all of the projects in the program.

Deemed savings, based on historical and verified data, are
applied to conventional energy efficiency measures
implemented in the program.

Statistical analyses of large volumes of metered energy
usage data are conducted.
www.schiller.com
12
Approaches for Determining Net
Energy Savings

The primary, but not exclusive, considerations that
account for the difference between net and gross savings
are free riders and participant and non-participant
spillover.

Approaches (categories):
• Self-reporting surveys
• Enhanced self-reporting surveys
• Statistical models that compare participants’ and
non-participants’ energy and demand patterns
• Stipulated net-to-gross ratios
www.schiller.com
13
Approaches for Determining Avoided
Emissions

Applying emission factors (e.g., pounds of CO2 per
MWh) to net energy savings

Using emissions scenario analyses, e.g., using
computer models to estimate the difference in emissions
from power plants with and without the reduced
electricity consumption associated with an efficiency
program.
www.schiller.com
14
Issues Addressed in NAPEE Guide

Defining evaluation goals and scale and which benefits to evaluate

Setting time frame for evaluation and reporting expectations

Setting spatial boundary for evaluation (i.e. what energy uses, emission
sources, etc. will be included in the analyses)

Defining baseline, baseline adjustments, and data collection requirements

Establishing a budget vis-à-vis expectations for quality of reported results

Selecting impact evaluation approaches for gross and net savings
calculations and avoided emissions calculations

Selecting who (or which type of organization) will conduct the evaluation
www.schiller.com
15
Evaluation Issues for Discussion

Context:


Efficiency is the first resource and a critical part of
cheap, reliable, clean and stable energy systems
Efficiency is the first mechanism for energy sector
climate change mitigation
• 50- 80% GHG emissions reduction is required for climate
stabilization
• EE has a key role according to IPCC, IEA, McKinsey and
other studies

Each jurisdiction's resource and climate impact
evaluation requirements are not policy-neutral (but the
Guide is)
www.schiller.com
16
Evaluation Issues for Discussion

How will efficiency savings be documented as real?
• What is additional?
• How does one attribute the savings to a particular activity?
• How accurate is accurate enough and how do you balance quality
of results and evaluation budgets



Will cross-jurisdictional, post-Kyoto, national or international
requirements (standards) be established? Is that a good idea?
How will the data, tools and technology for cost-effective
EM&V be developed and supported?
Who should do evaluation, the program administrator, the
regulator, and/or third-parties?
www.schiller.com
17
References - Sample Resources Program Evaluation


2007 NAPEE Model Energy Efficiency Program Impact
Evaluation Guide. www.epa.gov/cleanrgy/documents/evaluation_guide.pdf
2006 California Energy Efficiency Evaluation Protocols:
http://www.calmac.org/publications/EvaluatorsProtocols_Final_AdoptedviaRuling_06-192006.pdf

2004 The California Evaluation Framework.
http://www.calmac.org/publications/California_Evaluation_Framework_June_2004.pdf

2007 US DOE Impact Evaluation Framework for Technology
Deployment Programs.
http://www.eere.energy.gov/ba/pba/km_portal/docs/pdf/2007/impact_framework_tech_deploy_20
07_main.pdf

2006 International Energy Agency. Evaluating Energy
Efficiency Policy Measures & DSM Programmes. http://dsm.iea.org
www.schiller.com
18
References - Sample Resources M&V

2007 International Performance Measurement and
Verification Protocol (IPMVP). (www.evo-world.org)

2000 Federal Energy Management Program M&V
Guidelines. (http://ateam.lbl.gov/mv/docs/26265.pdf)

2002 ASHRAE Guideline 14 Measurement of Energy and
Demand Savings. (www.ashrae.org)
www.schiller.com
19
References - Sample Resources EM&V Websites

Evaluation reports and guidance documents, they can be
found at two web-accessible databases:
 CALifornia Measurement Advisory Council (CALMAC):
http://www.calmac.org.

The Consortium for Energy Efficiency’s Market Assessment and
Program Evaluation (MAPE) Clearinghouse:
http://www.cee1.org/eval/clearinghouse.php3.

Other Sites:
 Proceedings of the IEPEC Conference (http://www.iepec.org)
 ACEEE Summer Studies (http://www.aceee.org)
 Efficiency Valuation Organization (EVO) resource pages
(www.evo-world.org)
www.schiller.com
20
Time for
questions
Reprinted with permission
from Don Piraro
www.schiller.com
21