Transcript Comparison of 1994-1997 CA Evaluation Protocols, CA

Comparison of 1994-1997 CA
Evaluation Protocols, CA Framework,
IPMVP and CPUC Policy Manual*
A preface to group discussion
*In terms of how they define program gross energy and demand EM&V
requirements
Mary Sutter & Tim Caulfield
Equipoise Consulting Inc.
October 27, 2004
Discussion Overview


Purpose of this presentation
Overview of some documents used in California
evaluation of energy efficiency programs






CPUC Energy Efficiency Policy Manual Version 2
The California Evaluation Framework
Protocols and Procedures for the Verification of Costs,
Benefits, and Shareholder Earnings from Demand-Side
Management Programs (The “Protocols”)
International Performance Measurement and Verification
Protocol - Concepts and Options for Determining Water and
Energy Savings (IPMVP)
Comparisons
Group Discussion
2
Purpose of this Presentation





CALMAC is the appropriate venue for
substantive discussion on evaluation
Energy efficiency program evaluation in
California in a state of flux
New evaluation protocols are planned
Through review of past / present documents,
determine what are the most crucial
questions to discuss at this point in time
Narrow scope for now (Gross energy and demand impacts)
3
Energy Efficiency Policy Manual



Purpose: Sets the policy rules
in the development and evaluation
of energy efficiency (EE)
programs in California.
Broad set of objectives that
evaluation must meet.
Specific components indicated
with supporting evidence required
if not including in evaluation.
4
Policy Manual
The objectives below must have strong supporting arguments for omission from
evaluation plan

Policies for Programs with Energy Impacts

Evaluation must measure the level of energy and peak demand
savings achieved by a program.

Measure Cost-effectiveness
Up-front market assessments and baseline analysis (especially
for new programs)
Ongoing feedback, and corrective and constructive guidance
regarding the implementation of the program
Measure indicators of the effectiveness of specific programs,
including testing of the assumptions that underlie the program
theory and approach
Assess overall levels of performance and success of programs
Inform decisions regarding compensation and final payments
Help assess whether there is a continuing need for the program






5
Policy Manual
These components are required in the evaluation plan




Baseline Information
 Baseline data upon which to base energy savings measurement
 Perform study if none available or prove why cannot do study
Energy Efficiency Measure Information
 Description of EE measures in program
 Includes assumptions about important variables and unknowns
M&V Approach
 Reference appropriate IPMVP option
 Describe deviation from IPMVP
 Schedule for acquiring project-specific data
Evaluation Approach
 Questions to be answered through evaluation
 Evaluation tasks / activities
 Describe how evaluation will meet all policy objectives
6
The California Evaluation Framework



Purpose:
The California
Evaluation Framework (Framework)
provides a consistent, systemized,
cyclic approach for planning and
conducting evaluations of California’s
energy efficiency and resource
acquisition programs.
The primary purpose of impact
evaluation is to obtain the most
accurate and unbiased estimate of
energy and demand savings due to a
program.
Gross savings are calculated from
program participants relative to their
prior participation usage.
7
Framework – Gross Savings

Billing Analysis path



Based on statistical principals
Multiple methods / regression models
Engineering Analysis path



Based on basic rules of physics
Simple engineering models or building energy
simulations
M&V incorporated into field data collection (IPMVP)
8
Framework – Gross Savings Issues

Billing Analysis



Difficult / impossible for evaluation of 3rd party
programs.
Finding a non-participant group not effected by
any EE program (market noise)
Engineering Analysis


The most uncertain parameter may be the most
expensive to obtain
Cannot collect pre-retrofit measurements in many
cases
9
IPMVP


Purpose: The IPMVP
provides an overview of current best
practice techniques available for
verifying results of energy efficiency,
water efficiency, and renewable
energy projects.
This document can help in the
selection of the M&V approach that
best matches:



i) project costs and savings magnitude
ii) technology-specific requirements
iii) risk allocation between buyer and
seller (i.e., which party is responsible
for installed equipment performance
and which party is responsible for
achieving long term energy savings).
10
IPMVP
M&V – the process of determining savings using one of the four IPMVP
options
 Option A
 Engineering calculations with some parameters in algorithms
stipulated and other parameters onsite measured
 Option B
 Engineering calculations with all parameters in algorithm onsite
measured
 Option C
 Whole facility analysis with billing or metered data – from simple
pre/post comparison to regression analysis on a single building
over time
 Option D
 Computer simulation calibrated with hourly or monthly utility
billing data and/or end use metering
11
“Protocols”

Purpose: Protocols and
procedures to be used by the
IOUs to document and verify costs
and benefits of major DSM
program activities for shareholder
earnings and measurement
agreements for resource planning
purposes.
12
“Protocols”

Gross Impacts







Provides various types of models/approaches
acceptable and generally when they are
applicable
Sample design requirements for approaches
Reporting requirements
Documentation requirements
Measurement schedule
Net Impacts
Retention Studies
13
“Protocols”
Summary



Prescriptive protocols – Specify what, when
and how
Contain ability to request a waiver from
specified approaches and schedules;
reviewed by CADMAC.
Supplied relative surety that if protocols were
followed results would be acceptable.
14
Comparisons
EE Policy
Manual
CA
Evaluation
Framework
“Protocols”
IPMVP
Specific to energy efficiency programs
Yes
Yes
Yes
No
Protocols with known success parameters
and avenue for disputes
No
No
Yes
No
Energy impact evaluation procedures
No
Yes
Yes
Yes
Demand impact evaluation procedures
No
No
No
No
Program level sampling procedures
No
Yes
Yes
No
Net issues addressed
No
Yes
Yes
No
Specific reporting formats and procedures
No
No
Yes
No
Persistence impact evaluation procedures
No
No
Yes
No
Timing of evaluations discussed
No
Yes
Yes
No
15
Discussion Points
What are the relevant questions for our next set of protocols?






Who are the ultimate users of the energy and demand impacts,
what data is required and when is the information needed?
How can the protocols be structured such that you can put
resources where there is the most uncertainty?
Do evaluations require precise demand impacts or can we use
the Savings Goals GWh to MW conversion?
What are the ramifications of 3rd party programs being unable to
use billing analysis that includes nonparticipants?
Will evaluation budgets support yet-to-be-determined precision
requirements for energy/demand impact evaluation as well as
allowing for logic modeling and process evaluation?
Others?
16