Transcript SCE 2007 Process Evaluation Plans

Process Evaluation Plans for
2006-08 EE Programs
Outline for discussion with
CPUC Energy Division Staff
November 30, 2006
Tasks for All Process Evaluations
• Update program theory, document program activities,
and compare program operation with theory.
• Assess and make recommendations on program
documentation and processes needed to support energy
savings estimation.
• Investigate barriers to program participation and other
obstacles to program implementation.
• Evaluate how the program meets SCE portfolio needs.
• Recommend improvements and solicit implementer
feedback on usefulness of the recommendations.
11/30/2006
Southern California Edison Co.
2
Key Issues for Process Evaluations of
Major Programs
• The following slides identify a few key
issues that will be explored for each major
program.
• We solicit Energy Division’s input on
issues and concerns they have with
individual programs or evaluation plans.
11/30/2006
Southern California Edison Co.
3
SCE 2500
Appliance Recycling Program
• Issues with room air conditioner recycling
• Impact of increased rebate for freezers on the
level of freezer recycling
• Effectiveness of co-promoting program with
other SCE programs
• New implementation issues given two recycling
services contractors in the program
– Tracking databases
– End-to end process
– Quality assurance, control, and monitoring
mechanisms
11/30/2006
Southern California Edison Co.
4
SCE 2501
Residential Energy Efficiency Program
• Effectiveness of point of sale (POS)
– Incentives design, marketing, trade allies
– Product quality
• How the new electronic rebate application
is meeting customer needs
• Limited POS customer contact data
available that needs to be shared between
the process and impact studies
11/30/2006
Southern California Edison Co.
5
SCE 2502
Multifamily EE Rebate Program
• Three-fold increase in funding level
• Getting a wide variety of measure installations
in the programs
• A better handle on the installation practices for
the lighting measures
– Product quality, installation locations, documentation
• Effectiveness of increased program exposure to
contractor networks and multifamily trade circles
– Approaches for small, medium, and large properties
11/30/2006
Southern California Edison Co.
6
SCE 2505
California New Homes Program
• No plan available yet.
• Any issues that Energy Division or the
Master Evaluation Contractor would like to
see included?
11/30/2006
Southern California Edison Co.
7
SCE 2507
Comprehensive HVAC Program
• Start-up issues with the major, new component –
the AC Quality Program
– procedure development, recruitment of Verification Service
Providers (VSPs), development of contractor training material,
marketing etc.
• Program dependence on the midstream market actors
• Consistency across processes, quality of the service,
and level of results
• Program Tracking and Documentation
– evaluability assessment to identify gaps in program
data tracking and documentation affecting impact
evaluation, especially for early replacement
component
11/30/2006
Southern California Edison Co.
8
SCE 2508
Retro-Commissioning Program
•
Hold off on a formal process evaluation until 2008. Instead, conduct an
internal assessment in 2007, because the program manager is still making
many enhancements to the program operations
Other Proposed Activities
• Verification of baseline of a few early participants, to test program review
processes.
•
Measure Drop-Off Study, using 2007 program participants – Estimate the
average drop-off from the starting number of retro-commissioning
recommendations (and their energy savings) to the subset of
recommendations actually installed/implemented. This can be useful for
program planning and results forecasting.
•
Persistence Study: Identified by various published reports as an area of
concern. Can additional customer training improve it? We’re short of
funding to do a study of this.
11/30/2006
Southern California Edison Co.
9
SCE 2509
Industrial Energy Efficiency Program
As a new program, the Framework specifies an
examination of the following:
• Program design
• Management, staffing
• Operations,
• Support systems
• Use of best practices
• Tracking systems
11/30/2006
• Participant satisfaction
• Service-timelines
• Marketing outreach
• Program targeting
• Implementation issues
• Enrollment systems
Southern California Edison Co.
10
SCE 2509
Industrial Energy Efficiency Program
In addition, SCE will building on empirical
work to date by examining:
– How typical barriers to industrial energy
efficiency programs are impacting SCE’s
program
• implementation uncertainty, order fulfillment,
capital allocation criteria/strategy, productivity
concerns
– How SCE’s “Project Champion” structure is
mitigating these typical barriers
11/30/2006
Southern California Edison Co.
11
SCE 2510
Agricultural Energy Efficiency Program
Since this regrouping and tailoring of audit and
rebate offerings creates a largely new program,
the process study will examine:
•
•
•
•
•
•
Program design
Management and staffing
Operations
Support systems
Use of best practices
Tracking systems
11/30/2006
•
•
•
•
•
•
Participant satisfaction
Service timelines
Marketing outreach
Program targeting
Implementation issues
Enrollment systems
Southern California Edison Co.
12
SCE 2510
Agricultural Energy Efficiency Program
Furthermore, the study will build on 2003 and
2004-2005 SCE Pump Test Evaluation results by:
• Assessing SCE’s cost analysis
letter
• Reviewing customer tracking,
classification and identification
• Improving information delivery
between field testing personnel
and customer contact personnel
• Identifying and prioritizing
facilities or geographies in need
of testing
11/30/2006
• Discovering criteria or indicators to
guide field resource allocation
• Evaluating how traditional
marketing indicators (rate class,
previous participation) should
play in future participant sourcing
• Integrating 3rd party implementers
into program marketing
• Sharing information with 3rd party
implementers
Southern California Edison Co.
13
SCE 2511
Nonresidential Direct Install Program
• Explore additional measures for the program
– Program is lighting-dominated; other measures tried have had
problems
• Review and recommend marketing approaches that
could increase customer participation rates above
30%.
• Review/recommendations on inspection processes
• Review/recommendations on customer eligibility
• Review/recommendations on savings estimation
methods, given high variability across sites
• Review/recommendations on inspection processes
11/30/2006
Southern California Edison Co.
14
SCE 2512
Savings By Design
• No evaluation plan yet available.
• Any issues that Energy Division or the
Master Evaluation Contractor would like to
see included?
11/30/2006
Southern California Edison Co.
15
SCE 2513
Education, Training & Outreach
• Education, Training & Outreach Program
– Seven distinct program elements:







Energy Centers (CTAC & AgTAC)
Technology and Test Center (TTC)
Energy Design Resources (EDR)
Non-residential Remote Energy Audits (NRREA)
Mobile Education Unit (MEU)
Building Operator Certification (BOC)
Customer Language Efficiency Outreach (CLEO)
11/30/2006
Southern California Edison Co.
16
SCE 2513
Education, Training & Outreach
• Ensure program participant tracking systems support
indirect energy savings estimation
• Assess marketing outreach and program targeting
strategies, marketing collateral, training materials &
curriculum design
• Evaluate effectiveness of program delivery channels in
reaching intended targets
• Explore & apply “best practices” in adult ed.
• Investigate barriers to participation
• Assess participant satisfaction with programs
11/30/2006
Southern California Edison Co.
17
SCE 2515
Emerging Technologies Program
• Program Operations and Theory Alignment (crossprogram)
– Once an ET is adopted into a resource acquisition program, what
drives successful deployment?
• Baseline Knowledge and Awareness
– How do ETP stakeholders rate the ETP as a credible and timely
information source (as compared to DOE, ACEEE…)?
• Marketing Effectiveness
– Are the media and message used by ETP appropriate for the
stakeholders? Is the website being used effectively to
communicate assessment results?
• Stakeholder Needs Assessment
– Who are the stakeholders of ETP, and which of their needs is
ETP fulfilling?
11/30/2006
Southern California Edison Co.
18
SCE 2516
Statewide Codes & Standards Program
• No evaluation plan completed yet.
• Any issues that Energy Division or the
Master Evaluation Contractor would like to
see included?
11/30/2006
Southern California Edison Co.
19
SCE 2517
Business Incentives & Services Program
Process Evaluation Focuses:
•
•
•
•
•
•
Addition of the Audit Services Program
Encouragement of Comprehensive Projects
Inclusion of Added Load Projects
Integration of Demand Response Opportunities
The Green Building Module
Pilot Program For Joint Gas & Electric Audits
11/30/2006
Southern California Edison Co.
20
SCE 2518 – SCE 2525
Local Government Partnership Programs
• Best Practices
– A cross-program process evaluation will compare and contrast
operations to determine best practices.
– Factors delineated in the 2004-2005 Local Government Initiative
evaluation will be used to guide this best practices evaluation.
• Baseline Knowledge and Awareness
– How aware are local government officials and SCE customers of
SCE’s resource acquisition programs? How reputable are SCE
and govt officials as sources of EE information?
• Marketing Effectiveness
– How well do officials and customers understand the benefits of
participating in SCE resource acquisition programs?
• Customer Needs Assessment and Fulfillment
– Why did customers choose to participate? What were their
expectations, based on the marketing?
11/30/2006
Southern California Edison Co.
21
SCE 2526 – SCE 2530
Institutional Partnership Programs
• Best Practices:
– Program processes and results will be compared and contrasted
with each other as well as other notable institutional energy
efficiency partnership programs in the U.S.
• Education Effectiveness:
– These programs will train state agency staff to identify and scope
retrofit and RCx projects. Can the training courses be improved,
and can they be shared with CTAC and other RCx programs?
• Partner Needs Assessment and Fulfillment:
– How well do all partners share responsibilities? Are the
expectations and needs of the partners and program
stakeholders met?
• Communications/Staffing:
– The CDCR program will serve as the template for the State of
California Partnership. Partners will be interviewed throughout
the early period of the program implementation for early
feedback on program operations.
11/30/2006
Southern California Edison Co.
22
SCE 2531 – SCE 2548 - IDEEA
SCE 2549 – SCE 2552 - InDEE
• Small, innovative programs to fill gaps in existing
program offerings and test new technologies, with the
ultimate goal of “mainstreaming” successful efforts
– subject to the same benefit cost criteria as other programs
– the time frame may not be sufficient for new program options to
be fully explored
– cannot afford many implementation errors and still pass the
benefit-cost test.
• An “evaluability” assessment for each program
• Identify start-up issues
• Identify gaps in documentation, program tracking and
information
11/30/2006
Southern California Edison Co.
23
SCE 2554 – SCE 2556
Statewide Marketing Programs
• No current plans for process evaluation
• Any issues that Energy Division or the
Master Evaluation Contractor would like to
have addressed?
11/30/2006
Southern California Edison Co.
24