Transcript CAA Evaluation

Clean Air Agenda Horizontal Evaluation Roll-up:
Synopsis of Evaluation Results Available Up to
September 2010
Final
Prepared by Government Consulting Services
for Audit and Evaluation Branch, Environment Canada
November 8, 2010
Acronyms
ADAPT
Adaptation Theme
CARA
Clean Air Regulatory Agenda
CE
Clean Energy Theme
CT
Clean Transportation Theme
DFAIT
Department of Foreign Affairs and International Trade
DG
Director General
DGTLCC
Director General Theme Leads Coordinating Committee
DM
Deputy Minister
EC
Environment Canada
GHG
Greenhouse gas
HC
Health Canada
HRSDC
Human Resources and Skills Development Canada
IAQ
Indoor Air Quality Theme
IC
Industry Canada
ICC
Interdepartmental Coordinating Committee
INAC
Indian and Northern Affairs Canada
Page 2
INTL
International Actions Theme
M&A
Management & Accountability
NRC
National Research Council
NRCan
Natural Resources Canada
PHAC
Public Health Agency of Canada
TB
Treasury Board
TBS
Treasury Board Secretariat
TC
Transport Canada
UNFCCC
United Nations Framework Convention on Climate
Change
Introduction
•
This report has been prepared in fulfillment of a commitment to produce a horizontal roll-up of the results of the
evaluations done for the Clean Air Agenda (CAA) themes.
•
The roll-up is intended to help participating departments and agencies report on their CAA results to
Parliamentarians and Canadians, and to inform decisions regarding CAA funding and activities beyond 2011.
•
The report comprises four sections:

CAA Profile;

Methodology;

Findings; and

Observations.
Acknowledgements:
• The project team was led by Martine Perrault under the direction of the Environment Canada Evaluation Director,
Shelley Borys, and included Linda Lee, Gavin Lemieux and Katheryne O’Connor. The team would also like to
thank all the individuals from the following organizations who participated in the working group for this project and
well as contributed to the development of the evaluation reports that form the basis for this horizontal roll-up:
Environment Canada (EC) (including the Clean Air Agenda - Results Management Secretariat), Department of
Foreign Affairs and International Trade (DFAIT), Health Canada (HC), Indian and Northern Affairs Canada (INAC),
National Research Council (NRC), National Resources Canada (NRCan), Public Health Agency of Canada
(PHAC), and Transport Canada (TC). This report was prepared for Environment Canada by James Coflin from
Government Consulting Services (PWGSC).
Page 3
CAA Profile
This section provides an overview of the CAA under these headings:
• Background;
• Overview of Themes;
• CAA Program Components;
• Planned and Actual Expenditures by Theme;
• Agenda Logic Model; and
• Governance.
Page 4
Background
•
In the 2006 Speech from the Throne, the Government committed to taking measures to achieve tangible
improvements in our environment, including reductions in air pollutant and greenhouse gas (GHG) emissions.
•
Budget 2006 committed $1.7 billion, later increased to $2.2 billion, to the Clean Air Agenda (CAA) for the four-year
period from 2007-08 to 2010-11.
•
The CAA comprises:

the Clean Air Regulatory Agenda (CARA), an integrated regulatory approach to reducing air pollutant
(nitrogen oxides, sulphur oxides, volatile organic compounds, and particulate matter) and GHG emissions,
and

a series of program measures to:
i. support action by governments, individuals, organizations and businesses in reducing emissions of
GHGs and air pollutants, and
ii. to provide effective responses to climate change.
•
The CAA is a horizontal initiative comprising eight themes and 45 programs. (Note: The Agenda was expanded to
include the Clean Energy Dialogue as a component of the International Actions theme in 2009-2010).
•
A tabular overview of the eight themes is presented on the next page, followed by lists of the program
components within each theme, identifying the department(s) that is delivering the program.
•
Detailed information on the Agenda and the program components are available in the 2008-09 Environment
Canada Departmental Performance Report, Supplementary Table: Horizontal Initiatives: http://www.tbssct.gc.ca/dpr-rmr/2008-2009/index-eng.asp?acr=1481.
Page 5
Overview of Themes
2007-2011
Allocation
Theme
Lead department
Clean Air Regulatory Agenda (CARA)a
Environment Canada
Clean Energy
Natural Resources Canada
Clean Transportation
Transport Canada
Indoor Air Quality
Health Canada/National Research Council
Adaptation
Environment Canada
$115.9 M
International Actions
Environment Canada
$50.0 M
Partnershipsb
Environment Canada
$12.0 M
Management and Accountability
Environment Canada
$5.0 M
a
b
$347.3 M
$1177.0 M
$461.7 M
$23.0 M
This total does not include additional funds under a reprofiling of CARA that was approved in 2009.
Note: The Partnerships Theme was not launched.
Source: Environment Canada, 2008-2009 Departmental Performance Report, Supplementary Information, Table :
Horizontal Initiatives. http://www.tbs-sct.gc.ca/dpr-rmr/2008-2009/index-eng.asp?acr=1481. Unless otherwise
stated, all budget allocation numbers reported in this document are from this source.
Page 6
CAA Program Components 1
Clean Air Regulatory Agenda (EC)
1.
Industrial Sector Regulatory Actions (EC)
2.
Transportation Sector Regulatory Actions (EC, TC & NRCan)
3.
Consumer and Commercial Products Regulatory Actions (EC,
NRCan)
4.
Indoor Air Quality Management Actions (HC)
5.
Science in Support of Regulatory Activities & Accountability (EC, HC)
6.
Emissions Reporting (EC)
7.
Emissions Trading System (EC)
8.
Enforcement and Reporting on Progress (EC)
9.
Policy Analysis (EC)
Clean Energy (NRCan)
1.
ecoENERGY for Buildings and Housing (NRCan)
2.
ecoENERGY Retrofit (NRCan)
3.
ecoENERGY for Industry (NRCan)
4.
ecoENERGY for Renewable Power (NRCan)
5.
ecoENERGY Renewable Heat (NRCan)
6.
ecoENERGY for Technology (NRCan)
7.
ecoENERGY for Aboriginal and Northern Communities (INAC)
8.
Policy, Communications, Monitoring and Reporting (NRCan)
International Actions (EC)
1.
International Obligations (EC, DFAIT)
2.
International Negotiations (EC, NRCan, DFAIT)
3.
Asia-Pacific Partnership (NRCan, EC, IC)
4.
PM Annex (EC)
5.
Clean Energy Dialogue (EC, NRCan, DFAIT)
Partnerships (EC)
1. Community Partnerships (EC)
Clean Transportation (TC)
1.
ecoMobility (TC)
2.
ecoTECHNOLOGY for Vehicles (TC)
3.
ecoENERGY for Personal Vehicles (NRCan)
4.
ecoENERGY for Fleets (NRCan)
5.
ecoFREIGHT Partnerships (TC)
6.
National Harmonization Initiative for the Trucking Industry (TC)
7.
Freight Technology Demonstration Fund (TC)
8.
Freight Technology Incentives (TC)
9.
Marine Shore Power Program (TC)
10. ecoAUTO Rebate Program (TC, HRSDC)
11. Vehicle Scrappage Program (EC)
12. Analytical & Policy Support (TC)
Indoor Air Quality (HC/NRC)
1.
Indoor Air R&D Initiative (NRC)
2.
Radon Strategy (HC)
Adaptation (EC)
1.
Assist Northerners in Assessing Key Vulnerabilities and Opportunities
(INAC)
2.
National Air Quality Health Index and Forecast Program (HC, EC)
3.
Climate Change and Health Adaptation in Northern / Inuit
Communities (HC)
4.
Improved Climate Change Scenarios (EC)
5.
Innovative Risk Management Tools
6.
Regional Adaptation Action Partnerships (NRCan)
7.
Climate and Infectious Disease Alert and Response System to
Protect the Health of Canadians (HC, PHAC)
Management and Accountability (EC)
1.
Management & Accountability (EC)
Page 7
Planned and Actual Expenditures by Theme
($ million)
Theme
2007-08a
Approved
Clean Air Regulatory Agenda
2008-09a
Actual
Approved
Actual
63.8
42.2
76.0
62.6
101.5
81.6
206.9
202.5
98.8
92.6
219.4
160.4
Indoor Air Quality
5.5
4.6
6.4
6.4
Adaptation
7.3
5.7
26.0
20.3
13.2
7.8
10.7
9.7
Partnerships b
3.0
1.0
3.0
0.3
Management and Accountability
1.3
1.2
1.3
1.0
294.4
236.7
549.7
463.4
Clean Energy
Clean Transportation
International Actions
Total
Source: Environment Canada, 2007-08 and 2008-2009 Departmental Performance Report,
Supplementary Information, Table: Horizontal Initiatives.
a
Expenditures reported here may differ from those reported by individual departments or CAA
themes due to differences in reporting periods.
b Although
the Partnerships Theme program activities were not launched, some expenditures
were incurred during the program’s planning phase.
Page 8
Agenda Logic Model
V1.0
Health, economic and
environmental benefits for
Canadians have been realized
Long-Term Outcomes
(Government of Canada)
1
Risks to the health of
Canadians and the
environment resulting from
exposure to air pollution
have been reduced
Risks to communities,
infrastructure and to the health
& safety of Canadians
resulting from Climate Change
have been reduced
2
Trans-boundary
movement of air
pollutants has been
reduced
3
Indoor air quality
has been improved
Canada’s emission of
air pollutants has been
reduced
Innovations related to reducing air
pollution and addressing climate
change have maintained Canadian
competitiveness and provided
economic benefits.
5
6
7
4
Canadians take
actions in
response to
forecasted
levels of air
quality
International
Air quality
agreements
are consistent
with Canada’s
interest
11
12
Reduction
measures for
indoor air
pollutants and
radon exposure
have been
implemented
Targeted industrial
sectors have
reduced emissions
of GHGs and air
pollutants and
improved energy
efficiency
13
. Emissions of
GHGs and air
pollutants have
been reduced, and
energy efficiency
has improved in
Canadian homes
and buildings
14
8
Emissions of
GHGs, air
pollutants and
energy consumption
have been reduced
from modes of
transportation
15
Canadians and communities
have taken actions to reduce
their vulnerabilities from and
have adapted to predicted
impacts of Climate Change
Canada’s emission of
GHGs have been
reduced
16
Emissions of GHGs
and air pollutants
have been reduced
and energy efficiency
has improved from
the use of efficient
and environmentally
safe practices and
products
9
Emission of GHGs
and air pollutants
have been reduced
due to the use of
clean and
renewable energy
sources.
17
International trends on
climate change are
consistent with
Canada’s interests.
10
Canadian
communities and
user groups use tools
and information to
assess climate
change risks and
plan adaptation
strategies
18
19
International
climate change
agreements are
consistent with
Canada’s
interests
20
Intermediate Outcomes
(Clean Air Agenda)
Clean Air Regulatory
Agenda
Clean Transportation
Clean Energy
Partnerships
Adaptation
Indoor Air Quality
International
Management &
Accountability
Theme Outcomes
Regulations, Standards,
Guidelines,
Agreements
Financial incentives
Technology
Development &
industrial R&D
Research studies
Information & Decision
Support
Policy advice
Partnerships
& networks
Tools, training, processes
and systems
CAA Outputs
Page 9
Governance
•
EC is the lead department on the overall CAA and is responsible for the Clean Air Agenda-Results Management
Secretariat (CAA-RMS). In all, eight departments (EC; NRCan; TC; NRC; INAC; HC; PHAC; and DFAIT) are
responsible for themes and/or individual programs. Note: Two additional departments (Industry Canada and
Human Resources and Skills Development Canada) are partners in specific programs.
•
The CAA accountability structure is composed of these interdepartmental committees:

DM Committee.

ADM Steering Committee

DG Theme Management Committees

DG Theme Leads Coordinating Committee (DGTLCC)
•
The DGTLCC is the primary operational committee responsible for implementing the elements of the CAA,
comprising the leads from each of the eight DG Theme Management Committees.
•
The CAA-RMS is the main support structure for the CAA management and accountability theme. Its key functions
include:

Providing support and coordination for the horizontal management of the CAA across departments;

Maintaining a database of financial resource information;

Analyzing financial and non-financial information to support an assessment of performance across the
CAA, including achievement of results, use, allocation and sufficiency of resources, and risks; and

Providing relevant reports, as required, to the governance structure and TBS.
Page 10
Methodology
This section discusses the plan for the CAA evaluation under these headings:
• Scope and Coverage Limitations;
• Approach and Methodology;
• Theme and Program Evaluations Used for the Horizontal Roll-Up;
• Evaluation Issues and Questions; and
• Methodological Limitations.
Page 11
Scope and Coverage Limitations
Scope
•
•
The roll-up addresses the evaluation questions (page 15) set out in the 2008 horizontal evaluation plan.
It focuses on assessing theme and program level outputs and immediate outcomes (as reported in CAA program
and theme evaluations) that are contributing to the achievement of the CAA intermediate outcomes (see Logic
Model, page 9). The CAA long term outcomes were not assessed as part of this horizontal roll-up because they
are expected to occur in a longer time frame.
Coverage Limitations
• The roll-up covers 39 (87%) of the 45 CAA program components (see page 14).
• The combined four-year funding for the programs covered in this roll-up represents about 95% ($2.1 billion) of the
$2.2 billion CAA allocation (see page 14).
• Most of the contributing evaluations were conducted in 2009-10, thereby only covering 2 to 2½ years of
implementation of the CAA activities.
• Evaluations not included in the current roll-up will be reported separately when available. The current roll-up was
finalized without the benefit of those missing components due to a commitment to provide this horizontal report in
October 2010 to inform discussions on future CAA program activities.
Page 12
Approach and Methodology
Purpose
• This report, which examines the relevance and performance of the Clean Air Agenda, is intended to meet
reporting commitments and to contribute to ongoing clean air and climate change policy development.
Audience
• The CAA evaluation roll-up is intended to help meet the information needs of senior management at the
departments and agencies that have developed and delivered CAA themes and program components, as well as
at the Treasury Board Secretariat.
Methodology
• The sole line of evidence for this report is the content of the two thematic roll-up reports and 14 theme and
program level evaluations available for review by September 23, 2010. 2 This excludes evaluations that are still
pending, as specified on page 14.
• The following methods were used in the 14 theme and program evaluations that contributed information to this
roll-up:

key informant interviews (14),

literature review (6),

document review (14),

analysis of performance data (5),

analysis of secondary data (1),

case study (5),

survey (2), and

experts group (1).
• The evidence was extracted from the reports to produce crosswalk matrixes that mapped equivalent data from
multiple theme and program evaluations to fields defined by the CAA evaluation questions.
Page 13
Theme and Program Evaluations Used for Horizontal Roll-Up
Legend: ✓- Evaluation of this theme/program was included in this roll-up; X – Evaluation of this theme/program is in progress or planned.
Num ber of
Program s
Lead
Type
Included
4-year
Allocation a
1. Clean Air Regulatory Agenda (CARA)
9
EC
Them atic
2. Clean Energy
8




$347.3M
--$61.0M
$520.0M




X
$18.0M
$276.0M
$36.0M
$226.4M
$15.0M
$24.6M
Them es/Program s
NRCan
Them atic
2.1 ecoENERGY for Buildings and Housing
NRCan
Program
2.2 ecoENERGY Retrofit
NRCan
Program
2.3 ecoENERGY for Industry
NRCan
Program
2.4 ecoENERGY for Renew able Pow er
NRCan
Program
2.5 ecoENERGY Renew able Heat
NRCan
Program
2.6 ecoENERGY for Technology
NRCan
Program
INAC
Program
NRCan
Program
b



$461.7M
$23.0M
---


$14.0M
2.7 ecoENERGY for Aboriginal & Northern Communities
2.8 Policy, Communications, Monitoring and Reporting
3. Clean Transportation
12
TC
Them atic
4. Indoor Air Quality
2
HC
Them atic
5. Adaptation
7
EC
Roll-up
5.1 Assist Northerners in Assessing Key Vulnerabilities and Opportunities
INAC
Program
EC / HC
Program
5.3 Climate Change and Health Adaptation in Northern / Inuit Communities
HC
Program
5.4 Improve Climate Change Scenarios
EC
Program


$15.0M
NRCan
Program
X
$35.0M
X

$14.9M
$50.0M
-
$12.0M
$5.0M
$2,191.9M
5.2 National Air Quality Health Index and Forecast Program
5.5 Innovative Risk Management Tools
5.6 Regional Adaptation Action Partnerships c
5.7 Climate and Infectious Disease Alert and Response System
HC / PHAC
Program
6. International Actions
5
EC
Them atic
7. Partnerships (Not launched)
1
EC
--
8. Managem ent and Accountability (HMARF)
1
EC
Them atic
Total
a
45
$30.0M
$7.0M
- This total does not include additional funds under a reprofiling of CARA that was approved in 2009.
This program was not subject to a discrete evaluation as per NRCan’s evaluation plan.
c - These two programs extended their Terms and Conditions for one year and will thus expire in 2011-2012. As a result, the evaluation was pushed back until winter 2011.
b-
Page 14
Evaluation Issues & Questions
Relevance – Are CAA activities consistent with and do they contribute to addressing federal government clean air
priorities and the key environmental needs of Canadians?
1. Are CAA activities aligned with federal government priorities?
2. Are CAA activities connected with key environmental needs?
Success – Has the CAA achieved its intended outcomes?
3. To what extent has the CAA achieved its intended outcomes?
Cost-Effectiveness – Are the most cost-effective and efficient means being used to achieve outcomes?
4. Are there more cost-effective and efficient means of achieving CAA objectives?
5. How could the efficiency of CAA activities be improved?
Design & Delivery – Is the CAA designed and delivered in the best possible way?
6. Have each of the CAA activities been implemented, or are they on track to being implemented, as planned
and on time?
7. Is the CAA management and accountability structure in place and functioning as anticipated?
8. Is appropriate performance data being collected, captured and safeguarded? If so, is this information used
to inform senior management/decision makers?
9. What are the best practices and lessons learned from CAA activities?
Page 15
Methodological Limitations
Limitations affecting the horizontal roll-up
• Roll-up vs. horizontal evaluation: The large scale of CAA activities (i.e., $2.2B, 45 programs and 8 departments)
dictated the conduct of several theme and program level evaluations, and a subsequent horizontal roll-up of these
evaluation findings, rather than an umbrella evaluation of the overall CAA.
• Program maturity: Most evaluation evidence was collected during early program implementation and therefore
relies on limited performance data and/or data focusing on outputs rather than outcomes. Annual performance
reports were however developed for the first two years of the CAA.
• Lines of evidence: Many of the evaluations contributing to this report were heavily reliant on information drawn
from key informant interviews. The roll-up of the interview evidence cannot be considered highly reliable given the
open-ended nature of the interviews, the variations in questions, and the self-interest of a majority of the
respondents (about 46% of the 547 interviews conducted were with federal government employees managing
and/or delivering the programs).
• Level of Focus: It is difficult to draw conclusions about some of the CAA evaluation questions because of the level
of detail and focus of the contributing evaluations. Specifically, theme-level evaluations and theme-level roll-ups
tend to produce generalized or heavily qualified findings that cannot be quantified or consolidated.
Underlying Limitations
The following methodological and analytical limitations were reported by more than one of the theme evaluations. (The
number in parentheses indicates the number of evaluations identifying the limitation) :
• Program maturity (4): The programs were not completely implemented and it was too early to assess outcomes.
• Lines of evidence (3): Only two or three lines of evidence were used.
• Limited performance data (6): The available data were limited to output/activity level indicators.
• Policy processes (5): It was difficult to access/collect relevant output and outcome data on such processes.
• Cost data (4): Current and/or sufficiently detailed program and activity level financial data was not available.
Page 16
Findings
This section rolls-up the findings and conclusions from the program and theme-level evaluations that are relevant
to each horizontal-level evaluation question.
Page 17
Relevance
1. Are CAA activities aligned with federal government priorities?
Evidence
• Overall CAA activities and outcomes are seen to align with federal priorities as expressed in documents and
statements such as:

Budgets and speeches from the Throne;

Turning the Corner, the April 2007 Government of Canada plan to reduce greenhouse gases and air
pollution;

UNFCCC commitments, including the commitment under the Copenhagen Accord to a 17% reduction in
GHG emissions from 2005 levels;

Canada’s Performance 2008–09 that identifies “a clean and healthy environment” and “healthy Canadians”
as two of the government’s thirteen outcome areas;

A Climate Change Plan for the Purposes of the Kyoto Protocol Implementation Act, May 2010.
•
The evaluations confirmed that the program components they reported on are aligned with departmental priorities
and program activity architectures.
Finding
• CAA activities are aligned with federal priorities.
Page 18
Relevance
2. Are CAA activities connected with key environmental needs?
Evidence
• The CAA evaluation plan listed three indicators for this question: demonstration of the utility/rationale for CAA
activities; CAA programs remain relevant as new scientific evidence on air quality and climate change emerges;
and, evidence that CAA program architecture has adapted to new evidence and changes of government policy
and priorities.
• The theme and program evaluations, with the exception of the M&A theme evaluation, considered “needs” in the
course of assessing the relevance of programs and activities. However, the interpretation of the concept and the
kinds of evidence gathered varied. Some looked at key informant opinions about workloads or the responses of
beneficiaries and stakeholders as evidence of need, while others referred to research results and/or secondary
data sources to document air pollutant and GHG emission levels. The Clean Energy theme report demonstrated a
continued need for several of the theme’s program activities.
• Three of the thematic evaluations examined whether new scientific evidence had emerged that would raise
questions about the relevance of CAA activities – none was found.
• There were no discussions or reports of modified program architecture.
Finding
• Several of the theme and program evaluations provided evidence to demonstrate that CAA activities are
connected to key environmental needs.
Page 19
Success
3. To what extent has the CAA achieved its intended outcomes?
Approach
• The roll-up assessed the extent to which each CAA intermediate outcome (see Logic Model, page 9) has been
achieved by examining the “success” evidence from the evaluations for the themes that are expected to be the
major contributors to each outcome, as identified below:
CAA Intermediate Outcomes
CARA
CT
CE
ADAPT
IAQ
INTL

Canadians take actions in response to forecasted levels of air quality.
International air quality agreements are consistent with Canada’s
interests.

Reduction measures for indoor air pollutants and radon exposure have
been implemented.

Targeted industrial sectors have reduced emissions of GHG and air
pollutants and improved energy efficiency.


Emissions of GHGs and air pollutants have been reduced and energy
efficiency was improved in Canadian homes and buildings.
Emissions of GHGs, air pollutants energy consumption have been
reduced from modes of transportation.
Emissions of GHGs and air pollutants have been reduced and energy
efficiency was improved from the use of efficient and environment safe
practices and products.
Emissions of GHGs and air pollutants have been reduced due to the
use of clean and renewable energy sources.
Canadian communities and user groups use tools and information to
assess climate change risks and plan adoption strategies.
International climate change agreements are consistent with Canada’s
interests.
Page 20









Success
3. To what extent has the CAA achieved its intended outcomes?
Approach (continued)
• For each of the CAA intermediate outcomes, success information is presented under three headings:

The plans made to achieve the outcome, including which themes/programs were involved and the nature
of interventions -- incentives; regulations and agreements; or capacity building -- that are being used.

The term “capacity building” is used here to refer to activities that involve obtaining or distributing
information with the goal of changing individual and organizational behavior including research, public
information products and services, social marketing campaigns, training and educational activities, and
demonstration projects.

The evidence from the theme and program evaluations about the extent to which planned outputs have
been produced and intended outcomes achieved.

Findings that can be drawn from the available evidence. It is important to note here that outcome data
were not available for most of theme and program level evaluations because programs had not been fully
implemented when evaluated, or their outputs were only expected to have the impacts that will contribute
to key outcomes some months or years later. The findings sections that follow do not repeat this
qualification, but rather highlight where outcomes data are available.
continued......
Page 21
Success
3. To what extent has the CAA achieved its intended outcomes?
Estimate of Progress
•
An indicator (see example below) was developed to provide a high level estimation of progress made toward the
achievement of each CAA intermediate outcome based on the evidence reported by the theme and program
evaluations reviewed:

No outputs – no outputs because program implementation was delayed or terminated.

Partial outputs – outputs are being produced but not to planned levels, or work has been significantly
delayed.

Outputs delivered – most planned outputs are being delivered, but insufficient time has elapsed for their
contribution to CAA intermediate outcomes to have been experienced.

Outcomes likely achieved – given the extent of outputs delivered and program maturity (i.e., sufficient
time has elapsed since program implementation), and assuming that the CAA results logic is accurate, it is
likely that the intended intermediate outcome is being achieved. There was however no data collected in
the program or theme evaluation to assess whether the outcome was being achieved.

Outcomes achieved – there is reliable evidence that the intermediate CAA outcome has been achieved.
No
outputs
Partial
outputs
Outputs
delivered
Page 22
Outcomes
likely
Outcomes
achieved
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
Canadians take actions in response to forecasted levels of air quality.
The Plan
• The seven Adaptation theme programs were designed to improve Canada’s capacity to reduce health risks from
and adapt to a changing climate and air pollution. The program designers expected increased capacity to be
achieved through research and the development of tools, partnerships and standards. Research and tool
development is being done directly by the participating departments and by other organizations supported through
contribution funding.
• The theme targets: the North; human health; and infrastructure.
Evidence
• The Adaptation programs, with one exception, were launched in 2008-2009, not in 2007-2008, the first year of the
CAA. The exception is the AQHI which is the extension of a predecessor program.
• The theme evaluation, drawing on four program evaluations, reported that the programs were being implemented
as planned, although two of them were at risk of partial or delayed implementation.
• The available evidence showed that where implementation has begun, the programs are beginning to achieve
immediate theme-level outcomes: increased capacity to conduct science; greater collaboration; and increased
availability of and access to products.
Findings
• There is some preliminary, although weak, evidence that the Adaptation theme is making progress toward
achieving the theme’s immediate outcomes that are expected to contribute directly to this CAA intermediate
outcome.
Page 23
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
International air quality agreements are consistent with Canada’s
interests
The Plan
• The PM Annex component of the International Actions theme is working toward the negotiation of an annex to
the 1991 Air Quality Agreement. Under such an agreement, the U.S. and Canada would commit to reducing
emissions of PM (particulate matter) and its precursors. PM comprises air-bound, microscopic pollutants that
have negative impacts on air quality, the environment and human health.
Evidence
• The PM Annex initiative has two major outputs: i) partnerships and processes; and ii) formal and informal
consultations with domestic stakeholders.
• The International Actions theme evaluation reports evidence that some progress was made toward developing
partnerships and processes with the U.S. to establish the PM Annex.
• The evaluation found no evidence that stakeholders were consulted directly and it was not clear how Canada’s
position was developed or how the views of domestic partners and stakeholders have had an influence or been
meaningfully reflected.
• Changes in policy direction and positions on key issues in both Canada and the US have resulted in delays in
achieving the PM Annex.
Findings
• PM Annex activities have produced outputs related to partnerships and processes but there is little evidence to
show that consultations with domestic stakeholders have helped to inform the development of Canada’s
position in PM Annex negotiations. Such consultations are necessary to ensure the PM Annex is consistent with
Canada’s interests.
Page 24
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
Reduction measures for indoor air pollutants and radon exposure have
been implemented
The Plan
• The CAA expected to achieve this intermediate outcome through:

the Indoor Air Quality theme, which is focusing on capacity building activities including research on
improved ventilation, distribution and other indoor air quality solutions, construction of a indoor air
research laboratory, creation of a national committee on indoor air quality; the mapping of radon zones;
testing radon levels at federal sites and a radon public awareness campaign; and

The CARA theme that planned new federal guidelines (regulations) for indoor air contaminants, and a
national radon risk management strategy.
Evidence
• Because it was done during the third year of a four-year program and many of the outcomes are not anticipated
until the end of the four-year period, the IAQ evaluation examined the progress that had been made under the
theme’s two program components. It found that they were generally on track to deliver their planned outputs.
The notable exceptions are:

the programs have not delivered the planned radon public awareness campaign or the assessments of
the indoor air quality public awareness effort due to restrictions on the conduct of public opinion research;
and,

radon testing of federal buildings is behind schedule.
• The CARA evaluation reported that federal guidelines for indoor radon, carbon monoxide and ozone levels have
been published within planned time frames and that related work is generally on track.
Findings
• The Indoor Air Quality Theme and CARA have produced, or are on track to produce, the outputs that are
expected to contribute to the achievement of theme and CAA intermediate outcomes, but it is too early to expect
that the outputs have had their intended impact.
Page 25
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
Targeted industrial sectors have reduced emissions of GHG and air
pollutants and improved energy efficiency
The Plan
• The Industrial Regulations component of the CARA theme was designed to develop and implement sectorspecific GHG emissions regulations.
Evidence
• The CARA evaluation reports that the development of regulations for industrial greenhouse gas emissions, as
well as the development of emissions trading and offset systems, have been affected by “directional change in
policy.” The program, however, has undertaken many of the planned activities (consultations, research,
drafting instructions), but further work has been deferred.
Findings
• CARA has not yet had the opportunity to produce the outputs needed to achieve the GHG and air pollutant
emission reductions required for the achievement of this CAA outcome.
Page 26
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
Emissions of GHGs and air pollutants have been reduced and energy
efficiency was improved in Canadian homes and buildings
The Plan
• The Clean Energy theme offered homeowners, as well as small and medium –size organizations, incentives to
retrofit or construct energy efficient homes and buildings through two programs: ecoENERGY Retrofit ($805 M) and
ecoENERGY for Buildings and Houses ($61 M).
Evidence
• At the end of its first two years (2007-2008 and 2008-2009), the ecoENERGY Retrofit for Homes program had
already achieved 64-80% of its targeted range for GHG emission reductions and energy savings. Evidence suggests
that 84% of energy savings are directly attributable to the Program. Moreover, homeowners are undertaking double
the number of retrofits they had originally planned and 60% are motivated to undertake further measures outside of
the Program.
• At the end of its first two years (2007-2008 and 2008-2009), the Buildings and Houses program had achieved 6371% of its targeted range for energy savings and 80% of its GHG emission reductions and had effectively extended
the reach of NRCan’s energy efficiency programs in the residential sector. The program has also exceeded its target
of 5,000 trained energy advisor professionals in the construction of energy efficient new homes and buildings.
• The ecoENERGY Retrofit for Small and Medium Organizations program had achieved GHG emission reductions and
energy savings but not to the extent anticipated when targets for the programs were set. Furthermore, attribution of
this progress to the program’s financial incentive activities is only partial as some firms would have implemented
these changes on their own. For the most part, capacity building activities (e.g., training and design validation), are
increasing the capacity of commercial / institutional energy users to adopt energy management best practices and
implement energy efficiency measures. In turn, this contributes to the outcome of energy savings and GHG emission
reductions
Findings
• Evidence shows that, although they have not yet fully reached all of their intended targets, Clean Energy programs
are achieving GHG reductions and energy efficiency improvements in the Canadian homes and buildings sector.
Page 27
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
Emissions of GHGs, air pollutants, energy consumption have been
reduced from modes of transportation
The Plan
• Two themes are contributing to the goals of increasing fuel efficiency and reducing GHG and air pollutant
emissions within the personal and commercial transportation sectors:

The CARA theme is focused on developing and implementing a range of regulations and related
products; and

the Clean Transportation theme comprises capacity building and incentive programs intended to
increase the adoption of fuel efficient transportation technologies and behaviours, as well as plans
to establish voluntary emission reduction agreements with the air, railway and marine sectors of the
transportation industry.
Evidence
• The CARA theme evaluation found that the development of transportation-related regulations was
happening as planned and within scheduled time frames.
• The Clean Transportation theme evaluation found that, with small exceptions and some delays, the
initiative was delivering its planned activities and outputs. With respect to outcomes, it found evidence that
the ecoAUTO Rebate program , an incentive program, had contributed to an increased uptake of fuelefficient technologies in the personal vehicle market. There was also evidence of some increased
awareness of energy efficient transportation technologies among target audiences.
• With respect to the capacity building programs, both the theme evaluation and NRCan’s Evaluation of the
Transportation Sub-Sub Activity found the required outputs had been delivered but that outcomes could
not be assessed due to a lack of baseline and outcomes data and the challenges of attributing changes in
consumer and business behaviours to specific interventions.
• The Clean Transportation theme evaluation reports that the voluntary agreements with the air and railway
sectors have been renewed or maintained, but a similar agreement for the marine sector has not been
developed.
Findings
• By and large, the CAA transportation-related activities have delivered the regulatory, incentive and
capacity building outputs that are expected to contribute to the achievement of this CAA outcome.
Page 28
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
Emissions of GHGs and air pollutants have been reduced and energy
efficiency was improved from the use of efficient and environment safe
practices and products
The Plan
• It was expected that this outcome would be realized as a result of capacity building outputs such as public information
services, activities promoting smarter energy use, support for the development and deployment of clean energy
technologies, and fuel efficient driver training delivered by the Clean Energy and Clean Transportation themes, as well
as regulatory actions developed through the CARA theme.
Evidence
• The Clean Energy evaluation reports that awareness building activities and adoption of technologies under the
ecoEnergy for Industry program contributed to reduced emissions. At the time of the evaluation, the program was within
its target range for energy savings and GHG reductions. It however reports that, while the activities and programs
funded within the ecoEnergy for Technology Initiative lay the groundwork for increased deployment of clean power
generation, reduced emissions from this program will depend on the degree to which these technologies are deployed
on a large scale.
• The Clean Transportation theme evaluation and NRCan’s Evaluation of the Transportation Sub-Sub Activity found that
the planned capacity building outputs were being produced. The theme evaluation also highlighted the challenges of
attributing changes in consumer and business behaviour to specific capacity building interventions.
• With respect to regulatory interventions, the CARA theme evaluation reported that the planned development of energy efficiency standards for consumer products and regulations to reduce the emissions of volatile organic compounds
(VOC) in the manufacture and use of consumer and commercial products were on track, but that the impact of the
regulations would not be measurable until some years after they had come into effect. NRCan’s evaluation of Equipment
programming indicated that its Energy Efficiency Standards and Labelling Program contributes to increasing the
minimum energy efficiency of equipment manufactured by suppliers, and therefore purchased by consumers, thereby
contributing to reductions in energy use and associated GHG emissions.
Findings
• There is evidence that the Clean Energy, Clean Transportation and CARA themes have produced the regulatory and
capacity building outputs designed to increase the use of efficient and safe practices and products and some evidence
that this outcome is being achieved as a result. It is however too early to conclude on the achievement of this outcome
for 2 of the 3 themes involved.
Page 29
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
Emissions of GHGs and air pollutants have been reduced due to the use
of clean and renewable energy sources
The Plan
• The CAA expects to achieve this outcome through the incentives delivered by two Clean Energy theme
programs:

ecoENERGY for Renewable Power ($276.0 M) provides incentives for the producers of renewable power
(e.g., from wind, low impact hydro, biomass and solar photovoltaic); and

ecoENERGY for Renewable Heat ($36.0 M) uses contributions to commercial, industrial and institutional
users to encourage the installation of solar water and air heating systems. As more are installed, prices are
expected to fall and the consumption of conventional energy should decline.
Evidence
• By April 2009, Renewable Power projects with a total capacity of 3,100 megawatts had been commissioned or
were under construction – the program’s target is approximately 4,000 megawatts by March 31, 2011.
• During its first two years, the Renewable Heat Program supported 334 solar thermal projects – its target is 700
systems by March 31, 2011.
Findings
• NRCan’s Renewable Energy Evaluation concluded that, after two years, the renewable energy programs have
delivered planned outputs and are on track to reach most of their intended outcomes. In addition, it expressed the
opinion that many of the programs’ long-term outcomes will be achieved, although it is not possible to directly
attribute the achievement of these long-term outcomes to the programs due to interacting factors and
considerations.
Page 30
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
Canadian communities and user groups use tools and information to
assess climate change risks and plan adaption strategies.
The Plan
• Several of the Adaptation Theme programs included capacity development activities (i.e., the development and
distribution of tools and information such as guidelines, methodologies, data tables, models, and web pages) to
help researchers, government departments and agencies, businesses, community groups and others assess risks
and plan adaptive measures.
Evidence
• The Adaptation Theme evaluation reported that the evaluations completed for four of the seven programs indicate
that the programs were generally being implemented as planned and that Improved Climate Change Scenarios
may not be fully implemented within the program timeframe.
• Available information indicates that the use of adaptation information and products is most likely to be used by
groups who have an immediate interest in climate change impacts (e.g., scientists or individuals who are at-risk
from adverse health impacts).
Findings
• The evidence points to increased use of tools and information by groups with a clear interest in, or an incentive to
use, adaptation products and information.
Page 31
No
outputs
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
CAA Intermediate Outcome:
International climate change agreements are consistent with Canada’s
interests
The Plan
• The International Partnerships and Negotiations component of the International Actions Theme is the most
important contributor to the achievement of this CAA outcome. Its activities are intended to ensure that Canadian
environmental and economic interests are advanced through international negotiations. Note: For the purposes of
evaluation, “Canada’s interests” are those goals and interests defined by the Government of Canada in federal
negotiating instructions and formal policy statements.
Evidence
• The International Actions evaluation reported that the International Partnerships and Negotiations program had
produced planned outputs.
• The evaluation concluded that Canada’s interests are reflected in international climate change agreements,
although it is difficult to attribute this directly to the activities under the International Actions theme. As well, there
is clear evidence of enhanced consultation activities but they have not necessarily resulted in increased
consideration of stakeholder interests in Canada’s positions
Findings
• There is evidence that the International Actions theme has, at least partially, achieved this CAA intermediate
outcome.
Page 32
Success
Progress toward Outcomes: Summary
CAA Intermediate Outcomes
No
outputs
Canadians take actions in response to forecasted levels of
air quality.
International air quality agreements are consistent with
Canada’s interests.
Reduction measures for indoor air pollutants and radon
exposure have been implemented
Targeted industrial sectors have reduced emissions of
GHG and air pollutants and improved energy efficiency
Emissions of GHGs and air pollutants have been reduced
and energy efficiency was improved in Canadian homes
and buildings
Emissions of GHGs, air pollutants and energy
consumption have been reduced from modes of
transportation.
Emissions of GHGs and air pollutants have been reduced
and energy efficiency was improved from the use of
efficient and environment safe practices and products.
Emissions of GHGs and air pollutants have been reduced
due to the use of clean and renewable energy sources.
Canadian communities and user groups use tools and
information to assess climate change risks and plan
adoption strategies
International climate change agreements are consistent
with Canada’s interests.
Page 33
Partial
outputs
Outputs
delivered
Outcomes
likely
Outcomes
achieved
Cost-Effectiveness/ Alternatives
4. Are there more cost-effective and efficient means of
achieving CAA objectives?
Evidence
• Opinions about alternative more efficient approaches: Key informants were either satisfied that the current
program models were as or more efficient than alternatives, or they did not know of any realistic alternatives.
• Comparison of CAA activities to other comparable programs: Only the Clean Transportation Theme evaluation
reported the results of a comparative analysis. It found that comparable programs exist for most Clean
Transportation programs, but the availability of cost information for the comparable programs limited the analysis
to the ecoAUTO Rebate and the Vehicle Scrappage programs. The Vehicle Scrappage program seems to be
more efficient than United States and United Kingdom initiatives based on calculations of incentive cost per
vehicle scrapped and total cost per vehicle scrapped. A comparison of the ecoAUTO Rebate program and the
U.S. Cash for Clunkers program shows similar unit costs for administration. 3
• Opinions about whether CAA investments are a good use of public funds: While the evaluations do include key
informant suggestions for program refinements and adjustments, the overall tenor of the evidence is that program
staff and other stakeholders are generally satisfied that the programs are doing their job at reasonable cost and
that no realistic alternatives are available.
• Costing emission reductions: The Clean Transportation evaluation calculated a cost per tonne of GHG emissions
reduced based on forecasted GHG emissions reductions and original program budgets. The results suggest that
the Vehicle Scrappage and ecoAUTO Rebate programs are not cost-effective.
Finding
• While there is key informant and comparative evidence that CAA activities are efficient, calculations of the cost
per tonne of GHG emission reductions suggest that some programs, particularly incentive programs, are not costeffective.
Page 34
Cost-Effectiveness/ Alternatives
5. How could the efficiency of CAA activities be improved?
Evidence
• The evaluations sought key informant opinions about ways to improve the efficiency of the programs they were
associated with.
• The suggestions that were offered and reported tend to be specific to particular programs, even to particular
activities.
• A comparison of the evaluations failed to identify any suggestions that were repeated across themes and
programs.
• During the first two years of operation CAA theme actual expenditures were under budget (see page 8).
Finding
• There was no evidence of efficiency improvements that would be relevant to CAA activities generally.
Page 35
Design and Delivery
6. Have each of the CAA activities been implemented, or are
they on track to being implemented, as planned and on time?
Evidence
• Information about the implementation status of 40 of the 45 CAA programs is available.
• The majority of the programs have been implemented as planned, that is they are producing the expected
outputs and, generally, within the expected timeframes:

One program, Clean Transportation’s ecoAUTO Rebate program (TC) has been completed; and

Twenty-nine programs were identified as on track – delivering their outputs as planned.
• One (1) CAA program was not implemented as planned

i.e. Community Partnerships (EC).
• Work on two CARA initiatives – the Emissions Trading System (EC) and the Industrial Sector Regulatory Actions
(EC) – has been deferred.
• Seven programs have experienced challenges resulting in delayed or partial implementation:

The Clean Transportation programs ecoMobility (TC); Marine Shore Power Program (TC); National
Harmonization Initiative for Trucking Industry (TC), and ecoFREIGHT Partnerships on Freight (TC);

The International Actions initiatives PM Annex (EC) and Asia-Pacific Partnership (NRCan); and

The Radon Strategy (HC).
Finding
• The majority (30 of 40) of the CAA programs evaluated have been implemented as planned, six have been
implemented with significant schedule and/ or output changes, two were not implemented and two are described
as deferred.
Page 36
Design and Delivery
7. Is the CAA management and accountability structure in
place and functioning as anticipated?
Evidence
• The CAA management and accountability structure is the subject of a separate evaluation that concluded, in part,
that:

The M&A Theme has brought coherence to the horizontal reporting of CAA activities and results and has
emerged as a strong success story, in large part because of the CAA–RMS. Central agency
representatives said that it was a key model to follow for reporting on horizontal initiatives.

Roles and responsibilities for CAA management have been clearly defined, processes to support decision
making and coordination have been established, and results are being reported to Parliament and
Canadians in a transparent manner.

There was limited evidence, however, of policy and program adjustments being made within the CAA
horizontal decision-making structures. When needed, horizontal collaboration has occurred through other
more flexible channels.
• The general tenor of the management and accountability evaluation findings at the theme level tend to validate the
M&A theme evaluation conclusion: horizontal management structures have been implemented and play an
important role in reporting, but have limited roles in program and policy decision-making, in part due to limited
interdependencies between themes (see page 20). Instead, decision making operates on vertical lines of
accountability and, where there are interdependencies between programs, bilateral planning and coordination
solutions are used.
Finding
• The agenda and theme level horizontal management structures have been implemented as planned. Their
functioning in terms of CAA reporting responsibilities exceeds expectations, but they have not played any
significant role in decision-making.
Page 37
Design and Delivery
8. Is appropriate performance data being collected, captured and
safeguarded? If so, is this information used to inform senior
management/decision makers?
Evidence
• The CAA evaluation plan listed three indicators for this question:
• Populated performance data systems: The key informant and document review evidence reported by the theme
and program level evaluations suggests that the programs are meeting basic departmental and horizontal
reporting requirements.

The evaluations do not contain enough specific information to do a program-by-program analysis, but taken
together, their observations suggest that a minority of programs have fully functioning systems.

The most commonly identified issues were: the lack of an adequately structured and maintained system;
processes or systems that focus on activities and outputs, but lack key outcome data; and indicators that
are not relevant or appropriate.

Six (6) of the nine (9) that examined performance data questions made recommendations for
improvements to performance data.
• Decisions based on performance information: Only two evaluations reported on the use of performance data in
decision making: the International Actions evaluation observed that it was used for departmental reporting, but
little else, while the Clean Transportation document review and key informant interviews found examples of the
use of such data in program management decision making, as well as reporting.
• Baseline data: Only a few programs have collected or will be able to collect baseline data. As a result it will be
difficult, if not impossible, to assess the achievement of intermediate and longer-term outcomes whether the
outputs involved incentives, capacity building activities or regulations.
Finding
• It appears that CAA programs have established systems or processes to gather and maintain activity and output
data that supports program management, but some are not collecting the baseline and outcomes data needed to
measure and evaluate performance.
Page 38
Design and Delivery
9. What are the best practices and lessons learned from CAA
activities?
Evidence
• The following best practices and lessons learned are those at the Agenda level that cut across several CAA
themes or programs. Other best practices and lessons learned can be found in the CAA theme and program
evaluation reports.
• Best Practices: A best practice is a methodology or work process that has been demonstrated to achieve superior
results. The evaluations reported a number of best practices, but often fail to provided evidence to demonstrate
that the activity described was a superior solution.
• The best practices at the Agenda level reported by the Management and Accountability evaluation were:

Requiring approval of CAA performance reports by DMs of partner departments and agencies before the
reports are submitted to TBS.

The detailed charter, and the CAA–RMS implementation of the terms of the charter.

The approach of the CAA–RMS to ensure effective financial and performance data collection, compilation
and reporting (acknowledged by TBS to be a model of reporting on a horizontal initiative).
•
•
•
Lessons Learned: A lesson learned describes an aspect of a project or program that did or did not work well, and
identifies the risk of ignoring the lesson. In all, six of the evaluations reported a total 19 “lessons learned.”
The evaluations tended to provide only a brief description of the lesson without context or identification of risk.
While the lessons tended to be specific to a particular program or activity, it is worth noting that four of the
evaluations (CARA, Adaptation, Renewable Energy and IAQ) listed nine lessons learned that called for proactive
communication to and engagement of external stakeholders, including communities and individuals.
Finding
• The CAA–RMS led processes for financial and performance data compilation and reporting represents a best
practice for horizontal initiatives.
Page 39
Overall Observations
This section presents observations in two groups:
• Program observations: Observations related to overall CAA relevance and performance
• Evaluation observations: Observations related to technical aspects of horizontal evaluations
Page 40
Program Observations
Relevance
• The CAA programs are aligned with government priorities and are connected to key environmental needs.
Achievement of Outcomes
• Early evidence shows that a large majority of the planned activities are being implemented and delivered.
• As acknowledged in the 2008 CAA horizontal evaluation plan and the theme and program evaluations, the
impacts of most of the CAA programs will not be felt or measureable for some time, even if the program has been
fully implemented.
• Some outcome data for incentives programs (ecoEnergy for Retrofit, Renewable Energy, eco AUTO, and Vehicle
Scrappage) suggests that there is increasing uptake of energy efficiency products and services, fuel-efficient
technologies and clean energy sources, but there are also outstanding attribution challenges.
• With the exception of Clean Energy programs, there is mostly anecdotal evidence about the likely impacts of CAA
capacity building activities and, without baseline data and given restrictions on survey research, it will be difficult to
reliably assess those impacts in the future.
• Literature reviews have found international evidence about the effectiveness of regulatory interventions, but
evaluations of the Clean Air regulations have not yet been done.
Cost Effectiveness
• There is preliminary evidence that incentive programs, like the ecoAUTO Rebate program and the Vehicle
Scrappage program, may not be cost-effective when measured in terms of estimated emission reduction.
Design and Delivery
• The CAA horizontal management structures have enabled very successful horizontal reporting but have had
limited usefulness for program and policy decision-making in light of the limited interdependencies between CAA
themes and the maintenance of vertical lines of departmental accountability.
Page 41
Evaluation Observations
Horizontal Evaluation Frameworks
• The development of the horizontal evaluation plan provided some degree of consistency across the theme and
program level evaluations, but not enough to facilitate a strong roll-up of results. Future horizontal evaluation
planning might benefit from establishing requirements for all theme and program evaluations to address a small
number of core questions, with well defined indicators and comparable/equivalent data sources.
Performance Measurement
• The majority of the theme and program evaluations reported that they encountered problems because there was
limited outcome data. In many cases this was because the program had not reached the stage of its
implementation where outcomes could be measured, or because of methodological issues such as limitations on
public opinion research. In other cases, however, it may be that program managers did not collect and maintain
outcome data.
• Evaluation planning would benefit from scheduling early reviews of program performance measurement
frameworks and their implementation.
Common Terminology
• The roll-up of the theme and program evaluation results was complicated by different interpretations of some
concepts and terms such as needs, utility/rationale, unintended outcome, best practice, and lessons learned.
While variation in interpretation and understanding of these terms is common and not usually an issue with a
single evaluation, a horizontal evaluation plan would benefit from including a terminology lexicon to obtain
consistent evidence across several evaluation units and numerous evaluators.
Page 42
Endnotes
1.
2.
Descriptions of each of the CAA program components are included in the Environment Canada Departmental Performance Report, Supplementary
Table: Horizontal Initiatives. The 2008-09 supplementary table is published at http://www.tbs-sct.gc.ca/dpr-rmr/2008-2009/index-eng.asp?acr=1481.
The primary information sources for this evaluation roll-up were the following program and theme-level evaluation reports:
CARA: Evaluation of the Clean Air Regulatory Agenda -DRAFT (June 14, 2010), Audit and Evaluation Branch, Environment Canada.
CARA: Evaluation of the Equipment Sub-Sub Activity - Draft Final Report (June 28, 2010), NRCan Strategic Evaluation Division. (Note: This
evaluation covers NRCan’s Energy Efficiency and Standards Labelling Program, which is a sub-component of CARA under the Consumer and
Commercial Products Regulatory Actions program).
Clean Energy: NRCan’s Clean Energy Thematic Report - Draft (September 2010), NRCan, Strategic Evaluation Division.
Clean Energy: Renewable Energy Evaluation – Final Report (July 2010), NRCan, Strategic Evaluation Division.
Clean Energy: Evaluation of the Clean Electrical Power Generation (CEPG) S&T Sub-sub Activity (PAA 2.1.4.5) - Final Evaluation Report.
(September 17, 2010), NRCan, Strategic Evaluation Division.
Clean Energy: Evaluation of the Oil and Gas Sub-sub Activity. Draft Report (September 22, 2010), NRCan, Strategic Evaluation Division.
Clean Energy: Evaluation of Energy Efficiency for Industry, Housing and Buildings – DRAFT Report (September 15, 2010), NRCan, Strategic
Evaluation Division.
Clean Transportation: Government Consulting Services, Report on the Evaluation of the Clean Transportation Theme of the Clean Air Agenda
(July 2010), Transport Canada. (Note: This report covers the Clean Transportation theme programs delivered by Transport Canada and
Environment Canada).
Clean Transportation: Evaluation of the Transportation Sub-Sub Activity - Final Report (July 6, 2010), NRCan, Strategic Evaluation Division.
(Note: This evaluation covers NRCan’s participation the transportation sector regulatory actions and six programs, two of which are part of the
Clean Transportation theme).
Indoor Air Quality: Government Consulting Services, Evaluation of the Indoor Air Quality Theme of the Clean Air Agenda (June 2010), Health
Canada and National Research Council.
Adaptation: Evaluation of the Clean Air Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings -DRAFT
(July, 2010). Audit and Evaluation Branch, Environment Canada.
Adaptation: Evaluation of the National Air Quality Health Index Program (February 2010), Audit and Evaluation Branch, Environment Canada.
Adaptation: Evaluation of the Improved Climate Change Scenarios Program - Final Draft Report (June 2010), Audit and Evaluation Branch,
Environment Canada.
International Actions: Evaluation of the International Actions Theme of the Clean Air Agenda - Draft Final Report (June 2010), Audit and
Evaluation Branch, Environment Canada.
International Actions: Marbek | Stratos. Evaluation of the Canada-U.S. Clean Energy Dialogue - Draft Report, Audit and Evaluation Branch,
Environment Canada.
Management and Accountability: Evaluation of the Clean Air Agenda Management and Accountability Theme - Draft (Revised July 13, 2010),
Audit and Evaluation Branch, Environment Canada.
Page 43
Endnotes
3.
4.
It should be noted, though, that the United Kingdom and United States programs were adopted for economic stimulus reasons rather than to reduce
transportation emissions.
The NRCan Evaluation of the Equipment Sub-Sub Activity (contributing to the CARA Theme) made similar calculations for the two non-CAA programs it
reported on (equivalent data for the CAA programs was not available): (page 42)

The Accelerated Standards Action Program was expected to reduce GHG emissions by 2.8 Mt/year in 2010. In total, $28.4 million was spent
under the Accelerated Standards Action Program. Therefore, if the Accelerated Standards Action Program achieved its GHG emissions
reductions estimate, it will result in $10.14 per tonne of GHG emissions reduced in 2010, with reductions ongoing past 2010.

EnerGuide for Industry was expected to reduce GHG emissions by 1.4 Mt/year in 2010. In total, $2 million was spent under EnerGuide for
Industry. Therefore, if EnerGuide for Industry achieved its GHG emissions reductions estimate, it will result in $1.43 per tonne of GHG
emissions reduced in 2010, with reductions ongoing past 2010.
Page 44