Transcript ASD/CIVILIAN HUMAN RESOURCE MANAGEMENT

The State of Program Evaluation in
the Canadian Federal Government
Glenn Wheeler
Director, Results Measurement
Office of the Auditor General of Canada
October 17, 2001
Office of the Auditor General of Canada
Outline of Presentation
1. Background on Canadian System of Government
2. About the Office of the Auditor General of Canada
3. The Auditor General’s Approach to Evaluation
4. The Auditor General’s Audits of Program Evaluation
5. Emergence of Performance Reporting within the
Context of Program Evaluation
6. Lessons Learned from the Canadian Experience
Office of the Auditor General of Canada
1. Background on Canadian System of Government
Division of Powers
- a sovereign democracy:
- federation of 10 provinces and 3 territories
- a federal Parliament and 13 provincial/territorial
legislatures
- federal government - authority in all matters not specifically
reserved for provincial/territorial governments
- provincial/territorial governments - authority for education,
hospitals, property and local government
Office of the Auditor General of Canada
Federal Government
- over 100 departments and agencies responsible for areas
such as employment, national defence, fisheries, agriculture
and finance
- each department and agency has a minister who reports to
Parliament
- budgets approved annually
Office of the Auditor General of Canada
2. About the Office of the Auditor General of Canada
History
1878: Office of the Auditor General established
1931: Clarification of Auditor General’s role
1950: beginning of expansion of Auditor General’s mandate
1977: new legislation - the Auditor General Act
1994: legislation amended to allow Auditor General to allow
for tabling of up to 3 reports per year in addition to annual
report
1995: legislation amended to establish a Commissioner of the
Environment and Sustainable Development within the Office
Office of the Auditor General of Canada
Current Office
- new Auditor General, Sheila Fraser, for 10 year term
- headquarters in Ottawa with 5 regional offices
- staff of 520 auditors and support staff
Role
- The Auditor General aids accountability by conducting
independent audits of federal government operations
- these audits provide members of Parliament with objective
information to help them examine the government’s activities
and hold it to account
Office of the Auditor General of Canada
The Auditor General’s Role
Office of the Auditor General of Canada
Vision
- We are committed to making a difference for Canadians by
promoting, in all our work for Parliament, answerable, honest
and productive government that reflects a commitment to
sustainable development
Mission
- The Office conducts independent audits and examinations
that provide information, advice and assurance to Parliament.
We promote accountability and best practices in government
operations.
Office of the Auditor General of Canada
Auditor General Act
Section 7 (2) of the Auditor General Act requires the Auditor
General to call attention to anything he considers to be of
significance and of a nature that should be brought to the
attention of Parliament, including cases in which he has
observed:
a) accounts have not been properly maintained or public
money has not been fully accounted for;
b) essential records have not been maintained to ensure proper
allocation of revenue and to ensure that expenditures have
been made only as authorized;
c) money has been spent other than for the purposes for which
it was appropriated by Parliament;
Office of the Auditor General of Canada
d) money has been expended with due regard to economy and
efficiency;
e) satisfactory procedures have not been established to
measure and report on the effectiveness of programs; or
f) money has been expended without due regard to effects on
the environment.
Office of the Auditor General of Canada
3. The Auditor General’s Approach to Program Evaluation
–
–
–
–
–
Types of audits
- attest audits
- compliance audits
- value-for-money audits
A value-for-money audit is a systematic and objective
examination of government activities. Its scope includes the
examination of economy, effficiency and cost-effectiveness of
activities. It also examines procedures to measure
effectiveness. The subject of the audit can be a program, sectoral
activity, or a government-wide sectional area.
– A program evaluation is a disciplined assessment of government
programs and activities. It is based on independent, systematic
measurement and analysis, carried out to meet expectations set in
policy and standards, and publicly reported.
Office of the Auditor General of Canada
Interpretation
- although value-for-money audits and program evaluations
are similar, they are different in some ways
- the Auditor General does not conduct program evaluations
but rather value-for-money audits which examine procedures
used by departments to measure effectiveness (i.e., to
undertake program evaluations)
Office of the Auditor General of Canada
What the Auditor General expects when undertaking
audits on program evaluation
Value-for-money audits examine how departments plan and
undertake program evaluations of their programs, specifically:
1) the department’s evaluation policy
2) responsibility for program evaluation within the department
3) the organization of the evaluation branch within the
department
4) resources (people and funds) dedicated to program
evaluation
5) the number and quality of program evaluations completed
6) the mechanisms used to ensure quality of program
evaluations
7) the relevance and timeliness of program evaluations
8) how program evaluation findings are used
Office of the Auditor General of Canada
Why the Auditor General considers program evaluation to
be a crucial function in the federal government
1) to provide information to allow senior management to
better manage their programs
2) to provide evidence for departments concerning whether to
continue funding existing programs or re-allocate these funds
to other programs
3) to provide information to managers as these seek funding to
renew programs
4) to encourage accountability to Parliament for the results of
programs
5) to enhance public debate surrounding policy and program
options
6) to allow government to determine priorities
7) to encourage transparency in departments
Office of the Auditor General of Canada
4. The Auditor General’s Audits of Program Evaluation
Audits undertaken on the program evaluation function across
the federal government 1978 - first audit of evaluation function
1983 - update on implementation of 1978
recommendations
1993 - major government-wide audit on the state of
program evaluation in the federal government
1996 - examination of program evaluation in the context
of a changing public sector environment
2000 - reporting performance to Parliament
Office of the Auditor General of Canada
Federal Government Policy on Program Evaluation
1977 - original policy
1991 - revised policy
1994 - review policy (review, internal audit and program
evaluation
1995 - revised expenditure management system
2001 - new evaluation policy
Office of the Auditor General of Canada
1993 Audit - Key Observations
- program evaluation branches poorly funded, and often
subsumed within internal audit banches
- departments did not plan for evaluation work
- many program evaluations of smaller programs; fewer
evaluations of larger more important programs
- program evaluations often focussed on operational issues and
less likely to challenge the existence of a program
- limited use of program evaluation results
- program evaluation results not linked to resource allocation
or accountability reporting
Office of the Auditor General of Canada
1996 Audit - Key Observations
- program evaluation did not consistently provide
effectiveness information for government decision making and
broader accountability
- more larger programs were being evaluated than in 1993, but
undertaken in response to external demands
- scope and quality of program evaluation remained a concern
Office of the Auditor General of Canada
5. Emergence of Performance Reporting in Context of
Program Evaluation
Two key developments
1) revised expenditure management system
- in 1995 the federal government announced that under its
revised expenditure management system departments would
be required to submit an annual “Performance Report” to
Parliament
2) creation of new agencies
- since 1997 three new agencies have been created to
streamline key activities previously undertaken by many
departments (Canadian Food Inspection Agency, Parks
Canada, Canada Customs and Revenue Agency)
- new agencies required to submit annual reports
- legislation for the three agencies states that the Auditor
General annually shall “provide an assessment of the
fairness and reliability of the information about the
Agency’s performance”
Office of the Auditor General of Canada
2000 Audit - Key Observations
- department focus too much on listing program activities and
outputs and to little on reporting outcomes or results
- reporting makes too little use of evaluation findings on the
results that activities are accomplishing and on how programs
are contributing to outcomes
Office of the Auditor General of Canada
6. Lessons Learned from the Canadian Experience
Organizations must:
1) have a unit in place with resources (people and funds) and
expertise to undertake program evaluations
2) have a reporting structure in place so that the manager of
the program evaluation unit reports directly to the head of the
organization or its senior management committee
3) consider constructing a policy formalizing roles and
responsibilities with respect to program evaluation
4) plan to undertake program evaluations. Plans should cover
a one-year period at a minimum, and several years preferably
5) have a plan which prioritizes which programs should be
evaluated based on their importance and level of risk
Office of the Auditor General of Canada
6) have mechanisms built into the process to ensure the
quality of program evaluations and to ensure that they are
timely
7) ensure that program evaluation results are utilized by senior
management in making decisions regarding funding and
whether programs should be redesigned or discontinued
8) follow up with the management of programs evaluated to
determine whether evaluation recommendations are being
implemented
9) periodically review the performance of their program
evaluation units to determine the value that program
evaluation adds to the organization
Office of the Auditor General of Canada