CES National Conference Ottawa, June 2009 Reed Early, Manager Performance Audit

Office of the Auditor General of BC

2007

Performance Audit Redesign

Jan 2008 June 2008 Jan 2009

• • •

Need identification

• Current performance - On time/budget metrics length of projects increasing New Auditor General Training for new staff Example from New Zealand

Need identification con’t

Cheaper audits overall Better consistency, guidance, auditee relationships Faster process 6-9 months

Committee Structure

• Steering committee Core team Design team

• Input - Contracted CI experts - Focus groups - Executive direction • Project launch

-

Introduction and Training

Continuous Improvement

• A learned process – Not intuitive in corporate setting • Principles of Business Process Redesign – Be Bold – challenge everything – Eliminate re-work loops – Assume success and learn from our errors – Maintain control standards – Set indicators, monitor, improve

Redesign work

- Design Team looked for -

opportunities to improve

-

shorten or eliminate steps allow concurrent steps a consistent process Use technology wherever possible

Redesign - 54 steps

(was 150+)

Office of the Auditor General

Performance Audit Process #6 - May 2009

Identification

1 START Auditee Comms 2 Engagement Leader (EL): Meets Audit Team (AT), provides completed project template, & copies Admin Support re TENROX Needs further work 4 Proposal req’d?

Yes 4A EL: Informs ADM or equivalent of interviews 5 AT: Collects KOB conducts interviews; populates proposal template 6 AT: Completes proposal template, fwds to EL for review 7 EL: Meets AT, reviews proposal signs and fwds to AG 8 AG: reviews proposal with EL/AT Deliverable: completed Project Initiation Form 3 Audit Team Member ATM: Completes Project Initiation Form AG: stops audit Deliverable: proposal signed by AG and EL 9 Decision re proposal Yes 10 AG: signs proposal 11 EL: informs DM 12 Admin: TENROX update No Go Audit Plan Comm. Process No

Planning

13 AT: Arranges EL Meetings; Arranges Challenge Group: conducts research, arranges and conducts interviews; populates Audit Plan template 14 AT: Completes Audit Plan template, submits to EL Needs further work 15 AT: Meets with EL to review and revise audit plan 16 AT: Sends draft audit plan to Challenge Group 17 Challenge Meeting Process 18 EL: Meets AT, reviews plan, signs and fwds to AG 18A AG: Reviews plan with EL/AT Advisors Independence Checklist Process Interview Process Contractors Budget Deliverable: Audit Plan signed by AG and EL 19 Decision re plan?

19A Yes AG: signs plan No Go AG: postpones or cancels audit EQCR challenge - signoff *CONTD on next page* Admin: Drafts minutes and sends to AT AT: Reviews minutes and formulates response, fwds to EL, sets up meeting with EL to review response AT: Holds meeting with EL to review and finalize response AT: Sends to Challenge Group for information purposes Deliverable: Challenge Meeting minutes and response

Legend

AG = EQCR = Engagement Quality Control Reviewer EL = Auditor General Engagement Leader (AAG or Senior/Exec Director) AT = Audit Team [includes Subject Matter Expert, Writer, other Performance Auditor (internal or external) KOB = Knowledge of Business EA = Executive Assistant = Key Deliverable

Office of the Auditor General

Performance Audit Process #6 - May 2009

Conducting

*CONTINUED from previous page*

Reporting

20 Data Collection & Analysis Process 21 22 AT: Completes facts and findings section of Findings and Conclusions Template (FFCR), sends to auditee, sets up meeting 23 AT: Holds meeting, documents issues raised at meeting Deliverable: Key findings 25 Facts and findings completion process No 24 All relevant facts correct?

Yes 26 AT: Complete FFCR template, forward to EL 29 AT: Review and adjust report outline Included in audit plan as required EQCR challenge sign-off Deliverable: hyperlinked FFCR document 27 EL: Review FFCR document (and supporting evidence), develop review notes 30 Significant changes?

Yes Wait for EL review 31 28 AT: Takes action, updates FFCR document, sends to EQCR 32 AT: Meets with EL to review/confirm revised outline Deliverable: revised outline No 33a Report Writer: Begins drafting report 33b AT: Finalizes draft report, submits to EL

= DRAFT 1

34 EL: Read, meet with AT to discuss, make changes (all) as required

= DRAFT 2

36 EQCR challenge - reviews, signs-off, sends to AT 35 AT: Sends DRAFT 2 to EQCR, Edit and Communications and Advisors (if required) 37 Edit and Communications: Reads, meets to review, recommend changes to document, send to AT 38 Advisors: Review, send comments to AT 39a AT: Review proposed changes, make edits as required, writes AG bullets, sends to AG, EL, Comms and AT

= DRAFT 3

40 AG: Writes AG comments, sends to AT 41 AT: Meets with AG, EL, and AT to finalize, sends to auditee

= DRAFT 4

42 Auditee: Reviews Draft 4, provides comments 39b Write news release, Q&A, PAC 43 AT: Meet with EL, review auditee comments, make changes as required, add AG comments, send to AG, Auditee, Queen’s Printer, Comms Team

= DRAFT 5 (final draft)

, Deliverable: news release, backgrounder, Q&A, and PAC presentation 44 Receives for information purposes to prepare for 46 AG: release Queen’s Printers: Formats report; sends to ATM No 48 AT: Reviews QP, makes changes as req’d 45 Auditee: Prepares written response to report, sends to Audit Team 47 AT: Receives response from auditee, sends to QP, AG, EL and AT 49 Okay?

Yes 50 Exec Assistant: Receives, sends to Minister 53 Briefing Process 51 Queen’s Printers: Printing Process Includes Minister, Chair and Deputy of PAC, other stakeholders as required 52 Wait for 7 day period 54 EA: Public Release and Distribution Process Note: 7 days before release END Deliverable: Final report

Redesign tools

• • • • •

Templates and checklists “Guide on the Side” Consistent file structure (CaseWare) Analytic software (QDA Miner) Users manual (yet to be done)

Performance indicators

• • • • • Indicate areas to track Purpose is improvement (not acc’ty) Balanced scorecard measures A few measures Target 100% on each

Performance indicators con’t

Indicator Elapsed Time Billable Hours Quality Learning and Growth Clients Perspective Executive Perspective Data Source

Tenrox Tenrox project status report project status report post project status Auditor General’s Performance Post Audit Annual Survey of Executive

Measure

Project – elapsed time between first state date to deliverable, by phase i.e. start Planning to delivery of Plan Projects’ actual hrs by phase and total Project Staff Resource Utilization Use of all templates Use of all audit process steps Staff capacity Professional practices Auditee response to recent audit re performance of our office. Response by Exec to statements:

Public Report - Performance Indicators Summary

Office of the Auditor General of British Columbia Reporting Period:

Quarter 1 (Jan-Mar, 2000)

Completed By:

J Auditor - Data Test

Performance Summary: % Indicators Met (by Balanced Scorecard Group)

100% 100% 100% 100% 100% 100% 80% 60% 40% 20% 0%

Elapsed Time Quality Billable hours Learning and Growth Customer Perspective

• • •

Roll Out planning

Implementation plan for each tool, template, checklist – who, what, when, number of hours Monthly meetings to monitor implementation Overall communication/roll out

Implementation

(Jan 2009 to current)

Training incorporates all templates, tools, checklist, guides Continuous Improvement Working Group established (12x, 4x)

Continuous Improvement:

Questions to ask

• What is more important: flexibility or standardization and why? • What challenges would you anticipate having if you set out to streamline your process? • What do you think are the critical success factors for a process improvement project?

Improvement – so far

• First indicator baseline results –

Time/budget targets not met

–

Staff satisfied with training and templates but not following all steps

–

Suggestions for further improvements

–

“Consumer” Executive/auditee indicator results to come

Challenges

• Other changes happening in parallel: – Narrower more defined projects – Shorter reports – New communications team – Project management processes

Challenges con’t

Initiating and sustaining CI culture: – Inertia and natural resistance – CI Team, ourselves, and executive – Risk averse setting – Quality driven

Challenges (cont’d)

• •

It is Rocket Science!

Professional standards

•

Analytical, cautious staff

•

Getting key people to table

•

Flexibility versus standardization

•

Critical Success Factors

Executive support and leadership • Time and resource commitment • Recognition that change is needed

•

Critical Success Factors (cont’d)

Excellent external facilitator • Good design team: mix, competencies • Willingness to embrace continuous improvement philosophy

Reed Early [email protected]

C I