Transcript Draft cover 2005

The Canadian USOAP
Experience
The Canadian USOAP
Experience
Aim:
to share Canadian experience gained from the ICAO
audit process to provide other states with an insight that
may be helpful in preparing for upcoming audits
Outline
• Overview of the ICAO audit process
• Canadian preparation
• Canada’s Audit experience
• Lessons Learned
• Pitfalls to avoid
2
RDIMS/SGDDI
Overview of the ICAO audit process
Program Objective
(Assembly Resolution A35-6)
Assess State’s implementation of:
- Safety oversight, and
- Standards and Recommended Practices, associated
procedures and guidance material
3
RDIMS/SGDDI
ICAO auditing Philosophy cont’d
ICAO’s Comprehensive Systems approach:
Phase 1.
Review of State Aviation Activity Question and Compliance
Checklists to confirm Annex implementation and capability for Safety
Oversight
Phase 2.
On-site audit to confirm information and assessment of
State’s overall capability for Safety Oversight
4
RDIMS/SGDDI
Foundation Documents
• Rules of Conduct
-Memorandum of Understanding
o
• Audit documentation
•Pre-audit tools
- State Aviation Activity Questionnaire
- Compliance Checklist
•Protocol Checklists
- On site interview questions
5
RDIMS/SGDDI
Foundation Documents cont’d
Memorandum of Understanding
• Defines obligations for the audit
• Sent to State with notification of ICAO’s scheduled intention
- 9 months in advance of audit date
• States are expected to sign and return 90 days before the audit
6
RDIMS/SGDDI
Foundation Documents cont’d
MOU Highlights
• Scope to address all annexes except 9 and 17
• ICAO team members will have command of English and
at least one member will have command of the language chosen
by the State
• Audit will be conducted in accordance with “Safety Oversight
Audit Manual Doc 9735)
• ICAO to conduct the audit in accordance within a specified
timeline
7
RDIMS/SGDDI
Foundation Documents cont’d
State agrees:
- to provide necessary access to staff, documents,
state facilities;
- to provide transportation within the state;
- to provide required translation services
and office facilities;
- to submit updated version of SAAQ and
CC at least 90 days before the audit
- to establish a national coordinator; and
- to assist with accommodation arrangements.
8
RDIMS/SGDDI
Foundation Documents cont’d
Documentation tools
Preaudit
•State Aviation Activity Questionnaire
•Compliance Checklists
- Forms are available on ICAO web site
9
RDIMS/SGDDI
Foundation Documents cont’d
State Aviation Activity Questionnaire
Advance information required by ICAO for scheduling
and workload planning
Provides overview information on State Activity
organized into seven parts:
•General Administrative
•Legislative
•Organization
•Operational Activities
•Air Navigation Activities
•Aerodromes
•Accident and Incident Investigation
10
RDIMS/SGDDI
Compliance Checklists
• Advance information required by ICAO for assessing
compliance with SARPs
• Presents State with tool to identify differences to SARPs
• Update required 90 days in advance of an audit
11
RDIMS/SGDDI
Foundation Documents cont’d
Protocol Checklists
A tool for conducting the on site audit or internal self assessment
used to ensure:
•Transparency
•Consistency
•Standardization
Organized to conduct critical element assessment
in eight modules:
• Legislation
•
•
•
•
•
•
•
State Organization
Personnel Licensing
Operation of Aircraft
Airworthiness
Accident Investigation
Air Navigation
Aerodromes
12
RDIMS/SGDDI
Guidance Sources
ICAO intranet website provides access to:
• State Aviation Activity Questionnaire
• Compliance Checklists
• Protocol Checklists
• ICAO Annexes and Documents
• Completed audits on member states
ICAO Coordinators course
• Also listed on the web
ICAO Team leader for the State audit
• Makes a State/ICAO team for meeting both goals
13
RDIMS/SGDDI
Audit timeline - On site activity
Entry meeting
• Opportunity for introductions;
• ICAO provides explanation of process; and
• The State to provide a description of their organization.
Conduct of interviews
• Start with group interviews for Legislation
and Organization;
• Importance of time management
- Audit plans are on strict schedule
- State coordinators to ensure Protocol questions are
organized, assigned and prepared for the interviews
- ICAO team continues their work after State hours
- A complex audit will take up to two weeks
- Industry visits require State observer
14
RDIMS/SGDDI
Audit timeline - On site activity
Closing meeting
• meeting is for ICAO team to present
their preliminary findings
• no discussion of these findings is intended
• beginning of timeline for acceptance of findings
and the final report.
15
RDIMS/SGDDI
Audit Timeline - Post Audit
90 days
60 days
30 days
30 days
16
RDIMS/SGDDI
Canadian Preparation for the Audit
-Coordination
-Communications
-Arrangements for daily debriefs
-Training for State interviewees
17
RDIMS/SGDDI
Application of ICAO process in
The Pre-Audit phase Cont’d
Coordination
• three areas to be involved in coordination –
Civil Aviation, Transportation Safety Board
and Department of National Defense
• Started work on advance arrangements for:
» Accommodations and office facilities
» Security clearances
» Transportation arrangements for
industry and regional visits
18
RDIMS/SGDDI
Scope for Canadian auditing initiatives contin’d
Need for Internal Coordination System
- State organizational structure does not
parallel audit protocol organization or
ICAO annexes
- need to arrange and schedule appropriate
interview candidates
- need to support ICAO auditor during the
audit
19
RDIMS/SGDDI
Application of ICAO process in
The Pre-Audit phase Cont’d
Communications
- Need for close coordination between State Coordinator
and ICAO Team
- Resulted in effective coordination for planning and
resolution of Protocol question issues
- Introduction of Daily Audit debrief requirement
- Daily feedback for Management and Interviewees
- Triggers for providing follow-up clarification
- ICAO tool for managing “areas of concern”
20
RDIMS/SGDDI
Scope for Canadian auditing initiatives contin’d
Training for the State’s interviewees
- Improved understanding of their role and the
audit process;
- Guidance on role reversal from auditor to
interviewee; and
- Guidance on what are suitable questions.
21
RDIMS/SGDDI
Preparation by the Interviewee
• Protocol questions for specialty areas are
available on ICAO’s web site;
• Critical elements numbers on the protocol
questions confused the interviewees during
their preparation for the audit
22
RDIMS/SGDDI
Observations on the Canadian
Audit Experience
- description of ICAO’s Audit Plan;
- observations on the interviews;
- observations on the audit process; and
- a description of the Findings
23
RDIMS/SGDDI
Page from ICAO Work Plan
for OPS Specialist
24
RDIMS/SGDDI
A Description of ICAO’s Audit Plan
25
RDIMS/SGDDI
The Interview Process Contin’d
Observations on the Interviews
– ICAO’s lack of organizational understanding
lead to misdirection of questions and
inappropriate responses
– Interviewees were frustrated by auditors
changing the order of the Protocol questions as
well as by questioning from shaded areas
- T.C. frustration over amount of inputs that were
not addressed
– No use of non-international inventories
– Did not address Annex 10
26
RDIMS/SGDDI
Daily Debriefs
– Began on the end of second full day of interviews
• Attended by State coordinator and ICAO Team Lead
– Records provided to coordinators and ICAO
• Interviewees verified proper understanding and necessity
to provide additional information
• ICAO used to manage their follow-up issues and confirm
accuracy of record
– During the Daily Debriefs, ICAO used the terminology
“Area of concern” and at the closing meeting the term was
changed to “Preliminary Draft Finding”
27
RDIMS/SGDDI
The closing meeting
- Canada had all audit participants attend to ensure
full understanding of results and to undertake
immediate attention to corrective action.
- Each ICAO specialist summarized the preliminary
findings for their area
- Canada responded with commitment to follow
through on Findings.
- ICAO ended the meeting with the request to
complete a State Assessment form and the
commitment to continue using the “Daily Debrief”
as part of their audit process
28
RDIMS/SGDDI
Closing meeting Contin’d
Was the audit a success?
• From the state perspective;
– Were the logistics met?
• From the State and ICAO perspective;
– Was a proper understanding of the state
program developed to permit a satisfactory
“systems” assessment?
29
RDIMS/SGDDI
Findings
• ICAO auditors admitted that they had to dig deep
to find non-compliance
• There were “areas of concern” we disagreed with
and upon clarification were able to resolve
• Others included:
–
–
–
–
international conditions on licenses
Cabin crew flight and duty times
PPCs every two years
Notify ICAO of exemptions
30
RDIMS/SGDDI
Lessons Learned
1.
Being a “Beta” audit did not impact the final results
1. A shortage of time did not prevent preparation for the
audit
2.
Staff wanted more information on ICAO and its
requirements. Feedback was for more information on
ICAO role, terminology and a simplified version of the
audit process
3.
Need for state to adopt more proactive role in managing
interviews
1. Ensure assignment of Interviewers to appropriate
expertise in the organization
2. Promote requirement for advance preparation of
questions
31
RDIMS/SGDDI
Lessons Learned Contin’d
Lessons Learned (continued)
4. State response to “areas of concern” required
specific focus
– Too much material prevented clear understanding of
the state position
5. Communication plan was essential
– Resulted in team building and knowledgeable inputs
– Daily debrief supported clear understanding by all
participants
– Prevented development of inappropriate “areas of
concern”
32
RDIMS/SGDDI
Pitfalls in Advance of an Audit
1.
2.
Not using all the resources available to understand and
prepare for an audit.
- ICAO web, ICAO team leader, State auditing
expertise
Allowing communication barriers
- Ensure staff have clear roles that are understood in
the program
- Test the process for understanding and effectiveness
33
RDIMS/SGDDI
Pitfalls in Advance of an Audit Contin’d
3. Unprepared
interviewees
• Provide training on process and expected attitude
– Interviewees to understand the limit of their
responsibility to answer questions
– Don’t assume interviewees have knowledge of the
process
• Prepare in advance, formal responses to the protocol
questions and be prepared with supporting reference
documentation
• Address apparent regulatory or organizational protocol
issues with the time remaining before the actual audit
34
RDIMS/SGDDI
Potential Pitfalls During an Audit
1. Misunderstandings
– ICAO understanding of state organization impacts their
protocol management and accuracy of concerns
– State members must understand the limits of their
interview responsibilities
– Purpose of industry visits
35
RDIMS/SGDDI
Potential Pitfalls During an Audit Contin’d
2.
Giving too much information.
2.
3.
Not using the “Daily Debrief” system,
2.
4.
Know when to stop and how to focus the material to the key
point that will be understood
the feedback pulse for the audit
Accepting General Findings based on limited interviews
36
RDIMS/SGDDI
Summary
1.
2.
3.
4.
5.
Strong coordinator essential
Send coordinator on ICAO training
Select knowledgeable Branch coordinators
Short and direct answers to questions
Establish good communications with
ICAO team leader
6. Prepare interviewees including industry
37
RDIMS/SGDDI
Summary (continued)
7. Insist on daily debriefs
8. Escort auditors everywhere
9. Plan to have staff available at all times
during the audit including weekends and
late at night
10. Work with Auditors to ensure “Areas of
Concern” accurately reflect program
38
RDIMS/SGDDI
Questions?
39
RDIMS/SGDDI