Transcript Title of Presentation

Surveillance
Findings/Analysis/CAPs
ATAC 2010
1
Transport Canada Civil Aviation
Prairie Northern Region
•
•
•
•
•
•
David White
SMS Coordinator PNR
Oversight Coordination Division
Based in Edmonton, Alberta
(780) 495-5216
[email protected]
2
Contents
•
•
•
•
•
•
•
Define surveillance activities
Discuss surveillance procedures (SI-SUR-001)
Refer to areas of review, SMS, QAP,Ops Control
Analysing Issues identified during surveillance
Causal analysis (root cause)
Generating acceptable Corrective Action Plans (CAP)
Effective communication during the process
3
What is an Assessment?
• Assessment means the surveillance activity
conducted to evaluate effectiveness and level of
compliance with the CARs.
• Staff Instruction (SI) SUR-001
• Sometimes referred to as 17 PVIs.
4
What is a PVI?
• A systematic review of one or more SMS components of an
organization OR other regulated areas of a certificate holder.
• Used to determine if regulatory requirements are documented,
implemented, in use and effective within certified organizations.
• It is a means to focus on the effectiveness of management
systems & make judgments on performance.
• Quality Assurance (QA) and the Ops control system are
examples of management systems
5
Additional distinction between terms
• Assessment – Evaluation of entire SMS
– Limited to companies that meet SMS requirements
• PVI – Evaluation of regulated area of company’s
control system or specific SMS component
– Applicable to all companies
• Process Inspection – In-depth review of steps
to obtain specific output
• Our focus will be on Assessments/ PVIs here today.
6
Surveillance Activity Overview
• The activity consists of:
– Documentation review (preparation)
– On-site review (including interviews & sampling)
– Analysis
– Reporting
7
Surveillance Procedures in SUR-001
•
•
•
•
•
•
•
•
•
Notification
Team selection
PVI plan
Preliminary doc review
Documentation review
Team meeting
Sampling procedures
On-site procedures
PVI team on-site briefing
•
•
•
•
•
•
•
•
•
On-site review
Interviewing
Analysis
PVI worksheets
Findings
Exit meeting
PVI report
PVI CAP procedures
PVI follow-up & closure
8
Program Validation Inspection (PVI)
• PVI will review the following areas for certificate
holder without Quality Assurance (QA) Program:
– Part III - Airport operator’s obligations
– Part IV - Company operational control & supervision
– Part VII - Subpart 3 or 4 - Company operational control
– Part VII - Subpart 5 - Safety Oversight component
– Part VIII - Operating certificate requirements
9
Quality Assurance Program
• Regulations for certificate holders to have QA
– Will be implemented in phased manner
• PVIs for Parts III, IV, VII & VIII without fully
implemented QA Program use customized PVI
worksheets
• SUR-001 Appendix A contains expanded list of all
PVI worksheets organized by operating rule
10
Quality Assurance Program
• Certificate holders that have QAP:
– Inspectors use worksheet SUR-001 Appendix A Table 5.1
Quality Assurance
• This includes certificate pursuant to:
–
–
–
–
CAR 561
STD 566
CAR 573
Maintenance requirements for all Part IV & VII operators
11
Some Examples of Worksheets referenced in SI-SUR-001
(k) TABLE 3.1 – Reactive Processes
(l) TABLE 3.2 – Proactive Processes
(m) TABLE 3.3 – Investigation & Analysis
(n) TABLE 3.4 – Risk Management
(p) TABLE 5.1 – Quality Assurance
(r) Program Validation Inspection Worksheet – CAR 302
(s) Program Validation Inspection Worksheet – CAR 406
(t) Program Validation Inspection Worksheet – CAR 703/704
(u) Program Validation Inspection Worksheet – CAR 801
(ee) Program Validation Inspection Worksheet EASA
NOTE: These Worksheets are internal documents only
12
Surveillance Activity Responsibilities
• Convening Authority
– Identifies - Who
– Defines - Scope & depth
• PVI/Assessment Manager
– Plans
– Manages
• Team Member(s)
– Conduct
13
Documentation Review
• Doc review 4-6 weeks before on-site review –Assessment
• Doc review 1-2 weeks before on-site review –PVI
• Includes review of all files & docs relevant to certificate holder
and scope of the surveillance activity
• Ensure all reference manuals/docs are available & include latest
amendments
• Confirm revision status of any manual subject to review
• Review approved manuals for conformance to standard
• Identify areas that require further review during on-site.
– Develop specific questions for certificate holder and appropriate
management based on intelligence gathered during review.
14
“A general policy statement is not usually enough to satisfy
requirements. Document review team members must verify
that there is a policy in place and that it is support by an
appropriately documented procedure or process as required ”
• This varies dependent on the area being reviewed and the
type of certificate issued but is basically the “Write what you
do and do what you write” principle.
• TC may request additional documentation if the initial
submission does not meet the team’s requirement.
15
Onsite Activities
•Entrance Meeting
•Interviews ( Management and selected employees)
•Sampling
•Based on size, scope and complexity of the organization.
•Determined through process in SI-SUR-001
•Used in conjunction with interviews to determine regulatory
compliance
• Non-compliance issues Identified
•Purpose of the activity is to determine regulatory
compliance.
•Company briefing(s)
•Exit meeting
16
Surveillance Findings
• We will concentrate on PVI
examples today.
• Ineffective Quality Assurance
• Loss of Operational Control
• Examples of Non-compliance
with regulatory requirements.
• Examples of Non-compliance
with Company
Policy/Procedures and
processes.
17
Surveillance Reporting
• Report generated with findings.
• Required actions determined
through review with CA.
• Issues are typically handled
through the Corrective Action
Plan (CAP) process.
• Required additional actions are
based on the severity of noncompliance issues identified.
• Additional actions may involve
Certificate Action and/or
Enhanced Monitoring.
18
Review of Findings
• Recognizing the System
Failure
• Addressing the immediate
concerns
• Causal Analysis
• Planning the short-term
corrections
• Planning the long term
corrections
• Informing TC
19
Causal Analysis
SI-SUR-001, Section 10.4 CAP Evaluation …
A Root Cause Analysis of the Non-conformance:
The certificate holder’s analysis shall include a summary of
the root cause as well as any causal factors that may
have contributed to the non-conformance. Even though
the certificate holder may use a causal analysis method
that is not familiar to the CAP reviewer (e.g.: MEDA
process, “5 Why’s”, etc.), the certificate holder must be
able to demonstrate how they arrived at the root cause
and it should be clear what caused the nonconformance to occur.
20
Causal Analysis
• Causal Analysis seeks to identify and understand the
reasons why things are as they are and hence enabling
focus of change activity.
• Some Examples:
– The 5 Whys
– Meda
– Fishbone
• The key is just to keep asking until you get to a real 'aha'
that you can treat.
21
CAP Submission
•
•
•
•
•
•
TC Inspector Expectations
Causal Analysis
Appropriate Timelines
Documentation
Communication
SI SUR-001 Section 10.4 (2) states:
*** “The role of the principal inspector responsible for the
acceptance of the CAP shall be limited to the assessment of the
process used by the certificate holder in reaching conclusions
regarding the findings. It is not the inspector’s role to secondguess the CAP solutions, but rather to evaluate the process
used in developing those solutions.”***
22
CAP Acceptance and Follow-up
• Remember it is your Plan
• Monitor the implementation to ensure it is
implemented
• Communicate Issues if implementation not as per
plan.
• Document the issues and resolution with the
regulator.
• Be prepared for the Activity follow-up…Confirming
your systems are working not just that the examples
identified during the inspection have been
addressed
23
Surveillance Activity Closure
• Effective systems are a
continuing process.
• Your responsibility for day
to day compliance
• Systems will mature and
improve as we continue to
use them.
24
Thank-you
25