Slide 1

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 2

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 3

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 4

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 5

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 6

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 7

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 8

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 9

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 10

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 11

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 12

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 13

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 14

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 15

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011


Slide 16

Registration Management Committee (RMC)

9101:2009
Revision Workshop
Quality Management Systems Audit Requirements
for Aviation, Space, and Defense Organizations

“It’s All About Performing”
Tony Marino
The Boeing Company

Company Confidential

Other Party (OP) Assessor Workshop
San Diego, CA
January 20, 2011

1

Registration Management Committee (RMC)

9101 Team Charter
IAQG Initiative Study dated 30 Sept 2006 and presented at the IAQG
meeting in Seville, Spain on 5 Oct 06.

“Revise 9101 to align with the next revision of 9100 and upcoming ISO
17021, review scope and identify any additional stakeholder needs
and restructure the standard for a more process-based assessment
approach to auditing an organization’s quality management system.”

13 Members on the 9101 Team representing:
• Americas, Europe, and Asia-Pacific IAQG Sectors
• 6 Different Countries
• 8 IAQG member companies
• 4 Certification Bodies
San Diego, CA
January 20, 2011

2

Registration Management Committee (RMC)

9101 Major Changes
• One document covering 9100/9110/9120 standards.
• Elimination of “Scoring” and “Key Requirements” designation.
• Use of Customer feedback concerning QMS Organization
performance as an input for process oriented audits (e.g.,
OASIS, customer satisfaction/performance scores).
• In addition to conformity, also determination of Effectiveness
included.
• More emphasis on performance measuring.
• Introduction of the Objective Evidence Record (OER).

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

WHAT’S NEW?
• Expanded the Scope (QMS Audit Req’mts for Aviation, Space,
Defense Orgs)
• Eliminated scoring
• Eliminated use of the clause based questionnaire (replaced by
Objective Evidence Record)
• Greater emphasis on audit planning (more pre-work, on-site Stage
1 by Team Leader)
• Introduces audit methodologies
• Strengthened NCR requirements (standard form, containment,
single entry, recurring finding upgrade, 30-day C/A response
window)

San Diego, CA
January 20, 2011

4

Registration Management Committee (RMC)

WHAT’S NEW?
• Greater Focus on evaluation of process effectiveness (use of the
Process Effectiveness Assessment Record form for each audited
product realization process)
• Use of a QMS Process Matrix form
• Use of Standard Audit Report forms
• Provides Special Audits criteria (in response to a
customer/interested party request, scope change, certification
transfer)
• Replaces existing versions of 9101, 9111 and 9121

San Diego, CA
January 20, 2011

5

Registration Management Committee (RMC)

What has not Changed
• Determining conformity to the standard's.
• Nonconformity’s discovered during the audit are to be
documented.
• Based on information collected during the audit, the lead auditor
will have to draw conclusions on conformity and effectiveness of
the organizations quality management system.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

Development Model for Process
Based Auditing - 9101
Process Approach

OER

Audit
Records

Process-Based Auditing

NCRs
Organization:

5

Site/OIN:

Controls

2

1

Auditing
Company Name
3

Nonconformity Report (NCR)
4

Audit Report ID/type:

4

NCR Number:

6
7

SECTION 1 - DETAILS OF NONCONFORMITY:

8

Process/Area/Department:

9

Requirement/Clause No.(s):

Methods/Tools

Auditing
Company
Logo

Effectiveness Assessment Record
1

2

Report Number:

3

4

Issued Date:

Organization:
Site:

□ Core Process Assessed:
□ Support Process Assessed:
□ Foundation Process Assessed:

5

Issue Date:

10

PEARs

Classification (ma/mi):

6

11

Statement of Nonconformity:

12

Objective Evidence:

13

Due Date:

14

Auditor
NAME:

9100/9110/9120 Section Reference:

Inputs

Outputs

15

Signature:

Auditee Representative Acknowledgement
Name:
Signature:

7

Support Process(es) Assessed:

8

Statement of Effectiveness:

□The assessed core/support/foundation process has been determined to be effective.
□The assessed core/support/foundation process has been determined to be ineffective.
9

How was process effectiveness determined (methodology)?

7

SECTION 2- AUDITEE PLANNED ACTIONS (Attach separate sheet, as needed.)

16

Containment Action(s), including correction, with supporting completion date(s):

19

Root Cause:

21

Corrective Action(s), including supporting completion date(s):

15

24

17

Planned
Completion Date:

18

Actual
Completion Date:

20

Cause Code:

22

Planned
Completion Date:

23

Actual
Completion Date:

Resources

10

Objective Evidence, Observed Conditions, Data, and Comments to Support Effectiveness
Determination

Auditee Representative
Date:
Name and Signature:
Auditor Signature for
Date:
Acceptance of C/A(s):
SECTION 3 - DETAILS OF AUDITOR VERIFICATION OF ACTION:

11

Assessment Team Leader

12

Signature and Date

7/25

7/26

SECTION 4 - NCR CLOSURE (Auditor Name
/Signature/Date):

27

Approved by Audit Team Leader
(Name/Signature/Date):

Signature indicates that all relevant requirements (including requirements for support processes identified in Block #7)
have been assessed to determine effectiveness of the core or support process identified in Block #5.

Performance Focused
Evidence of conformity
San Diego, CA
January 20, 2011

Evidence of effectiveness

Registration Management Committee (RMC)

What is the Effectiveness of an
Organization?
• Process not implemented; planned results are not achieved?

• Process implemented; planned results are not achieved and
appropriate actions not taken?

• Process implemented; planned results are not achieved, but
appropriate actions being taken?

• Process implemented; planned results are achieved?
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What is Effectiveness?
Extent to which planned activities are realized and planned results achieved
(Reference ISO 9000 Quality management systems – Fundamentals and
Vocabulary)

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to the Auditor?
Will use client and client customer performance data to plan audit
activities.
Review and examine an organization’s “Effectiveness” from:
• The customer viewpoint; Satisfaction and performance.
• Organizational leadership goals and objectives.

• Operational processes of the organization.
Construct and follow “audit trails”.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

What will this Mean to an Organization?
A QMS Organization should be able to…
• Demonstrate conformity of the QMS Organization to the standard.
• Show process mapping (identified, sequence and interaction).

• Explain process measures identified:
– Who reviews the data?
– Targets defined?
– Targets are met?

– Are the measures actionable?
– How is the data used?

• Show actions taken when product quality and on-time delivery
performance targets are not met.

The ultimate measure of a QMS Organizational effectiveness –
Customer Satisfaction
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

As an OP Assessor You will See Change in
the Auditor’s Planning, Conduct and Results
• Influences in the planning of the audit. (9101 4.2.1)
• Knowledge of the client processes, sequence and interaction.
(9101 4.1.2.5)
• Looking at organizational and process performance. (9101 4.2.2.1)

• Testing defined measures (targets defined and are they
actionable).
• Appropriate actions taken when defined measures are not met.
(9101 4.1.2.1)
• Development of audit trails based on data and performance (9101
4.1.2.4)
• Make up of the audit package. (9101 Appendices)
San Diego, CA
January 20, 2011

12

Registration Management Committee (RMC)

Exercise
• Witness Audit
• Auditor is evaluating metric/scorecard package for
Management Review. You (OP Assessor) observe a
scorecard for 3 programs from customer “X” reflects Red in
quality for each program for the past 3 months. Auditor
makes inquiry about frequency of Management Reviews,
minutes and action logs. No questions regarding customer
“X” asked at this moment in time. Thanks the MR and moves
on the next part of the audit plan. No further inquiry is made
throughout the course of the audit. Audit concluded with no
NCR’s.
• As an observer, what issue might be here?

San Diego, CA
January 20, 2011

13

Registration Management Committee (RMC)

The Gains
• Emphasizes process approach within organizations.
• Shifts energies from completing the clause based questionnaires
to determining and documenting conformity and effectiveness of
processes.
• Provides useful information to stakeholders concerning process
performance and effectiveness.

San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

9101 SUMMARY
• 9101D represents a significant advancement to aerospace
auditing. Although auditing for effectiveness has always been
an expectation of the standard and to customers, the reality is
that most auditors focus on process conformity to procedural
requirements overlooking process results.

• Customers have long complained about “certified” suppliers
that continually have poor performance issues (on-time and/or
quality). This standard addresses performance in a way that
has not been approached before.

• This change sets a stage for a new journey and as with any
journey, there will be challenges. We must stay on course.
San Diego, CA
January 20, 2011

Registration Management Committee (RMC)

QUESTIONS

?
San Diego, CA
January 20, 2011