Transcript Slide 1

The ISEAL Assurance Code
ISEAL Conference June 10 2011
Patrick Mallet, ISEAL Credibility Director
Paddy Doherty, ISEAL Code Development Manager
Initiating the Consultation
• Scope agreed by Stakeholder Council
– Based on 4 month consultation process
– Prioritization of issues through survey, key person interviews
• Background Research
– Carried out by Richard Bradley – ‘quotes throughout’
– Background information and issues to consider
• Consultation Process
– Steering and Technical Committees will meet later in June
– Your first opportunity to provide input to Code content
– First draft by September, approval in June, 2012
Consultation Findings – In Brief
Identify the top 4 challenges to credible assurance
Auditor competence
Resource constraints
Governance
Standard quality
Product traceability
Transparency
Managing large producer groups
Other
Certification decision making
Access for small CBs
0
10
20
30
40
% choice
50
60
70
Consultation Findings – In Brief
Which of these challenges could be addressed by the
Assurance Code?
Auditor competence
Governance
Transparency
Standard quality
Managing large producer groups
Certification decision making
Resource constraints
Product traceability
Other
Access for small CBs
0
10
20
30
40
% choice
50
60
70
Proposed Scope
• Include some or all of the following issues
– Auditor Competence - screening, training, qualification, calibration
and monitoring
– Audit implementation – minimum requirements for good practice +
guidance notes to ISO 17065, 17021
– Transparency – additional requirements (beyond ISO) where needed –
‘transparency can reduce the need for excessive rigour’
– Standard quality – consistent interpretation of standards
– Accessibility – deals with the challenges of cost and access and will
include innovative options
• Complementary to ISO standards (17011, 17065, 17021)
• Requirements apply to scheme-owner; & CBs and ABs where
appropriate
Issues to consider for the Assurance Code
• Some standards schemes require compliance (and are
satisfied) with ISO standards while others do not
• One-size does not fit all – defining a ‘cascade’ of assurance
options appropriate for the scheme and its stage of
development
• ISO standards are good at management systems for
consistency, competency, and impartiality, but do not cover
the ‘soft’ issues important to ISEAL members
• Who is responsible for which activity is not always clear eg:
training auditors, monitoring CBs
• Certification can deliver additional benefits besides the
‘assurance’ (thus changing the cost/benefit ratio)
Findings from the Research
• Interviewees for this project identified common issues
– Few could define a minimum level of certainty that they wanted
an assurance program to deliver.
– All agreed that some form of risk assessment was required in a
sampling program to focus audit attention on higher risk
activity.
– While many used a formula to determine sampling numbers
(e.g. 10% or square root), none had a statistical explanation as
to why they used that formula.
– Most used judgmental sampling rather than statistical sampling.
– The cost of each program having its own different audit
management systems was significant, and there was interest in
how collaboration could take place.
More Findings from the Research
• “ISEAL members are taking innovative approaches which
could be used as models to develop materials for the
Assurance Code – especially if a ‘cascade’ of verification
requirements was developed”
• “Most schemes do not have measures of overall performance,
and do not understand sampling or risk assessment processes
well”
• “The role of technology in certification is increasing, and the
pace of change is accelerating. The Assurance Code needs to
be able to accommodate these changes, and could be used to
hasten or restrict them”
External Trends and Advances
• Strong trend amongst other programs (e.g. ISO, food safety) to place
increased emphasis on personnel competency and potential for personnel
certification.
• IFOAM’s participatory guarantee system encourages self and/or peer
assessment. Successful when local stakeholders are fully involved .
• GlobalG.A.P. operates a certification integrity program, in which staff
repeat both accreditation and certification audits to compare results –
checks if outcomes achieved and calibrates accreditors and certifiers.
• European product conformity system (the CE mark) has a number of levels
of possible assurance, to which products assigned due to risks of failure.
• Audit technology is rapidly developing, and the Assurance Code should
take those changes into account.
Choice of Assurance Models
Level of Confidence
• What level of risk is acceptable?
‘Traditional’
certification
system
Personnel
certification
only
Self
declaration
Traditional
system with
personnel
certification
Self declaration with
3rd party verification
by random sample
Self declaration
with 2nd party
verification
Lowest
Low middle
Cost
high middle
highest
Risk-based Approaches
• Certification as a risk management programme
• Audit risk is the risk that the audit will not provide an accurate
conclusion as to client conformity
• Expressed by multiplying three factors:
– Control risk – the risk that the client does not know that their
system is non-conforming
– Inherent risk – risks associated with the client, the industry or
culture
– Detection risk – the risk that the audit will miss non-conformities
if they exist
Sampling
• Sampling is inherent in certification but may not be explicit
• Sampling used in choice of who to audit, how frequently and
what to audit – focusing auditing on higher risk activity
• Most systems use judgmental or non-statistical sampling
– Limitations on conclusions that can be drawn
• Sampling within an audit can be performed in differing ways:
– Representative, at random (acceptance sampling)
– Focused on finding problems to be corrected (corrective sampling)
– Sampling the important issues to protect scheme (protective
sampling)
– Preventing client from predicting sample, thereby lowering audit
risk (preventive sampling)
Risk and Sampling Options
• Many programs require CBs to “perform a risk analysis”, with
little or no instruction as to how this should be performed or
what evidence of analysis is required
• The Assurance Code could set out a standardised risk
assessment program to be followed by certification scheme
owners and by CBs. This could include methods for identifying
hazards and risk analysis, and may include sections on
identification and selection of risk controls, and on monitoring
of effectiveness.
• Should the Assurance Code define how a risk assessment is to
be performed and assign responsibility for performing it?
Audit Performance
• Audit performance, and hence the credibility of assurance, is
the sum of CB management and auditor competency. As
many have commented, our approach to auditor competency
is weak – our concentration is on CB systems. Perhaps a
rebalancing is required?
Now
Competent
systems
Competent
personnel
Competent
systems
Competent
personnel
Later
Auditor Competence
• Personnel competencies:
– Can describe qualifications required, or
– Can describe what an individual must be able to do (outcomes)
• Latter approach is recognised as being more reliable and is
being more widely adopted
• Those evaluating personnel competency can follow ISO17024,
a standard for personnel certification bodies
• Increasing numbers of schemes are using established
certifiers such as IRCA or RABQSA for this purpose
• Possible benefits for ISEAL members to adopt a cohesive
approach to certification of personnel to avoid duplication,
allow people to work across programmes and reduce costs
Auditor Competence Options
• As well as setting competency requirements, an Assurance
Code could consider minimum requirements for auditor
experience as an auditor, and for auditor supervision and
continuing professional development
• If the Assurance Code includes personnel certification
requirements, other system requirements may be able to be
lessened
• An Assurance Code could set out a generic process for
competency evaluation. If it did so, it should consider basing
processes on ISO 17024 requirements
• It may be efficient to have a central (common) registration /
accreditation programme for auditors
Audit Implementation Options
• Audit software – auditors use templates that ask questions
based on inputted information (RA Tourism is pleased with
this approach)
• Common requirements for audit systems: software, reporting
frameworks
• Common methodology for risk-assessment & sampling
• Certification scheme owners should consider their strategic
objectives before deciding on which sampling strategy to
follow during audits. An Assurance Code could set out
examples of sampling strategies to be followed for differing
types of objectives
Audit Technologies
• Audit technologies have been developing rapidly, enabling the
following
– Workflows built into software - checklists change based on
responses
– Options to select descriptions of how the client achieving
conformity, beyond yes / no
– Logic rules ensure complete audits and identify inconsistencies
– Information on risk used to change audit frequency or intensity
– New reporting tools, combined with faster hardware, increase
ability to extract information from data
– Operating costs and response times are lowered
– Use of mobile phones for data transfer allows relatively low
cost, almost ubiquitous access
Accessibility
• A cascade of increasing verification requirements could have
appeal. The Assurance Code could describe the verification
requirements needed at each level within the cascade
• eg: depending on x, you are required to:
- Comply with ISO Standards; or,
- Second-party certification combined with selective auditing or,
- Self declaration combined with peer review and risk-based
sampled third-party audits; or
- Another level of assurance (eg: certification of persons)
• Risk assessment to reduce frequency of audits – select the
control option that gives best control at reasonable cost
Costs and Accessibility
• Simplistic financial model for a CB with 300 certificates
• Assumes surplus remains constant at 5%
Cost reduced or size increased
Accreditation fees
Auditor salaries, audit time or number
of audits
Drop both accreditation fees and
auditor salaries, audit time or number of
audits
Increase CAB size
Reduction by
50%
100%
20%
50%
20%
40%
Fee drops by
5.25%
10.5%
7%
17.5%
9%
18.25%
50%
100%
400%
5.5%
17%
25.75%
Costs and Accessibility
• Implications of the model:
• To create a 10% drop in fees charged to clients, one of the
following would be needed:
– Accreditation fees would need to drop to zero
– Audit salaries, frequency or duration would need to drop by
30%
– Accreditation fees and audit salary costs or audit frequency or
audit duration (or a combination of the last three) would all
need to drop by 23%
– CB volume would need to increase by 65%
• Simplest method of lowering fees with no impact on
credibility is increased throughput in each CB
Standard quality options:
• Good practice in crafting standards that provide for consistent
interpretation
• Requirements for guidance and support to auditors to ensure
consistent application of the standard
Transparency
• Alternate assurance systems should include requirements for
transparency (beyond what ISO standards require)
– Public client list
– Public list of de-certifications
• Certification scheme owners using 17065 as a base for their
programs could consider aligning how they present CB
requirements to match the layout and format of 17065
• Current accreditation processes focus on a limited number of
issues related to systems, competency and organisational
behaviours. The Assurance Code may need to consider
whether it should widen the AB’s brief to include whether or
not strategic objectives, including outcomes, are met
Issues to Resolve
• The greatest challenge will be the discussion of “how sure do we
want to be?” – once this is known, choice of assurance models
becomes easier
• Some stakeholders are demanding more rigour in assurance while
others feel the costs outweigh the benefits – how to reconcile?
• What is the balance in the Code outputs between requirements and
guidance?
• Technology and knowledge could be combined to have a scheme run
without traditional CBs (certification of auditors)
• Guidance for capacity-building (delivery of knowledge) in the audit
(adding value to assurance to change the cost/benefit ratio)