Transcript Qualification Verification

SQA’s Approach to Quality
Assurance of Assessment
Matthew McCullagh
Quality Manager
Welcome
欢迎
Why are we here?
 To introduce you to SQA’s New Approach to
Quality Assurance
 To help you understand the key changes to
Quality Assurance process
 To help you become familiar with:
– The new Quality Assurance Criteria
– The new Confidence Statements
– How visits are planned, conducted and
reported
Thursday 6th September 2012
9.00
Coffee and Registration
9.30
Introduction, New Approach to QA Overview,
Aims and Objectives of Training Days
10.10
Preparation for Qualification Verification Activity,
Attitudes and Behaviours, KPMs
11.00
Break
11.15
Preparation for Qualification Verification Activity,
Attitudes and Behaviours
12.00
Planning for Qualification Verification Activity – the theory
12.30
Lunch
1.30
Conducting the Qualification Verification Visit – the theory
2.45
Break
3.00
Introducing the Quality Assurance Criteria
3.20
Quality Assurance Criteria Workshop
4.30
Close
Friday 7th September 2012
9.00
Coffee and Registration
9.30
Feedback: The Quality Assurance Criteria
11.15
Break
11.30
Decision Making
12.30
Lunch
1.30
Communicating Decisions
2.00
Theory of Completing Excel Reports/Action Plans
2.30
Break
2.45
Writing effective Verifier Reports
3.45
Case Study Assignment
4.00
Summary and Q & A
4.30
Close
SQA’s Approach to QA
 SQA is moving to introducing a new approach to Quality
Assurance of qualifications delivered in China
 It is based on SQA’s
Principles of Quality Assurance:
Open, fair and transparent
Aimed at risk reduction
Proportionate
Intelligence led
A shared responsibility
Constantly reviewed and improved
What is the new approach to
Quality Assurance
SQA now operates a transparent, intelligence-led, risk
based model for quality assurance of approved centres
and potential centres delivering our HND qualifications in
China.
Timeline of Implementation
of New Approach in China
September
2011
Overview
to Centre
staff in
Beijing
and
Xiamen
March
2012
Launch of
new
approach
May
2012
Overview to
EVs at
central
verification
events
September
2012
Specialist
training for
Centre
staff and
EVs
September
2012-2013
Implementation new
approach to QA
New Quality Criteria
 Five Categories:
–
–
–
–
–
Management of a centre
Resources
Candidate Support
Assessment & Verification
Records/Data Management
 All signposted to four QA processes
 All pre-rated as High/Medium/Low impact
 All supported by possible sources of evidence
The New Approach to Quality
Assurance: The Four Processes
 Systems Approval
 Qualification Approval
 Systems Verification
 Qualification Verification
High/Medium/Low Impact Criteria
 All criteria are important
 However, some are more important than others
 Those that are most important are rated as High
Impact criteria
 Those that are less important are rated as Low
Impact criteria
 In between these two are medium impact
criteria.
 Failure to meet a High Impact criteria will have a
greater impact on the outcome of the visit
Sufficiency of evidence
 No Systems Verifier or Qualification Verifier can
change the impact rating of a criteria.
 For each criterion you verify, you must decide
whether the centre has presented you with:
– Sufficient evidence to meet the criterion
– Some, but insufficient evidence to meet the
criterion
– No evidence to meet the criterion
Good Practice and Guidance
 As part of the verification visit, you should
identify good practice that is evident in the
centre.
 You should also give advice and guidance to
help centre staff improve the delivery of our
qualifications
 This is a mandatory part of your role as a
Qualification Verifier
Sufficiency of evidence
 We will support you and help you to understand
what sort of evidence centres can provide to
meet each criterion
 Once you have made your decision regarding
the sufficiency of evidence, SQA staff will then
calculate our level of confidence
 This will depend on the impact level of the
criteria and the sufficiency of the evidence
presented
Introduction of Confidence Statements
In relation to Qualification
Verification
High level of confidence in the maintenance of
SQA standards within this Verification Group
Broad confidence in the maintenance of SQA
standards within this Verification Group
Reasonable confidence in the maintenance of SQA
standards within this Verification Group, although
moderate risks exist within the following
categories:
(insert appropriate category)
(insert appropriate category)
Minimal confidence in the maintenance of SQA
standards within this Verification Group as
significant risks exist within the following
categories:
(insert appropriate category)
(insert appropriate category)
No confidence in the maintenance of SQA
standards within this Verification Group as severe
risks exist within the following categories:
(insert appropriate category)
(insert appropriate category)
Confidence
Statement
High Level of Confidence
Broad Confidence
Reasonable Confidence
Minimal Confidence
No Confidence
In relation to Systems Verification
High level of confidence in the systems that support the
maintenance of SQA standards within this centre
Broad confidence in the systems that support the maintenance
of SQA standards within this centre
Reasonable confidence in the systems that support the
maintenance of SQA standards within this centre, although
moderate risks exist within the following categories:
(insert appropriate category)
(insert appropriate category)
Minimal confidence in the systems that support the
maintenance of SQA standards within this centre as significant
risks exist within the following categories:
(insert appropriate category)
(insert appropriate category)
No confidence in the systems that support the maintenance of
SQA standards within this centre as severe risks exist within
the following categories:
(insert appropriate category)
(insert appropriate category)
How Confidence Statements are
calculated
 Confidence statements are given for the
outcome of each Systems Verification and
Qualification Verification visit for each of the
categories of criteria verified
 These are based on the sufficiency of the
evidence provided to the you
 A summary statement of confidence is also
given, based on the confidence statements for
each category
What if non-compliances are
identified?
 Where there is No evidence or insufficient
evidence to meet a criterion, you must agree
with centre staff, during the visit:
– the action the centre needs to take in order to
become compliant.
– The evidence they need to produce
– Where to send this evidence
– The date by which the action must be taken
What happens then?
 SQA will ask you to consider the evidence the
centre sends us and ask you to consider
whether it is Sufficient or insufficient.
 Depending on the sufficiency of the evidence,
the Confidence Statement will be re-calculated.
Sanctions
 If the outcome of a verification visit results in a
Confidence Statement of Reasonable, Minimal
or No Confidence, SQA staff may decide to
place a sanction on a centre until our confidence
increases to Broad or High.
 Our confidence may decrease if a centre fails to
meet action points agreed during a verification
visit.
 Sanctions will only be used where necessary.
Sanctions
In relation to Qualification
Verification
Confidence
Statement
In relation to Systems Verification
High Level of Confidence
Entry in Action Plan
Entry in Action Plan
Broad Confidence
Suspension of specific qualification certification:
by verification group
by qualification
Suspension of all existing qualification approval
Reasonable Confidence
Suspension of centre certification
Suspension of approval application by verification
group
Suspension of certification for all qualifications with assessed
components
Suspension of direct certification claim status:
by verification group
by qualification
Suspension of qualification approval application - whole
centre
Suspension of specific qualification approval:
by verification group
by qualification
Withdrawal of specific qualification approval:
by verification group
by qualification
Withdrawal of all existing specific qualification approval
Minimal Confidence
No Confidence
Withdrawal of centre approval
Benefits of new approach for
centres
 Open and transparent to all
 Clear guidance available to verifiers
 Proportionate response where issues are
identified
 Focused on Good Practice and improvement as
well as compliance
Questions
SQA’s Approach to Quality
Assurance of Assessment
David Pirnie
Lead Verifier
Welcome
欢迎
Qualification Verification
The Qualification Verifier role activities





Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report
Preparing for Qualification
Verification
The Qualification Verifier role activities





Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report
Being prepared
“The best preparation for good work
tomorrow is to do good work today”.
Elbert Hubbard
Preparing for Qualification
Verification
Before carrying out any activities, verifiers are
required to have a full understanding of their role
and responsibilities. This involves acquiring
knowledge and keeping it up to date.
Preparing for Qualification
Verification
Required knowledge – three key areas:
 Knowledge of your subject area
 Centre roles and centre responsibilities
 Qualification Verification
Preparing for Qualification
Verification
Knowledge of your subject area:
 Have a common interpretation of the Unit
Standards including the Evidence Requirements
for each unit you have been assigned to verify
 Know how to access Arrangements Documents,
Instruments of Assessment and Marking
Schemes to gain a full understanding of content
Preparing for Qualification
Verification
Knowledge of centres roles and responsibilities:
 Centre Co-ordinators
 Assessors
 Internal verifiers
Preparing for Qualification
Verification
Knowledge of the Qualification Verifier role:




Role Profile
Sources of support
Verification policies and procedures
Values
QIIPS:
Quality, Integrity, Innovation, Partnership
& Service
Sounds like:
“QUIPS” – a dictionary definition:
“A clever remark often prompted by the
occasion”.
SQA Values
 Quality
 Integrity
 Innovation
 Partnership
 Service
SQA Values
Quality
As custodians of the standards we verify, we
continually strive to maintain consistent quality in
their assessment.
SQA Values
Integrity
 We use open and honest communication with
internal and external customers (centres),
promoting transparency, and building trust with
others.
 We take a standardised approach to verification
as laid down by SQA and agreed by consensus
within our Qualification Verification team, even if
our personal opinions may at times conflict.
SQA Values
Innovation
 We view unfamiliar approaches to assessment
and verification with an open mind, whilst
ensuring quality is maintained.
 We believe that creativity should be seen not as
a threat to quality assurance but as an
opportunity for improving upon existing practice.
SQA Values
Partnership
 We understand that by working in partnership
with SQA centres, we will achieve common
goals of excellence and consistency in
assessment.
SQA Values
Service
 We maintain a professional approach with SQA
centres at all times, regardless of the challenges
we might face in carrying out our work.
Preparing for Qualification
Verification
More information on the preparation phase of
Qualification Verification can be found in the first
section of SQA’s New Approach to Quality
Assurance: Guidance on Visiting Verification For
HN Qualifications in China
Workshop
Values Exercise
Planning for Qualification
Verification
The Qualification Verifier role activities





Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report
Planning
“Failing to plan is planning to fail”.
Alan Lakein
Visit Planning: Sampling
SQA cannot always verify all HN Units, for all
candidates, on every visit to a centre, therefore a
sampling strategy has to be adopted.
Purpose of sampling:
 To establish that there is a common
interpretation of the standards within each
centre and that the assessment and verification
systems allow valid, reliable and fair assessment
decisions to be made.
Visit Planning: Applying a sampling
frame
SQA is responsible for systematically sampling on
a percentre basis:







Selected Units
Assessment decisions and practice
Methods/Instruments of assessment
Problematic/revised units
Locations where assessment takes place
Evidence of candidates across Units
Documented evidence of policies/procedures
(implementation)
Visit Planning
SQA (China Office) agrees optimum date (s) for
visiting verification. This is an agreement between
SQA, each centre and each QV
SQA selects Units to be verified and communicates
this to each centre (copying to SQA China Office)
SQA (China Office) confirms to each QV,
the centre(s) they should visit, the date of each visit,
and the Units they should verify
Qualification Verifier contacts centre to confirm visit
date(s) and travel arrangements
New Approach to Quality Assurance
Welcome Back
欢迎回来
Carry out Qualification Verification
The Qualification Verifier role activities





Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report
Carry out Qualification Verification
Discussion point
Consider the verification activities you perform
from the time of arrival until the end of the visit.
Contribute to the discussion by letting us know
what kind of techniques/approaches you have
used that you have found particularly beneficial.
Carry out the visit
Prepare yourself for the
visit
Agree agenda and running
order of visit
Conduct sampling activities
Judge the sufficiency of
evidence
Conduct sampling activities







Complete Units (all Outcomes complete)
Incomplete Units (some Outcomes complete)
Evidence not yet internally verified
Assessment/verification decisions
Assessment and verification practice
Interviews with assessors/verifiers/candidates
Referencing of evidence to standards
Introducing the Quality Criteria
Introducing the New Approach to Quality
Assurance Criteria
Introducing the Qualification Verification Guidance
Introducing the Quality Criteria
In your groups, discuss and familiarise yourselves with the criteria you have been allocated.
Formulate a list of summary points on flip chart paper to help you explain the purpose of each
criterion to your colleague verifiers. The summary should also include possible sources of
evidence.
In formulating your points, you are asked to use SQA’s New Approach to Quality Assurance:
Guidance on Visiting Verification for HND Qualifications in China as your information source.
Timings:
30 minutes discussion and research
20 minutes flip-charting your points
Please appoint a spokesperson(s) to present the flip-charted points which will be presented in the
first session tomorrow morning.
Each group will be assigned a facilitator to answer any questions/queries you may have.
Questions
The New Approach to Quality
Assurance
Welcome Back
欢迎回来
The New Approach to Quality
Assurance
Introducing the the new approach to
quality assurance criteria
feedback
Criteria 4.2
4.2
The centre must provide documented
evidence to ensure that assessments
are valid, reliable, equitable and fair.
Signed pre-delivery
checklist.
Standardisation
meeting minutes,
internal audit, review
records
Criteria 2.3
2.3
Records must be maintained to
provide evidence that the centre has
sufficient competent staff who have
the necessary qualifications,
occupational experience and
understanding to support the
assessment and internal verification
of qualifications being offered in the
centre.
CVs/staff info
sheets, CPD records,
copies of relevant
certificates, list of
current assessors
and internal
verifiers, changes to
the deployment of
assessors/internal
verifiers
Criteria 4.8
4.8
Outcomes of External quality
assurance must be disseminated to
appropriate staff and any action
points must be monitored against
agreed timescales.
Signed distribution
list, corrective action
log/report, minutes
of meetings
Criteria 3.2
3.2
Candidates' development needs and
prior achievements (where
appropriate) must be matched against
the requirements of the award. These
must then be related to agreed
personal action/training plans.
APL evidence, skills
profile, training
needs analysis,
assessment plans,
reviews
Criteria 4.6
4.6
The centre must comply with requests
for access to premises, records,
information, candidates and staff for
the purpose of external quality
assurance.
Documented
procedure for
handling QA visits,
roles and
responsibilities, site
selection checklists,
permission for SQA
QA representatives
to obtain access
Criteria 2.7
2.7
There must be evidence of initial and
on-going reviews of accommodation,
equipment and reference, learning
and assessment materials.
Current Health &
Safety certificates
itinerary checklists,
procurement
records, library
contents, system for
supporting eassessment
Records of review
Criteria 4.3
4.3
Evidence of candidates' work must be
accurately and consistently judged by
assessors against SQA's
requirements.
Assessor reports,
records of
achievement,
standardisation
meeting minutes, IV
reports, Qualification
Verification reports
Criteria 4.1
4.1
The centre's documented assessment
and verification procedures must be
implemented and the subsequent
assessment/verification practices
reviewed, the results of which must
be recorded and actioned.
Minutes of
assessor/internal
verifier meetings,
internal audit, review
records
Criteria 4.7
4.7
Candidate evidence must be retained
in line with SQA requirements.
Candidate evidence,
documented
retention policy
Criteria 4.5
4.5
The centre must take steps to ensure
that assessment evidence is the
candidate's own work.
Candidate
disclaimer,
Candidate and staff
induction materials:
assessor/IV roles
and responsibilities
Criteria 5.6
5.6
Comments/queries about the
qualification specification,
assessment guidance, qualification
verification or related SQA matters
must be resolved and recorded.
Correspondence file,
emails,
standardisation
meeting minutes,
notes of action
points and
resolution
Preparing for Qualification
Verification
The Qualification Verifier role activities





Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report
Making verification decisions
 Deciding on the sufficiency of evidence can be
likened to making an assessment decision
 Making an assessment decision takes account
of the overall strength of the evidence provided
 One source of evidence will be sufficient at
times. At other times, a number of sources will
have to be considered, dependent on each
centre’s approach to meeting quality assurance
requirements and to some extent the structure of
each criterion
One evidence source could be
sufficient in some cases for this
criterion:
Candidate’s development needs and prior achievements (where appropriate) must
be matched against the requirements of the award. These must then be related to
an agreed personal action/training plans.
The following are examples only, and do not constitute a
complete list of all possible types of evidence to support the
above criterion.
APL/RPL/APA evidence
Initial/diagnostic/formative assessment, skills profiles, training needs
analyses
Personal development/action plans, competence reviews, learner
agreement/contracts, assessment plans
More than one evidence source is
likely to be considered for this
criterion:
The documented assessment and verification procedures must be implemented
and the subsequent assessment/verification practices reviewed, the results of
which must be recorded and actioned.
The following are examples only, and do not constitute a complete list
of all possible types of evidence to support the above criterion.
Records of assessment
Records of internal verification/audits, monitoring of assessor/verifier
practice
Reviews of assessment and verification practices
The main guiding principles for
making verification decisions
Decisions should be based:
 on the sufficiency of evidence relating to each
criterion
 only on the criteria provided
The evidence examples given in the guidance
document are just that. They are not a shopping
list of evidence sources that all centres must have.
Awareness of things that could
influence on our decisions
Own
experience
Own
standard
s
Our
confide
Previous
centre
perform
-ance
-ence
Attitude of
centre
staff
Own
mindset
(closed)
Insufficient/no evidence = action
points
 Sufficient evidence - this means that the centre has provided
evidence that fully meets the criterion (no action points required)
 No evidence – this means that the centre has not provided any
evidence in support of the criterion (action points required)
 Insufficient evidence – this means the centre can provide some
evidence in support of the criterion (action points required)
Agreeing action points
Is there anything wrong with this action point?
“You should submit evidence of having
disseminated the results of Qualification
verification visits to relevant staff”
Action points should be SMART
 Specific: Centres should be clear in terms of the action
they need to take to close-off the action point.
 Measurable: How will SQA or the centre know that the
action point has been met, what will be the measure of
success?
 Achievable: The centre must have sufficient time in
which to achieve the agreed action. Please factor in time
it takes for SQA to receive, process and edit the report
and send it to the centre.
 Relevant: The action must directly relate to the criterion
 Time bound: Agree a specific and realistic.
date for the action to be completed by.
Questions
New Approach to Quality Assurance
Welcome Back
欢迎回来
The stages of Qualification
Verification





Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report
The stages of communicating
verification decisions
Plan to deliver balanced
feedback
Explain your verification
decisions
Plan to deliver balanced feedback
Plan to include:
 Good practice
 Recommendations
 Action points (where appropriate)
Consider:
 Location
 Order of feedback and emphasis
 Words and tone
Explaining your verification
decisions
Where action points are given, ensure you:
 have explored all possible evidence sources
 gather all information to support your decisions
 can justify your decisions - clearly relating them
to criteria requirements
 Give the centre an opportunity to ask questions and
clarify points
Communicating Decisions
Role play exercise:
David as a Qualification Verifier feeding back at the end of a
Qualification Verification visit to Matthew who is the SQA Coordinator.
Please note your observations of the following on your
observation sheet:
 The content of the Feedback
 Attitudes and behaviours of the Qualification Verifier and
the SQA Co-ordinator
 Any possible improvements
The stages of Qualification
Verification





Prepare for verification
Plan verification
Carry out verification
Communicate verification decision
Complete verification report
The stages of completing the
Qualification Verification Report
Ensure report content matches your
feedback
Justify your verification decisions
Write clear action points
Check report prior to
submitting to SQA
The Qualification Verification Visit
Report
The Criteria Pages
The Required Action Log
The Summary Worksheet
Recommendations and Good
Practice Worksheet
Writing Effective Verifier Reports
 SMART action points
 Report writing guidance available
– www.sqa.org.uk/files ccc/Writing for SQA part
A.pdf
 Write report as soon after visit as possible
 Checking content thoroughly
Assignment: Report Writing
Assignment will be emailed to you by Thursday 20th
September.
You will be asked to:
 Review a Case Study
 Complete a Qualification Verification Visit Report
 Return to [email protected] by Friday 12th October
Assignment: Report Writing
SQA will provide you with:
 Guidance document on how to complete a
qualification verification visit report
 Feedback on each report to provide support
byFriday 26th October 2012
Summary
 To introduce you to SQA’s New Approach to
Quality Assurance
 To help you understand the key changes to
Quality Assurance process
 To help you become familiar with:
– The new Quality Assurance Criteria
– The new Confidence Statements
– How visits are planned, conducted and
reported
Questions