The National Cancer Peer Review Programme ‘Internal Validation (IV)’

Welcome and Introductions

Aims of Today

To promote an understanding of the Internal Validation process within the National Cancer Peer Review Programme To enable Trust and Network Managers to implement the Internal Validation process with confidence To enable Trust and Network Managers to cascade Internal Validation process to others within their organisations

Learning Outcomes

• Understanding of the Internal Validation purpose • Understand the new earned autonomy in relation to Internal Validation • Understand the key stages of the Internal Validation process • Have knowledge of different models for Internal Validation • Have confidence to complete an Internal Validation report • Have confidence to train others in their organisations and know where to access support and advice

Session 2 The National Cancer Peer Review Programme

Why have we Changed?

Lighter touch approach Greater focus on internal quality assurance Responsiveness to NHS changes Continued emphasis on clinically-led peer review Better use of resources A greater emphasis on outcomes

The NCPR Process

Peer Review Visits

Targeted

Externally Verified Self Assessments

Sampled

Validated Self-Assessments

(annual)

All Teams

Internally Validated Self Assessments

• Completion of an annual self assessment by the team/service who deliver the particular cancer service. • Internal validation of the self assessment should be undertaken by the host organisation or coordinating body for that service. • It is not mandatory to internally validate a service which is subject to a peer review visit but would be seen as good practice.

External Verified Self Assessments

• An external check of selected internally validated self assessments led by the zonal cancer peer review coordinating teams. • This check will take the form of a desktop exercise. • The schedule for which topics are to be externally validated each year can be found on the CQuINS website. • This process will ensure that every team/service will be externally verified at least once every five years.

Peer Review Visits

• The schedule of peer review visits will be agreed with each cancer network by the end of December.

• The teams/services selected for a peer review visit will be informed by the 31 st December each year. • Each visiting cycle will commence in May and be completed by the following March.

The National Schedule

April-Aug Self Assessment By End of Sept Internal Validation of self-assessments By End of Nov External Verification 31 Dec Notification of Visits Jan Preparation for Visits May-March Review Visits April-Aug Self Assessment By End of Sept Internal Validation of self-assessments By End of Nov External Verification

Schedule 2010/2011

Schedule 2010/2011

Session 3 Earned Autonomy in Relation to IV

Delivery Specification Guides Delivery Specification Guides are available on the CQuINS resource page

Earned Autonomy

• In recognition that some teams/services have achieved a good standard of internal quality assurance governance and in line with the cancer reform strategy to reduce the burden of inspection, the concept of ‘Earned Autonomy’ will be introduced for the IV process during 2010.

• A team/service who have earned autonomy do not have to be subject to IV.

Teams are still required to perform an annual self assessment and identify immediate risks, serious concerns, concerns and good practice.

The Criteria for Earned Autonomy in Relation to Peer Review

1. The team/service must have received a peer review visit report in the previous year indicating that the compliance against the measures was 75% or greater and have no immediate risks or serious concerns; and 2. The teams/services compliance against the measures in the previous year and the current year must be greater than 75%; and 3. The teams/services compliance against the measures should be no less than the previous year and there should be no immediate risks or serious concerns identified within the self assessment.

The Criteria for Earned Autonomy in Relation to External Verification

1.The team/service must have received an External Verification (EV) report in the previous year indicating that the IV report was ‘confirmed’ i.e. Green; and 2.The teams/services compliance against the measures in the previous year and the current year must be greater than 75%; and 3.The teams/services compliance against the measures should be no less than the previous year and there should be no immediate risks or serious concerns identified within the self assessment.

Session 4 Clinical Lines of Enquiry (CLE)

Pilot for Breast and Lung service 2010/2011

Details of Clinical lines of Enquiry: • Identification of clinical indicators • Data • Clinical Lines of enquiry (See the delivery Specification guide on CQuINS and evidence guides).

Identification of Clinical Indicators

• Discussion with SSCRG Leads, NCIN and Medical Directors • Outcome of the work has been to develop clinical indicators for Breast and Lung • It is the intention to feedback and review the Clinical Indicators at the NCIN Site Specific Clinical Reference Groups on an annual basis.

Data

• Where National data is available this will be provided to both the review teams and MDTs. • A proforma will be provided for local data, which should be returned to the Zonal Teams and the National Cancer Peer Review Senior Information Analyst by the end of the Self-Assessment period.

• The proforma’s for Breast MDT and NSSG Local data are available in the Delivery specification on CQuINS.

Clinical Lines of Enquiry

• A briefing sheet on the relevance of these clinical indicators. • This will structure and ensure consistency of the discussions about the data on a Peer Review visit.

• A commentary on the clinical lines of enquiry should be included in the PR and IV Report.

CLE in reports

• Question 3 for an NSSG: ‘Can you demonstrate that you have effective processes in place for evaluating services across the network and identifying priorities for improvement?’; and • Question 4 for MDTs ‘Can you demonstrate how you are continuously improving your service ?’

Session 5 IV: Purpose, Schedule and Timetabling

Purpose of Internal Validation

• To ensure accountability for the self assessment within organisations and to provide a level of internal assurance; • To develop a process whereby internal governance rather than external peer review is the catalyst for change; that the organisation is using the self assessments for its own assurance purposes; • To confirm that, to the best of the organisation’s knowledge, the assessments are accurate and therefore fit for publication and sharing with stakeholders; • To identify areas of good practice that could be shared.

Topics for IV in 2010

Who is subject to IV?

• IV to be completed and uploaded onto CQuINS by the end of September –

All teams who have undertaken self assessment as per the National schedule except teams due for peer review visit or teams who have earned autonomy

• IV not required –

Teams due for peer review visit

NB some organisations may choose to IV these teams and this would be seen as good practice

Outcomes of Self Assessment & Internal Validation

• Self assessment – Compliance against all measures – Answer to 4 key questions – Comparable data and report on CQuINS – Ownership of above by team • Internal validation – Quality assurance of self assessment – Revised compliance against all measures – Organisational assessment of team against 4 key questions – Ownership of process by organisation – Identification of risks & concerns

Session 6 Models of IV

Validation – What we Expect

The process is agreed within the organisation The process adopted has agreement with the commissioners within the locality and the cancer network Accountability for the self assessments is confirmed by agreement of the chief executive of the organisation There is commissioner and patient / carer involvement within the process The process and outcome of the validation is reported on the nationally agreed proforma.

Validation – Suggested Approaches

Desk-Top Review

Small panel review and validate assessment

Panel Review

Small panel review assessment Meet with representatives of the MDT/NSSG to discuss key issues and finalise validation

Earned Autonomy

Completion of the earned autonomy form IF MEET ALL CRITERIA

Suggested IV Panel Membership

• Cancer services manager • Lead cancer clinician and or Lead cancer nurse • Local commissioner and or Network manager • Patient or carer

Advice on Involving Patient/Carers

They should be nominated by either the Network or Locality User Group They should not be involved in validation of an MDT that has provided their care or treatment They should not be in current treatment and normally be at least 2 years post initial diagnosis/treatment They must be supported in clearly understanding what is being asked of them

Who Validates?

Service

MDT Cross Cutting Service Locality Group NSSG Network Cross Cutting Group Network Board PCTs

Responsibility for Validation

Host Trust Host Trust Host Network Management Team Host Network Management Team Host Network Management Team External verification by zonal cancer peer review coordinating team Host Network Management Team

Session 7 Process of IV

Planning and Effective Scheduling of IV

• Location - suitable room, IT facilities if required • Selection of the panel – ensuring availability on the day • Training – use of existing reviewers, use materials on CQuINS • Ensure the data for Clinical Lines of Enquiry is available • Do what can been done before the meeting review of the evidence, ensure all other evidence is available.

A Review of the Self Assessment and Supporting Evidence

Either before or during the IV panel meeting. – Check the compliance of each individual measure and record any changes.

– Complete the assessment on CQuINS – Identify any risks or concerns – Note any evidence seen and not uploaded on CQuINS either on the assessment spreadsheet or on the IV report form Discussion at the IV panel should confirm any unresolved issues.

Completion of the IV Report Form

After the panel meeting the IV report form should be completed • A full commentary should be included against the key questions • This should correlate with the measures compliance.

• For Breast / Lung team IV report forms should include comment on the Clinical Lines of Enquiry (CLE)

Delivery Specification Guides and Evidence Guides

Dealing with Risks, Concerns and Good Practice

The IV report form should clearly identify any: –

Immediate risks

– An issue that is likely to result in harm and requires immediate action.

– –

Serious Concerns

– An issue that could compromise the quality or outcome of patient care

Concerns

– An issue that affects the delivery or quality of the service –

Good Practice

– relates to the service and can be either innovative, common practice undertaken very well or significant achievements'.

Evidence must support these and details of actions to resolve them should be included .

Details of the IV Process

• Details of the how the IV was undertaken should be included on the report form.

• Details of who attended the IV should be clearly stated on the form along with the role they fulfil e.g. Patient, Commissioner, Lead Clinician

Feedback to Teams/Services

• The IV panel should give initial feedback at the panel, and • The IV report should be discussed with the team / service reviewed

Sign Off and Uploading the IV Report Form

• Prior to publication the report can not be viewed on CQuINS • Check to ensure it is suitable for publication • The final IV report form should have the agreement of the chief executive • Finally publish on to the CQuINS website

Process of IV Exercise 1

Exercise 1

In groups discuss your experiences of the IV process during 2009.

Identify 2 examples of what went well and what you would change next time

Session 8 Completing IV Reports

Validation – The Process

Agreed Validation Process takes place Further clarification may be sought on some issues / opportunity of re-submission of specific evidence Validation proforma agreed Validated compliance recorded on CQuINS Validation proforma uploaded

The Validation Proforma

• Will be a public document • Will form basis of Annual Peer Review Report for those teams not subject to external review • Handbook contains guidance on identifying Immediate Risks, Serious Concerns and Concerns

Validation – Key Tips:

Remain objective Have you got the evidence?

Difficult if you are too close to the team Be wary of making quick assumptions based on local knowledge Don’t be confrontational Maintain developmental feel Teams can resubmit Phased Approach Always praise Will take time to ‘perfect’ Everyone needs to be clear about their role PROVIDE FEEDBACK

Completing IV reports Exercise 2

Exercise 2

Information available: • 2 example reports (for you to critique) • Supportive information – Operational Policy – Workplan – Annual Report – Spreadsheet of compliance • Blank IV report for you to complete

Recording Internally Validated Assessment on CQuINS

Session 9 Purpose of External Verification

External Verification – The Purpose

Verify that self-assessments are accurate Check consistency across organisations Ensure that a robust process of self-assessment and internal validation has taken place Provide a report on performance against the measures and associates issues relating to IOG implementation Identify teams or services who will receive an external peer review visit in accordance with the selection criteria.

When does External Verification Take Place?

• Nationally selected topics each year within which all teams self-assessments will be externally verified • In addition, random selection • Every team will be externally verified at least once every 5 years

External Verification – The Process

Desk top review of validated assessment undertaken by Zonal Quality Team Review of accuracy of self-assessment Zonal Team may request further information Zonal Team will have access to specialist clinical input and patient/carer input

External Verification – The Outcome

Signed off by Quality Director and or Clinical Lead If organisation unhappy with outcome there will be the opportunity for dialogue with a view to finding a solution External Verification will indicate if the IV is confirmed (Green), confirmed with exceptions (Amber) or if the IV is unconfirmed (Red).

National proforma uploaded to CQuINS / published

Annual Meeting with Network

• December each year • The purpose of the meeting will be to; – inform the Zonal Team of key issues within the Network such as implementation of Improving Outcomes Guidance, Service Configuration changes – agree the teams to be visited and schedule for the following year.

Selecting Teams for a Peer Review Visit

New Teams/services or where new measures have been published Milestones not met for implementation of an IOG as agreed with CAT Immediate Risks identified at previous peer review visits that have not yet been resolved Requests from organisations i.e. SHAs, local and specialist commissioners, PCTs, Networks, Acute Trusts % compliance with measures within lowest performance grouping Concerns regarding rigor of Internal Validation Stratified random sample based on % compliance (if available capacity) Revisits

Session 10

Training & support for review panels

Ensure that members of the IV panel are Informed and appropriately trained for the IV process: • The training materials from today, as well as reviewer training and user training are all available on CQuINS.

• Why not send IV panel members on NCPR reviewer training?

• A List of trained reviewers is available from the Zonal Team • Ask the Zonal Team is they can help with Training

Support

• Consider what support panel members need on IV. • A team leader / chair should guide them through the process – This could be a cancer managers or network manager.

• User should be reimbursed their out of pocket expenses.

Session 11 Close and Evaluation

Aims of Today

To promote an understanding of the Internal Validation process within the National Cancer Peer Review Programme To enable Trust and Network Managers to implement the Internal Validation process with confidence To enable Trust and Network Managers to cascade Internal Validation process to others within their organisations

Close

Thank You

Please Complete your Evaluation form Have a Safe Journey Home....