Transcript national audit of internal quality control practice.
FOCUS 2011
NATIONAL AUDIT OF INTERNAL QUALITY CONTROL PRACTICE in UK.
David Housley. Luton and Dunstable NHS Trust.
Teresa Teal. National ACB Audit Group.
INTRODUCTION • A previous audit of IQC within the Thames Region in 2006 showed significant variation in practice.
• 29 returns were received.
• Some areas of very poor performance were identified.
• 9 audit standards were developed and published 2008. Housley D G
et al
; Ann Clin Biochem; 45: 135 –139.
• AIM: To re-audit IQC practice nationally and compare to previous audit data from a single area.
METHODS • Questions based on the previous audit, with minor modifications to improve clarity, facilitate electronic data gathering and to increase focus on audit standards. • Questionnaire designed using SurveyMonkey TM • Distributed by e mail in March 2011 via ACB office with a hyperlink to the questionnaire. • Circulated to one individual in each UK hospital.
• Respondents were instructed to restrict answers to routine chemistry systems only (not endocrine or esoteric).
Q. 1 & 2 • 86 responses received.
• ANALYTICAL SYSTEMS IN USE; – 41% Roche – 16% – 8% – 14% – 13% – 8% Beckman - Coulter * Olympus * Abbott Siemens Ortho • *note change of ownership since previous audit.
Q.3
• How frequently is internal quality control material run through your analyser (assume standard core hours (weekday) with no analytical, technical or procedural problems that require extra analysis)? • Please answer for serum / plasma assays only.
• 53% • 64% • 6% At set time intervals At fixed times of the day After a set number of samples or tests – Responses ranged from hourly through to 24 hourly. – Most common responses ranged from 4 to 8 hourly.
86 answered question
Q.4
• Please state what your lab considers to be its core working hours and whether you operate a shift system or on-call system outside of this period?
• Core working hours of labs are variable but most common answers approximated too: – 62% 09:00 – 17:30 – 26% – 3% – 5% – 4% 08:00 – 20:00 08:00 – 18:00 07:00 – 22:00 24/7 shift 85 answered question
Q.5
• Does frequency of IQC outside of weekday core hours differ to the frequency within core hours?
24.5% YES
75.5% NO (21% previously) Mostly this is a reduction in the frequency of IQC, but significant evidence of
discretionary
testing still remains.
Q. 6 • Does your laboratory have a policy for IQC practice outside of weekday core hours ?
• – YES = 58% (30% previously)
NO = 42%
86 answered question
Q.7
• According to your laboratory sample throughput, what is longest run length (number of patient specimens processed between IQC assessment) for a plasma / serum sodium measurement ?
600 400 200 0 1400 1200 1000 800 Roche Beckman Abbott Olympus Siemens Ortho 9000 8000 7000 6000 5000 4000 3000 2000 1000 0 Roche Beckman Abbott Olympus Siemens Ortho 59 answered question
Q.8
• Does your laboratory preferentially utilise IQC material from a third party source if it is available ?
• YES = 85% • NO = 15% • Unchanged from previous regional audit.
85 answered question
Q.9
• Which of the following are used routinely (every IQC assessment)?
• 44% • 27% • 61% Unassayed (no values assigned) material Assayed (quoted values used) Assayed (values assigned by lab) If you assign your own values, how many measurements are made to derive the target and SD ?
Replicates Number of labs 100 3 80 1 50 3 40 1 30 10 25 1 20 23 10 6
Q.10
• How are the target concentrations for IQC material chosen ? – 60% – 81% – 86% What is available on the market Clinically important decision levels To cover low, normal and high values – 42% – 35% Detection limits of assay To assess areas of poor assay performance – 5% Other reasons – custom made material, stability, lot size, cost.
85 answered question
Q.11
• Are multi-rules (e.g. Westgard rules) used as the primary form of QC data evaluation for accepting / rejecting QC ?
YES = 60.5% NO = 39.5% Free text replies showed a lot of confusion .
Very similar percentages to previous regional audit.
57 answered question
Q.12
• If you do use multi-rules (e.g. Westgard rules), do you apply the same rules to all analytes, or do you use method (analyte) specific rules ? • 89.5% Same for all • 10.5% Method specific • Free text replies showed a lot of confusion.
Q.13
•
Do you use single 2 SD rules ?
• 55% • 45% YES NO (76% in previous regional audit) • Fewer laboratories using single 2SD rules.
• Free text replies varied from ‘all analytes’ to single analytes, most commonly sodium / ISE.
85 answered question
Q.14
• How are your control limits set ?
– 92% Data generated within laboratory – 46% Cumulative data – 31% Manufacturer derived ranges – 11% Peer group or EQA data – Varies with different analytes.
Q.15
•
Do you consider the IQC data on your main chemistry analyser easy to access ?
• 84% • 16% YES NO 86 answered question
Q.16
•
What is the typical response when IQC data suggests a method is out of control (e.g. failed QC) ?
– 67% – 73% – 87% –
6%
Repeat using existing control material Re-run using new control material Re-calibrate then re-run control Over-ride then continue with patient samples – Free text comments from all.
– Would depend on whether 2SD being used as a warning or multi rules are violated.
– Question could have been more specific.
• Q.17
What grade / type of staff make decisions about whether to accept or reject IQC data in real time ?
6% 5% 95% 96% 82% 44% 26% 20% 36% 32% Trainee / MTO Qualified (not registered) BMS 1 BMS 2 BMS 3 BMS 4 Locum BMS Quality manager Clinical Scientist Consultant / Specialist registrar Very similar results to previous regional audit.
85 answered question
• Q.18
If IQC data showed that a method was out of control (i.e. QC had failed), what grades of staff would make the decision to continue processing patient samples ?
18% WOULD NEVER HAPPEN IN MY LAB 0% 1% 15% 61% 65% 40% 3.5% 20% 49% 50% MTO / Qualified (not registered) Trainee BMS 1 BMS 2 BMS 3 BMS 4 Locum BMS Quality manager Clinical Scientist Consultant / Specialist registrar 85 answered question
Q.19
•
How often is out of control (non-ideal) IQC accepted (e.g. in order to ensure work is completed) ?
– 7% – 7% – 2% – 55% – 26% – 5% Daily Weekly Monthly Rarely Never Other – Other reasons were ‘known problem’; ‘reagent or calibrator change’; ‘depends on clinical significance’; ‘only for non-critical tests such as protein or bilirubin’; ‘we know we get fliers’.
84 answered question
Q.20
•
What grade of staff monitors overall IQC performance ?
– 29% Quality manager – 86% Senior in charge of section – 53% Clinical scientist – 61% BMS3 or BMS 4 – 26% Consultant 85 answered question
Q.21
•
Do you formally compare performance between multiple analysers (same site or across sites)?
• 71% • 29% YES NO • Very similar to previous regional audit.
• Very wide range of statistical processes in use.
84 answered question
Q.22
•
If your service is split between geographically distant sites (e.g. different hospitals) are there barriers that prevent IQC performance being compared for the same type of analysers located on the different sites ?
• 20% • 80% YES (34% in previous regional audit) NO • Barriers include lack of time, poor IT, staff pressures, time pressures, poor cooperation. 60 answered question
Q.23
•
If an assay is judged not to be performing adequately (poor IQC performance), what systems are used to prevent patient results being released ?
– 79% – 33% – 32% – 39% Chemistry is disabled Data not transferred to LIS Results not validated Relies on operator not to report 85 answered question
Q.24
•
Please state how long your laboratory stores IQC data for. If the data is not readily retrievable for the whole duration, please additionally state how long it is in a readily retrievable form?
– 7% – 10% – 13% – 1% – 24% – 1% – 28% – 16% <1 year <3 years 5 years 7 years 10 years 30 years Indefinite For lifetime of analyser 76 answered question.x
Q.25
•
Does your audit trail allow you to identify retrospectively for any patient sample, the IQC data/performance relating to analysis?
• 89% YES • 11% NO • Free text comments gave as little as 2 hours and one week and several said a month.
• Many said indefinitely or for lifetime of analyser.
Q.26
•
Does your laboratory use six sigma ?
• 8% • 92% YES (3% in previous regional audit.) NO 85 answered question
Q.27
•
Does your laboratory hold regular dedicated IQC meetings ?
• 68% • 32% YES NO • Very similar results in previous regional audit.
Frequency ranged from daily up to 6 monthly. Monthly meetings are the most common scenario.
Q.28
•
Is IQC a standing agenda item on governance or management meetings ?
• 81% • 19% YES NO (72% in previous regional audit.) •
Is IQC discussed at dedicated meetings or general senior management meetings ?
– 55% Both – 14% – 26% – 6% Dedicated meeting only Senior management meeting only Neither dedicated meeting or senior management meeting 86 answered question
STANDARD 1
• IQC policies should cover a 24 hour period.
– 25% of labs still show clear evidence of having IQC practices that vary according to the time of day or whether samples are being run during core hours versus ‘on-call’ or other shift.
There is an apparent improvement compared to the previous audit in which 70% had no policy, BUT Remaining labs should develop their policies for 24/7.
Labs should review the validity of IQC testing practice out of hours if it differs to the routine day.
Discretionary IQC testing out of hours MUST BE discouraged.
STANDARD 2
• Target values and ranges for IQC material should be assigned locally for each instrument. • Manufacturer target levels and ranges should not be used.
• Significant variability remains in the way that labs establish the target value and SD ranges for IQC materials. • This is unchanged since the previous audit.
STANDARD 3
• Laboratories should use IQC material from a third party source.
• The proportion of labs not using third party QC material remains unchanged since the previous audit.
STANDARD 4
• Use EP5-A2 or equivalent to determine IQC values within each individual laboratory.
• There is clear evidence of significant variability in the way that labs assign target values and ranges to QC material. Since this is vital to the ability of a IQC process to detect true error, this should be addressed.
STANDARD 5
• Encourage method specific rules (single and multi) based on the required quality of the assay, and known method bias and coefficient of variation.
• There has been an apparent increase in the number of labs that utilise method specific rules, but scope for significant further work.
• Other concerns (see later)
STANDARD 6
• Discourage inappropriate use of single rules, but when used, ensure they have the required degree of error detection.
• 55% stated they use single 2SD rules, compared to previous 76%. This is an apparent improvement, but again concerns (see later).
STANDARD 7
• Review grades of staff accepting and rejecting IQC. • The use of trainee and other staff who are not state registered to accept and reject QC should be discouraged or they must be supervised.
• Still approximately 10% of labs are allowing either trainees or non-registered staff to make key decisions. This is unchanged since the previous audit.
• A weakness of the audit is that we do not know about the degree of supervision, but we suggest labs review and ensure appropriate supervision is in place or change of practice occurs.
STANDARD 8
:
• If IQC policies are robust, and the correct IQC with appropriate error detection is used, failed IQC should not be accepted (e.g. to continue processing patient samples).
•
ONLY 18%
of respondents said continuing analysis would never happen.
• This remains very disappointing. The reasons quoted are as unconvincing as they were in the original audit.
STANDARD 9
• For analysers that have multiple modules, each module should be assessed individually with IQC on each assessment.
• Not measured in either audit, but principle still proposed.
DISCUSSION POINTS IQC testing intervals are still very variable. It is unclear how much relates to lack of confidence in methodology versus poor IQC planning. This is reflected in the large variation in the number of sodium measurements that labs are making between IQC measurements. Scope for standardisation.
There is a need for our professions to work with manufacturers of IQC material to refine the target concentrations used.
DISCUSSION POINTS Some free text answers imply a lack of understanding of IQC. For example, some of the labs that stated they use single 2SD rules are using multi-rules. Most of the labs that state they use method specific rules are not.
• Would there be any value in the ACB and/or IBMS designing a questionnaire to evaluate core knowledge and understanding of IQC within the professions ?
There is lack of clarity on what statistical tests best allow services operating on multiple sites to compare data across their whole service see recent paper.
DISCUSSION POINTS 40% of labs still rely on human intervention to prevent patient results being released when IQC indicates a method is out of control. This should be minimised in favour of automated IT systems.
Several labs to need to review the length of time that IQC data is stored for – RCPath guidance exists.
6% of labs that do not formally discuss IQC at either a dedicated meeting or management / governance meetings should review the feasibility of doing this.
COMMENTS FROM WESTGARD • Analytical failures can be invisible to the laboratory until they escalate to become serious. Maintaining quality in the light of workload pressures is essential.
• The UK are doing more for IQC than in the US. • What the UK have done is very helpful and we would be keen to ensure labs around the world can learn from it.
• Interested to use their website as a forum to disseminate and discuss findings in the form of articles and essays.
• They have data from a UK lab and a European lab that shows introduction of a robust IQC programme can reduce the number of calibrations by 50% and can reduce expenditure on IQC material by 50% (reduction in number of controls, repeat runs and calibrations).
IQC: planning and implementation strategies. James Westgard.
• Ann Clin Biochem. 2003. 40. 593-611.
• Do not use 2SD control limits.
• Do not use the same control rules for all tests.
• Do select IQC procedures on the basis of the quality required and precision and accuracy of the method.
• Do minimise false rejections in order to maximise response to real problems when they occur.
• Do build in the error detection necessary to detect medically important errors by selection of appropriate control rules and numbers of control measurements.
Clinical Biochemistry.
Leeds General Infirmary. UK.
• Ann. Clin. Biochem. March 2011. P136-146.
• Jassam N, Lindsay C, Harrison K, Thompson D, Bosomworth MP, Barth JH.
• The implementation of a system for managing analytical quality in networked laboratories.
• Defined quality requirements for 71 analytes and developed software to monitor analytical indices.
• 5 sites, single manufacturer.
• 9 general chemistry, 7 immunoassay analysers.
The way forward
• Adopt a total IQC programme. • IQC policies should cover 24 hours.
• Establish IQC procedures that minimise false rejections and maximise the detection of medically important errors.
• Encourage method specific IQC rules.
• Define control limits based on own data and reject results which are our of control.
• Preferentially use IQC from a third party.