Transcript Document
Chemometrics Method comparison
Department of Chemical Pathology, University of Pretoria,
Dr R Delport 2003
Chemometrics Performance characteristics that are taken into account:
Precision, Accuracy, Interference, Working Range, and Detection Limit.
http://www.westgard.com/lesson21.htm
Chemometrics
To carry out a good method validation study, you need to do the following:
• Define a quality requirement for the test in the form of the amount of error that is allowable, preferably an allowable total error, • Select appropriate experiments to reveal the expected types of analytical errors, • Collect the necessary experimental data, • Perform statistical calculations on the data to estimate the size of analytical errors, • Compare the observed errors with the defined allowable error, and • Judge the acceptability of the observed method performance. http://www.westgard.com/lesson21.htm
Chemometrics
An experimental plan can be formulated by:
• Recognizing the types of errors that need to be assessed for this test and method, • Identifying the appropriate experiments and the amount of data needed to estimate those types of errors, then • Organizing these experiments to perform the quick and easy ones first and the ones taking more time and effort last. http://www.westgard.com/lesson21.htm
Chemometrics
Types of errors
•
Imprecision or random errors,
•
Inaccuracy, bias, or systematic errors, which can be of two types
•Constant systematic error or •Proportional systematic error. http://www.westgard.com/lesson21.htm
Chemometrics
The
dashed line
in the middle of the figure represents ideal method performance where the test method and the comparative method give exactly the same results. The
bottom line
in the figure shows the effect of a proportional systematic error, where the magnitude of the error increases as the test result gets higher.
The
top line
shows the effect of a constant systematic error, where the whole line is shifted up and all results are high by the same amount. Note that these results will also be subject to the random error of the method, therefore the
actual data points
would scatter about the line as illustrated in the figure. The range of this scatter above and below the line provides some idea of the amount of random error that is present.
http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Description of different experiments.
Replication experiment
• Provides information about random error • Is performed by making measurements on a series of aliquots of the same test samples within a specified period of time, usually within an analytical run, within a day, or over a period of a month. • Preliminary experiment involves determining within-run imprecision. • Final experiment requires at least 20 working days to provide estimate of the total imprecision, which includes within and between run components.
http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Description of different experiments.
Interference experiment
• Provides information about the constant systematic error caused by the lack of specificity of the method. • One test sample is prepared by adding the suspected material to a sample containing the analyte. • A second aliquot of the original sample is diluted by the same amount with solvent, then both samples are analyzed by the test method and the difference determined.
http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Description of different experiments.
Recovery experiment
• Provides information about the proportional systematic error caused by a competitive reaction. • Test sample is prepared by adding a standard solution of the analyte being tested to an aliquot of a patient specimen. • A baseline sample is prepared by adding an equal amount of the solvent used for the standard solution to a second aliquot of the same patient specimen. • The two samples are then analyzed by the test method and the amount recovered is compared to the amount added. http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Description of different experiments.
Comparison of methods experiment
• Is primarily used to estimate the average systematic error observed with real patient samples. • Can also reveal the constant or proportional nature of that error. • A series of patient specimens are collected and analyzed by both the test method and a comparative analytical method. • The results are compared to determine the differences between the methods, which are the analytical errors between the methods. http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Walking tour of the plan.
Get the method working and establish an operating protocol.
• Set up the instrument, prepare the reagents, calibrate the methods, and obtain results from test samples. • Check the standards and be sure the method is properly calibrated, otherwise calibration errors will show up throughout the experimental studies.
http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Walking tour of the plan.
Determine the working range.
• The working range will vary from test to test and must be defined as part of the specifications for the method. • Check by analyzing a series of solutions, in duplicate or triplicate, covering the concentrations range of interest. • If detection limit is a critical characteristic, it may be assessed at this time or in the next phase of preliminary experiments.
http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Walking tour of the plan.
Determine within-run imprecision, recovery, and interference.
• The replication experiment might include 20 samples of two or three materials whose concentrations closely match the medical decision levels of interest for the tests. • Interference experiments should test common problems such as hemolysis, lipemia, and high bilirubin. • Recovery experiments assess whether there are any competitive reactions due to the matrix or other materials in the native specimens. http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.
• Fresh patient specimens, and stored specimens. • Minimum of 40 well-chosen patient samples should be tested over a minimum of 5 working days. • Distributed one-third in the low to low-normal range, one third in the normal range, and one-third in the high abnormal range. • Method acceptability should be judged on the basis of the sizes of the random, systematic, and total analytical errors. http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Walking tour of the plan.
Difference plot
Comparison of methods experiment.
• Used as initial graph if the two methods are expected to show one-to-one agreement, • Displays the difference between the test minus comparative results on the y-axis versus the comparative result on the x-axis, such as shown in the accompanying figure.
http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Walking tour of the plan.
Difference plot
Comparison of methods experiment.
• Used as initial graph if the two methods are expected to show one-to-one agreement, • Displays the difference between the test minus comparative results on the y-axis versus the comparative result on the x-axis, such as shown in the accompanying figure.
• Differences should scatter around the line of zero differences, half being above and half being below.
• Repeat measurements if indicated.
http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.
Correlation coefficient • The correlation coefficient is a statistic that is almost always calculated and reported to describe the results from a comparison of methods study. • The new or "test" method values are plotted on y-axis and comparison values on x-axis. http://www.westgard.com/lesson21.htm
Chemometrics Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.
• A value of 1.000 indicates perfect correlation between the results of two methods. • Other statistics (such as slope, intercept, and standard deviation of the residuals) can also be calculated from the same data to estimate the size of errors occurring between the methods.
http://www.westgard.com/lesson15.htm
Random error
Chemometrics Analytical errors
Random error, RE, or imprecision
• Can be either positive or negative. • Direction and exact magnitude cannot be predicted. • Imprecision is quantitated by calculating the standard deviation (SD) from the results of a set of replicate measurements. http://www.westgard.com/lesson15.htm
Random error
Chemometrics Analytical errors
Random error, RE, or imprecision
• Can be either positive or negative. • Direction and exact magnitude cannot be predicted. • Imprecision is quantitated by calculating the standard deviation (SD) from the results of a set of replicate measurements. • As the SD often increases as the concentration increases, the coefficient of variation (CV) is calculated to express the SD as a percentage of the mean concentration from the replication study. • Maximum size of a random error is commonly expressed as a 2 SD or 3 SD estimate to help understand the potential size of the error that might occur.
http://www.westgard.com/lesson15.htm
Chemometrics Analytical errors
Systematic error
Systematic error, SE, or inaccuracy
• Is always in one direction. • A systematic shift displaces the mean of the distribution from its original value. • In contrast to random errors that may be either negative or positive and whose direction can not be predicted, systematic errors are in one direction and cause all the test results to be either high or low.
http://www.westgard.com/lesson15.htm
Chemometrics Analytical errors
Systematic error
Systematic error, SE, or inaccuracy
• How high or how low can be described by the bias, which is calculated as the average difference, or the difference between averages, between the value by the "test" method and a "comparative" method in a comparison of methods experiment. • Alternatively, the expected systematic difference may be predicted from the equation of the line that best fits the graphical display of test method values on the y-axis vs comparative method values on the x-axis.
http://www.westgard.com/lesson15.htm
Chemometrics Analytical errors
Systematic error
Systematic error, SE, or inaccuracy
• How high or how low can be described by the bias, which is calculated as the average difference, or the difference between averages, between the value by the "test" method and a "comparative" method in a comparison of methods experiment. • Alternatively, the expected systematic difference may be predicted from the equation of the line that best fits the graphical display of test method values on the y-axis vs comparative method values on the x-axis.
• SE may stay the same over a range of concentrations, in which case it can also be called constant error , or it may change as concentration changes, in which case it can be called proportional error.
http://www.westgard.com/lesson15.htm
Total error
Chemometrics Analytical errors
Total Error, TE,
• It is the net or combined effect of random and systematic errors, as shown in the accompanying figure. http://www.westgard.com/lesson15.htm
Total error
Chemometrics Analytical errors
Total Error, TE,
• It is the net or combined effect of random and systematic errors, as shown in the accompanying figure. • It represents a "worst-case" situation, or just how far wrong a test result might be due to both random and systematic errors. • Because laboratories typically only make a single measurement for each test, that measurement can be in error by the expected SE, or bias, plus 2 or 3 SD, depending on how you quantitate the effect of RE.
http://www.westgard.com/lesson15.htm
Test Bias plots
NCCLS EP9-A example dataset Method: Comparative v Test
Performed by
Analyse-it Software, Ltd.
Chemometrics
analysed with: A nalyse-It + Clinical Labo rato ry v1.40
Date
1 February 1999
n
300 250 200 Identity line A =B 150 100 50 0 0 100 200
Mean of Method - Com parative
300
Bias 95% CI 95% limits of agreement Lower Upper
40 -0.1
-2.3 to 2.2
-13.7
13.6
95% CI
-17.4 to -10.0
9.9 to 17.3
300 250 200 150 100 50 0 0 100 200 300
Chemometrics
15 10 5 0 -5 -10 -15 0 100 200
Mean of Method - Com parative
300 Zero bias 0 5 10 15
Chemometrics
15 10 5 0 -5 -10 -15 0 100 200
Mean of Method - Com parative
300 Zero bias