Transcript Does your Grain Calibration Need to be Updated?

Does your grain calibration need to be updated?
Benoit Igne1 ([email protected]), Glen R. Rippke1 ([email protected]), Lance R. Gibson2 ([email protected]),
Charles R. Hurburgh, Jr 1 ([email protected])
1
Department of Agricultural and Biosystems Engineering, Iowa State University, Ames, Iowa.
2 Department of Agronomy, Iowa State University, Ames, Iowa.
Various pathways of diffuse
reflectance
13th IDRC Conference
Results
Introduction
7
7
6
6
5
5
4
3
4
Cross Validation
Next year
25% Calibration set
10% Calibration set
1
2002
2003
0.00
0.73
0.06
0.06
0.01
2.02
0.07
0.06
2004
0.00
0.21
0.06
0.04
3-year average
0.04
0.99
0.06
0.06
LS-SVM Regression
2003
2004
2002
2005
2003
2004
Cross
validation
Validation set
next year
25% calibration
set
10% calibration
set
2002
2003
2004
0.01
0.64
0.03
0.06
0.02
2.29
0.04
0.05
0.00
0.31
0.05
0.02
3-year average
0.01
1.08
0.04
0.04
2005
Years
 No significant difference between all the validation methods using a part of the
calibration set for both regression methods over the four years (α = 0.05).
 Significant difference between validation set of next year sample and 25% of the
calibration set (α = 0.05) for 2002. No significant difference among the others.
 Significant difference in 2003 and 2004 between all validation methods based on
same year sample and next year sample (α=0.05).
 No significant difference between both regression method (α = 0.05).
Data collection
 Triticale samples from 2002 to 2005 (Iowa, USA).
10% calibration
set
Cross Validation
Next year
25% Calibration set
10% Calibration set
Years
Materials and Methods
25% calibration
set
0
2002
 Propose objective criteria to determine when an update is needed.
Validation set
next year
3
1
0
 Evaluate calibration validation scenarios.
Cross
validation
2
2
Objectives
PLS - Regression
LS-SVM Regression Models
PLS-Regression Models
RPD values
The US Department of Agriculture has established a 3-year rule based on
cumulative bias which can freeze calibrations for an extended period even if
calibration updates are needed.
What about bias evolution?
Validation scenarios
RPD values
The update of a calibration is a costly process whose outcomes are not easily
predictable. If new samples have high variability, the new model can be less
accurate than before. Once the model is developed, a calibration transfer process
is often required. In commercial applications, users’ model need to be updated as
well as their instruments restandardized.
 Based on a bias-evolution, the decision to update the calibration would occur only
through a next year validation set samples (update if three-year average bias
exceeds ± 0.3%).
 The same trend is observable for both regression methods.
Comparison of 10% and 25% validation sets
 Foss Infratec™ 1241 (transmittance instrument).
 Crude protein analysis by AACC Method 46-30 (combustion nitrogen).
using a LECO CHN-2000 Analyzer.
8
25%
10%
Preprocessing methods
 2nd derivative (Savitzky-Golay 5-point window, 3rd order polynomial).
6
6
5
5
4
3
1
1
0
0
Software
 MATLAB v. 7.04 (The MathWorks, Inc.).
 PLS_Toolbox 3.5.4 (Eigenvector Research, Inc.).
 LS-SVMlab Toolbox (Suykens et al., 2002).
2004
2005
 Bias-based decision is non-efficient.
3
2
2003
 Calibrations developed through year n and validated on the same calibration pool
can lead to misinterpretations of the performance of the calibration if n+1 year is
spectrally different.
4
2
2002
25%
10%
7
RPD values
 Least Squares Support Vector Machine Regression (LS-SVM R).
RPD values
Calibration methods
Discussion
8
7
 Partial Least Squared Regression (PLS-R).
RPD mean values for 25% and 10% validation set scenarios
by LS-SVM R method
RPD mean values for 25% and 10% validation set scenarios
by PLS-R method
 RPD-based decision gives a better state of the calibration. Objective criteria with n+1
year validation samples would lead to the following rules:
2002
Years
2003
2004
2005
Years
Col 2
Col 4
- Better RPD  update can be planned: few risks of poorer accuracy of the
calibration after update.
 No significant difference between errors given by both regression methods
(α=0.05).
- Equivalent RPD  all the sample variability is included in the calibration: no need
to change it.
 No significant difference between the error of both validation methods (α=0.05).
Additional data would help clarifying this issue.
- Lower RPD  update necessary. Attention is needed in the choice of the new
calibration samples to avoid the addition of noise or yearly outliers.
 JMP v. 6.0.0 (SAS Institute Inc.).
Validation methods
 Cross-validation (leave-one-out).
 Validation set by holding out 25% and 10% of the calibration set.
 Validation set from year n+1.
Evaluation parameter
 Relative Performance Determinant (RPD) using the standard error of
prediction (SEPd) corrected for bias.
Conclusions
 Between similar years, all validation methods gave the same results, no matter the regression method used. While predicting a dissimilar year, the use
of a next year sample validation set gave more realistic information on the performance of the calibration.
 The use of a small validation set (removed from the calibration set) can lead to wrong conclusions about the calibration performances.
 Validation criteria must be adapted to the situation but bias-based decisions are likely not optimal.
 The establishment of strict rules is not feasible. The combination of all the parameters presented and discussed here can help to take the best decision.