Transcript 4th ISTA Proficiency Test on GMO Testing on Soybean

International Seed Testing Association
ISTA’s Position, Experience and Activities in
the Field of Biotech Trait Detection
ISTA GMO TF
B.R. Kahlert
S. Grégoire, C. Haldemann, K. Remund, J.L. Laffont
and many others
Outline
What is ISTA?
ISTA and of GM Seed Testing
Accreditation
Proficiency Tests
Performance Data Evaluation
Audit
ISTA can help, ISTA helps to share
Workshops, Seedcalc, Website
Objectives of the
International Seed Testing Association
• The primary purpose of the Association is to develop, adopt and
publish standard procedures for sampling and testing seeds,
and to promote uniform application of these procedures for
evaluation of seeds moving in international trade.
• The secondary purposes of the Association are actively to
promote research in all areas of seed science and technology, to
participate in conferences and training courses aimed at
furthering these objectives, and to establish and maintain liaison
with other organisations having common or related interests in
seed.
ISTA: Laboratories Worldwide
More than:
•
160 laboratories,
•
200 personal members,
•
originating from 70 countries supporting the
association (Government mandates)
ISTA’s major Products & Services
• International Rules for Testing Seed
– Sampling, Purity, Other Seeds, Germination,
Tetrazolium, Seed Health, Species and Variety
Testing, Moisture, Coated Seed, Vigour, X-ray
– seeds of agricultural & vegetable; flowers,
spice, herb & medicinal;
tree & shrubs
• Proficiency Test Programme
• Laboratory Accreditation Programme
• International Seed Lot Certificates
ISTA’s Vision & Strategy in
Testing of GM Seeds
• ISTA Seed Testing Laboratories are competence
centres for detection, identification and quantification
of GM seeds
• ISTA focuses its activity on developing and
maintaining a system that ensures reliability and
accuracy of results all over the world, when testing
for GM seeds – Performance Based Approach
defined in the Position Paper:
ISTA’s Strategy Regarding Methods for the Detection, Identification and Quantification of
Genetically Modified Seeds in Conventional Seed Lots
adopted by the Executive Committee Nov. 2001
ISTA’s Actions in
Testing of Seed with Specified Trait(s)
1.
2.
3.
4.
5.
Chapter in ISTA Rules for testing of seeds with
specified trait(s)
Implementation of Laboratory Accreditation under
Performance Based Approach
Organising proficiency tests on detection and
quantification of GM seeds in conventional seed
Establishing a system for Performance Data
Evaluation for the Presence of Seed with Specified
Trait(s) in Seed Lots and Specified Trait Purity
Provide technical help for the laboratories (training
courses, workshops, Seedcalc7, ...)
1. Chapter in the ISTA Rules
•
•
•
Rules for the testing of seeds with specified traits
including the detection, identification and
quantification of GM seeds came into force in
February 1st, 2006
The Rules do not contain specific methods, but
define the general principles in testing and
reporting results of such tests
Laboratories having methods for GM seed testing
in their scope of accreditation can report the test
results on the International Seed Analysis
Certificate
2. Conditions for ISTA Accreditation under
the Performance Based Approach
Laboratory ...
•
Has established method(s)
•
Defines scope of accreditation
•
Participates successfully in the ISTA Proficiency
Test Programme
•
Submits performance data
•
Has an audit
3. Organisation of Proficiency Tests
AIMS
•
•
•
to evaluate the performance of participating
laboratories
to collect data to be used to set performance
standards
To increase experience for ISTA and participating
laboratories
Satisfactory performance in proficiency tests is a
prerequisite for attaining or maintaining the status
as an accredited laboratory
Laboratories can participate to ISTA proficiency tests,
even if they do not seek accreditation
Design of the 6th Proficiency Test
•
•
•
•
•
•
•
•
crop:
Brassica napus L.
# of samples per lab:
10
# of negative samples: 3
# of positive samples:
7
0.3%:
2
0.6%:
3
1.2%:
2
event:
GT73
sample size:
3300 seeds
Qualitative and Quantitative tests
the laboratory chooses the method
Qualitative Test: The detection of
GM seeds in samples
49 labs classified each of the 10 samples as either positive or negative.
4 laboratories reported false results. These are 8% of the laboratories.
Sample classified as false...
All samples
489
12
Negative samples
147
8
Positive samples
342
4
0.3% GM seeds
97
2
0.6% GM seeds
147
1
1.2% GM seeds
98
1
0%
20%
40%
60%
80%
% of samples tested
# of samples reported as false
100%
2.4%
5.4%
1.2%
2.0%
0.7%
1.0%
Sub-sampling Quantification
Laboratory’s estimated results and its mean
3.0
3.0
Spiking level 0.3%
Estimate of % GM seeds
2.8
3.0
Spiking level 0.6%
2.8
2.6
2.6
2.4
2.4
2.4
2.2
2.2
2.2
2.0
2.0
2.0
1.8
1.8
1.8
1.6
1.6
1.6
1.4
1.4
1.4
1.2
1.2
1.2
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
1
2
3
4
5
6
7
8
9
10
11
12
Spiking level 1.2%
2.8
2.6
0.0
1
2
3
4
5
6
7
8
9
10
11
12 laboratories reported the test plan...
• the size of sub-samples (in number of seeds)
• the number of sub-samples tested
• the number of positive sub-samples
 used to compute the result
12
1
2
3
Labs
1
2
3, 4
5, 6
7, 8
9, 10, 11
12
4
5
6
7
8
# of subsamples
~300
~90
~30
20-25
15
5-10
2-3
9
10
11
12
size of
sub-sample
10
30
~110
~150
210-220
330-550
Quantitative Test
23 laboratories reported the GM content of the positive test samples in %
of GM seed by number or by mass or in % DNA copies
3.0
2.8
2.6
2.4
2.2
% GM seeds
2.0
1.8
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Spiking
level (%)
0.3/0.33
0.6/0.67
1.2/1.35
19
20
21
22
23
Mean (%)
0.38
0.80
1.57
Standard
deviation
0.22
0.40
0.90
Variation
coeff.
59%
50%
58%
Summary PT06
PT06 - Qualitative Results:
• The percentage of laboratories reporting correct
results for all samples is higher as in the previous
tests
• >90% of labs made no missclassification,
• >97% of samples are reported correctly
PT06 - Quantitative Results:
• The results show similar variation for the the spiking
level (VC = ~50%)
• Most laboratories overestimated the true level
Design of the ISTA Proficiency Tests
PT01
30
# of
samples
per lab
negative
12
positive
18
0.1%
0.2%
0.3%
0.5%
0.6%
0.7%
1.0%
18
1.2%
1.4%
1.5%
2.0%
4.0%
Events
T25/Mon810
(maize)
Sample
300
size
Qualitative
X
Sub-sampl.
Quant. test
-
PT02
10
PT03
12
PT04
12
PT05
12
3
7
-
3
9
3
3
9
3
-
3
9
2
X
(X)
(X)
X
X
X
X
X
X
3
4
Mon810
(maize)
3000
X
(X)
(X)
PT06
10
3
7
2
3
3
3
3
3
2
3
3
3
T25/Mon810 GTS-40-3-2 GTS40/A2704
GT73
(maize)
(soybean)
(soybean)
(rapeseed)
1500
3000
3000
3000
110 laboratories from 42 countries
X
X
X
PT01 – PT06: Qualitative Tests
% laboratories
Percentage of laboratories reporting results correctly or falsely:
100
90
80
70
60
50
40
30
20
10
0
PT01:
PT02:
PT03:
PT01
PT02
PT03
PT04
PT05
PT06
% laboratories
Percentage of samples reported correctly or falsely:
100
90
80
70
60
50
40
30
20
10
0
PT01
correct results
PT04:
PT05:
PT06:
PT02
PT03
false negative
both false positive and false negative
PT04
PT05
false results
false positive
PT06
maize,
MON810 & T25;
1%
maize;
MON810;
0.7% & 1.4%
maize;
MON810 & T25;
0.2%, 2% & 4%
soybean;
GTS40;
0.1%, 0.5% & 1.0%
soybean;
GTS40 & A2704;
0.2%, 0.5%, 1.0%, 1.5%
rapeseed;
GT70;
0.3%, 0.6% & 1.2%
PT01 – PT06: Quantitative Test
(Overall) mean of the quantitative test results for each spiking level
6.00
% mean GM content
5.00
4.00
3.00
2.00
1.00
0.00
0.1
0.2
0.2
GTS40 MON810 GTS40
+T25
0.3
GT70
0.5
0.5
GTS40 GTS40
0.6
0.7
1.0
1.0
1.0
1.2
1.4
1.5
2.0
4.0
GT70 MON810 GTS40 GTS40 A2704
+T25
GT70 MON810 GTS40 MON810 MON810
+T25
+T25
+T25
soya
maize
soya
rape
soya
soya
rape
maize
soya
soya
soya
rape
maize
soya
maize
maize
PT04
PT03
PT05
PT06
PT04
PT05
PT06
PT02
PT04
PT05
PT05
PT06
PT02
PT05
PT03
PT03
PT01 – PT06: Quantitative Test
Variation coefficient of the quantitative test results
100%
90%
% mean GM content
80%
70%
60%
50%
40%
30%
20%
10%
0%
0.1
0.2
0.2
GTS40 MON810 GTS40
+T25
0.3
GT70
0.5
0.5
GTS40 GTS40
0.6
0.7
1.0
1.0
1.0
1.2
1.4
1.5
2.0
4.0
GT70 MON810 GTS40 GTS40 A2704
+T25
GT70 MON810 GTS40 MON810 MON810
+T25
+T25
+T25
soya
maize
soya
rape
soya
soya
rape
maize
soya
soya
soya
rape
maize
soya
maize
maize
PT04
PT03
PT05
PT06
PT04
PT05
PT06
PT02
PT04
PT05
PT05
PT06
PT02
PT05
PT03
PT03
Summary PT01-PT06
Qualitative Results:
Better results when
• only one transgenic event involved
• transgenic events are well known
Better results for negative samples than for positive samples
Quantitative Results:
Reported values close to the true value
• when spiking level between 0.2% and 1.5%
• when transgenic event and reference material are well known
How to evaluate the performance of
laboratories in ISTA Proficiency Tests?
A
B
No problem has been detected in this test
There are small problems, but no specific look or
action is suggested to the participant
C
Problems, ISTA indicates there might be things to
consider by the laboratory to explain or correct
things
BMP Below Minimum of Performance, ISTA indicates
that the results were poor and the laboratory
need to find explanations and to improve/correct
In-round rating: Presence/absence
3 suggested rating systems
Example with 10 samples in a Test
1/10
2/10
3/10
4/10
10%
20%
30%
40%
Rate
Fixed Number of
misclassified samples
Rate
Percentage of
misclassified samples
Rate
Percentage of
misclassified samples
A
0 errors
A
0% - 5%
A
0% - 6%
B
1 or 2 errors
B
>5% - 10%
B
>6% - 20%
C
3 errors
C
>10% - 20%
C
>20% - 30%
BMP
More than 3 errors
BMP
>20%
BMP
>30%
System1
System2
2 mistakes
2/10=20%
System3
BMP
C
Yes
Yes
Too many sample results are
too far from truth
No
The average results by spiking
level are not accurate
No
B
Yes
There are too many
inaccurate sample
results
No
In-round Rating:
Principles
for Quantification
A
BMP
C
B
Yes
Yes
More than ½ of the sample results
outside [½ true level; 2 x true level]
No
Probability to observe a
more extreme value of the sum
of absolute spiking levels z-scores when
assuming the laboratory provides
accurate results
is  0.01
No
Yes
More than 1/6 (~17%) of
the sample z-scores outside
[-2; +2]
(missing sample values are
counted as outside)
No
A
PT06 Rating
Quantitative and Sub-sampling
Quantitative Tests (35 labs)
50
50
45
45
40
40
35
35
30
30
# of Labs
# of Labs
Qualitative Test (49 labs)
25
20
25
20
15
15
10
10
5
5
0
0
A
B
Rating System 1
C
Rating System 2
BMP
Rating System 3
A
B
Rating System 1
C
Rating System 2
BMP
Rating System 3
RS1: seed number
RS2: mass of seed
RS3: median
PT01-PT06: In-round Rating
20
0
0
0
0
45 50 39 53 59 48
45 50 39
20 25
PT05
0
PT04
53 59 48
PT06
20
PT05
20
PT04
20
PT03
40
PT02
40
PT01
40
PT06
40
PT03
60
PT02
60
PT01
60
PT06
60
PT05
80
PT04
80
PT03
80
PT02
80
PT01
100
50 56 34
PT06
20 25 50 56 34
PT05
0
100
0
20 25
50 56 34
0
20
25 50
PT06
45 50 39 53 59 48
100
PT05
PT01
45 50 39 53 59 48
100
PT04
0
PT04
0
PT03
0
PT03
0
PT02
20
PT02
20
PT01
20
PT06
20
PT05
40
PT04
40
PT03
40
PT02
40
PT01
60
PT06
60
PT05
60
PT04
60
PT03
80
PT02
80
PT06
80
PT05
80
PT04
100
PT03
100
PT02
100
PT01
100
PT01
% of laboratories which reported an A, B, C or BMP
Qualitative Results
Quantitative Results
56 34
PT01-PT06: Overall Rating
Rating System for 6 PTs based on the in-round rating values
Qualitative Results
One test
rating
A
B
C
BMP
One test
Score Value
5 points
4 points
3 points
0 points
Overall rating
on six tests
A
B
C
BMP
Range on
# of labs
six tests
(min. 4 rounds) 6 5 4
28-30 points
10
10 - 27-21points
14
2 12 16-20 points
8
- 2 6
<16 points
2
- - 2
Overall rating
on six tests
A
B
C
BMP
Range on
# of labs
six tests
(min. 4 rounds)
28-30 points
0
27-21points
5
16-20 points
5
<16 points
9
Quantitative Results
One test
rating
A
B
C
BMP
One test
Score Value
5 points
4 points
3 points
0 points
6 5
- - 5
- 1
-
2
4
-
4
7
4. Performance Data Evaluation
for presence of seed with specified trait(s) in seed lots
and specified trait purity
•
•
•
Laboratory can choose an established and
reliable method
The «Performance Based Approach» is used
instead of «one ISTA validated method»
(not published in the ISTA Rules)
Laboratory needs to demonstrate to ISTA
accreditation team its ability to perform well with
the method routinely used
Performance Data Evaluation
Scope of ISTA Accreditation
Performance approved method to be included in lab’s
scope of accreditation
• must be defined in terms of method used, the species
tested, and the specified trait is to be determined
Testing
principle
Attribute/
target
PCR
35S Promotor
Rt-PCR
Primer
35S-1/35S-2
35S-F/35S-R
35S-F/35S-R
Lec R/Lec-F
Qual./quant
specificity
Species
qual. screening
LOD/
quant. range [%]
0.03
Glycine max
quant. screening
0.1 - 3.0
Performance Data Evaluation
Relevant PD Documents
Evaluation of Performance Data for each combination
species, test method and trait!
For some combinations a PD has to be performed
(on seeds, defined number of samples, defined spiking
levels)
For other combinations suitable alternative Performance
Data may be used
(any other data providing effidence that accuracy and
repeatability criteria are met)
Performance Data Evaluation
Quantification of Trait(s)
Range: 0.1% to 3%
7 levels * 4 samples
Performance Data Evaluation
Accuracy: departure from true value
Repeatability: variability of results by level
7
Example of
use of the
data sent
by the
laboratory
Accuracy: Departure from true value
Repeatability: variability of results by level
Deviation
and
variability
by level
Performance Data Evaluation
Grading performance
8.2
8.3
Audit
• Auditors assess whether implementation aspects are
appropriately addressed and whether practice
complies with the documented procedures
• To that effect, training, equipment management,
facilities, reporting and running the overall quality
managements system will be assessed
• A complete practical demonstration of any
performance approved method will not be required
5. ISTA Workshops on GM Seed Testing
period
workshop title
location
2002 –
2006
ISTA/FAO Training Course on
Varietal Verification and GMO
Detection (8)
Argentina, South
Africa, Thailand,
Slovenia, Egypt,
China, Jamaica,
Turkey
2004 –
2005
ISTA Workshop on Statistical
Aspects of GMO Detection (3)
France, USA,
Argentina
ISTA Variety Testing Workshop Way to the future
2003
ISTA Open Forum Discussion on
(Semi)-Quantitative GMO Testing
Switzerland
http://www.seedtest.org
ISTA
Web
site