Transcript QA Sensory Evaluation Methodologies

Services Provided by LCBO Quality Assurance
Leading Sensory Evaluation
Services
LC Quality Assurance
VQA Appellation System
VQA Mandate
Appellation of Origin System
Quality Standard
Control of use of specified terms, descriptions and designations
associated with the VQA appellation system
LC Quality Assurance
Role of Sensory Evaluation Testing
Quality of Beverage Alcohol Products




Sensory Quality
Chemical Composition
Microbiological Stability
Packaging and Labelling Standards
LC Quality Assurance
Role of VQA Sensory Evaluation Testing
Scope

Free of technical faults / defects

Typicity of varietal character – for varietal wines


“A wine bearing varietal designation shall be assessed to
determine if the varietal designation for the wine exhibits the
predominant character of a wine produced from the
designated grape variety/varieties” (VQA Rules)
Typicity of the wine category: Late Harvest, Icewine,
Nouveau, Sparkling – Traditional Method, Icewine
Dosage, Botrytized Wine (VQA Rules)
LC Quality Assurance
Role of VQA Sensory Evaluation Testing

“A wine shall be deemed to have passed the taste test if
a majority of the members of the Tasting Panel
determine:


That the wine’s attributes fairly reflect the viticultural
and oenological quality standards established in O.Reg.
406/00 (Rules) without defects or flaws; and
That the wine is representative of quality wines of the
stated category” (VQA Rules)
LC Quality Assurance
Sensory Panels


Grading Panel – 4 panel
groups of 5 panellists each
28 Members (20 regular
panellists, 8 alternate
tasters)
LC Quality Assurance
Panel Member Selection Criteria
Panel members are LCBO Product Consultants
who work in retail stores within metropolitan
Toronto area
Continuous training/ development
opportunities
LCBO Product Knowledge I, I, III
Certification from the Wine and Spirit
Education Trust
Master of Wine Certification
LC Quality Assurance
Panel Member Selection Criteria
Panel members are subjected to annual :
Sensory evaluation testing
Product knowledge testing
25 % of the questions are VQA specific
Demonstrated professionalism – Sensory Evaluations
Code of Conduct
LC Quality Assurance
Panel Groups Assembly Criteria:
Consistent
Performance
Reproducible
Results
Balanced
Panels
Results from the annual testing
Testing performance history
Previous experience (as a Grading Panel member, from
the industry)
Performance monitoring data
Sensory sensitivities
LC Quality Assurance
Panel Training
Training Elements
Aroma, flavour recognition
Product category recognition
Varietal character recognition
Regional character recognition
Winemaking techniques
Technological influences
Defect identification
LC Quality Assurance
Sample Presentation
Blind sample presentation:
Samples are presented without identifying
markings
Uniform sample presentation

Sample information: Varietal composition, vintage
year, wine category, method of production (where
applicable: sparkling wines, icewine dosage, etc)
LC Quality Assurance
Sensory Evaluation Method
Sensory Evaluation Grading System
LC Quality Assurance
Sensory Evaluation Grading System


Client
Satisfaction


Superior objectivity – system
design eliminates bias
Wider system applicability–
system design equally applies to
all wine categories
Efficient and effective calculation
of results – time savings, accuracy
Data Analysis Tools & Traceability
of results – enabled by automated
data collection and quantification
of the results
LC Quality Assurance
Sensory Evaluation Grading System
Sensory evaluation grading
system elements:

New sensory evaluation grading
method
Integrating technology:
• Automated data collection
• Automated data quantification, analysis
and reporting
LC Quality Assurance
Sensory Evaluation Grading System
Sensory characteristics applicable to all types of beverage alcohol
products.




Appearance and Colour
Aroma (primary and secondary) and Bouquet
Taste
Harmony
Appearance
Harmony
Colour
Aroma
LC Quality Assurance
Bouquet
Taste
Sensory Evaluation Grading System
Sensory characteristics and attributes are evaluated
using five grading categories.
Category
Excellent
Very Good
Good
Satisfactory
Unsatisfactory
Quality
Level
Outstanding
/Exceptional
Superior/
Very Correct
Typical
Weak/Not at
full potential
Faulty /
Defective
LC Quality Assurance
Grading Form
Characteristics &
Attributes
Grading Categories
Excellent
Very Good
Appearance & Colour
x
Aroma&
Bouquet
Correctness
x
Intensity
x
Taste
Good
Quality
x
Correctness
x
Intensity
Harmony
Satisfactory
x
Finish
x
Quality
x
LC Quality Assurance
x
Comments
Unsatisfactory
Innovative Technology
Data collection
Data quantification
Data analysis
Data reporting
LC Quality Assurance
Data Collection – Tablet PCs
Automated
Interactive
Flexible-Wireless
Pen or/and keyboard
Handwritten notes
LC Quality Assurance
New system implemented on June 1 2004; Comparison for periods June December 2003 and 2004
45
40
Scores
Frequency, %
Scores Frequency, 2003
Scores Frequency, 2004
Difference 04-03
2004
35
30
2003
25
20
15
10
5.9
5.7
5
4.5
3.5
0.3
1.6
0
-5
<12.0
12.0 - 12.4
-3.2
12.5 - 12.9
-3.0
13.0 - 13.4
13.5 - 13.9
14.0 - 14.4
14.5 - 14.9
-4.0
-10
-15
-11.4
LC Quality Assurance
15.0 - 15.4
15.5 - 15.9
>16.0
Scoring Intervals
(0-20 Scoring Scale)
QA of Sensory Evaluation Results
Quality Sensory Evaluation Result
Quality
Assurance
System
Controls
Verification
Tasting
Performance
Monitoring
LC Quality Assurance
Proficiency
Programs
QA of Sensory Evaluation Results
VQA Tasting protocol (VQA Rules)
• 2nd Bottle tasting
Grading System Controls:
•
•
•
•
•
•
Security of panellist registration
Forced completeness of the assessment
Ratings cannot be changed, once finalized
Sample evaluation cannot be redone, once completed
Validation questions in the grading questionnaire
Calculation validation based on statistical measures
Automated calculation and reporting
LC Quality Assurance
QA of Sensory Evaluation Results
Verification Tasting - an independent sensory evaluation of the products in conditions
identical to the tasting panel.
The verification process has a quality assurance role and provides a reference value that is
then compared with the panel results to identify any discrepancies in the assessment.
Verifiers – qualified Quality Assurance tasters. A tasting session may require up 2
Verifiers.
Verifiers’ results are typically not included in the calculation, unless discrepancy situation.
LC Quality Assurance
QA of Sensory Evaluation Results
Panellists’ performance is monitored for each tasting session
Measures:
• Outliers Frequency, % - measure of rating accuracy
• Rating Rank – measure of rating bias (high, low, trends)
• Sensitivity data - missed defects, good detection, hyper-sensitivity
Feedback on performance – Feedback Report
Provides tools to help identify training opportunities
LC Quality Assurance
QA of Sensory Evaluation Results
Panellist Feedback
Report
Tasting Session:
Panellist:
Total Number of Samples:
Total Number of Outliers:
Outliers Frequency, %
Rank, Descending
4-Apr-2005 2:23 PM
Panellist Name
26
4
15.4
2.9
Panellist Ratings vs. Panel Median
15
14
Grading Scores
13
12
11
10
9
Panelist Values
8
Product Result
(Median)
7
6
1
2
3
4
5
6
LC Quality Assurance
7
8
9
10
11
Samples
12
13
14
15
16
17
18
19
20
21
QA of Sensory Evaluation Results
Internally designed to measure quality and consistency of the sensory evaluation
assessments
System
Designed to measure:
• System performance
Panel
• Panel Performance
Panellist
• Panellist Performance
Typical measures: repeatability, reproducibility, bias, defects identification, etc.
LC Quality Assurance
Internal Proficiency Testing
Objective:
Assessment of results repeatability & reproducibility
Product Categ.: Ontario Wines
Methodology:
Assessment of panel results for three Ontario wines presented as blind duplicates.
Wines are of sound quality and similar quality levels.
Sensory Results
Average
SD
Sample 1
Sample 2
Sample 3
Replicate 1
Replicate 2
Replicate 1
Replicate 2
Replicate 1
Replicate 2
13.76
1.24
13.80
0.57
13.86
0.36
13.56
1.02
13.98
0.69
13.64
1.30
LC Quality Assurance
Internal Proficiency Testing
Repeatability
Repeatability Results: p-Values * (2-tailed test, 95 % Confidence Level)
Sample 1 Sample 2 Sample 3
1. Panel level
No significant difference between replicate results
0.8711
0.1895
0.3044
2. Panel groups level: No significant difference between replicates
Group A:
0.4382
0.2809
0.0759
Group B:
0.4814
0.5232
0.5746
Group C:
0.7688
0.5137
0.2235
Group D:
0.9653
0.5291
0.0823
3. Panellist level:
Only one taster exibited significant variability of the results.
Source of variation: Sample position in the line-up in reference with the sensory profile of
adjacent samples.
LC
* Interpretation of results:: If the p-Value is lower
than 0.05,
then there is a signifficant difference at a 95% CL
Quality
Assurance
Internal Proficiency Testing
Reproducibility
Sample
Results, Median Value (reporting value)
Replicate 1
Replicate 2
Replicate 1
Replicate 2
Replicate 1
Replicate 2
Replicate 1
Replicate 2
Group A Group A Group B Group B Group C Group C Group D Group D Panel
Nov. 8/04 Nov. 8/04 Nov. 10/04 Nov. 10/04 Nov.29/04 Nov.29/04 Nov.17/04 Nov.17/04
Ave.
Sample 1
13.2
13.6
14.6
14
13.4
13.7
13.8
13.8
13.8
Sample 2
13.8
13.8
13.6
13.6
13.7
13.7
14
14.4
13.6
Sample 3
14.4
13.6
13.8
14.4
13.7
14.2
13.2
14
13.6
Average SD:
Panel
SD
0.39370
0.25981
0.40552
0.35301
No significant difference between reported results, i.e. median values (p-Value: 0.649671) at a 95%
confidence level
LC Quality Assurance
Internal Proficiency Testing
Reproducibility
Results, Median Value (reporting value)
Sample
Sample 1
Sample 2
Sample 3
Grading QA Panel Group A Group B Group C Group D
Sept. 16/04 Sept.13/04 Sept.15/04 Sept.20/04 Sept.22/04
Panel
Average* (Ref. Panel)
12.9
13.8
13.8
13.5
13.4
13.2
14.4
14.2
14.2
14.4
14.8
14.2
10.7
10.6
10.2
9.4
9.8
12
Average:
Grading
Panel SD
0.25000
0.28284
1.14746
0.56010
* Average of individual panellists' score.
No significant difference between panels' (i.e. all 5 panels) reported results (i.e. median value), at a
95% confidence level (p-value = 0.99605)
LC Quality Assurance
Sensory Evaluation Grading System
Summary






A superior method for performing sensory evaluations.
Eliminates many sources of bias.
Customized for use with all beverage alcohol products.
Flexibility (data collection, quantification & reporting).
Provides significant time savings in data management.
Provides tools to analyze panellist results to help
identify training opportunities.
LC Quality Assurance
Leading
the Sensory
Evaluation of
Beverage Alcohol
LC Quality Assurance