Transcript Standard Verification System for Long-range Forecasts (SVSLRF) Richard Graham, Met Office Hadley Centre.

Standard Verification System for Long-range
Forecasts (SVSLRF)
Richard Graham, Met Office Hadley Centre. With acknowledgements to all GPCs, LCSVSLRF and members of the WMO Expert Team on Extended and Long-range Forecasting
(ET-ELRF)
WMO/THORPEX/WCRP sub-seasonal to seasonal implementation planning meeting, WMO HQ, Geneva,
2-3 December 2011
© Crown copyright Met Office
Lead Centre for Standard Verification of
Long-range forecasts (LC-SVSLRF)
http://www.bom.gov.au/wmo/lrfvs/
Jointly operated by
Bureau of Met. BoM) & Met.
Service of Canada (MSC)
David Jones, Bertrand Denis
• repository of global SVSLRF scores for GPC forecasts
• Core of SVS: ROC maps and diagrams, reliability diagrams, MSSS
• documentation on verification scores (Attachment II-8 of the WMO
manual on the GDPFS)
• observation datasets for verification, and code for calculating the
SVSLRF scores
SVSLRF: skill measures
Probability forecasts (for tercile categories):
• Relative Operating Characteristics (ROC): diagrams and
scores (area under ROC curve)
• Reliability diagrams and sharpness diagrams
Deterministic forecasts (ensemble mean):
• Mean Square Skill Score (MSSS) and decomposition
terms (correlation, variance, bias) – climatology as
reference
Variables:
• T2m, precipitation, SST, Nino3.4 index
Level 1: ‘bulk’ scores and diagrams
for regions/indices (for admin.)
• ROC curves and scores; reliability diagrams
and freq histograms; MSSS
• Variables/indices: T2m, precip, Nino3.4
• Regions: Tropics (20N to 20S), N. Extratropics
(20-90N); S. Extratropics (20-90S)
• Seasons: MAM, JJA, SON, DJF (Nino3.4:
calendar months)
• Lead-times: all available (but not > 4 months for
T2m and precip; not > 6 months for Nino3.4)
Level 2: scores at grid points for
generating skill maps (for users)
• ROC scores; MSSS (and decomposition)
• Variables/indices: T2m, precip, SST
• Regions: global, each defined grid point
• Seasons: 12 rolling 3-month seasons
• Lead-times: all available (but not > 4 months)
• Stratification: Where sufficient hindcast years,
stratify into seasons with El Niño active and
seasons with La Niña active
Lead Centre SVSLRF
Reliability/sharpness
diagrams
ROC curves
Courtesy: David Jones (BoM)
ROC score
maps
Sample verification: ROC
scores for SON precipitation,
1-month lead
Melbourne
Toulouse
Washington
above
below
0.6
Tokyo
Some issues
• Each GPC calculates own scores and submits to
LC-SVSLRF. Centralised verification would help
standardisation (e.g. in category definition,
verification datasets used) and speed processing;
• Good practice = all forecast products have
corresponding verification products – big task, we
are still catching up to a degree;
• Only GPC minimum variable set are required to be
verified (T2m, precip, SST, Nino3.4) – but forecasts for
other variables (e.g. Z500) are available
• No verification yet of individual month forecasts (available
on LC-LRFMME website)
• No verification yet of the LC-LRFMME multi-model
combinations
Thank you! Any questions?