Transcript Operational Seasonal Forecast Systems: a view from ECMWF Tim Stockdale The team: Franco Molteni, Magdalena Balmaseda, Kristian Mogensen, Frederic Vitart, Laura Ferranti European Centre for.

Operational Seasonal Forecast Systems:
a view from ECMWF
Tim Stockdale
The team: Franco Molteni, Magdalena Balmaseda, Kristian Mogensen,
Frederic Vitart, Laura Ferranti
European Centre for Medium-Range Weather Forecasts
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
1
Outline
• Operational seasonal systems at ECMWF
 System 3 - configuration
 System 3 – products
 System 3 – skill measures
• EUROSIP
 ECMWF, Met Office and Météo-France multi-model system
• Some (relevant) issues in seasonal prediction
 Estimating skill and model improvement
 Cost effective systems
 Multi-model systems, data sharing policy
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
2
Sources of seasonal predictability
 KNOWN TO BE IMPORTANT:
o
o
o
o
El Nino variability
Other tropical ocean SST
Climate change
Local land surface conditions
-
biggest single signal
important, but multifarious
especially important in mid-latitudes
e.g. soil moisture in spring
 OTHER FACTORS:
o
o
o
o
o
o
Volcanic eruptions
- definitely important for large events
Mid-latitude ocean temperatures
- still somewhat controversial
Remote soil moisture/ snow cover
- not well established
Sea ice anomalies
- local effects, but remote?
Dynamic memory of atmosphere
- most likely on 1-2 months
Stratospheric influences
- solar cycle, QBO, ozone, …
 Unknown or Unexpected
- ???
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
3
ECMWF operational seasonal forecasts
• Real time forecasts since 1997
 “System 1” initially made public as “experimental” in Dec 1997
 System 2 started running in August 2001, released in early 2002
 System 3 started running in Sept 2006, operational in March 2007
• Burst mode ensemble forecast
 Initial conditions are valid for 0Z on the 1st of a month
 Forecast is created typically on the 11th/12th (SST data is delayed up to
11 days)
 Forecast and product release date is 12Z on the 15th.
• Range of operational products
 Moderately extensive set of graphical products on web
 Raw data in MARS
 Formal dissemination of real time forecast data
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
4
ECMWF System 3 – the model
• IFS (atmosphere)
 TL159L62 Cy31r1, 1.125 deg grid for physics (operational in Sep 2006)
 Full set of singular vectors from EPS system to perturb atmosphere initial
conditions (more sophisticated than needed …)
 Ocean currents coupled to atmosphere boundary layer calculations
• HOPE (ocean)
 Global ocean model, 1x1 mid-latitude resolution, 0.3 near equator
 A lot of work in developing the OI ocean analyses, including analysis of
salinity, multivariate bias corrections and use of altimetry.
• Coupling
 Fully coupled, no flux adjustments, except no physical model of sea-ice
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
5
System 3 configuration
• Real time forecasts:
 41 member ensemble forecast to 7 months
 SST and atmos. perturbations added to each member
 11 member ensemble forecast to 13 months
 Designed to give an ‘outlook’ for ENSO
 Only once per quarter (Feb, May, Aug and Nov starts)
 November starts are actually 14 months (to year end)
• Back integrations from 1981-2005 (25 years)
 11 member ensemble every month
 5 members to 13 months once per quarter
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
6
NINO3.4 SST anomaly plume
ECMW F forecast from 1 Oct 2010
Monthly mean anomalies relative to NCEP adjusted OIv2 1971-2000 climatology
System 3
1
0
0
-1
-1
-2
-2
Anomaly (deg C)
1
APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN
2010
2011
Forecast issue date: 15 Oct 2010
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
7
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
8
Other operational plots for
DJF 2010/11
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
9
Tropical storm forecasts
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
10
Performance – SST and ENSO
NINO3.4 SST rms errors
Rms error of forecasts
has been systematically
reduced (solid lines) ….
192 start dates from 19870101 to 20021201
Ensemble sizes are 5 (0001), 5 (0001) and 5 (0001)
Fcast
Fcast
S3S3
Fcast Fcast
S2 S2
Ensemble sd
Fcast S1 Fcast S1 Persistence Persistence
1
0.8
Rms error (deg C)
.. but ensemble spread
(dashed lines) is still
substantially less than
actual forecast error.
0.6
0.4
0.2
0
0
1
2
3
4
5
6
Forecast time (months)
NINO3.4 SST anomaly correlation
Workshop
Sub-seasonal
Seasonal
Prediction, Exeter, 1-3 December 2010
wrt on
NCEP
adjusted OIv2to
1971-2000
climatology
11
More recent SST forecasts are better ....
NINO3.4 SST rms errors
NINO3.4 SST rms errors
168 start dates from 19940101 to 20071201
Ensemble size is 11
156 start dates from 19810101 to 19931201
Ensemble size is 11
Fcast S3
Persistence
0.8
0.8
Ensemble sd
Rms error (deg C)
Rms error (deg C)
1
0.6
0.6
0.4
0.4
0.2
0.2
0
Persistence
Fcast S3
Ensemble sd
1
0
1
2
3
4
5
6
0
7
0
1
2
3
4
5
6
1994-2007
1981-1993
NINO3.4 SST anomaly correlation
NINO3.4 SST anomaly correlation
wrt NCEP adjusted OIv2 1971-2000 climatology
wrt NCEP adjusted OIv2 1971-2000 climatology
1
0.9
0.9
Anomaly correlation
Anomaly correlation
1
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
7
Forecast time (months)
Forecast time (months)
0
1
2
3
4
5
6
7
0.4
0
1
2
MAGICS 6.12n cressida - net Mon Mar 9 11:58:09 2009
3
4
5
6
7
Forecast time (months)
Forecast time (months)
MAGICS 6.12n cressida - net Mon Mar 9 11:59:56 2009
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
12
NINO3.4 SST rms errors
324 start dates from 19810101 to 20071201
Ensemble size is 5: predictability limit for finite sample
Fc f3yi/m3
Persistence
1
Rms error (deg C)
0.8
0.6
At longer leads, model
spread starts to catch up
0.4
0.2
0
0
1
2
3
4
5
6
7
8
9
10
11
12
13
11
12
13
Forecast time (months)
NINO3.4 SST anomaly correlation
wrt NCEP adjusted OIv2 1971-2000 climatology
1
Anomaly correlation
0.9
0.8
0.7
0.6
0.5
0.4
0
1
2
3
4
5
6
7
8
9
10
Forecast time (months)
MAGICS 6.12n cressida - net Thu Jul 16 17:10:57 2009
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
13
How good are the forecasts?
Deterministic skill: DJF ACC
Temperature: actual forecasts
Temperature: perfect model
Perfect-model Anomaly Correlation Coefficient for CodOecmfE0001S003M001 with 11 ensemble members
Near-surface temperature
Hindcast period 1981-2003 with start in November and averaging period 2 to 4
Anomaly Correlation Coefficient for CodOecmfE0001S003M001 with 11 ensemble members
Near-surface temperature
Hindcast period 1981-2003 with start in November and averaging period 2 to 4
-1
-0.9
-0.8
-0.7
-0.6
-0.4
-0.2
0.2
0.4
0.6
0.7
0.8
0.9
1
-1
-0.9
-0.8
-0.7
-0.6
-0.4
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
-0.2
0.2
0.4
0.6
0.7
0.8
0.9
14
1
How good are the forecasts?
Deterministic skill: DJF ACC
Precip: actual forecasts
Precip: perfect model
Perfect-model Anomaly Correlation Coefficient for CodOecmfE0001S003M001 with 11 ensemble members
Precipitation
Hindcast period 1981-2003 with start in November and averaging period 2 to 4
Anomaly Correlation Coefficient for CodOecmfE0001S003M001 with 11 ensemble members
Precipitation
Hindcast period 1981-2003 with start in November and averaging period 2 to 4
-1
-0.9
-0.8
-0.7
-0.6
-0.4
-0.2
0.2
0.4
0.6
0.7
0.8
0.9
1
-1
-0.9
-0.8
-0.7
-0.6
-0.4
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
-0.2
0.2
0.4
0.6
0.7
0.8
0.9
15
1
How good are the forecasts?
Probabilistic skill: Reliability diagrams
Reliability diagram for CodOecmfE0001S003M001 with 11 ensemble
members
Reliability
diagram for CodOecmfE0001S003M001 with 11 ensemble member
Precipitation anomalies below the lower tercile over tropical band
(land and
sea points)
Near-surface
temperature
anomalies above the upper tercile over Northern
Hindcast period 1981-2003 with start in May and averaging period
2 to 4
Hindcast
period 1981-2003 with start in November and averaging period 2 to
Tropical
precip
< with
lower
tercile,
NH extratrop
temp
upper
tercile,
DJF
Threshold
estimated with
a kernel>method
for the
PDF
Threshold
estimated
a kernel
methodJJA
for the PDF
Skill sc
Brier s
Reliab
Resolu
Sharpn
ROC s
Skill scores and 95% conf. intervals ( 1000 samples)
Brier skill score:
-0.012 (-0.054, 0.027)
Reliability skill score: 0.916 ( 0.894, 0.933)
1.0 skill score: 0.073 ( 0.051, 0.097)
Resolution
Sharpness:
0.104 ( 0.098, 0.110)
ROC skill score:
0.306 ( 0.257, 0.351)
( 0.316, 0.296)
1.0
0.8
0.8
Rel
Res
BS
Observed frequency
Observed frequency
Sharp
1.0000
0.6
0.6
1.0000
0.025
0.020
0.015
0.1000
0.4
0.005
0.0100
0.000
0.0100
-0.005
0.2
0.2
Sharp/Rel/Res
0.4
0.010
Brier score
Sharp/Rel/Res
0.1000
-0.010
0.0010
0.0010
-0.015
-0.020
0.0
0.0
0.2
0.4
0.6
Forecast probability
0.8
1.0
0.0001
0.0
0.0
0.2
0.2
0.4
0.6
0.4
0.8
Forecast
Forecast probability
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
0.6
-0.025
1.0
0.8
0.0001
1.0
probability
16
How good are the forecasts?
Probabilistic
skill:
Reliability diagrams
Reliability diagram for
CodOecmfE0001S003M001
with 11 ensemble members
Near-surface temperature anomalies above the upper tercile over Europe (land and sea points)
Hindcast period 1981-2003 with start in November and averaging period 2 to 4
Threshold estimated with a kernel method for the PDF
Europe: Temp > upper tercile, DJF
Skill scores and 95% conf. intervals ( 1000 samples)
Brier skill score:
-0.092 (-0.217, 0.005)
Reliability skill score: 0.895 ( 0.771, 0.953)
Resolution skill score: 0.013 ( 0.003, 0.058)
Sharpness:
0.073 ( 0.065, 0.080)
ROC skill score:
0.077 (-0.076, 0.232)
( 0.113, 0.041)
1.0
0.8
Observed frequency
Sharp
Rel
Res
BS
1.0000
0.025
0.020
0.6
0.015
0.1000
0.4
0.005
0.0100
0.000
Brier score
Sharp/Rel/Res
0.010
-0.005
0.2
-0.010
0.0010
-0.015
-0.020
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Forecast probability
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
0.0001
0.2
0.4
0.6
-0.025
1.0
0.8
Forecast probability
17
EUROSIP
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
19
Single model
Multi- model
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
20
DEMETER: multi-model vs single-model
BSS
Rel-Sc
Res-Sc
Reliability diagrams (T2m > 0)
1-month lead, start date May, 1980 - 2001
0.039
0.899
0.141
0.039
0.899
0.140
0.095
0.926
0.169
-0.001
0.877
0.123
0.065
0.918
0.147
-0.064
0.838
0.099
0.047
0.893
0.153
0.204
0.990
0.213
multimodel
Hagedorn et al. (2005)
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
21
Some (relevant) issues in seasonal
prediction
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
22
Tentative results from ECMWF S4
Z500 Anom. correlation S3(11)-ERA Int 1989-2008 DJF
135°W
Global
rms acc:
0.612 NH:0.331
TR:0.783
SH:0.389
90°W
45°W
0°
45°E
90°E
(11 members, 20 years)
135°E
ACC
0.9
60°N
60°N
0.8
30°N
30°N
0.6
0.4
0°
0°
0.2
System 3
-0.2
30°S
30°S
60°S
60°S
-0.4
-0.6
-0.8
-0.9
135°W
90°W
45°W
0°
45°E
90°E
135°E
Z500 Anom. correlation ffcf(11)-ERA Int 1989-2008 DJF
135°W
Global
rms acc:
0.613 NH:0.294
TR:0.793
SH:0.387
90°W
45°W
0°
45°E
90°E
135°E
ACC
0.9
60°N
60°N
0.8
30°N
30°N
0.6
Cy36r4 - T159L62
0.4
0°
0°
0.2
-0.2
30°S
30°S
60°S
60°S
-0.4
-0.6
-0.8
-0.9
135°W
90°W
45°W
0°
45°E
90°E
135°E
Fisher z transform diff S3(11)-ffcf(11) 1989-2008 DJF
135°W
90°W
sigma:
0.343 0°mean: -0.0154
45°W
45°E
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
90°E
135°E
z
23
Z500 Anom. correlation ffky(6)-ERA Int 1989-2008 DJF
135°W
Global
rms acc:
0.624 NH:0.346
TR:0.804
SH:0.371
90°W
45°W
0°
45°E
90°E
135°E
ACC
0.9
60°N
60°N
0.8
30°N
30°N
0.6
0.4
0°
Alternate stochastic physics
0°
0.346 vs 0.294
0.2
30°S
30°S
-0.4
-0.6
60°S
60°S
-0.8
A real improvement, now
scoring better than S3
-0.9
135°W
90°W
45°W
0°
45°E
90°E
135°E
Z500
Int 1989-2008
1989-2008 DJF
DJF
Z500Anom.
Anom.correlation
correlationfg4m(5)-ERA
ffn5(6)-ERA Int
135°W
135°W
Global
rms
0.629
TR:0.819
SH:0.338
Global
rms acc:
acc:
0.605 NH:0.342
NH:0.288
TR:0.795
SH:0.332
90°W
45°W
0°
45°E
90°E
90°W
45°W
0°
45°E
90°E
135°E
135°E
ACC
ACC
60°N
60°N
60°N
60°N
0.9
0.9
0.8
0.8
30°N
30°N
30°N
30°N
0.6
0.6
T159L91, plus revised
stratospheric physics
0.4
0.4
0°
0°
0°
0°
0.2
0.2
-0.2
-0.2
30°S
30°S
30°S
30°S
60°S
60°S
60°S
60°S
-0.4
-0.4
Only 5 members, but score of
0.342 is much better than L62
-0.6
-0.6
-0.8
-0.8
-0.9
-0.9
135°W
135°W
90°W
90°W
45°W
45°W
0°
0°
45°E
45°E
90°E
90°E
135°E
135°E
Z500
Anom.
correlation
Int 1989-2008
1989-2008DJF
DJF
Fisher
z transform
difffg2d(5)-ERA
ffky(6)-ffn5(6)
135°W
135°W
Global
rms Workshop
acc:
0.614
NH:0.313
TR:0.812
SH:0.287
sigma:
0.343
0.0512
90°W
45°W
0° mean:
45°E
90°E
on Sub-seasonal
90°W
45°W
0°
45°E
90°E
135°E
to Seasonal
Prediction, Exeter, 1-3 December 2010
135°E
24
-0.2
30°S
30°S
60°S
60°S
-0.4
-0.6
-0.8
-0.9
135°W
90°W
45°W
0°
45°E
90°E
135°E
Z500 Anom. correlation fg79(11)-ERA Int 1989-2008 DJF
135°W
Global
rms acc:
0.646 NH:0.39
TR:0.819
SH:0.409
90°W
45°W
0°
45°E
90°E
135°E
ACC
0.9
60°N
60°N
0.8
30°N
30°N
T255L91
0.6
0.4
0°
0°
0.2
-0.2
30°S
30°S
60°S
60°S
-0.4
Score is now 0.390, cf 0.294
for T159L62
-0.6
-0.8
-0.9
135°W
90°W
45°W
0°
45°E
90°E
135°E
Z500
Anom.
correlation
fgcn(11)-ERA Int 1989-2008 DJF
Fisher
z transform
diff fgcn(11)-fg79(11)
135°W
135°W
Global
rms acc:
0.627
NH:0.273
TR:0.819
SH:0.381
sigma:
0.343
90°W
45°W
0°mean: -0.0457
45°E
90°E
90°W
45°W
0°
45°E
135°E
135°E
90°E
ACC
z
60°N
60°N
60°N
60°N
0.9
1
0.8
0.8
30°N
30°N
30°N
30°N
0.6
0.6
0.4
0.4
0°
0°
0°
0°
0.2
0.2
-0.2
-0.2
30°S
30°S
30°S
30°S
60°S
60°S
60°S
60°S
-0.4
-0.4
-0.6
-0.6
-0.8
-0.8
-0.9
-1
135°W
135°W
90°W
90°W
45°W
45°W
0°
0°
45°E
45°E
90°E
90°E
Workshop
on Sub-seasonal
Global
rms
acc:
0.646 NH:0.39
TR:0.819
SH:0.409
90°W
45°W
0°
45°E
90°E
Score is 0.273
From the best to the worst!
(Also other fields)
135°E
135°E
Z500 Anom. correlation fg79(11)-ERA Int 1989-2008 DJF
135°W
T255L91, with alternate
stochastic physics
to Seasonal
Prediction, Exeter, 1-3 December 2010
135°E
25
Possible interpretations
 Statistical testing suggests differences are real, for this 20 year period
 Different model configurations give different model “signals” in NH winter
 Hope was that hemispheric averaging would increase degrees of freedom
enough to make scores meaningful
 Hypothesis 1: this is not true - a given set of signals gets a given score
for the 20 year period, but this is of no relevance to expected model skill
in the future, and cannot be used for model selection.
 Hypothesis 2: Some model configurations really do better capture the
“balance” of processes affecting NH winter circulation, even if it is via
compensation of errors. Better to choose the model with the better score.
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
26
Choosing a model configuration
• Encouraging that some configurations give good results
• Higher horizontal and vertical resolution are consistently
positive
• Model climate is much improved, again resolution clearly
helps
• Forecast skill??
• How should we weight seasonal forecast skill?
• What other tests should we use for a model?
Links to extended/monthly forecast range??
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
27
Cost effective systems
• Back integrations dominate total cost of system
 System 3:

3300 back integrations
(must be in first year)
492 real-time integrations (per year)
• Back integrations define model climate




Need both climate mean and the pdf, latter needs large sample
May prefer to use a “recent” period (30 years? Or less??)
System 2 had a 75 member “climate”, System 3 has 275.
Sampling is basically OK
• Back integrations provide information on skill
 A forecast cannot be used unless we know (or assume) its level of skill
 Observations have only 1 member, so large ensembles are much less
helpful than large numbers of cases.
 Care needed eg to estimate skill of 41 member ensemble based on past
performance of 11 member ensemble
 For regions of high signal/noise, System 3 gives adequate skill estimates
 For regions of low signal/noise (eg <= 0.5), need hundreds of years
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
28
Data policy and exchange issues
• Present data policy
 In Europe, constrains the free distribution/exchange of seasonal forecast
data
 Policy is not fixed in stone, and may evolve over time
• Science
 Want to make sure that scientific studies are hindered as little as possible
 CHFP is main research project on seasonal prediction; data policy has
been OK, resources for data exchange were long a sticking point.
 High level support for new projects may be helpful
• Real-time forecasts
 Some data can be used by /supplied to WMO
 Need to ensure that it is enough
 Need to ensure that important “public good” applications are supported
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
29
Conclusions
• Seasonal prediction still exciting and challenging
• Mid-latitude skill and reliability still need much work
• Higher resolution seems helpful
• Testing/assessing/selecting models needs to cut across
timescales
• Coordinated experimentation has potential to be valuable,
beyond CHFP
• Careful design will make it easier for operational centre’s
to participate
Workshop on Sub-seasonal to Seasonal Prediction, Exeter, 1-3 December 2010
30