Transcript Evaluation of the ability of Numerical Weather Prediction models run in support of IHOP to predict the evolution of Mesoscale Convective Systems (Koch)

Evaluation of the ability of Numerical
Weather Prediction models run in
support of IHOP to predict the evolution
of Mesoscale Convective Systems
Steve Koch, Linda Wharton, Andy Loughe
Forecast Systems Laboratory
Bill Gallus, Jeremy Grams
Iowa State University
Beth Ebert
Australian Bureau of Meteorology
Background
(Weisman, Skamarock, and Klemp, 1997: The resolution dependence of explicitly
modeled convective systems, Mon. Wea. Rev., 125, 527-548)
1.
Results from 3D midlatitude squall-line
simulations suggest that 4 km grid size is
needed to reproduce much of the
mesoscale structure and evolution of
MCSs seen in 1-km simulations.
2.
Evolution at resolutions coarser than 4 km
is characteristically slower, due largely to a
delayed strengthening of the cold pool.
3.
Cold pool is crucial to the evolution of an
MCS into an upshear-tilted mature
system.
4.
An overly strong mesoscale circulation
and overprediction of precipitation results
at resolutions coarser than 4 km.
MCS: Mesoscale Convective System
Objectives of this Study
1.
Determine forecast precipitation properties for each type of observed MCS for
the 12-km NWP models run during IHOP (Eta, MM5, and WRF). Operational
models in the U.S. use resolutions of 12-20 km presently, not 4 km.
2.
Traditional “measures-oriented” verification statistics (RMS, bias, etc.) severely
penalize an incorrectly located precipitation system that may be forecast with
only small positional or shape error, yet have practical forecast utility. We use an
“object-oriented” verification technique in this study.
3.
For each MCS type, we obtain systematic model performances by using the
Ebert-McBride (2000) technique to determine the fractional contribution of
forecast precipitation displacement, intensity, and shape errors.
4.
The EM technique was implemented in the Real-Time Verification System
(RTVS) at FSL and changes were made to the EM code as needed to make it
applicable to mesoscale systems in the central U.S.
Ebert-McBride (EM) Verification Technique
EM Technique takes maximum of observed and forecasted rain at all points
and determines Contiguous Rain Areas (CRAs) exceeding specified isohyet
FCST
FCST
OBS
CRA BOX
Expanded CRA box
Forecast is permitted to shift within expanded CRA box by userdefined amount, until either RMSE is minimized or correlation
coefficient maximized
Application of the EM technique to
6h accumulated precipitation
ending at 0600 UTC 13 June 2002
Strategy using Ebert-McBride technique
•
Eta, WRF, and MM5 12-km runs from IHOP period were analyzed using
the EM Contiguous Rain Area (CRA) technique
•
Observed CRAs were assigned a morphology based on 2-km
composite reflectivity radar images (30 minute time resolution)
•
We classified the morphology of MM5 and WRF model convective
systems using hourly reflectivity output from the models. Only sixhourly output was available for the Eta model.
•
Required that an MCS meet the following criteria for at least 3 hours:
30 dBZ (~3 mm/h) over at least a 100 x 100 km area
40 dBZ (~13 mm/h) over at least a 50 x 50 km area
Improvements to EM CRA code
1.
Percent of grid points allowed to shift off domain reduced from
50% to 0.1% for RMSE minimization and to 25% for correlation
coefficient maximization (removes problem of displacements
usually being off edge)
2.
Plots changed to show entire expanded CRA region, with shifted
forecast overlaid on observed rainfall chart
3.
Correlation Coefficient maximization seems to produce more
reasonable results than minimization of RMSE, but the error
decomposition required development of a new decomposition (four
terms) based on Murphy (1995)
4.
Critical mass threshold for 24 hr periods was reduced by a factor
of 4, allowing a greater number of smaller CRAs to be identified
Improvements to EM CRA code
5.
Increasing threshold amount from .25 to .50 inch (13 mm) per 6 hours
helps to identify distinct systems, but statistics are then computed over
too small an area (so we kept the threshold at .25 inch)
6.
Because models at best only resolve 6Dx features, a Lanczos filter was
introduced to filter observations – creating patterns more similar to that
forecast in the models
7.
Sensitivity tests of tuneable parameters such as expansion of CRA
domain, and minimum size (grid points) of CRA/observed rainfall area
were performed, but results did not indicate the need for changes
8.
CRA statistics for error decomposition continue to be computed over
union of observed, forecasted and shifted forecast of rain. Volume and
average rain rate are determined just over appropriate portion of CRA
MCS Morphology Classification
•
Linear
o
Linear (CL)
o
Linear Bowing (CLB)
o
Linear (CL & CLB) sub-classifications (Parker and Johnson 2000):

o
Squall Line Developmental Types (Bluestein and Jane 1985):

•
•
Trailing Stratiform (TS), Leading Stratiform (LS), Parallel Stratiform (PS)
Broken Areal (BA), Broken Line (BL), Backbuilding (BB), Embedded Areal (EA)
Non-linear
o
Continuous Non-Linear (CNL)
o
Discontinuous Areal (DA)
o
Isolated Cells (IC)
Orographically Fixed (OF)
In the 6-hour CRA window, if multiple types were observed, then the type dominating most
of the time was used.
If multiple systems were observed in one CRA, then the system with greater temporal,
spatial, and rain volume was used.
Number of Linear MCSs in Radar Data
120
111
100
89
80
61
60
38
40
28
20
0
20
19
9 8
CL
CLB
General
TS
LS
4
PS TS_PS LS_PS
Squall
3
BA
BL
BB
EA
Development
Number of Nonlinear MCSs in Radar Data
140
121
120
96
100
80
CNL
DA
IC
60
37
40
20
0
Rain Volume
Pre-MM5
•The version of the MM5 model
(“Pre-MM5”) run in real-time
during IHOP by FSL exhibited
very large wet biases.
•These results compelled FSL to
make major changes to the “Hot
Start” diabatic initialization, both
during the field phase and for
several months thereafter.
• Those changes resulted in a
removal of the bias for linear
MCSs, but a slight wet bias was
maintained with non-linear MCSs
8
7
6
5
4
3
2
1
0
7.49
4.67
3.08
2.46
1.21
0.51
All
Linear
Non-Linear
3.50
3.00
2.50
2.00
MM5-fcst
MM5-obs
Eta-fcst
Eta-obs
1.50
1.00
0.50
0.00
CL
CLB
CNL
DA
IC
Maximum Rainfall
3.5
3.0
2.5
in/6-hr
• The post-MM5 overpredicts
rainfall maxima for all MCS
categories except CNL
2.0
1.5
MM5-fcst
MM5-obs
1.0
0.5
0.0
CL
CLB
CNL
DA
IC
2.5
• Conversely, the Eta
underpredicts rainfall maxima
for all categories except IC
in/6-hr
2.0
1.5
1.0
Eta-fcst
Eta-obs
0.5
0.0
CL
CLB
CNL
DA
IC
Frequency Classification of MCSs
MCS Type
Radar (%)
Eta (%)
MM5 (%)
WRF (%)
CNL
31
33
36
34
DA
25
25
10
19
IC
10
4
3
1
Total Nonlinear
66
62
49
54
CL
29
33
42
38
CLB
5
5
9
8
Total Linear
34
38
51
46
Distribution of Forecast Rainfall
Errors by Error Type for the Five
Basic MCS categories
Displacement Errors: ETA
Displacement Errors: MM5
Displacement Errors: WRF
Conclusions
• Displacement vectors (polar plots):
o None of the models displays a strongly preferred (or systematic) direction
and magnitude of displacement vectors, either for any particular MCS or
between MCS classes, except for the linear CL and CLB types, which
were forecast too slowly (north or northwest) by the WRF and MM5
models
o This may suggest cold pools for squall line systems were forecast to be
too weak, which if true is consistent with the idealized study of squall lines
by Weisman et al. (1997)
• Decomposition of rainfall errors (histograms):
o Overall, for all three models and all MCS types, the largest contributors to
total MSE are conditional bias and pattern errors, with volume error
consistently being smallest
o This was a major problem for CLB type, suggesting poorly forecast bowing
structures and with the wrong intensity for the convective lines
o Nonlinear systems (CNL and DA) were forecast with the least error,
though all three models did display a significant bias for these systems