Transcript Data Assimilation Research Testbed Tutorial

Open Source Ensemble Kalman Filtering:
the Data Assimilation Research Testbed - DART
Tim Hoar, Jeffrey Anderson, Nancy Collins,
Kevin Raeder, Hui Liu, Glen Romine
NCAR Institute for Mathematics Applied to Geophysics
10 years in 12 hours
Sunday Afternoon (4 hours):
•Introductions / making teams
•Configure environment
•DART WWW-site
•Download DART
•DART_LAB – Matlab-based exercises to learn DA
Monday Morning (4 hours):
• Recap of yesterday / questions
• Select chapters of the DART tutorial
• Moving from toy models to large models
Monday Afternoon (4 hours):
•Diagnostics
•Testing Strategies … 1 observation, please.
•Real Observations
•CLM
CAHMDA-V July 2012 pg 2
Sunday Afternoon
Introductions: I need to know where to start and what to
cover. As you introduce yourself, please let me know how
much experience you have with each of the following:
•Unix/Linux command line
•Shell programming (i.e. csh, ksh, sh)
•vi / emacs / kedit / …
•Matlab / NCL / IDL / NCO / R
•Ensemble Data Assimilation Theory
I cannot give individual attention to this many people during these
exercises. We will need to make teams of 3 that will need to work
together during this tutorial. We need to make teams that have some
experience in each of the skills listed above. When I tell you to go stand
in a corner, do not take it personally! I mean no disrespect!
CAHMDA-V July 2012 pg 3
Configuring your *nix environment on “x50”
•
•
•
•
•
•
•
•
•
•
Your $HOME directory …
Customizations … ‘dotfiles’
Compilers, Libraries, inconsistencies …
Your shell … csh, sh, bash, ksh, tcsh …
Your commands … $PATH, aliases …
Your favorite $EDITOR
Remote logins, X forwarding … Windows
Batch jobs
Being nice – lots of people on one machine – “top”
Being graceful – orphaned processes
CAHMDA-V July 2012 pg 4
DART “home page”
http://www.image.ucar.edu/DAReS/DART
The most useful (to me) pull-down menus:
•
•
•
•
Getting Started
Documentation
Diagnostics
Miscellany: Platform-specific Notes
CAHMDA-V July 2012 pg 5
Download DART
•Register for the DART code – really.
•Actually download the code
• we will cheat
•“svn” - making modifications with NO FEAR
•DART file tree / schematic
•DART documentation
•DART tutorial
•Review DART interface requirements …
•DART build mechanism – mkmf
•DART_LAB
CAHMDA-V July 2012 pg 6
What is Data Assimilation?
Observations combined with a Model forecast…
+
=
… to produce an analysis.
Overview article of DART:
Anderson, Jeffrey, T. Hoar, K. Raeder, H. Liu, N. Collins, R. Torn, A. Arellano, 2009:
The Data Assimilation Research Testbed: A Community Facility.
Bull. Amer. Meteor. Soc., 90, 1283–1296. doi:10.1175/2009BAMS2618.1
CAHMDA-V July 2012 pg 7
Ensemble Filter for Large Geophysical Models
1. Use model to advance ensemble (3 members here)
to time at which next observation becomes available.
Ensemble state
estimate after using
previous observation
(analysis)
Ensemble state
at time of next
observation
(prior)
CAHMDA-V July 2012 pg 8
Ensemble Filter for Large Geophysical Models
2. Get prior ensemble sample of observation, y = h(x), by
applying forward operator h to each ensemble member.
Theory: observations
from instruments with
uncorrelated errors can
be done sequentially.
CAHMDA-V July 2012 pg 9
Ensemble Filter for Large Geophysical Models
3. Get observed value and observational
error distribution from observing system.
CAHMDA-V July 2012 pg 10
Ensemble Filter for Large Geophysical Models
4. Find the increments for the prior observation ensemble
(this is a scalar problem for uncorrelated observation errors).
Note: Difference between
various ensemble filters is
primarily in observation
increment calculation.
CAHMDA-V July 2012 pg 11
Ensemble Filter for Large Geophysical Models
5. Use ensemble samples of y and each state variable to linearly
regress observation increments onto state variable increments.
Theory: impact of
observation increments on
each state variable can be
handled independently!
CAHMDA-V July 2012 pg 12
Ensemble Filter for Large Geophysical Models
6. When all ensemble members for each state variable
are updated, there is a new analysis. Integrate to time
of next observation …
CAHMDA-V July 2012 pg 13
DART_LAB: Matlab-based tutorial
• DART/DART_LAB/presentations
•
•
•
•
Section 1 … the 1D perspective
Section 2 … impacting an unobserved state variable
Section 3 … sampling error and localization
Section 4 … perturbed observations (EnKF)
• DART/DART_LAB/matlab
•
•
•
•
Section 1 … gaussian_product, oned_ensemble, oned_model
Section 2 … twod_ensemble, run_lorenz_63, run_lorenz_96
Section 3 … run_lorenz_96
Section 4 … oned_ensemble, twod_ensemble, oned_model,
run_lorenz_63 and run_lorenz_96 all allow selection of EnKF.
I’m going to focus on
CAHMDA-V July 2012 pg 14
Ensemble Filter for Large Geophysical Models
A generic ensemble filter system like DART just needs:
1. A way to make model forecasts;
2. A way to compute forward operators, h.
CAHMDA-V July 2012 pg 15
That’s all for today ….
CAHMDA-V July 2012 pg 16
Monday morning
Introductions and lies … which of these is true about me?
“I have seen all 7 continents.”
“I am a competitive square dancer.”
CAHMDA-V July 2012 pg 17
Monday morning
•
•
•
•
Questions from yesterday
netCDF : ncdump, ncview
Matlab customizations for DART …
DART tutorial (in the interest of time, we’re skipping a lot
– at your leisure go back and be complete)
•
•
•
•
•
•
•
•
•
Sections 1,2,3 in their entirety (but quickly)
Section 4: skip to pg 29
Section 5: only pg 15
Section 7: introduces lorenz_96 (L96) pg 10
Section 8: sampling error (L96) pgs 10,13
Section 9: inflation (L96) after pg 15
Section 11: building DART
Section 14: observation quality control
Section 18: not knowing the truth
• Scripting for standalone executables – large models.
CAHMDA-V July 2012 pg 18
Monday Afternoon
• Diagnostics
• State-space (useful if you know the truth)
• Observation-space (useful in general)
• link_obs example with dev/POP
• obs_diag example wth dev/WRF
• Testing Strategies
• Lorenz_96 – perfect model experiment
• Observation sequence file creation
• Ameriflux data
• CLM/DART
•
•
•
•
•
CESM multi-instance facility
Modifying the run script
CLM variable specification – picking a state vector
Adding new DART kinds/types
Observation operators
CAHMDA-V July 2012 pg 19
Reasons to NOT reinvent the wheel
DART has proven methods to address the most common (and some
not-so-common) issues affecting the performance of ensemble filters:
•Inflation
Anderson, J. L., 2009
Spatially and temporally varying adaptive covariance inflation for ensemble filters.
Tellus A, 61, 72-83 doi:10.1111/j.1600-0870.2008.00361.x
Anderson, J. L., 2007.
An adaptive covariance inflation error correction algorithm for ensemble filters.
Tellus A, 59, 210-224. doi:10.1111/j.1600-0870.2006.00216.x
•Novel algorithms
Anderson, J.L., 2010
A Non-Gaussian Ensemble Filter Update for Data Assimilation.
Monthly Weather Review, 138, 4186-4198, doi:10.1175/2010MWR3253.1
Anderson, J. L., 2007
Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter.
Physica D, 230, 99-111. doi:10.1016/j.physd.2006.02.011
•Parallelization
Anderson, J., Collins, N., 2007.
Scalable Implementations of Ensemble Filter Algorithms for Data Assimilation.
Journal of Atmospheric and Oceanic Technology, 24, 1452-1463. doi:10.1175/JTECH2049.1
CAHMDA-V July 2012 pg 20
More reasons to NOT reinvent the wheel
•Diagnostics
•
•
•
Many routines/methods are provided to explore the performance of the assimilation.
Native ability to explore ‘value’ of different types of observations
Immediate ability to perform ‘perfect model’ experiments
•Documentation
•
•
•
•
•
Each code module has a companion HTML document to describe its use and purpose.
http://www.image.ucar.edu/DAReS/DART
All documentation/code available online
Workshop materials
Self-paced tutorials included in the download
•Portable, tested on many platforms
•
•
•
•
Free, open source.
Too many platforms/compilers to bother listing.
Distributed and maintained with subversion.
Can exploit, but does not need MPI.
•Humans!
• Tim, Nancy, Jeff, Kevin, Hui, Glen …all reached at [email protected]
CAHMDA-V July 2012 pg 21
Creating the initial ensemble of CLM.
Replicate what we have N times.
Use a unique (and different!) realistic DATM for each.
Run them forward for “a long time”.
model time
“spun up”
“a long time”
Getting a proper initial
ensemble is an area of
active research.
We don’t know how much spread we
NEED to capture the uncertainty in the
system.
CAHMDA-V July 2012 pg 22
The ensemble advantage.
You can represent
uncertainty.
In a free run,
the ensemble spread
frequently grows.
With a good assimilation:
ensemble spread
ultimately remains stable
and small enough to be
informative
observation times
CAHMDA-V July 2012 pg 23
O(1 million)
atmospheric obs are
assimilated every
day.
Atmospheric Reanalysis
Assimilation uses 80
members of 2o FV CAM
forced by a single ocean
(Hadley+ NCEP-OI2)
and produces a very
competitive reanalysis.
1998-2010
4x daily
is free and
available.
Contact
[email protected]
500 hPa GPH
Feb 17 2003
Contours
5200m:5700m by 100
CAHMDA-V July 2012 pg 24
Code to implement all of the algorithms
discussed is freely available from:
http://www.image.ucar.edu/DAReS/DART
[email protected]
CAHMDA-V July 2012 pg 25
DART is used at:
Public domain software for
ensemble Data Assimilation
– Well-tested, portable, scalable,
extensible, free!
Models
– Toy to HUGE
Observations
– Real, synthetic, novel
An extensive Tutorial
– With examples, exercises,
explanations
People: The DAReS Team
43 UCAR member universities
More than 100 other sites
Moving towards coupled assimilation
for earth system models.
Tim Hoar, Nancy Collins, Kevin Raeder, Jeffrey Anderson,
NCAR Institute for Math Applied to Geophysics
Data Assimilation Research Section
Steve Yeager, Mariana Vertenstein, Gokhan
Danabasoglu, Alicia Karspeck, and Joe Tribbia
NCAR/NESL/CGD/Oceanography
Hypothesis: Need Ensemble of Atmospheres to Force
Ensemble Assimilation for Ocean
• Case 1: 23 POP members
forced by a single atmosphere.
• Case 2: 48 POP members
forced by 48 CAM/DART
analyses.
• Case 2 Generates additional
ocean spread, improved
analyses.
500 hPa GPH
Feb 17 2003
DATM
2D forcing
from CAM
assimilation
DART
Obs
3D state
POP
Coupler
3D restart
2D forcing
Current POP Assimilation
from within the Climate
Earth System Model CESM
World Ocean Database T,S observation counts
These counts are for 1998 & 1999 and are representative.
FLOAT_SALINITY
FLOAT_TEMPERATURE
DRIFTER_TEMPERATURE
MOORING_SALINITY
MOORING_TEMPERATURE
BOTTLE_SALINITY
BOTTLE_TEMPERATURE
CTD_SALINITY
CTD_TEMPERATURE
STD_SALINITY
STD_TEMPERATURE
XCTD_SALINITY
XCTD_TEMPERATURE
MBT_TEMPERATURE
XBT_TEMPERATURE
APB_TEMPERATURE
68200
395032
33963
27476
623967
79855
81488
328812
368715
674
677
3328
5790
58206
1093330
580111
• temperature observation error standard deviation == 0.5 K.
• salinity observation error standard deviation == 0.5 msu.
Ensemble Spread for Pacific 100m XBT
Spread of the
“climatological”
ensemble
Twice as much!
Small spread!
100m Mooring Temperature RMSE – Pacific
POP/CAM as good
or better RMSE
Coupled Free Run
POP forced by observed
atmosphere (hindcast)
23 POP 1 DATM
48 POP 48 CAM
Physical Space: 1998/1999 SST Anomaly from HadOI-SST
Fully coupled
assimilation will
need data from
all models at the
same time
DART
Obs
CAM
POP
Coupler
CLM
This is a very
CESM-centric view of
fully coupled data
assimilation.
CICE
DART works with many geophysical models
Global Atmosphere models:
CAM
Community Atmosphere Model
NCAR
CAM/CHEM
CAM with Chemistry
NCAR
WACCM
Whole Atmosphere Community
Climate Model
NCAR
AM2
Atmosphere Model 2
NOAA/GFDL
NOGAPS
Navy Operational Global
Atmospheric Prediction System
US Navy
ECHAM
European Centre Hamburg Model Hamburg
Planet WRF
Global version of WRF
JPL
MPAS
Model for Prediction Across
Scales (under development)
NCAR/DOE
DART works with many geophysical models
Regional Atmosphere models:
WRF/ARW
Weather Research and
Forecast Model
NCAR
WRF/CHEM
WRF with Chemistry
NCAR
NCOMMAS
Collaborative Model for
NOAA/NSSL
Multiscale Atmospheric Simulation
COAMPS
Coupled Ocean/Atmosphere
Mesoscale Prediction System
US Navy
CMAQ
Community Multi-scale Air Quality
EPA
COSMO
Consortium for Small-Scale
Modeling
DWD
DART works with many geophysical models
Ocean models:
POP
Parallel Ocean Program
DOE/NCAR
MIT OGCM
Ocean General Circulation
Model
MIT
ROMS
Regional Ocean Modeling
System (under development)
Rutgers
MPAS
Model for Prediction Across
Scales (under development)
DOE/LANL
Land Surface models:
CLM
Community Land Model
(under development)
NCAR
DART works with many geophysical models
Upper Atmosphere/Space Weather models:
ROSE
NCAR
TieGCM
Thermosphere Ionosphere
Electrodynamic GCM
GITM
Global Ionosphere
Thermosphere Model
NCAR/HAO
Michigan