Transcript Document

Variational cloud retrievals
from radar, lidar and
radiometers
Robin Hogan
Julien Delanoë
Nicola Pounder
University of Reading
Introduction
• Best estimate of the atmospheric state from instrument synergy
– Use a variational framework / optimal estimation theory
• Some important measurements are integral constraints
– E.g. microwave, infrared and visible radiances
– Affected by all cloud types in profile, plus aerosol and precipitation
– Hence need to retrieve different particle types simultaneously
• Funded by ESA and NERC to develop unified retrieval algorithm
– For application to EarthCARE
– Will be tested on ground-based, airborne and A-train data
• Algorithm components
– Target classification input
– State variables
– Minimization techniques: Gauss-Newton vs. Gradient-Descent
– Status of forward models and their adjoints
• Progress with individual target types
– Ice clouds
– Liquid clouds
1. New ray of data: define state vector
Use classification to specify variables describing each species at each gate
Ice: extinction coefficient , N0’, lidar extinction-to-backscatter ratio
Liquid: extinction coefficient and number concentration
Rain: rain rate and mean drop diameter
Aerosol: extinction coefficient, particle size and lidar ratio
2. Convert state vector to radar-lidar resolution
Often the state vector will contain a low resolution description of the profile
3. Forward model
3a. Radar model
Including surface return
and multiple scattering
3b. Lidar model
Including HSRL channels
and multiple scattering
4. Compare to observations
Check for convergence
Not converged
Retrieval
framework
Ingredients developed before
In progress
Not yet developed
6. Iteration method
Derive a new state vector
Either Gauss-Newton or
quasi-Newton scheme
3c. Radiance model
Solar and IR channels
5. Convert Jacobian/adjoint to state-vector resolution
Initially will be at the radar-lidar resolution
Converged
7. Calculate retrieval error
Error covariances and averaging kernel
Proceed to next ray of data
Target classification
• In Cloudnet we used radar and lidar to provide a detailed
discrimination of target types (Illingworth et al. 2007):
• A similar approach has been used by Julien Delanoe on CloudSat and
Calipso using the one-instrument products as a starting point:
• More detailed classifications could distinguish “warm” and “cold” rain
(implying different size distributions) and different aerosol types
Example from the AMF in Niamey
94-GHz radar reflectivity
Observations
532-nm lidar backscatter
Forward
model at final
iteration
94-GHz radar reflectivity
532-nm lidar backscatter
Results: radar+lidar only
Retrievals in
regions where
radar or lidar
detects the cloud
Retrieved visible extinction coefficient
Retrieved effective radius
Large error
where only one
instrument
detects the
cloud
Retrieval error in ln(extinction)
Results: radar, lidar, SEVERI radiances
Delanoe & Hogan (JGR 2008)
TOA radiances
increase
retrieved optical
depth and
decrease particle
size near cloud
top
Retrieved visible extinction coefficient
Retrieved effective radius
Cloud-top
error greatly
reduced
Retrieval error in ln(extinction)
Unified algorithm: state variables
• Proposed list of retrieved variables held in the state vector x
State variable
Representation with height / constraint
A-priori
Ice clouds and snow
Visible extinction coefficient
One variable per pixel with smoothness constraint
None
Number conc. parameter
Cubic spline basis functions with vertical correlation
Temperature dependent
Lidar extinction-to-backscatter ratio
Cubic spline basis functions
20 sr
Riming factor
Likely a single value per profile
1
Liquid clouds
Liquid water content
One variable per pixel but with gradient constraint
None
Droplet number concentration
One value per liquid layer
Temperature dependent
Rain rate
Cubic spline basis functions with flatness constraint
None
Normalized number conc. Nw
One value per profile
Dependent on whether from
melting ice or coallescence
Melting-layer thickness scaling factor
One value per profile
1
Rain
Aerosols
Extinction coefficient
One variable per pixel with smoothness constraint
None
Lidar extinction-to-backscatter ratio
One value per aerosol layer identified
Climatological type
depending on region
Ice clouds follows
Delanoe & Hogan
(2008); Snow &
riming in
convective clouds
needs to be added
Liquid clouds
currently being
tackled
Basic rain to be
added shortly; Full
representation later
Basic aerosols to
be added shortly;
Full representation
via collaboration?
The cost function
• The essence of the method is to find the state vector x that
minimizes a cost function:
The forward model H(x)
predicts the observations
from the state vector x
ny
J 
i 1
Some elements of x
are constrained by an
a priori estimate
 yi  H (x)
2

2
yi
Each observation yi is
weighted by the inverse of
its error variance
nx

i 1
 xi  bi 
2

2
bi
+ Smoothness
constraints
This term penalizes
curvature in the
extinction profile
Gauss-Newton method
•
See Rodgers’ book (p85): write the cost function in matrix form:
J
•
1
y  H (x)T R 1y  H (x)  1 x  a T B 1 x  a 
2
2
Define its gradient (a vector):
x J  HT R 1y  H (x)  B1 x  a
•
…and its second derivative (a matrix):
2x J  HT R 1H  B1
•
Approximate J as quadratic and apply this:

x i 1  x i   J
2
x

1
x J
– Advantage: rapid convergence (instant convergence for linear problems)
– Another advantage: get the error covariance of the solution “for free”
– Disadvantage: need the Jacobian matrix of every forward model: can be
expensive for larger problems
Gradient descent methods
•
Just use gradient information:
xi 1  xi  x J
– Advantage: we don’t need to calculate the Jacobian so forward model is cheaper!
– Disadvantage: more iterations needed since we don’t know curvature of J(x)
– Use a quasi-Newton method to get the search direction, such as BFGS used by
ECMWF: builds up an approximate form of the second derivative to get
improved convergence
– Scales well for large x
– Disadvantage: poorer estimate of the error at the end
•
Why don’t we need the Jacobian H?


J obs  HT R1y  H (x)
Typical convergence behaviour
– The “adjoint” of a forward model takes as input the vector {.} and outputs the
vector Jobs without needing to calculate the matrix H on the way
– Adjoint can be coded to be only ~3 times slower than original forward model
– Tricky coding for newcomers, although some automatic code generators exist
Forward model components
• From state vector x to forward modelled observations H(x)...
Ice & snow
Liquid cloud
Rain
Aerosol
Jacobian matrix
x
H=y/x
Lookup tables to obtain profiles of extinction, scattering
& backscatter coefficients, asymmetry factor
Ice/radar
Ice/lidar
Ice/radiometer
Liquid/radar
Liquid/lidar
Liquid/radiometer
Rain/radar
Rain/lidar
Rain/radiometer
Aerosol/lidar
Aerosol/radiometer
Computationally expensive matrixmatrix multiplications: the most
expensive part of the entire algorithm
Sum the contributions from each constituent
Radar scattering
profile
Lidar scattering
profile
Radiative transfer models
Radar forward
modelled obs
Lidar forward
modelled obs
Radiometer
scattering profile
H(x)
Radiometer forward
modelled obs
Gradient of radar
measurements
with respect to
radar inputs
Gradient of lidar
measurements
with respect to
lidar inputs
Gradient of radiometer
measurements with
respect to radiometer
inputs
Jacobian part of radiative transfer models
Equivalent adjoint method
• From state vector x to forward modelled observations H(x)...
Ice & snow
Liquid cloud
Rain
Aerosol
x
Gradient of cost function (vector)
J/x=HTy*
Lookup tables to obtain profiles of extinction, scattering
& backscatter coefficients, asymmetry factor
Ice/radar
Ice/lidar
Ice/radiometer
Liquid/radar
Liquid/lidar
Liquid/radiometer
Rain/radar
Rain/lidar
Rain/radiometer
Aerosol/lidar
Aerosol/radiometer
Vector-matrix multiplications: around
the same cost as the original forward
operations
Sum the contributions from each constituent
Radar scattering
profile
Lidar scattering
profile
Radiative transfer models
Radar forward
modelled obs
Lidar forward
modelled obs
Radiometer
scattering profile
H(x)
Radiometer forward
modelled obs
Adjoint of radar
model (vector)
Adjoint of lidar
model (vector)
Adjoint of radiometer
model
Adjoint of radiative transfer models
Scattering models
• First part of a forward model is the scattering and fall-speed model
– Same methods typically used for all radiometer and lidar channels
– Radar and Doppler model uses another set of methods
Particle type
Aerosol
Radar (3.2 mm)
Aerosol not
detected by radar
Liquid droplets
Mie theory
Rain drops
T-matrix: Brandes
et al. (2002)
shapes
Ice cloud
T-matrix (Hogan et
particles
al. 2010)
Graupel and hail Mie theory
Melting ice
Wu & Wang
(1991)
Radar Doppler
Aerosol not
detected by radar
Beard (1976)
Beard (1976)
Thermal IR, Solar, UV
Mie theory, Highwood
refractive index
Mie theory
Mie theory
Westbrook &
Heymsfield
TBD
TBD
Baran (2004)
Mie theory
Mie theory
• Graupel and melting ice still uncertain; but normal ice is decided...
Radiative transfer forward models
•
Computational cost can scale with number of points describing vertical profile
N; we can cope with an N2 dependence but not N3
Radar/lidar model
Applications
Speed
Jacobian
Adjoint
Single scattering: b’=b exp(-2t)
Radar & lidar, no multiple scattering
N
N2
N
Platt’s approximation b’=b exp(-2ht)
Lidar, ice only, crude multiple scattering
N
N2
N
Photon Variance-Covariance (PVC)
method (Hogan 2006, 2008)
Lidar, ice only, small-angle multiple
scattering
N or N2
N2
N
Time-Dependent Two-Stream (TDTS)
method (Hogan and Battaglia 2008)
Lidar & radar, wide-angle multiple scattering
N2
N3
N2
Depolarization capability for TDTS
Lidar & radar depol with multiple scattering
N2
•
•
•
•
N2
Lidar uses PVC+TDTS (N2), radar uses single-scattering+TDTS (N2)
Jacobian of TDTS is too expensive: N3
We have recently coded adjoint of multiple scattering models
Future work: depolarization forward model with multiple scattering
Radiometer model
Applications
RTTOV (used at ECMWF & Met Office)
Infrared and microwave radiances
N
Two-stream source function technique
(e.g. Delanoe & Hogan 2008)
Infrared radiances
N
N2
LIDORT
Solar radiances
N
N2
•
•
Speed
Jacobian
Adjoint
N
Infrared will probably use RTTOV, solar radiances will use LIDORT
Both currently being tested by Julien Delanoe
N
Gradient constraint
A. Slingo, S. Nichols and J. Schmetz,
Q. J. R. Met. Soc. 1982
• We have a good constraint on the gradient of the state variables
with height for:
– LWC in stratocu (adiabatic profile, particularly near cloud base)
– Rain rate (fast falling so little variation with height expected)
• Not suitable for the usual “a priori” constraint
• Solution: add an extra term to the cost function to penalize
deviations from gradient c:
2
J
 dxi

 
c

i  dz
Example in liquid clouds
• Using simulated observations:
– Triangular cloud observed by a 1- or 2-field-of-view lidar
– Retrieval uses Levenberg-Marquardt minimization with Hogan and
Battaglia (2008) model for lidar multiple scattering
Optical depth=30
Footprint=100m
Footprint=600m
Two FOVs: very good performance
One 10-m footprint: saturates at optical depth=5
One 100-m footprint: multiple scattering helps!
One 10-m footprint with gradient constraint: can
extrapolate downwards successfully
Progress
• Done:
– C++: object orientation allows code to be completely flexible:
observations can be added and removed without needing to keep track
of indices to matrices, so same code can be applied to different
observing systems
– Code to generate particle scattering libraries in NetCDF files
– Adjoint of radar and lidar forward models with multiple scattering and
HSRL/Raman support
– Radar/lidar model interfaced and cost function can be calculated
• In progress / future work:
– Interface to BFGS algorithm (e.g. in GNU Scientific Library)
– Implement ice, liquid, aerosol and rain constituents
– Interface to radiance models
– Test on a range of ground-based and spaceborne instruments
– Test using ECSIM observational simulator
– Apply to large datasets of ground-based observations…