Slide 1

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 2

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 3

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 4

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 5

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 6

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 7

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 8

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 9

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 10

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 11

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 12

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 13

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 14

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 15

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 16

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 17

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 18

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 19

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 20

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 21

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 22

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 23

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 24

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 25

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 26

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 27

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 28

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 29

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 30

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 31

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 32

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 33

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 34

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 35

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 36

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 37

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 38

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 39

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 40

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 41

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 42

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 43

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 44

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 45

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 46

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 47

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 48

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 49

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 50

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 51

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 52

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 53

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 54

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 55

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION


Slide 56

International Conference on Environmental Observations,
Modeling and Information Systems ENVIROMIS-2004
17-25 July 2004, Tomsk, Russia

Mathematical modeling of natural and
anthropogenic change of regional climate
and environment
V.N. Lykosov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected]

Climate System
• 1. ATMOSPHERE – the gas envelope of the Earth (oxygen, nitrogen,
carbon dioxide, water vapor, ozone, etc.), which controls the solar radiation
transport from space towards the Earth surface.
• 2. OCEAN – the major water reservoir in the system, containing salted
waters of the World ocean and its seas and absorbing the basic part of the
incoming solar radiation (a powerful accumulator of energy).
• 3. LAND – surface of continents with hydrological system (inland waters,
wetlands and rivers), soil (e.g. with groundwater) and cryolithozone
(permafrost).

• 4. CRYOSPHERE – continental and see ice, snow cover and mountain
glaciers.
• 5. BIOTA – vegetation on the land and ocean, alive organisms in the air,
water and soil, mankind.

The Climate System(T. Slingo, 2002)

19
37
-46
19
39
-48
19
41
-50
19
43
-52
19
45
-54
19
47
-56
19
49
-58
19
51
-60
19
53
-62
19
55
-64
19
57
-66
19
59
-68
19
61
-70
19
63
-72
19
65
-74
19
67
-76
19
69
-78
19
71
-80
19
73
-82
19
75
-84
19
77
-86
19
79
-88
19
81
-90
19
83
-92
19
85
-94
19
87
-96
19
89
-98

C degrees

Annually mean air temperature in Khanty-Mansiisk for the time
period from 1937 to 1999 (Khanty-Mansiisk
Hydrometeocenter).

0

-0,5

-1

-1,5

-2

-2,5

-3

Decades

Air temperature. May 2003 (Khanty-Mansiisk
Hydrometeocenter)

30

25

C degrees

20

15

10

5

0
1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Среднесуточная температура
Средняя температура

Максимальная температура
Норма

Features of the climate system as physical object-I
• Basic components of the climate system – atmosphere and
ocean – are thin films with the ratio of vertical scale to
horizontal scale about 0.01-0.001.
• On global and also regional spatial scales, the system can be
considered as quasi-twodimensional one. However, its density
vertical stratification is very important for correct description
of energy cycle.
• Characteristic time scales of energetically important physical
processes cover the interval from 1 second (turbulence) to
tens and hundreds years (climate and environment variability).
• Laboratory modelling of such system is very difficult.

Features of the climate system as physical object-II
• It is practically impossible to carry out specialized physical
experiments with the climate system.
• For example, we have no possibility to “pump” the atmosphere by the
carbon dioxide and, keeping other conditions, to measure the system
response.
• We have shirt–term series of observational data for some of
components of the climate system.

• Conclusion: the basic (but not single) tool to study the climate system
dynamics is mathematical (numerical) modeling.
• Hydrodynamical climate models are based on global models of the
atmosphere and ocean circulation.

Objectives of climate modeling


To reproduce both “climatology” (seasonal and monthly means) and
statistics
of
variability:
intra-seasonal
(monsoon
cycle,
characteristics of storm-tracks, etc.) and climatic (dominated modes
of inter-annual variability such as El-Nino phenomenon or Arctic
Oscillation)



To estimate climate change due to anthropogenic activity



To reproduce with high degree of details regional climate: features of
hydrological cycle, extreme events, impact of global climate change
on regional climate, environment and socio-economic relationships



Fundamental question (V.P. Dymnikov): what climatic parameters
and in what accuracy must by reproduced by a mathematical model
of the climate system to make its sensitivity to small perturbations of
external forcing close to the sensitivity of the actual climate
system?

Computational technologies
• Global climate model (e.g. model with improved spatial resolution
in the region under consideration) implemented on computational
system of parallel architecture (CSPA)
•

Methods of “regionalization”:

1) statistical approach (“downscaling”);
2) hydrodynamical mesoscale simulation ( e.g., mesoscale model
ММ5) requires CSPA;

3) large-eddy simulation of geophysical boundary layers (requires
CSPA)
• Assessment of global climate change and technological impact on
regional environment

Observational data to verify models
• 1) ECMWF reanalysis ERA-15 (1979-1993 г.г.), ERA-40 (1957-2001)
http://www.ecmwf.int/research/era
• 2) NCEP/NCAR (National Center of Environment Protection/National
Center of Atmospheric Research, USA), 1958-1997,
http://wesley.wwb.noaa.gov/reanalysis.htm
• 3) Precipitation from 1979 to present time
http://www.cpc.ncep.noaa.gov/products/globalprecip

• 4) Archive NDP048, containing multiyear data of routine observations on
225
meteorological
stations
of
the
former
USSR
http://cdiac.esd.oml.gov/ftp/ndp048
• Numerical experiments with modern global climate models produce a
large amount of data (up to 1Gb for 1 month). This requires special
efforts for its visualization, postprocessing and analysis.

where

Large-scale hydrothermodynamics of the atmosphere
u
1
RT   


 
 f 
tg   v 


  Fu
dt
a
a cos    
  



du

u

 f 
tg 
dt
a


dv

1  
RT   

  Fv

 u  
a












t

dT



dt






 
u

v  


   FT  
    
c p  

t
a
cos



a




RT




t

Subgrid-scale
processes
parameterization

,

 u
  v cos     

 

 0,
a cos    




dt

dt

RT

,

1

dq

d

 

,



u

 F q  ( C  E ),



a cos   



v 
a 

 




.

,

Parameterization of subgrid-scale processes
• Turbulence in the atmospheric boundary layer, upper ocean layer
and bottom boundary layer
• Convection and orographic waves
• Diabatic heat sources (radiative and phase changes, cloudiness,
precipitation, etc.)
• Carbon dioxide cycle and photochemical transformations
• Heat, moisture and solute transport in the vegetation and snow cover
• Production and transport of the soil methane
• Etc.

T.J. Philips et al. (2002). Large-Scale Validation of
AMIP II Land-Surface Simulations

Table 2. Model codes and features of the sixteen AMIP2 models analysed in Zhang et al. (2002)
Code

Resolution

Land-surface components

Soil model

No. of layers

No. of layers

Model

in soil temp.

in soil moist.

Country

calculations

calculations

Canopy representation

complexity
A

T42L18

bucket

const. canopy resistance

3

1

CCSR, Japan

B

T63L45

force-restore

intercept. + transpiration

2

2

CNRM, France

C

4x5 L21

multi-layer diffusion

intercept. + transpiration

24

24

INM, Russia

D

T159L50

multi-layer diffusion

intercept. + transpiration

4

4

ECMWF, UK

E

T63L30

multi-layer diffusion

intercept. + transpiration

4

3

JMA, Japan

F

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

NCAR, USA

G

T62L18

multi-layer diffusion

intercept. + transpiration

3

2

NCEP, USA

H

T42L18

multi-layer diffusion

intercept. + transpiration

2

3

PNNL, USA

I

3.75x2.5 L58

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UGAMP, UK

J

3.75x2.5 L19

multi-layer diffusion

intercept.+transpiration+CO2

4

4

UKMO, UK

K

T47L32

multi-layer diffusion

intercept. + transpiration

3

3

CCCMA, Can

L

4x5 L20

multi-layer diffusion

intercept. + transpiration

2

3

GLA, USA

M

T42L30

multi-layer diffusion

intercept. + transpiration

3

3

MRI, Japan

T42L18

multi-layer diffusion

intercept.+transpiration+CO2

6

6

SUNYA, USA

4x5 L24

bucket

no

1

1

UIUC, USA

4x5 L15

bucket

no

1

1

YONU, Korea

N
O
P

The Taylor diagram for the variability of the latent heat flux at the
land surface as follows from results of AMIP-II experiments
(Irannejad et al., 2002).

Sensitivity of the climate system to
small perturbations of
external forcing
(invited lecture at the World Climate Conference,
Moscow, 29 September – 3 October, 2003)
V.P. Dymnikov, E.M. Volodin, V.Ya. Galin, A.S.
Gritsoun, A.V. Glazunov, N.A. Diansky, V.N. Lykosov

Institute of Numerical Mathematics RAS, Moscow

INM coupled atmosphere - ocean general
circulation model
AGCM
- Finite difference model with spatial resolution 5°x4° and 21 levels in sigmacoordinates from the surface up to 10 hPa.
- In radiation absorption of water vapour, clouds, CO 2, O3, CH4, N2O, O2 and
aerosol are taken into account. Solar spectrum is divided by 18 intervals, while
infrared spectrum is divided by 10 intervals.
- Deep convection, orographic and non-orographic gravity wave drag are considered
in the model. Soil and vegetation processes are taken into account.
||
Non-flux-adjusted coupling
||

OGCM
-The model is based on the primitive equations of the ocean dynamics in spherical
sigma-coordinate system. It uses the splitting-up method in physical processes and
spatial coordinates. Model horizontal resolution is 2.5°x2°, it has 33 unequal levels in
the vertical with an exponential distribution.

The climate model sensitivity to
the increasing of CO2
CMIP - Coupled Model Intercomparison Project
http://www-pcmdi.llnl.gov/cmip

CMIP collects output from global coupled ocean-atmosphere general
circulation models (about 30 coupled GCMs). Among other usage, such models
are employed both to detect anthropogenic effects in the climate record of the
past century and to project future climatic changes due to human production of
greenhouse gases and aerosols.

Response to the increasing of CO2
CMIP models (averaged)
INM model

Global warming in CMIP
models in CO2 run and
parameterization of lower
inversion clouds
T - global warming (K), LC - parameterization
of lower inversion clouds (+ parameterization
was included, - no parameterization, ? - model
description is not available). Models are ordered
by reduction of global warming.
M odel

T

LC

N C A R -W M
GFDL

3.77
2.06

?
-

LM D

1.97

-

CCC

1.93

-

UKM03

1.86

-

CERF

1.83

-

C C SR

1.75

-

C S IR O

1.73

+

G IS S

1.70

-

UKM O

1.59

-

BM RC

1.54

+

ECHAM 3

1.54

-

MRI

1.50

-

IA P

1.48

+

N C A R -C S M

1.26

+

PCM

1.14

+

IN M

0.99

+

NRL

0.75

+

Mesoscale non-hydrostatic modeling
• MM5 - Penn State/NCAR Mesoscale
Modeling System
http://www.mmm.ucar.edu/mm5
• Program code: Fortran77, Fortran90, C.
Hybrid parallelization (shared and
distributed memory) + vectorization
• Documentation
• Implemented by V. Gloukhov
(SRCC/MSU) on MVS-1000M

International Conference on
Computational Mathematics ICCM-2004,
21-25 June, 2004, Novosibirsk, Russia

Large-Eddy Simulation of
Geophysical Boundary Layers on
Parallel Computational Systems
V.N. Lykosov, A.V. Glazunov
Russian Academy of Sciences
Institute for Numerical Mathematics, Moscow
E-mail: [email protected], [email protected]

Geophysical Boundary Layers (GBLs) as
elements of the Earth climate system
• Atmospheric Boundary Layer HABL ~ 102 - 103 m
• Oceanic Upper Layer
HUOL ~ 101 - 102 m
• Oceanic Bottom Layer
HOBL ~ 100 - 101 m

GBL processes control:

• 1) transformation of the solar radiation energy at the atmosphereEarth interface into energy of atmospheric and oceanic motions
• 2) dissipation of the whole Earth climate system kinetic energy

• 3) heat- and moisture transport between atmosphere and soil (e.g.
permafrost), sea and underlying ground (e.g. frozen one).

Dynamic structure of GBLs
Three types of motion:

• totally organized mean flow
• coherent semi-organized structures (large
eddies and waves)
• chaotic three-dimensional turbulence

Turbulence in PBLs

Rough surface
● Large scales
● Stratification
●

Synoptical variations

Boundary-Layer flows

Energy range
Inertial range

Dissipation range

Differential formulation of models
Models are based on Reynolds’ type equations obtained after spatial averaging of Navier-Stokes equations and added by equations of heat and moisture (or salt):

Turbulent Closure

Equation for turbulent kinetic energy (of subgrid-scale motions):

Equation for turbulent kinetic energy dissipation:

Constraint on maximal value of sub-grid turbulence length scale:

l

c E



3/2

,

Numerical scheme
• Finite-difference approximation on “C” grid

• Explicit time scheme of predictor-corrector type (Matsuno scheme)

Calculation of tendencies
Velocity components
diffusion
Coriolis force
gravity

Calculation of
eddy viscosity and diffusion
coefficients

advection
pressure gradient
Temperature, moisture, salinity

Calculation of the Poisson
equation R.H.S.
Solver for the Poisson equation

diffusion
advection
ТКЕ,
ТКЕ
dissipation

ТКЕ, ТКЕ dissipation
diffusion
advection
production, dissipation,
non-linear terms

Summing up
of tendencies
Input-output,
post-processing

Boundary conditions

velocity components

temperature,
moisture, salinity

ТКЕ, ТКЕ dissipation

PARALLEL IMPLEMENTATION
• Parallel version of models is developed to be
mainly used on supercomputers with distributed
memory
•Procesor-to-processor data exchange is realized
with the use of MPI standard
• non-blocked functions of the data transferreceive
•3-D decomposition of computational domain

• on each time step, processes are co-exchanged
only by data which belongs to boundary grid
cells of decomposition domains
• The Random Access Memory (operative
memory) is dynamically distributed between
processors (the features of FORTRAN-90 are
used)
• Debuging and testing of parallel versions of
models is executed on supercomputer MVS1000М of Joint Supercomputer Center (768
processors, peak productivity - 1Tflops)

“Extreme” case:

Domain 768 x 768 x 256 (= 150 994 994) grid points
574 processors
3D decomposition (12 x 12x 4)
MGD Poisson solver
15 Gbytes of memory
1 step of scheme ~ 20 s of computer time.

3%

6%

exchanges

7%
d iffu s io n

2%

10%

a d v e c tio n o f s c a la rs
a d v e c tio n o f m o m e n tu m
tu rb u le n t c lo s u re

58%

1%

b o u n d a ry c o n d itio n
a d d itio n a l p ro c e d u re s

13%

P o is s o n e q u a tio n
in c lu d ig e x c h a n g e s

Spectra of kinetic energy calculated using results of large-eddy simulation of the
convective upper oceanic layer under different spatial resolution (m3)

von Karman votrex street behind a round cylinder, Re =200: top - from
М. ван Дайк (1986). Альбом течений жидкости и газа, bottom - model results

Turbulent flow between buildings
Wind 2.24 м/s

Lagrangian transport of fine-dispersive particle tracer
 Large number of particles in turbulent flow generated by the LES-model of the atmospheric boundary layer.
The calculation of trajectories is carried out simultaneously with numerical integration of the hydrodynamic
part of the model.
● It is assumed that each of particles has non-zero mass and size. It is possible to sort particles in few groups
accordingly to their size and density.
 The code of tracer transport is parallelized with the use of MPI. The data exchange between processors is
realized with the help of non-blocking transfers and takes place on the background of basic calculations.
 Using results of calculations, the spatial distribution of the particle tracer concentration is calculated for each
time step.
 On parallel computational system this algorithm allows to simultaneously calculate trajectories of tens
millions particles. The time which is needed to calculate the particles transport is significantly less then the
time of the ABL model integration.
● Equation of the particle motion in the air flow:



m x i  f i   i 3 mg ,

where xi (i=1,2,3) – particle coordinates, m – mass of particle, f – drag force.
● To calculate f , the Stokes formula is used:



f i  6  r  (V i  x i )

An example of the particle transport by the turbulent flow between buildings
Wind

Particle concentration

Particles are ejecting near the surface (along the dotted line). The maximal number
of particles is about 10 000 000.

Thermodynamics of shallow
Reservoir (Stepanenko & Lykosov,
2003, 2004)

Ea

U
H,LE

1) One-dimensional approximation.
Snow
2) On the upper boundary: fluxes
of momentum, sensible and latent heat,
solar and long-wave radiation
are calculated
On the lower boundary: fluxes are
prescribed

3) Water and ice: heat transport
Snow and ground: heat- and moisture
transport
U – wind velocity
H – sensible heat flux
LE – latent heat flux
S – shirt-wave radiation
Ea – incoming long-wave radiation
Es – outgoing long-wave radiation

“Upper” ice
Water

“Lower” ice
Ground

Es

S

Mathematical formulation
- for water and ice:
ñ

T

  T
2



t

h 
2

 c

2

dh   T
dt h  

 c

1 dh  T
h dt  



I
z

,

 

z
h

- for snow:

T

с



t

W
t


z


 

z



T
z

  LF fr ,

 F fr .

- temperature

- liquid water

- for ground:

c

T

W
t
I
t

t




z

  T
W


  T
 c  T  W
      L i Fi ,
z 
z
z



W

W
z




z

- temperature

 Fi ,
- liquid water

 Fi .
- ice

- heat
conductivity

Kolpashevo

Simulated snow surface temperature versus measured one on meteorological
station Kolpashevo (1961)
Температура поверхности снега, 01.61
EДанные натурных измерений
EРезультаты моделирования
0

Quality of snow surface temperature
reproduction is an indicator of quality of
heat transfer parameterization in the
“atmospheric surface layer – snow” system.

Температура, С

-10

-20

-30

-40

-50

Accuracy of temperature measurements
on meteorological station is about 0.5 ℃.

0

5

10

15

20

25

30

35

Время, дни

Т е м п е р а т у р а п о в е р х н о с т и с н е г а , К о л п а ш е в о , 0 2 .1 9 6 1

-5

Тем пература, С

-1 0

-1 5

-2 0

-2 5

-3 0

Д а н н ы е н а т ур н ы х э кс п е р и м е н т о в

-3 5

Р е з ул ь т а т ы м о д е л и р о в а н и я

-4 0
0

5

10

15

В рем я, дни

20

25

30

Syrdakh Lake

Температура, С
-4

-3

-2

-1

0

1

2

3

4

5

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 11.Вертикальный профиль температуры в оз. Сырдах,
апрель 1977 г.

Температура, С
3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

0

1

Глубина, м

2

3

4

наблюдения
эмпир. параметр.
e-параметр.

5

Рис. 13.Вертикальный профиль температуры в оз. Сырдах,
июнь 1977 г.

Ground temperature under Syrdakh Lake
(modeling results)

As follows from results of numerical experiments, the talik is stably existing
during all of integration time (20 years, 1965-1984). Its depth varies from 1.2 to 2
meters under the lake bottom.

THANK YOU
for YOUR ATTENTION