Transcript Document
Changes at ECMWF Walter Zwieflhofer Head of Computer Division CAS 2001, Annecy Slide 1 ECMWF Operational system (October 2001) Data assimilation 4D-Var with a 12-hour data window Horizontal resolution: 120 km for inner and 40 km for outer loops Global atmospheric forecasts Forecasts to 10 days from 12 UTC at 40 km resolution Ensemble forecasts to 10 days with 51 members at 80 km resolution Four forecasts per day to 3 days to provide boundary conditions for some of the ECMWF Member States (40 km, 3D-Var) Ocean wave forecasts Global forecasts to 10 days at 40 km resolution European waters forecasts to 5 days at 15 km resolution Seasonal forecasts Global forecasts to 6 months (ensemble with 30 members) CAS 2001, Annecy Slide 2 ECMWF Forecasting-system changes since 1999 19r2 Increased resolution in stratosphere 22r3 New error variances (for Jb and Jo) 21r1 Assimilation of raw TOVS/ATOVS radiances additional ATOVS data, improved land-surface, 21r4 New error statistics for 4D-Var Jb, SSMI wind data, sea-ice and LW radiation corrected use of humidity observations, increased 23r1 12-hourly 4D-Var PBL resolution, improved cloud, convection and 23r3 T511/159 resolution, second ATOVS satellite, orography. increased angular resolution in wave model 22r1 Improved SSM/I QC and second SSM/I satellite CAS 2001, Annecy Slide 3 ECMWF 23r3 0.0 23r1 0.0 S Hem 22r3 0.5 22r1 0.5 21r4 1.0 23r3 1.0 23r1 1.5 22r3 1.5 21r4 2.0 21r1 2.0 19r2 2.5 21r1 Mean reduction in day-one error measured in pre-operational trials 19r2 N Hem 2.5 Evolution of forecast skill for northern and southern hemispheres CAS 2001, Annecy Slide 4 ECMWF Error distribution for Day 5 (Europe) CAS 2001, Annecy Slide 5 ECMWF Annual-mean RMS errors against analyses from WMO scores CAS 2001, Annecy Slide 6 ECMWF Planned improvements 2002 Increase of the inner loop resolution of 4D-Var from 120 km to 80 km 2003-2004 Assimilation of cloud and rain in 4D-Var Higher vertical resolution in free troposphere and lower stratosphere Assimilation of AIRS data Operational running of a Severe Weather forecasting system 2005-2006 CAS 2001, Annecy Assimilation of IASI data Slide 7 ECMWF Computer configuration – October 2001 CAS 2001, Annecy Slide 8 ECMWF PrepIFS ECMWF tool to prepare forecast and data assimilation experiments Uses the operational configuration of the forecasting system and allows researcher to introduce modifications to: input data (including the selection of observations) the model code the resolution to be used and many other parameters Written Java, with a client part that can be installed on remote machines Allows registered external users to configure and submit experiments via the Internet CAS 2001, Annecy Slide 9 ECMWF Major changes to the computing environment Desktop systems (procurement completed in 2000) Decision was taken to replace all proprietary UNIX workstations with PCs running LINUX over a period of three years Data Handling System – DHS (procurement in progress) A scalable system that will be able to support the expected growth over 6+ years is required Invitation to Tender for a new system was issued earlier this year Supercomputer – HPC (procurement in progress) Fujitsu systems are rented and the contract ends on 31 Dec 2002 Invitation to Tender for a replacement to cover the period 2003-2006 was issued in March this year CAS 2001, Annecy Slide 10 ECMWF DHS procurement Archive holds currently some 400 TB of data (primary data) and has been growing by some 60% per annum Two services need to be supported at ECMWF: ECFS, a large-scale UNIX-like filesystem for general use MARS, a customised meteorological archival and retrieval system Vendors were asked to offer a scalable storage system and were given the option to offer a replacement for the ECFS and/or MARS software layer DHS and HPC procurements were separated because: Lifetime of the data handling systems tends to be considerably longer than that of the HPC systems Allows a simpler HPC procurement not affected by the complexities associated with the replacement of the archive system CAS 2001, Annecy Slide 11 ECMWF HPC procurement ECMWF asked for a phased introduction of resources covering the period 2003-2006, with a parallel run during the second half of 2002 The specifications allowed for some flexibility, enabling vendors to adapt their response to their product development plans. This included the option to propose a 3-month shift of the service period The funds available were published Vendors were required to commit to performance increases, expressed as factors, over the performance of the Fujitsu systems installed at ECMWF Requirements for disk space, I/O and network bandwidth were expressed as function of these factors CAS 2001, Annecy Slide 12 ECMWF The Benchmark Suite IFS tests (for the performance commitments) 4D-VAR (T511/255 L60; 2, 4 or 6 copies depending on PHASE) DETERMINISTIC FORECAST (T799 L90; 2 or 4 copies) EPS (T399 L60; 50 or 100 copies) Burton Smith taxonomy: Size * g >> 2 * O + L Suitable for both T-type and C-type systems IFS model runs well on all architectures 4D-Var is very sensitive to communication bandwidth and has sections on which scalar machines do well Resolutions representative for the workload planned for 2003-2006 were used CAS 2001, Annecy Slide 13 ECMWF Observations Vendors were allowed to optimize the codes However, changes that would affect the readability or portability were not permitted Some vendors used a hybrid MPI / OpenMP approach Some vendors found that such a hybrid approach did not improve the results obtained by using MPI only Percentages of peak: 4D-Var scaled not as well the deterministic forecast and the EPS Scalar machines: 3% to 12%, depending on test/product Vector machines: 14% to 38%, depending on test/product Rule of thumb for ECMWF’s codes was confirmed: ~10% for scalar architectures and ~30% for vector architectures CAS 2001, Annecy Slide 14 ECMWF Status of both procurements The Tender Evaluation Boards have completed their work and have made recommendations to the ECMWF Director The Director has awarded both contracts, subject to approval by ECMWF Council Documents detailing the respective procurement process and the proposed solutions have been sent to the technical and financial committees of the ECMWF Council These committees will meet in November to consider ECMWF’s proposals and to make recommendations to Council Council will meet in December and (hopefully) authorize the Director to sign the appropriate contracts CAS 2001, Annecy Slide 15 ECMWF Changes over the next 18 months New Servers New Network New HPC system Linux PCs New DHS CAS 2001, Annecy Slide 16 ECMWF