Transcript Document

EO GRID Processing on Demand
ESA grid activity report on 2007
luigi.fusco,
guoqing li
ESRIN, European Space Agency (ESA)
Presented for WGISS-23, 2007-5-23
Summary
Brief introduction to ESA EO
satellites
EO community and GRID
requirements
EO GRID Processing on Demand (G-POD)
– Operational experience
– Future vision and plans
ENVISAT product’s examples
Key international EO programmes
The International Charter on Space and Major
Disasters
 Initiated by CNES and ESA, joined by CSA, NOAA, ISRO, CONAE,
JAXA, USGS and DMC
 Unified system of space data acquisition & delivery in case of
natural or human-made disasters
 Data delivery to civil protection agencies, emergency & rescue services;
UN cooperating body since 2003
 Operational since 2000: 24 hrs on-duty-operator, data resources from
all Charter members
 Recent Examples of Charter Activation:
Bam Earthquake 2003
Darfur Crisis 2004
Tsunami Catastrophe 2004/2005
Hurricane Katrina 2005
Cyclone Kyrill 2007
Key international EO programmes
GLOBAL MONITORING FOR ENVIRONMENT
AND SECURITY (GMES)
European independence in critical data sources for
environmental monitoring and security
and
the European contribution to the Global Earth
Observation System of Systems (GEOSS)
The Earth Science world
Coupled and interdisciplinary processes
Cryosphere
Complex web of sensor
Field
Biosphere
Complex data analysis
Optimal
Trajectory
First guess
Noisy
observations
time
Some Earth Science digital data
requirements
 Global, regional, local applications
– Alternative use of the data at different resolution
 Large historical distributed archives
– Long term data and knowledge preservation issues
 Near real-time access to data
– For processing, value adding and dissemination
 Integration with models to provide long term trends
and forecast
– Data assimilation
 Integrate different data sources
– Standardisation, Virtual Organisation, …
 Need to link data to technical information and
scientific results
– Need to keep/transfer the historical Knowledge
Why GRID at ESA for EO ?
 Operational issues
– Envisat generate > 500GBytes of data per day…
– EO archives are “scattered”
– Algorithm evolutions require recurrent reprocessing
– More efficient use of existing infrastructures
 Scientists / user
– Need near-real-time access, historical and multi-sources
– have the processing algorithms
– require large investments to handle the data on their site
 GRID can solve the equations…
– move processors close to the data
– reduce dissemination costs and effort
– evolutions benefit to all at once
GRID as a common shared platform for collaborations
EO GRID Processing on Demand (G-POD) Objectives
 Provide a “user-segment” environment
– Put data & processors together
– Allow “on-demand” processing of the data
 Offer scientists a “production” lab
– Focus on algorithms
– Reuse housekeeping functions (e.g. catalogue, software tools)
– Bridge gap from “prototype” to “production” processor
 Offer scientists a “collaboration” environment
– Share tools and functions
– reuse output of other processors
– IPR is kept by the scientist
 Open and scalable
– host “any” processor
– bring more resources (other grids) and data (other archives)
together
The ESA EO GRID environment
 Computing and
Storage Elements
– Over 200 Working Nodes
– Over 100 TB on-line store
– Middleware: LCG 2.6, GLOBUS 4,
gLite3
– Link to external CE and SE (e.g.
campus, EGEE…)
 Data Interfaces
– GS products Rolling Archives
(ENVISAT, MSG)
– MODIS NRT products over Europe
– Access to AMS SatStore
– Some accesses to NASA and other
external data providers
 Network
–
–
–
–
Gbit LAN
64-192 Mbps to GARR
HiSEEN WAN (e.g. to PACs/PDHS)
MEGALAB (soon 20-40 Gbps in
Frascati area)
 Security
User certification via ESACERT
underway
 Software resources on-line
– IDL, Matlab, BEAT, BEAM, BEST,
CQFD, Compilers, public domain
image processing utilities
– Catalogue queries and data
provision functions
– Data viewers
Grid on Demand Infrastructure
SOAP
Clients
C#, Delphi,
PHP, JAVA
Grid-On-Demand Core Elements
Earth Observation Grid on-Demand Portal
(Web Service)
SOAP
SOAP
User Session Manager
and Security
Task Scheduler
OpenGIS Web Services
Muis Gateway
(Web Service)
Users Data
Folder
Data Reprocessing
Service Operator
Task
Database
Auxiliary
GIS Data
Task Manager
(Submission and Status)
•Login
HTTP SOAP
ESA GS Operat’s
HTTP
MUIS Query and Rendering
Service
EOLI
Servlets
Browse Cache
System
Task Factory
(I/O Interface)
MUIS-C
Envisat
Catalogue
•Define Job
HTTP
Task
Database
SOAP
Jobs/User
References
Job Factory
SOAP
AMS Service
(Archive Listing)
SOAP Service
Job
Templates
Application
Programs
AMS
FTP
Temporary
Environment
Cache System
Storage
Auxiliary FTP
Server
GLOBUS/LCG
Local GRID Resources
MyProxy
Server
Proxy
Certificates
(ESTEC)
Certificate
Authority
Job Manager
Temporary
Environment
External Clients
CRLs
To other GRIDs
(e.g. CNR)
Application Interface
Components
ESA Cluster
Nodes
To CNR, other GRIDs
MyProxy
MyProxy
Grid Gateway
Rolling Tera
Storage
Envisat PDS
FTP
Authentication Services
and Session Manager
GIP/CIP
AMS/API
EO GridEngine
(Web Service)
Web Client
HTTP-based request
HTTP
SOAP
CRL download
User access tools
•Monitor
Task Flow
Definition
AMS/API
•Submit Job
EPIS
SOAP
SFTP
DVD
FTP/SFTP
Operational Procedure
3rd-Party Elements
Product Input Flow
Only one
environment
for operations
and new dev
Grid-On-Demand
Core components
Product Output Flow
I/F to external GRID resources via research networks
EO Data on-line in GRID storage
 ASAR
– ASAR GM: Feb 2004 onwards
– ASAR Medium Resolution: June 2005 onwards
– Occasional High Resolution products
 MERIS
– Level 1,2 RR: full mission
– Samples of FR Level1
 AATSR
- Level 1: full mission
- Level 2: since mid 2005
 Atmospheric
– GOME Level 1,2: 1996 onwards
– GOMOS, MIPAS, SCIAMACHY full archives from D-PAC
 Third-party data
– MSG SEVIRI: last 12 months
– MODIS NRT over Europe
More to come …
Web Portal Interface
Temporal/spatial selection of
products
Job definition, submission and
live status monitoring
Customisable result visualization
interfaces
Access to output products and
documentation
Experience so far
Wide ranging applications supported
–Systematic productions of high-level
products (e.g. level-3)
–On-demand Data access and visualisation
–Algorithm and data validation activities
• software test-beds (ESA toolboxes e.g. BEAT,
BEAM, BEST)
• Processor test-beds
–Internal Research projects
Over 40 projects supported, more to
come!
Routine Production
 MERIS Level-3 Products NRT generation
– Joint ESA collaboration with ACRI (France),
JRC/Ispra (European Commission) and
Brockmann Consult (BEAM)
– 11 monthly products published on-line
http://earth.esa.int/meris/level3
 Daily ASAR GM mapping of Antarctica
– Internal Development in operations since 2005
– Daily Generation of 400-m resolution mosaics
and publish to ESA Web Map Server
http://ssems1.esrin.esa.int:8080/map/context.php
 EarthCare Simulator
– Collaboration with Estec and Univ. of Valencia (Spain)
– Preparatory activity of the mission EarthCARE
– 2.5 105 simulations carried out using ESA and CNR
GRIDs
 River and Lake Processor
– ESA/Montfort University (UK) collaboration
– Accurate River and Lake heights measurements in
NRT from satellite altimetry (RA2)
– products published online http://earth.esa.int/riverandlake
 Meris True-Color Mosaics
– 9km resolution global Monthly mosaics of MERIS data
On demand processing
 ASAR on Demand
– Integrated environment for SAR
processing
– Binds separate functionality into higherlevel applications (flood monitoring, coregistration, mosaicing, etc)
 Imager
– Multi-mission imaging tools
– Data selection, user-defined processing,
image rendering, geo-coding,
visualisation
 MIRAVI Geo-toolbox
Geocoding of MERIS full resolution
images produced by MIRAVI real-time
service
 Aeromeris
– Fast extraction over user-area of pixels
and statistics from the complete MERIS
level-2 product archive
– Output to Excel, Google Earth, XML
Algorithm Validation
 Volcanoes Monitoring by
Infrared
– Fast Extraction of AATSR thermal
anomalies over > 200 volcanoes
– Under validation, Operational service
planned for early 2007
 GRIMI-2
– EADS/Astrium MIPAS Level-2 reference
processor used to validate the operational
PDS processor
– Test data generation: 300 profiles (100GB)
processed in 1hour
 YAGOP
– Alternative GOMOS O3 and Temp level-2
processor (internal research)
– Validation against operational ESA
products
 RAIES Processor
– Assessment of the 2000 Hz individual
echoes of Envisat/RA2
New Research applications
 MERIS/GOME inter-comparison
– Validation of Envisat/MERIS cloud
fraction and water vapour measurements
against ERS-2 GOME
 Var2Grid
– ESA/DMI cooperation (DK)
– 3DVar data assimilation (T and specific
humid.) with CHAMP/GPS products
(Background ECMWF ERA40)
– Internal prototype development
– Extraction of Sodium profiles in
mesosphere from Sciamachy
– Collaboration with ESO for the next
generation telescope ground site survey
 ALOS PRISM DEM Prototype Processor
– Internal Research (ESA Trainee)
– Image co-registration of PRISM views
and height reconstruction by image
matching between PRISM triplet images
Mountains around Shizuoka,
Japan
 SCIA-SODIUM
Earth Science G-POD
Call for Proposals
http://eopi.esa.int/G-POD
Promote the access and use of
EO mission data available at ESA,
offering on-line access to products
with attached computing
infrastructure and tools to assist the
generation of “scientific added value
products”
– new earth science algorithms
– Build network of competences;
promote e-collaboration
– transition from science
algorithms to new user
products…
 First call published beginning
June 2006 on EOPI web
 11 Proposals selected in first call,
implementation ongoing
 First very promising results
expected mid 2007
 Next call in few months
YAGOP: Yet Another GOMOS Processor
• Validation period: 11 day
ensemble of GOMOS
data (Sep., Oct., Dec.
2002)
• total set of selected occ. events
GOMOS: 1719
MIPAS: 4099
ECMWF: 1719
GOMOS to MIPAS temperature profile
comparison (relative errors shown in Fig.)
Time – Space coincidences criteria for
analysis: 3h&300Km
GRID is very good for
such analysis:
1 generic grid service
near future GRID plans
 Operational Opportunities
– Routine generation of “standard” high level products
– Extend infrastructure to other facilities and across
establishments
 Consider potential use of GRID
–
–
–
–
In future EO missions preparation
GMES Service Elements
Contribute to GEOSS architecture
As a mean to get the science community more involved in
ESA programmes
 Earth Science presence in European einfrastructure initiatives
– Strong participation to new EC FP7 initiatives such as einfrastructure, digital repositories, knowledge
preservation, e-content...
Knowledge
Infrastructures
GRID based RTD plans
CWE Applications
DL Applications …
… as Knowledge
Infrastructure
PECS
GRID
Grid ES Community
Applications
ES Grid on Demand
in operation @ ESA
Thank you!
The ESA EO GRID Team
More information at
http://eogrid.esrin.esa.int