Challenges in the provision of large data sets to multiple user
Download
Report
Transcript Challenges in the provision of large data sets to multiple user
MONITORING WEATHER
AND CLIMATE FROM SPACE
Challenges in the provision of large data
sets to multiple user communities
14-16 July 2014, Boulder CO
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
1
Mission Statement
We are the European Organisation for the Exploitation
of Meteorological Satellites (EUMETSAT).
EUMETSAT is an intergovernmental organisation and was
founded in 1986. Our purpose is to supply weather and climaterelated satellite data, images and products
• 24 hours a day,
• 365 days a year
to the National Meteorological Services of our Member and
Cooperating States in Europe, and other users worldwide.
www.eumetsat.int
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
2
EUMETSAT Headquarters (Darmstadt, Germany)
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
3
EUMETSAT Data Services - High Level Overview
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
4
EUMETSAT Data Services - Data Flows
EUMETSAT GROUND INFRASTRUCTURE
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
5
EUMETSAT Earth Observation Portal
Single Access Point
One stop shop
Data discovery
User registration (>10.000)
Subscription for data
services:
Near-real time data
Online Data Access
Long Term Archive
Additional
Image Gallery (WMS/ WCS)
User Notifications
Operational Real-Time
Services Status Indicator
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
6
EUMETCast Dissemination
Near Real Time data delivery to users in 3 continents via Satellite
4
3
500+
201-500
51-200
11-50
2-10
1 User
0 User
Users (2/2014)
3,280 users worldwide,
4168 stations
5 new users per month
EUMETCast Americas
SES 6, C-BAND
EUMETCast Africa
EUTELSAT 5 WEST A, C-BAND
EUMETCast Europe
EUTELSAT 9 A, KU-BAND
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
7
EUMETCast Dissemination:
Integrating terrestrial dissemination
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
8
MULTICAST ACCROSS GÉANT and NREN
INFRASTRUCTURE
Exploring the multicast usage on
GÉANT infrastructure in Europe
and their world wide connectivity;
2011 Proof of concept on terrestrial
multicast on GÉANT test bed
2012 3 month feasibility study on
terrestrial multicast via GÉANT
2013 Multicast test to KMA Korea
2014 Multicast test to NOAA
(Boulder)
2014 Start of 2 year prototype
emulating operational scenarios of
multicast across Europe and our
overseas partners using GÉANT,
NREN, I2 and Teint infrastructure.
EUMETCast
Platform
(MMDS)
EUMETCast
User Stations
Multicast
GEANT/
NRENs
Multicast
...
Multicast
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
9
Reference
Stations
EUMETCast Terrestrial real-time monitoring
-
Multiple Multicast channels
SSM mode
20-30Mbit/s
FEC ¾
Closed loop monitoring based on successful file receipt
Packet/channel based ack/nack retransmission possible but not yet enabled
Acknowledgements: N-WAVE&NOAA Alex Hsia, David Hartzell
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
10
EUMETCast Dissemination:
Low cost, off the shelf Receiving Station
DVB Router
Laptop/ PC
Antenna
DVB PCI Card
Software
EKU
DVB USB
Typical total cost of a station < 2000 Euros
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
11
EUMETSAT Long Term Archive
Archived data starting 1981
All EUMETSAT Satellite data
including reprocessed data
> 1 Petabytes in the archive
for MTP, MSG, Metop
Retrieval/Ingestion: 10 to 1
Long-Term Data Preservation
in practice since 1995;
Disaster resilient - 3rd Copy
Access via EUMETSAT EO
Portal
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
12
What we are working on…
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
13
Outlook – Enhance Future abilities in terms of
•
•
•
•
•
Scalability
Flexibility,
Responsiveness to emerging needs,
Resource efficiency
While aiming for:
• Better sharing of resources,
• Avoidance of data islands,
• Reducing costs
• Fit the organisational structure
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
14
Many aspects to consider => Challenge
Viewing Angle
15
EUM/TSS/VWG/14/762641, 7 July 2014
Outlook - Product delivery and management
• Generation of user tailored products
• Efficient for write once, read many times
• Suitable products for user needs and
dissemination infrastructure (bandwidth)
• Avoid/ minimise processing where possible
• Efficient for slicing
• Usage of Format evolutions
• Secure the raw data (golden copy)
• Initially smaller products which increase over
time
• Reprocessing to re-build formats
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
16
Outlook - Bringing data to Users
• Adaptive and flexible services to deliver data to
users/ communities (e.g. EUMETCast)
• Community driven formats optimised for
dissemination mean
• Serving large user groups from different
communities
• Managing data policies in accordance with data
provider requirements via a data policy and
legal framework support by the related
technical mechanisms
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
17
Outlook - Bringing Users to data
Standardisation activities in relation to Discovery,
Access and Retrieval (DAR) of data
Strong focus on interoperability and ease of user
experience
One stop shop experience for discovering services and data
Facilitating the Interoperability of portals by exploiting standards
(OGC, INSPIRE)
Enhance Long Term Archive, Online Data Access
Outreach activities
Exploiting capabilities to execute user algorithms on data
Post-Processing, Hosted Processing
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
18
Outlook - Information to Users
• Delivering Information to users rather
than data
• Processing where the data is located
• Service Interfaces instead Facility
interfaces
• Application of (private) cloud principles
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
19
Thank you for your attention
Improving
Data Mobility & Management for International Climate Science, [email protected] & [email protected]
20