Transcript OOI CI Overview and Status

Ocean Observatories Initiative
OOI CI Overview and Status
Matthew Arrott
DMAC-ST
Washington DC, Jan 18-19, 2012
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
1
Agenda
• Review of the OOI CI Construction Objectives
• Introduction of Release 2 Functional Objectives
• Computational Infrastructure Objectives
• External Observatory Integration
• What will OOI-CI mean to me
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
2
OOI CI Construction Objectives
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
3
OOI Project Scope
Building/deploying science infrastructure for
understanding a changing water planet
• Multi Dimensional Engineering
Concern
• Social
• Organizational
• Functional
• Procedural
•
Technical
Cyberinfrastructure fulfills the integrative mechanism
across these dimensions
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
4
The Hubble Telescope for the Oceans
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
5
The Hubble Telescope for the Oceans
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
6
Core Science & Education Capabilities
• Interactive Ocean Observing (R3 - R4)
• Interactive Ocean Modeling & Data Assimilation (R3 - R4)
• Discipline-Driven Semantic Organization of Data (R3 - R4)
• Automated Data Product Generation (R1 - R3)
• Interactive Instrument Network (R1 - R3)
• Integrated Observatory Management (R1 - R2)
• User-Driven Integration of Resource (R1 - R3)
Observatory
Activity Model
7
OOI CI Release 2 Kickoff
May 2, 2011
OOI Integrated Observatory Product
•The fully operational research observatory will meet the following
•Goals:
•
•
•
•
•
•
•
•
•
•
•
•
Continuous observations at time scales of seconds to decades
Spatial measurements from millimeters to kilometers
Sustained operation during storms and other severe conditions
Real-time or near-real-time data as appropriate
Two-way transmission of data and remote instrument control
Power delivery to sensors between the sea surface and the seafloor
Standard plug-n-play sensor interface protocol
Autonomous underwater vehicle dock for data download/battery recharge
Access to deployment and maintenance vehicles that satisfy the needs of
specific observatories
Facilities for instrument maintenance and calibration
A management system that makes data publicly available
An effective education and outreach program
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
8
Extending the Traditional Data Access Model
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
9
Multi-Scale Research Specific Ocean
Laboratories
10
Release Schedule
Inception Phase
Elaboration Phase
Construction Phase
Transition Phase
11
OOI Review Year 2
May 16-20, 2011
Release Schedule by Subsystem
Inception Phase
Elaboration Phase
Construction Phase
Transition Phase
12
OOI Review Year 2
May 16-20, 2011
CI Components Developed for R1
• Python Capability Container
• Java Capability Container Access Library
• Web UI Platform
• Exchange Messaging System
• Distributed State Infrastructure
• Resource Registry Framework
• Science Data Persistence and Transport Format
• Data Publish-Subscribe Framework
• Event Notification Framework
• Instrument Agent Framework
• Elastic Processing Unit
• Virtualized Cloud Management Tools
13
OOI Review Year 2
May 16-20, 2011
Capabilities of the Managed Instrument Network
• Operate Marine Observatories
• Operate Platforms and Instruments
• Manage Instrument Lifecycle
• Test and Troubleshoot Instruments
• Acquire Data and Generate Data Products
• Search Data
• Visualize Data
• Manage the Integrated Observatory Network
14
OOI Review Year 2
May 16-20, 2011
R1 Integrated Observatory Network Web UI
https://confluence.oceanobservatories.org/display/CIPUB/OOI+Review+2011+Demo
s
15
OOI Review Year 2
May 16-20, 2011
Release 2 Functional Objectives
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
16
Observatory Operations & Administration
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
17
User Experience Incremental Development Plan
• CI Release-1: Data Distribution Network
• Provide framework for discovery and interaction based
on data-intensive users’ needs
• CI Release-2: Managed Instrument Network
Science
Experts
• Provide support for operations and maintenance of
Resource
Providers
marine networks based on RSN and CGSN needs
General Science Users
• CI Release-3: On Demand Measurement Processing
Scientific Collaboration
• Provide dynamic analysis and visualization tools based
on science communities’ user needs
• CI Release-4: Interactive Ocean Observatory
• Provide mission control and collaboration support based
on science communities’ needs
18
OOI Review Year 2
May 16-20, 2011
R2 Product Description: List of Use Cases
UC.R2.01 Acquire External Data
SourceUC.R2.02 Derive Data
ProductUC.R2.03 Produce Real-Time QC
DataUC.R2.04 Browse to Get Data
ProductUC.R2.05 Register and Connect
InstrumentUC.R2.06 Command
InstrumentsUC.R2.07 Direct Instrument Access
IIUC.R2.08 Manage Instrument
LifecycleUC.R2.09 Activate Instrument
DriverUC.R2.10 Manage Marine
PlatformUC.R2.11 Operate Marine
ObservatoryUC.R2.12 Deploy Agents On
Remote PlatformUC.R2.13 Acquire Data From
InstrumentUC.R2.14 Monitor an
InstrumentUC.R2.15 Qualify Instrument
InterfaceUC.R2.16 Install Instrument
Automatically
UC.R2.20 Annotate ResourcesUC.R2.21
Transform Data in WorkflowUC.R2.22 Version
ResourceUC.R2.23 Ingest Dataset
SupplementUC.R2.24 Search for
ResourceUC.R2.25 Advanced Resource
SearchUC.R2.26 Navigate Resources and
MetadataUC.R2.27 Manage Replicated
ArchiveUC.R2.28 Manage Resource
MetadataUC.R2.29 Integrate External Data
Source
19
UC.R2.30 Define InteractionUC.R2.31
Define New ServiceUC.R2.32 Conduct
NegotiationUC.R2.33 Enroll in an
OrgUC.R2.34 Share an Org
ResourceUC.R2.35 Share Affiliated Orgs'
ResourcesUC.R2.36 Create an
OrgUC.R2.37 Control Service
InteractionsUC.R2.38 Define Resource Life
CycleUC.R2.39 Manage ION
UsersUC.R2.40 Monitor ION
ResourcesUC.R2.41 Recover Failed
ProcessUC.R2.42 Define Resource
PolicyUC.R2.43 Operate Message Brokers
DM
COI
CEI
S&A
A&S
UX
UC.R2.17 Define Visualization
MethodUC.R2.18 Visualize Data
ProductUC.R2.19 Produce Matlab
Visualization
OOI Review Year 2
May 16-20, 2011
UC.R2.53
View Modeler-Submitted
ProductsUC.R2.54
Access NEPTUNE CA Data
Ops
EOI
UC.R2.55 Manage Help
TicketUC.R2.56 Monitor ION
NetworkUC.R2.57 Configure Start
Page
UC.R2.44 Put Services Anywhere
EasilyUC.R2.45 Replicate Activated
ServiceUC.R2.46 Operate Integrated
SystemUC.R2.47 Deploy Versioned User
ProcessUC.R2.48 Schedule User-Defined
ProcessUC.R2.49 Deploy Distributed
ProcessesUC.R2.50 Define Scaling
PolicyUC.R2.51 Define Execution
EngineUC.R2.52 Manage ION Processes
UC.R2.58 Display Arbitrary
ResourceUC.R2.59 Generate New
Screen
Managed Instrument Network
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
20
Sensor Set #1 Instrument Agents
Class
#
Family
Instrument Class Long Name
ADCPS
16 Water Velocity
Velocity_profile_600m
CTDGV
24 Conductivity, Temperature, Depth
CTD_glider
CTDMO
135 Conductivity, Temperature, Depth
CTD_mooring
CTDPF
30 Conductivity, Temperature, Depth
CTD_profiler
DOSTA
61 Dissolved Oxygen
oxygen_dissolved_stable
FDCHP
7 Direct Covariance Flux
flux_direct_cov_HP
FLORD
35 Fluorometer
Fluorometer_two_wavelength
METBK
18 Meteorology
Meteorology_bulk
NUTNR
22 Nitrate
nutrient_Nitrate
OPTAA
35 Optical Properties
attenuation_absorption_optical
PCO2A
10 Partial Pressure CO2
pCO2_air-sea
PCO2W
25 Partial Pressure CO2
pCO2_water
PHSEN
33 Partial Pressure CO2
pH_stable
PREST
3 Seafloor Pressure
pressure_SF_tidal
SPKIR
22 Optical Properties
spectral_irradiance
VEL3D
32 Water Velocity
Velocity_point_3D_turb
VELPT
21 Water Velocity
Velocity_point
WAVSS
10 Surface Wave Spectra
wave_spectra_surface
ZPLSG
4 Zooplankton Sonar
21
OOI Review Year 2
May 16-20, 2011
plankton_ZP_sonar_global
Sensor Set #2 Instrument Agents
Class
#
BOTPT
Family
3 Seafloor Pressure
Instrument Class Long Name
pressure_bottom_tilt
CAMDS
12 Cameras x
CAMHD
1 Cameras x
camera_digital_video_HD
HYDLF
5 Hydrophone
Hydrophone_LF_passive
MASSP
2 Mass Spectrometer
mass_spectrometer
OBSBB
3 Ocean Bottom Seismometer
Seismometer_BB_triaxial_accel
OBSBK
2 Ocean Bottom Seismometer
Seismometer_BB_triaxial_keck
OBSSP
8 Ocean Bottom Seismometer
Seismometer_shortperiod
PPSDN
1 Seafloor Fluid and DNA
DNA_particulate
RASFL
1 Seafloor Fluid and DNA
watersample_chem_trace_H2S_pH
THSPH
1 Seafloor Properties
Temp_H2_H2S_pH
TMPSF
1 Seafloor Properties
Temperature_seafloor
TRHPH
1 Seafloor Properties
Temp_resist
22
OOI Review Year 2
May 16-20, 2011
camera_digital_still_strobe
Sensor Set #3 Instrument Agents
Class
#
Family
Instrument Class Long Name
ADCPA
15 Water Velocity
velocity_profile_mobile_asset
ADCPT
17 Water Velocity
Velocity_profile_300m
CTDAV
3 Conductivity, Temperature, Depth
CTDBP
18 Conductivity, Temperature, Depth
CTD_bottom_pumped
DOFST
22 Dissolved Oxygen
oxygen_dissolved_fastresp
FDCLP
FLORT
3 Direct Covariance Flux
42 Fluorometer
CTD_AUV
flux_direct_cov_LP
Fluorometer_three_wavelength
HPIES
2 Inverted Echo Sounder
IES_pressure_velocity
HYDBB
6 Hydrophone
Hydrophone_BB_passive
NUTR4
3 Nutrients
nutrient_four_channel
OSMOI
2 Seafloor Properties
watersample_chem_trace
OTISF
1 Seafloor Properties
flow_benthic
PARAD
30 Optical Properties
PRESF
9 Seafloor Pressure
pressure_SF
VADCP
2 Water Velocity
Velocity_profile_50m_turb
ZPLSC
8 Zooplankton Sonar
plankton_ZP_sonar_coastal
23
OOI Review Year 2
May 16-20, 2011
PAR
CI Architecture for Instrument Integration
24
OOI Review Year 2
May 16-20, 2011
Data Processing and Product Generation
25
OOI's Sensor Life Cycle
OOI Final
System
Design
Core
Sensor
List
Core
Sensor
Specs
Sensor
Working
Group
Core
Sensor
RFPs
Common
Sensor
Selections
COL
COL
RSN
RSN
CGSN
Deploy at
Sea
RSN
CI
System
Integration
Test
CI
Integration
Test
CI
CI
COL
26
RSN
CGSN
CGSN
CI
CI
CGSN
Procure
Common
Sensors
OOI Review Year 2
May 16-20, 2011
Test
w/Instrumen
t Test Kit
Encode
Algorithms
Develop
QC/Xform
Algorithms
RSN
CI
Integrate w/
Sensor
Agent
CI
CGSN
Develop
Sensor
Drivers
Instrument Development Kit
• Scheduled for Release-2
• Includes:
• Logical Test Facility
Workbench (dry testing)
• Marine Specific System
Test Facility (wet testing)
• Configurations for RSN
and CGSN observatories
• Access and Management
Portals for Interactive
access
27
OOI Review Year 2
May 16-20, 2011
Computational Infrastructure Objectives
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
28
National Network Deployment (Year 3)
29
OOI Review Year 2
May 16-20, 2011
CyberPoPs and Network Infrastructure
• Acquisition Point CyberPoPs:
• Portland, OR
• Woods Hole, MA
• Distribution Point CyberPoPs:
• Seattle, WA
• Chicago, IL
• San Diego, CA
• McLean, VA (optional)
• Engineering Center:
• San Diego, CA
• Network Infrastructure
• Dedicated 10GE loop San
Diego, Portland, Seattle,
Chicago; branch to McLean
(optional)
• Dedicated 1GE connection to
Woods Hole
30
OOI Review Year 2
May 16-20, 2011
San Diego Engineering Center
• Purchased and deployed
San Diego CyberPoP and
network equipment (see
rack drawing)
• Located in UCSD’s Atkinson
Hall server room 1101
(secure and protected)
• In use for
• Release-1 production and
QA
• Management tools
(Confluence, Jira, etc)
31
OOI Review Year 2
May 16-20, 2011
San Diego CyberPoP and Network Equipment
32
OOI Review Year 2
May 16-20, 2011
Acquisition Point CyberPoP Deployments (Y3)
• Sites
• Portland
• Woods Hole
• Function
• Data acquisition from
marine observatories
• Real-time data processing
and OOI data product
generation
• Data preservation
• High availability
33
OOI Review Year 2
May 16-20, 2011
CyberPoP Physical Layout (Year 3 Deployments)
• Sites
• Seattle
• Chicago
• Function
• Content distribution (web
servers, data servers,
messaging)
• Interconnects with major
national and international
network providers (layer 2
peering)
• Links to compute clouds
34
OOI Review Year 2
May 16-20, 2011
Network Peering Integration
Seattle
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
Chicago
35
Multi-Tier Messaging Federation
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
36
Multi-Site Service Network Deployment
37
External Observatory Integration
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
38
IOOS Integration Model
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
39
External Observatory Integration Progress
• Identification of datasets and data sources needed to
support early adopter group (ESPRESSO/Rutgers,
HiOOS/UHawaii) numerical model workflows
• Classification of datasets by type and representation
• Development of Dataset Agents for automatic data and
metadata ingestion
• Translation of data/metadata into the Common Science
Data Format
• Streaming of data packets and metadata update
notifications
40
OOI Review Year 2
May 16-20, 2011
External Observatory Integration
• Exemplar: MARACOOS (Rutgers University)
• Experimentally substitute part of the scientists’ data assimilation workflow preceding
numerical model execution
41
OOI Review Year 2
May 16-20, 2011
External Observatory Integration: Master Dataset List
• Datasets
• NAVY : NRLSSC (DAP) : 1
• NOAA : NDBC SOS (HTML) : 34
• NOAA : NDBC SDF (DAP) : 1
• NOAA : PFEG (DAP) : 1
• Rutgers : tashtego (DAP) : 2
• UCSD : HFRNET (DAP) : 1
• UH : SOEST (DAP) : 1
• USGS : WaterService (HTML) : 26
42
OOI Review Year 2
May 16-20, 2011
“What will OOI-CI mean to me”
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
43
Regional Data Provider & Manager
• Effective mechanism to reliably at scale distribute your data
• to your user community and data management/preservation
facility
• Independently publish and receive data
• in a wide variety of syntactic formats and community vocabularies
• Publish and receive as near-real-time data streams and as historical
data sets
• Effective mechanism to reliably at scale aggregate and disseminate
data from your publishers and to your end user communities
44
OOI CI Release 2 Kickoff
May 2, 2011
Participating Agency
• Participate as providers, managers and consumers
• Anticipated some requirements for publishing on the OOI
as a separate Agency will require coordination with NSF
• Requirements are being addressed as a part of the IOOS
OOI Integration effort
45
OOI CI Release 2 Kickoff
May 2, 2011
How does one “join” – approval / certification?
• Open data policy network for consumers of data
• Anticipate publishers required to register and certified
• Data policies and rights management being established
• Open data policies and content limitations to science and
education data content
• Publishing requires acceptance of some form of open use policy
• Consuming requires some form of “as is”; “at your on risk”; “with
attribution of source if republished” policies
46
OOI CI Release 2 Kickoff
May 2, 2011
Relationship with IOOS, NEPTUNE and others observatories
• External Observatory Integration subsystem within the OOI
• Specifically focused on providing bi-directional integration
interfaces to IOOS, NEPTUNE-Canada and WMO (aka GTS)
• Integration continues on into the Operations and Maintenance
phase of the OOI as a user/observatory “on boarding” capability
• Provide the training and support to communities that wish to
integrate with OOI
47
OOI CI Release 2 Kickoff
May 2, 2011
OOI define standards for them to “plug in”?, e.g SensorML
• OOI provides a wide variety of syntactic formats and community
vocabularies by which to interface with the OOI
• OOI and the user community will be able to extend these interfaces
• Capability facilitated by architectural choice of using controlled set
of canonical data models into and out of which all data are
transformed
• New transformations are added, “plug ins”, to the network for a new
format and/or vocabulary to one of the controlled data models
48
OOI CI Release 2 Kickoff
May 2, 2011
Will a region be able to host an instance of the OOI “stack”?
• The OOI “Integrated Observatory Network” is a federation architecture
• Comprising participants from multiple domains of authority operating from their local
domain of authority
• The OOI “stack” exists at a couple of levels:
• It is a messaging protocol
• It is a component software implementation that implements the messaging protocol
and process management containers (think modern web server) that publish and
consume services
• “Capability Containers”, can be and are written in multiple languages
• achieve their interoperability through use of the common OOI messaging protocol
• OOI message protocol and any of the OOI supplied Capability Containers will be
provided under one of the standard open source licenses
49
OOI CI Release 2 Kickoff
May 2, 2011
Thanks!
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
50