Transcript IDN TASK TEAM Presented by Mr. Ken McDonald

IDN TASK TEAM
Presented by Mr. Ken McDonald
Notes
from the
Cordoba
meeting


MD9.4 Software Release Demonstrated
Primary action items assigned to the IDN Task
Team
Completed tasks:
(1) set up GEOSS portal for WGISS to review
(2) prepared related service/tool (SERF)
descriptions for GEOSS-related services.
(3) sent the URL to all of WGISS for the
Conference Calendar (provided on IDN
website-see “calendar” tab).
(4) updated lyrics of ‘We WGISS’ and captured
into an audio file.
IDN Website – Current Homepage
New IDN Map
New IDN Web Site Header
New tabs that link Data Sets, Data Services
and the Conference Calendar. Action (3) from Cordoba
Status of IDN Site
• IDN Website continues to use Zope/Plone.
• Discussion List: Please use Interop to
provide feedback on:
–
–
–
–
DIF extension for Models
Renaming of Keyword Fields
Geographic locations Hierarchy
Platform/ Instrument Restructure
Numerical Model Fields Proposed (2002/03)
•
Anne Sophie Archambeau from IPSL, France proposed an addition to
the DIF to handle numerical output data sets:
Group: Numerical_Experiment
Model_Name:
Model_Version:
Model_URL:
Model_Configuration:
Model_Resolution:
Model_Calendar:
Group: Model_Integration_Period
Start_Date:
Stop_Date:
End_Group
Simulation_Name:
Initial_Conditions:
Perturbation:
Imposed_Boundary_Conditions:
End_Group
•
First proposed in
2002
Interop discussions on the proposed new fields are summarized in the
CEOS IDN Interop Newsletter #15 for April 2003:
http://gcmd.gsfc.nasa.gov/pipermail/interop/2003-April/000011.html
Global Organization for Earth System Science
Portal (GO-ESSP) – June 2005 Meeting
New Proposal for numerical extensions to the DIF
• DIF describes datasets at discovery level.
•
Where simulations are involved, discovery metadata needs more
information than is available in the existing schema.
•
New DIF extension schema for numerical model output proposed
by the British Atmospheric Data Center (BADC) can be found at
http://ndg.badc.rl.ac.uk/kwiki/ndg/index.cgi?DIFExtensionV005
• Additional information and draft examples can be found at
http://ndg.badc.rl.ac.uk/kwiki/ndg/index.cgi?ProposedChangesToDIF
Comments are welcome!
A new model keyword list
Global Coupled Climate System Models
Global General Circulation Models (GCM)
Earth System Component Models (Global or Regional)
Process Models: Clouds and Precipitation
Phenomenological/Decision Making Models
Geophysical Models (as in Earth Geophysics)
Solar-Atmospheric Space-Weather Models
Energy Balance Models
Radiative-Convective Models
A higher level of hierarchy?
Diagnostic and Simulation Models
Data Assimilation Models
Forecast, Prediction, Projection (Scenario’s) Models
Monitoring and Early Warning Systems
New DIF/SERF Changes
• Parameters: will be renamed to Science Keywords:
• Service Parameters: will be renamed to Service
Keywords:
• Keyword: will be renamed to Ancillary Keyword:
• Add field to Temporal Coverage for Relative Time:
Group: Temporal Coverage
Start_Date: [yyyy-mm-dd]
Stop_Date: [yyyy-mm-dd]
Relative_Start_Date: -x (where x= # days ago)
End_Group
Relative time will allow for:
– Auto updating of temporal coverage for near-real time data.
– Specifying number of days data available.
Instrument and Platform
Clean-up
• Implement an option to list all instruments and platforms
alphabetically for searching.
• Expanded database table structures for Instrument and
Platform information and categories.
• Extend Supplemental Information to include expanded
Instrument and Platform information.
• Feedback Welcome!
Instrument and Platform
Clean-up
• Instrument database information can include:
– Instrument Name (currently exists as short name > long
name
– Categories: types, subtypes (can be used in web interface for
browsing. For example:
• Earth Viewing vs. Space Viewing vs. in situ, etc.
• Wavelength region (microwave vs. visible, etc.)
– Channels (if appropriate)
– Spectral information; instrument information.
• Platform database information can include:
– Platform name (currently exists as short name > long name
– Categories: type of platform (e.g. Landsat Series)
– Instrument names (if appropriate): link to instrument
database table.
– Platform orbital and launch information.
Instrument Examples
Group: Instrument
Instrument_Category: Earth Remote Sensing Instruments
Instrument_Type: Passive Remote Sensing
Instrument_Subtype: Sounders
Short_Name: AIRS
Long Name: Atmospheric Infrared Sounder
Group: Spectral Information
Wavelength_Keyword: Infrared
Wavelength_Keyword: Visible
Number_Channels: 21
Spectral_Coverage: 0.41-15.4 micrometers
Spectral_Resolution: 1200
IFOV: 0.185 - 1.1 degrees
Radiometric_Resolution: 2.3-13.5 km
End_Group
Group: Description
The AIRS instrument ….
End_Group
Online_Resource: http://www-airs.jpl.nasa.gov/
Creation_Date: 2005-08-25
Revision_Date: 2005-08-25
IDN_Node: USA/NASA
End_Group
Instrument Examples
Group: Instrument
Instrument_Category: In Situ Instruments
Instrument_Type: Trace Gas Detectors
Instrument_Subtype: Chromatographs (?)
Short_Name: ACATS
Long Name: Airborne Chromatograph for Atmospheric Trace Species
Group: Spectral Information
Wavelength_Keyword: (r)
Number_Channels:
Spectral_Coverage:
Spectral_Resolution:
IFOV:
Radiometric_Resolution
End_Group
Group: Description
ACATS is....
End_Group
Online_Resource: http://www.cmdl.noaa.gov/hats/airborne/acats.html
Creation_Date:
Revision_Date:
IDN_Node: USA/NOAA
End_Group
Platform Example
Group: Platform
Platform_Category: Earth Observing
Satellite
Platform_Series: EOS
Short_Name: Aqua
Long_Name:
Group: Instruments Aboard
Instrument_Short_Name: MODIS
Instrument_Short_Name: AMSR-E
Instrument_Short_Name: AMSU-A
Instrument_Short_Name: CERES
Instrument_Short_Name: AIRS
Instrument_Short_Name: HSB
End_Group
Group: Launch
Launch_Date: 2002-04-02
Launch_Site: Vandenberg AFB, CA
End_Group
Group: Orbit
Orbit_Altitude: 705 km
Orbit_Inclination: 98.2 degrees
Equator_Crossing:
Period: 99 minutes
Repeat_Cycle: 16 days
Perigee: 673 km
Apogee: 686 km
Orbit_Type: Sun Synchronous
Swath_Width_Range:
End_group
Coverage: Global
Group: Description
The Aqua satellite....
End_Group
Creation_Date: 2005-08-25
Revision_Date:
Node Reports
0:035
JCADM Report
Antarctic
Master
Directory
Home Page
Antarctic Master Directory
NADC Portals
•
•
•
•
•
•
•
•
•
•
•
•
•
The alias format is http://gcmd.nasa.gov/portals/portal_name/
Examples of alias: http://gcmd.nasa.gov/portals/amd_ar/
Belgium Antarctic Program created January 2003
Finnish Antarctic Program created January 2003
Centro de Datos Antárticos, Argentina created August 2003
Canadian Polar Commission/Canadian Committee for Antarctic Research
created October 2003
Swiss Committee on Polar Research created November 2003
US Antarctic Data Centers/US Antarctic Data Coordination Center created
February 2004
Japan National Institute for Polar Research created February 2004
Estonian Antarctic Data Center created July 2004
Centro Nacional de Datos Antárticos, Spain created February 2005
Instituto Antartico Uruguayo created February 2005
Italian Antarctic Program created February 2005
Institut Polaire Francais Paul Emile Victor, France created June 2005
Centro Nacional de Datos Antarticos, Chile created July 2005
Antarctic Master Directory
Content Statistics – Growth 2002 through 2005
Demo Mapserver for Ukraine
00:55
GCMD Ukraine/CEOS Prototype Mapserver
http://mapserver.gsfc.nasa.gov/website/CEOS/
Goals:
•
•
•
Incorporate geospatial data from various sources: ESRI, NOAA,
USDA, and NASA
Provide users information to familiarize themselves with the
Ukraine region.
Provide direct links to metadata and geospatial data sets.
GIS Layers:
•
•
Thematic Layers: Political Boundaries, Roads, Water Bodies,
Cities.
Raster Layers: World Land Cover, World Ecoregions, MODIS
Imagery.
GCMD Prototype Mapserver Interoperability
•
Access to GCMD’s Web Mapping Services (WMS) through the
ESRI OGC Connector.
•
OGC WMS compatible clients can access GCMD’s map services.
GCMD Mapserver
User Friendly Desktop GIS
(WMS Client)
Direct Access to Map Service & Metadata
Layers Link to Metadata
Data Query
Ecosystems of the Kiev Region
New Keywords since Last CEOS
Meeting
PART C
1:05
New/Modified Earth Science
Keywords Since May 2005
• NEW Terms and Variables for SunEarth Interactions
– Solar Energetic Particle Properties
– Solar Energetic Particle Flux
– Ionosphere/Magnetosphere Dynamics
• Some Existing Variables Modified
New Sun-Earth Interactions Keywords
Earth Science > Sun-Earth
Interactions
Ionosphere/Magnetosphere Dynamics:
Solar-Energetic Particles:
Energy Deposition
Particle Composition
Particle Density
Particle Distribution Functions
Particle Speed
Particle Temperature
Total Electron Flux
Solar Energetic Particle Flux
Alpha Particle Flux
Electron Flux
Proton Flux
Ion Flux
Heavy Nuclei Flux
Neutral Particle Flux
Sub-Atomic Particle Flux
New TERM
New Variable
Aurorae
Electric Fields/Electric Currents
Geomagnetic Forecasts
Geomagnetic Indices
Ion Chemistry/Ionization
Magnetic Fields/Magnetic Currents
Magnetic Storms
Plasma Waves
Solar Wind
Solar Activity:
Corona Holes
Coronal Mass Ejections
Coronal Properties
Solar Active Regions
Solar Flares
Solar Imagery
Solar Irradiance
Solar Oscillations
Solar Prominences/Solar Filaments
Solar Radio Wave Emissions
Solar Synoptic Maps
Solar Ultraviolet Emissions
Solar Velocity Fields
Solar X-Ray Emissions
Sunspots
Space Science/Astronomy
Keywords
• Added new Category > Space Science to accommodate space
science/astronomy researchers in Antarctica supporting JCADM
and the NSF Polar Program.
– Space Science Category keywords can be selected from the docBUILDER
tools, but are not yet available for searching for data.
• Need assistance from IDN in defining
Space Science/Astronomy Topics, Terms and Variables.
Search Hits on Topic Spectral/Engineering
2000
1800
1600
1400
1000
800
600
400
200
n02
M
ar
-0
2
M
ay
-0
2
Ju
l02
Se
p02
N
ov
-0
2
Ja
n03
M
ar
-0
3
M
ay
-0
3
Ju
l03
Se
p03
N
ov
-0
3
Ja
n04
M
ar
-0
4
M
ay
-0
4
Ju
l04
Se
p04
N
ov
-0
4
Ja
n05
M
ar
-0
5
M
ay
-0
5
Ju
l05
0
Ja
Hits
1200
Drop-off was an artifact
of the statistics and not
a real decline in usage.
Month
Spectral/Engineering
Detailed Variable Review
• Staff is currently reviewing all Detailed
Variables for spelling, clarity, and
duplication for consistency and validity.
– Sections completed include Atmosphere,
Paleoclimate, Solid Earth, Sun-Earth, Human
Dimensions, Oceans, and Land Surface.
New Data Center, Location,
Instrument, and Platform
Hierarchies for Improved Search
Multi-year Effort
Comments Welcome
Data Centers and Locations
• Data Center “Bucket” clean-up has been
completed and implemented!
• Location “Bucket” has been completed and
implemented!
– Database hierarchies rather than buckets will be
implemented for easier location navigation and
maintenance.
• Data Centers and Locations can be searched
hierarchically.
New Data Center Hierarchy Search
New Data Center Hierarchy Search
Shows long name
For all data center names
When you place your
mouse over the data
center name
New Location Hierarchy Search
New search box for location
Search or map date search
New Location Hierarchy Search
Select
Continent
Choose from among top level
location categories
New Location Hierarchy Search
Breadcrumbs show you where
you are in the hierarchy
Select Europe
Continent category is subdivided
into individual continents
New Location Hierarchy Search
Breadcrumbs
Select Eastern
Europe
Category Europe is further
subdivided into individual
countries and regions
New Location Hierarchy Search
Breadcrumbs
Select Ukraine
Category Eastern Europe
is further subdivided into
individual countries and
regions
New Location Hierarchy Search
Breadcrumbs
Results for search on
“Ukraine” are displayed
URL Content Type Cleanup
Objective:
• Provide an accurate account of related information in the
DIF and SERF.
• Provide “Direct Access” to data URL to get quick access to
data.
1. GET DATA
2. WEB MAPPING SERVICE (WMS)
• Provide “Direct Access” to Services
1. GET SERVICE (Online Clients/Web Services)
• Provide “Supplemental Information” that describes URL
Content Types.
URL Content Type Keywords
Current List
ASSOCIATED DATA SET(S)
BDP METADATA
CALIBRATION/VALIDATION DATA
DIAL DATA SERVER
DISC-AIRS
DISC-MODIS
DISC-OMI
DODS_DIR
DODS_FILESERVER
DODS_LAS
DODS_URL
EML METADATA
FGDC METADATA
GCMD MAPSERVER
GET DATA
GET SERVICE
NBII METADATA
NOMADS DATA SERVER
PROJECT HOME PAGE
REFERENCE MATERIALS
RELATED DATA SET DESCRIPTION
RELATED DATA SET METADATA
RELATED SERVICE METADATA
SOAP API
SOFTWARE PACKAGES
UDDI REGISTRY
WEB COVERAGE SERVICE (WCS)
WEB FEATURE SERVICE (WFS)
WEB MAPPING SERVICE (WMS)
New Proposed List
GET RELATED DATA SET(S)
GET RELATED SERVICE(S)
GET EXTENDED METADATA RECORD
VIEW RELATED DATA SET DESCRIPTION
VIEW RELATED SERVICE DESCRIPTION
GET DIF METADATA RECORD
GET SERF METADATA RECORD
VIEW PROJECT HOME PAGE
VIEW RELATED REFERENCE MATERIAL
GET WEB FEATURE SERVICE (WFS)
GET WEB COVERAGE SERVICE (WCS)
GET WEB MAPPING SERVICE (WMS)
GET DATA
GET SERVICE
GET SOFTWARE PACKAGES
DIF/SERF POPULATION
Ja
n99
Ap
r-9
9
Ju
l-9
9
O
ct
-9
9
Ja
n00
Ap
r-0
0
Ju
l-0
0
O
ct
-0
0
Ja
n01
Ap
r-0
1
Ju
l-0
1
O
ct
-0
1
Ja
n02
Ap
r-0
2
Ju
l-0
2
O
ct
-0
2
Ja
n03
Ap
r-0
3
Ju
l-0
3
O
ct
-0
3
Ja
n04
Ap
r-0
4
Ju
l-0
4
O
ct
-0
4
Ja
n05
Ap
r-0
5
Ju
l-0
5
# DIFs
18000
12000
Total DIFs January 1999 - August 2005
16000
14000
WHFC children
DIFs
split of Aqua
MODIS DIFs from
Terra MODIS DIFs
10000
new Antarctic DIFs
8000
6000
Date
Ja
nFe 03
bM 03
ar
Ap 03
r
M -03
ay
Ju 03
n0
Ju 3
lAu 03
gSe 03
pO 03
ct
N -03
ov
D 03
ec
Ja 03
nFe 04
bM 04
ar
Ap 04
r
M -04
ay
Ju 04
n0
Ju 4
l
Au -04
gSe 04
pO 04
ct
N -04
ov
D 04
ec
Ja 04
nFe 05
bM 05
ar
Ap 05
r
M -05
ay
Ju 05
n0
Ju 5
lAu 05
g05
# DIFs
Cumulative DIF Population 2003-2005
16500
16000
15500
15000
14500
14000
13500
13000
12500
12000
11500
Month
Ja
nFe 03
bM 03
ar
Ap 03
r
M -03
ay
Ju 03
n0
Ju 3
lAu 03
gSe 03
pO 03
ct
N 03
ov
D 03
ec
Ja 03
nFe 04
bM 04
ar
Ap 04
r
M -04
ay
Ju 04
n0
Ju 4
lAu 04
gSe 04
pO 04
ct
N 04
ov
D 04
ec
Ja 04
nFe 05
bM 05
ar
Ap 05
r
M -05
ay
Ju 05
n0
Ju 5
l05
# New/Revised
New and Revised Supplemental Descriptions
200
180
160
140
120
100
80
60
40
20
0
Month
Data Centers New2
Sensors New4
Data Centers Revised2
Sensors Revised4
Sources New3
Campaigns New5
Sources Revised3
Campaigns Revised5
Ja
nFe 03
bM 03
ar
Ap 03
r
M -03
ay
Ju 03
n0
Ju 3
lAu 03
gSe 03
pO 03
ct
N 03
ov
D 03
ec
Ja 03
nFe 04
bM 04
ar
Ap 04
r
M -04
ay
Ju 04
n0
Ju 4
lAu 04
gSe 04
pO 04
ct
N 04
ov
D 04
ec
Ja 04
nFe 05
bM 05
ar
Ap 05
rM 05
ay
Ju 05
n0
Ju 5
lAu 05
g05
Month
New, Revised and Deleted DIFs 2003-2005
1800
1600
1400
1200
1000
800
600
400
200
0
# DIFs
DIFs New
DIFs Deletes
Revised
DIF Population by Earth Science
Topic
SUN-EARTH INTERACTIONS
1%
SOLID EARTH
6%
SPECTRAL/
ENGINEERING
6%
PALEOCLIMATE
2%
AGRICULTURE
5%
ATMOSPHERE
17%
OCEANS
13%
BIOSPHERE
16%
LAND SURFACE
12%
HYDROSPHERE
7%
HUMAN DIMENSIONS
10%
CLIMATE INDICATORS
1%
CRYOSPHERE
4%
AGRICULTURE
ATMOSPHERE
BIOSPHERE
CLIMATE INDICATORS
CRYOSPHERE
HUMAN DIMENSIONS
HYDROSPHERE
LAND SURFACE
OCEANS
PALEOCLIMATE
SPECTRAL/ENGINEERING
SUN-EARTH INTERACTIONS
SOLID EARTH
DIF Population by Node
CLIVAR
0%
NCAR
1%
UN
7%
NOAA
18%
G3OS
4%
USDA
7%
CIESIN
2%
AMD
32%
USGS
8%
CONAE
1%
INPE
0%
ESA/ESRIN
2%
UNEP/GRID
6%
CSIRO
0%
CCRS
7%
DLR
1%
JST
1%
RAS
0%
JAXA
2%
PNRA
1%
NEONET
0%
CNES
0%
NOAA
USDA
CIESIN
USGS
CONAE
INPE
CCRS
ESA/ESRIN
DLR
CNES
NEONET
PNRA
RAS
JAXA
JST
CSIRO
UNEP/GRID
AMD
G3OS
CLIVAR
UN
NCAR
DIF Population by Node, August
2005
UN
4%
CLIVAR
0%
AMD
19%
NCAR
1%
G3OS
3%
NASA
41%
UNEP/GRID
3%
CSIRO
0%
JST
1%
NEONET
0%
PNRA
1%
RAS
0%
JAXA
1%
USGS
5%
CNES
0%
DLR
0%
USDA
4%
CIESIN
1%
ESA/ESRIN
1%
CCRS
4%
INPE
0%
NOAA
11%
CONAE
0%
NASA
NOAA
USDA
CIESIN
USGS
CONAE
INPE
CCRS
ESA/ESRIN
DLR
CNES
PNRA
RAS
JAXA
JST
CSIRO
UNEP/GRID
AMD
G3OS
CLIVAR
UN
NCAR
NEONET
IDN SERF Population
Total SERFs In GCMD
Jan. 2001- August 2005
1400
1200
SERF Totals
1000
800
600
400
200
0
Jan-01
M ay-01
Sep-01
Jan-02
M ay-02
Sep-02
Jan-03
M ay-03
Month
Sep-03
Jan-04
M ay-04
Sep-04
Jan-05
M ay-05
Distribution of SERFs
Metadata Handling
3%
Models
12%
Data Analysis &
Visualization
32%
Reference and Information
Services
10%
Hazards Management
3%
Environmental Advisory
8%
Education/Outreach
15%
Data Analysis & Visualization
Education/Outreach
Hazards Management
Metadata Handling
Data Management/
Data Handling
17%
Data Management/Data Handling
Environmental Advisory
Reference and Information Services
Models
Distribution of SERF Usage
Models
15%
Data Analysis & Visualization
21%
Metadata Handling
10%
Data Handling
12%
Information and Reference
Services
10%
Hazards Management
9%
Environmental Advisory
12%
Education/Outreach
11%
User Activity (Service Parameter Hits)
August 2002 – January 2005 Possible result
of several GIS
conferences
2000
Introduction
of the new
web page
1800
1600
Indexed by
GOOGLE
Number of Hits
1400
1200
1000
800
600
400
200
0
Aug-02
Nov-02
Feb-03
M ay-03
Aug-03
Nov-03
Feb-04
M ay-04
Aug-04
Nov-04
Feb-05
Months
Data Analysis & Visualization
Education/Outreach
Hazards Management
Metadata Handling
Data Manamgent/Data Handling
Environmental Advisory
Reference and Information Services
Models
M ay-05
WEB USAGE
IDN Usage by Domain Type 2005
.us
1%
international
23%
.gov
3%
.edu
7%
.org
1%
.com
19%
numeric
23%
.net
22%
.mil
1%
.gov
.edu
.org
.com
.net
.mil
numeric
international
.us
IDN Usage by Domain 2002-2005
14000
Cache opened to
Internet robots
12000
8000
Introduction
of the new
web page
6000
4000
2000
Ja
n05
M
ar
-0
5
M
ay
-0
5
Ju
l-0
5
Ju
l-0
4
Se
p04
N
ov
-0
4
Ja
n04
M
ar
-0
4
M
ay
-0
4
Se
p03
N
ov
-0
3
Ja
n03
M
ar
-0
3
M
ay
-0
3
Ju
l-0
3
Se
p02
N
ov
-0
2
0
Ja
n02
M
ar
-0
2
M
ay
-0
2
Ju
l-0
2
Users by Domain
10000
Month
.gov
.edu
.org
.com
.net
.mil
numeric
international
.us
# Web Page Hits Since January 2001
6000000
5000000
Introduction
of the new
web page
3000000
2000000
1000000
0
Ja
n03
M
ar
-0
M 3
ay
-0
3
Ju
l-0
3
Se
p03
N
ov
-0
3
Ja
n04
M
ar
-0
M 4
ay
-0
4
Ju
l-0
4
Se
p04
N
ov
-0
4
Ja
n05
M
ar
-0
M 5
ay
-0
5
Ju
l-0
5
#hits
4000000
Cache opened to Internet
Search robots
month
Month
Aug-05
Jul-05
Jun-05
May-05
Apr-05
Mar-05
Feb-05
Jan-05
Dec-04
Nov-04
Oct-04
Sep-04
Aug-04
Jul-04
Jun-04
May-04
Apr-04
Mar-04
Feb-04
Jan-04
Dec-03
Nov-03
Oct-03
65000
Sep-03
Aug-03
Jul-03
125000
Jun-03
May-03
Apr-03
Mar-03
Feb-03
Jan-03
Sessions
# Web Sessions Since January 2003
145000
Cache opened to Internet
Search robots
105000
85000
Introduction
of the new
web page
45000
25000
DIF/SERF SEARCH/RETRIEVAL
Searches by Controlled Keyword
Sun-Earth Interactions
2%
Solid Earth
7%
Spectral/Engineering
5%
Agriculture
10%
Atmosphere
15%
Paleoclimate
4%
Oceans
16%
Biosphere
8%
Land Surface
11%
Climate Indicators
5%
Hydrosphere
6%
Cryosphere
4%
Human Dimensions
7%
Agriculture
Atmosphere
Biosphere
Climate Indicators
Cryosphere
Human Dimensions
Hydrosphere
Land Surface
Oceans
Paleoclimate
Spectral/Engineering
Sun-Earth Interactions
Solid Earth
Update on Most Current Software
Middleware Searching Architecture
Global Change Master Directory
Central Index
GeoConnections (4%)
JCADM (18%)
NOAA (11%)
UNEP (4%)
GCMD Search Engine
NASA Contribution (42%)
EOS DAACs, ESIP,
NASA Centers
CEOP , GLOBEC, GOFC, GOSIC, IAI,
JAXA, NCAR, NOAA, OPeNDAP, UN,
USDA, WDC,
Harvesting Interfaces
(e.g. OAI-PMH)
Data sharing nodes
Portals API
Interfaces
Data contributed through
various portal collaborations
Tools
Other Data Contributions
User Search
Data contributed by staff, universities,
federal and local agencies, international
agencies and organizations, or
commercial providers
Middleware Search/Retrieval Component
(Integration of spatial, freetext, and controlled queries)
Multi-Layer Search Component
delegates
1) User performs
query
7) Refines search
3) joins
Controlled Vocabulary Database Layer
Freetext (Lucene) Database Index Layer
Spatial Database Index Layer
2)
Controller
Cache
4) Set of unique IDs
Search
Results
6) Returns to user
Result
Processor
5) Title set information, brief
summary, dataset links, etc.
Ingest of Metadata: Sequence diagram
Interaction among User, OPS/docBUILDER and MD Server
Data
provider
Start docBUILDER
MD Server
Click on Location Field
Get Location Keywords
Click on Data Center Field
Get Data Center Keywords
Get Personnel
load
load
OPS/
validate
docBUILDER
Unrecognized Personnel
Set Personnel
Unrecognized Personnel
Set Personnel
Unrecognized
Instrument Keyword
Unrecognized
Instrument Keyword
Set instrument Keyword
Set instrument Keyword
Commit to database
User Action
Software Action
GCMD Architecture
Web Application Server (Tomcat)
Full Text
Search Interface
Keyword
Interface Search
Tool/
Third Party
Clients
(e.g. OPeNDAP)
Metadata
Authoring Tool
Dataset
API
Services
API
Controlled
Vocabulary
API
Published
Interfaces
Open
API
Portal
Interfaces
Middleware
Security Layer
Operations
RMI/SOAP
Subscription
Service Interface
Spatial/Temporal
Search Interface
Query Engine
Lucene
RDMS
Spatial
XML Loader
Data Object Model
DIF
SERF
Keywords
Persistent Framework Database
Core Transaction Peer
HTTP
Data Retrieval Flow Diagram
Data Access Systems
search data sets
search results
OPeNDAP
(DODS)
refine results
refined search results
view full record
Global Change
Master Directory
MODIS
full data set description
request data
requested data set
Other
Local Database Agent
DB
MD Server
Local Database Agent
NCAR
DB
MD Server
Local Database Agent
MD Server
DB
Local Database Agent
Network
OAI Network
Local Database Agent
HTTP
Server
(Jetty/Tomcat)
OAI
Repository
OAI
Harvester
File
Based
Repository
New Content
Incoming
Queue
Mirrors Table
docBUILDER
Metadata
Controlled
Vocabulary
(Authoring Tool)
OAI/PMH (Open Archives InitiativeProtocol for Metadata Harvesting)
Capabilities
OAI-PMH
Repository
Identify
ListMetadataFormats (DIF,
FGDC, Dublin Core)
 ListIdentifiers
 Get Record
 ListRecords


GCMD
Metadata
Controlled
Vocabulary
Identifies
protocol;
supported
formats
Retrieves DIF
Metadata
identifiers,
records in XML
OAI-PMH
Harvester
New Portals added since March 2005
 CEOS Services Portal
 GEOSS
 Data Portal
 Services Portal
 Arctic Portal
 Arctic France Portal
AMD France Portal
 AMD Chile Portal
 NOAA NCDC Portal
(Current total: 55)
New Features added since April 2005
• New Keyword interface
– richer location hierarchy
– richer data center hierarchy
• MD Server upgrade
– support for data resolution
"valids”
– started finalizing DB schema
with support for platform and
instrument bucket changes
• docBUILDER
– Standalone and HTML version released
• New collaborative portals
– 7 portals added
• New Geospatial One Stop
projects added
– Includes NASA sun-earth projects
• Top 10 DIF/SERF based on
monthly stats
– “User’s Choice” box on home page
features highlights for the month
Data Resolution “Valids”
• Reminder! DIF authors can now populate their DIFs by selecting horizontal,
vertical and temporal resolution in docBUILDER.
• Vertical Resolution refers to both Altitude and Depth resolution.
Horizontal Resolution Range:
<1m
1 m - < 30 m
30 m - < 100 m
100 m - < 250 m
250 km - < 500 m
500 m - < 1 km
1 km - < 10 km
10 km - < 50 km
50 km - < 100 km
100 km - < 250 km
250 km - < 500 km
500 km - < 1000 km
> 1000 km or > 10 degrees
Vertical Resolution Range:
<1m
1 m - < 10 m
10 m - < 30 m
30 m - < 100 m
100 m - < 1km
> 1km
Temporal Resolution Range:
< 1 second
1 second - < 1
minute
1 minute - < 1 hour
Hourly - < Daily
Daily - < Weekly
Weekly - < Monthly
Monthly - < Annual
Hourly Climatology
Daily Climatology
Pentad Climatology
Weekly
Climatology
Monthly
climatology
Annual
Annual
climatology
Decadal
Climate Normal
(30-year
climatology)
MD9.4.0 Release: Highlights
New Free-Text
New
Search Capability powered by
secondary refinement capabilities by keywords or free-
text for data sets.
New Data
Center and Location hierarchies.
New brief
excerpt of record summary display with results and
search term highlighting within the metadata records.
New customized
features.
portal interfaces with additional navigation
New spatial and temporal resolution range keywords within the
data resolution fields now available for population (but not for search until a
majority of records contain this information).
PostgreSQL
• Database setup scripts are being customized to
work with an open source database.
– Scripts are being updated to work with
PostgreSQL 7.4.7 database.
– Currently testing MD Server setup with
PostgreSQL.
– Install instructions will be rewritten to make
installation as easy as possible for Nodes.
New docBUILDER
Highlights of New docBUILDER Tool
(Changes the users will see)
• Offers option to use personalized templates.
• Supports three-color coding for required, recommended
and optional fields.
• Provides automatic population of Data Center /Service
Provider URLs.
• Allows direct editing of the XML document.
• Offers more intuitive interface for Personnel and Data
Center fields.
• Incorporates spell-checker for all fields and full document.
= Both
= HTML
= Standalone
Highlights of New docBUILDER Tool
(Changes “behind the scenes”)
• Complies fully with XML Schema (previous version only
supported DTD).
– Provides capability of rendering custom authoring tool
according to the XML schema
– Renders customized widgets based on XML element names
association.
• Contains new business logic with a completely rewritten
business layer, resulting in much faster access to XML
documents and valids.
• Separates business logic from presentation logic, making
changes to presentation much easier.
– Refactored framework using Struts and JSP allowing for
complete portal customizations.
HTML docBUILDER
docBUILDER
HTML docBUILDER
(Opening Screen)
HTML docBUILDER
(Main Screen)
Menu with options
to save, email, etc.
Legend
Checkboxes
indicating progress
Editable text
area
HTML docBUILDER
(Data Resolution Field)
Field Name
Field
Description: short
description of the filed,
including the type of
information expected.
Add/Remove
field: user can click
the green (+) to add a
new field, or the red
(–) to remove that
field.
Link to full field
description: link to
the DIF/SERF Writer's
Guide page that fully
describes the field,
including characters
allowed, definition,
format, etc.
HTML docBUILDER
(Science Keywords Field)
Subset list selector
Variable level
filter: if checked,
removes all items
with Variable level
List search: filters
list as you type, keeping
only those items that
match.
Option to add
detailed variable:
allows user to add
detailed variables to
any selected items in
the list
HTML docBUILDER
(Personnel Field)
Personnel
Lookup link:
when clicked,
presents the user
with all personnel
currently in the
GCMD DB. After
the user selects a
person and clicks
the Accept button,
the information is
then populated in
the form on the
Personnel page.
HTML docBUILDER
(Data Center Field)
Data Center Lookup
link: allows the user to select
from a current list of data
centers, then populates the Data
Center form with this info,
including URL.
MD9.5 Plans for the future
MD9.5 Projected Release: December 2005
New refinement capabilities for services by keywords or free-text.
Keyword subscription service.
Relative Time Capability for data sets.
Automated update reminders.
Platform/Instrument Upgrade and Clean-up.
Open Archives Initiative (OAI) - Protocol for Metadata Harvesting
(PMH) substitution for Local Database Agents (LDA)
Central Repository for very small data sets.
Upgraded spatial search.
Integrate SOAP API.
Server software upgrade to handle search on temporal/spatial resolution.
This will be dependent on adequate population of the resolution field
by authors.
Keyword Subscription Service
• Receive notifications of updates, removals,
and additions to keywords for:
–
–
–
–
–
Data Centers/Service Providers
Platforms
Instruments
Locations
Projects
Search Refinements for SERF MD9.5
• Refine by:
–
–
–
–
–
–
–
–
Service Providers
Instruments
Platforms
Projects
Science Keywords
Service Keywords
Parent SERFs
Child SERFs
Close-up of Refinement
Methods to
refine by
Users can also refine
by Parent and
Children SERFs