Transcript Document 7897139

EDG Valids Activities
Patrick Agbu & Frank Corprew
EOS Data Gateway Science and Operations
[email protected]
CEOS Subgroup Meeting – CINTEX Split
September 9-13, 2002
Global Science & Technology, Inc.
1
Introduction
• To enable and maintain interoperability among
EOS Data Gateway (EDG) and other catalog
systems, e.g., ECS Data Model, ECHO, GCMD,
and CINTEX among others.
• To this end, the science and operations team have
established a procedure for managing attribute
keywords (Valids).
Global Science & Technology, Inc.
2
Justification
• Valids are the vital components to the
operation of the EDG.
• Utilized for the building of search queries.
• Must be resubmitted on a regular basis by a
data provider every time their data holdings
change.
Global Science & Technology, Inc.
3
Guidelines
• Proper management of Valid keywords
ensures interoperability with other catalog
systems by using standard and agreed upon
terms.
• Using and sharing keywords from reputable
sources, e.g., the GCMD.
• Controlling some keywords and checking
for compatibility with established ones.
Global Science & Technology, Inc.
4
Clients
• Local EDG clients are hosted at data centers both
nationally and internationally.
• The Public and Hidden Operational clients are the
two clients that require the most frequent update
of Valids and Valid-related information.
• A regular and frequent schedule of Valid
processing is needed to maintain and synchronize
the operational EDG client and the various EDG
servers.
Global Science & Technology, Inc.
5
Operational Requirements
• Components used in the EDG Valids ingest
process include:
–
–
–
–
–
Submission interface
Various computer systems
Processing utilities
Valids files
Related files and information.
Global Science & Technology, Inc.
6
Processing Tool
• The first component of the EDG Valids ingest process is
the EOS Data Gateway (EOS) Valids Processing (EVP)
interface.
• The EVP interface is a web-based ingest/processing and
testing tool for EDG Valids, and Valid-related files.
• Valids and Valid-related information can be submitted
independent of one another.
• EVP handles updates to dependent Valids, definitions,
package information, and extended criteria.
Global Science & Technology, Inc.
7
8
EVP Access
• There is only one username and password assigned to
each data center/server. If more than one person will be
accessing EVP from a particular data center/server, please
coordinate among staff.
• The address of the operational EDG Valid Processing
(EVP) interface is:
http://pilsner.gsfc.nasa.gov/evp-doc/index.html.
Global Science & Technology, Inc.
9
Processing Streams
• There are 2 Valids processing streams:
1) Operation stream (performed twice weekly)
2) Test stream (1-2 day turnaround, perhaps longer for
a NEW Data Center) or custom processing
• Flow:
TEST --> PREOPS --> OPS
• Clients: Pre-Ops, Public, Pre-Ops Hidden, Hidden
Global Science & Technology, Inc.
10
Processing Modes
• Handles all modes of Valid processing and testing,
consisting of:
- public modes (ops and pre-ops),
- hidden modes (ops and pre-ops), and
- test modes (ECS-TS1, ECS-TS2, ECS-OPS, VATCTS1, PVC-TS1, PVC-TS2, and PVC-OPS).
• EVP handles complete replacement of Valids, deletions,
and additions.
VATC = Verification Test Center
PVC = Pretest Verification Center
Global Science & Technology, Inc.
11
12
13
14
15
Testing and Checking
• EVP can be used for preliminary testing of Valids
against your server before a need to promote.
• EVP will check any Valid or Valid related files for
syntax, semantics, and content.
Global Science & Technology, Inc.
16
Schedule
• Processing of Valids is now done on a twice weekly basis,
rather than a 2-week cycle.
• Valids received between Thursday and COB Monday will
be processed and released by the following Wednesday.
• Valids received between Tuesday and Wednesday COB,
will be processed and released by the following Friday.
Global Science & Technology, Inc.
17
Updates
• EVP can only handle one deletion action, or one addition
action per processing cycle.
• Replacement can be performed repeatedly, so long as each
successive replacement incorporates previous
modifications.
• Updates to a Data Center’s dependent Valids in RCS occur
after final ingest processing for a particular cycle. At that
time, updated files are then accessible for further
submission actions (deletion, addition, and replacement) in
EVP.
Global Science & Technology, Inc.
18
Test client
• The EDG client invoked by EVP is a test instance of the
client, visible only to the data submitter, after Valids and
Valid related information have passed syntax, semantic,
and content checks by EVP.
• Files to be uploaded, either from UNIX platforms, or Pc's
should be text formatted files only, in ODL. Microsoft
word format, or other non text formats will not work.
Global Science & Technology, Inc.
19
Promotion
• The use of EVP to upload and test Valids does not
make them directly visible in the various EDG
clients (hidden, public, test).
• Conversely, if one desires to test only one’s Valids,
but do not wish to have them processed
operationally by the EDG Science and Operations
group at that time, ‘Request for Promotion’ should
not be chosen within the EVP interface.
Global Science & Technology, Inc.
20
Last Chance for Correction
• When the ‘Request for Promotion’ is made, a template
with prepopulated information is presented. Additional
explanation of any additions, deletions, or changes made to
any Valid and Valid-related files since the last submission
is appended below the prepopulated information.
• EVP can be used to test Valid submission against either
preoperational or operational server. If a request for
promotion of Valids for either preoperational or operational
mode is submitted for an operational data center, the
submission will be treated as an operational submission.
Global Science & Technology, Inc.
21
File format
• Complete replacement of dependent Valids, in Object
Description Language (ODL), are to be submitted in their
entirety.
• Dependent Valid additions should be submitted in
complete odl groups for each data product.
• Dependent Valid deletions should be submitted as a text
file with one dataset on each line, and each dataset name
enclosed in parenthesis, and prefixed with
DATASET_ID =
Global Science & Technology, Inc.
22
File Formats (cont)
• Packaging information, in ODL, is to be submitted in its
entirety as a text file, regardless of the type of change
made since last submission.
• Definition files, in ODL, are submitted separately as a text
file. Only new or modified definitions are submitted for
processing.
• Extended Attributes (Extended Valids) information, in
ODL, are submitted separately as a text file. Only new or
modified extended search attributes are submitted for
processing.
Global Science & Technology, Inc.
23
ODL Requirements
• Required fields and character limits
–
–
–
–
–
–
Dataset_id (85 chars)
Parameter (80 chars)
Sensor (30 chars)
Source (30 chars)
Processing_level (2 chars)
MD_ENTRY_ID (31 chars)
Global Science & Technology, Inc.
24
ODL Requirements (cont)
• Optional fields and character limits
– Campaign (80 chars)
– Specialized_criteria_avail (None)
Global Science & Technology, Inc.
25
Definition Guidelines
• Syntax includes ATTRIBUTE, KEYWORD, and
DEFINITION fields.
• Definitions should be clear and concise; maximum length
should be from 500 to 1000 characters; most definitions
are much shorter.
• Include units in parameter definition when appropriate.
• Acronyms may be used in definitions, but they must be
defined on first usage.
• Only new or changed definitions should be submitted.
Global Science & Technology, Inc.
26
Extended Criteria
• In the EDG, search fields that are common across all (or
most) Data Sets are supported as core search attributes.
Many data centers may also want to provide searching on
additional attributes, which may be common across several
datasets, or specific to one.
• The implementation provides for extended search criteria
as defined by individual data centers, and allows data
centers to coordinate in designing specialized search
criteria that apply to some subset of the EDG datasets.
Global Science & Technology, Inc.
27
Extended Criteria (cont)
• Because the implementation calls for the EDG client to
construct a custom search screen for each extended search
criterion, we rely on the Valids information provided in the
SPECIALIZED_CRITERIA group.
• When a user selects standard search criteria, only the
extended search criteria applicable to those datasets
making use of such extended criteria will be available.
• If there are conflicts within the user's final search criteria,
the client will notify the user of the incompatibilities
before the search is sent.
Global Science & Technology, Inc.
28
Extended Search
• Path/Row is treated as a geographic specification, not an extended
search. Search by Granule ID is implemented as a separate search
type. Thus, all extended search criteria will be compatible with
geographic search criteria.
• Unique search criteria (i.e., not variants) will be the only
members of their categories.
• Extended search definitions are submitted by data providers as
text file, using ODL objects for each SPECIALIZED_CRITERIA
group corresponding to each extended search criteria.
Global Science & Technology, Inc.
29
Advantages
• Alerts by email indicate that Valids have been
submitted for processing in EVP.
• Gives us a record of submission in the event of
system or disk failures.
• Indicates the changes from last submission and
assists us in verifying that processing is done
correctly.
• Gives us a record of input file name along with
system generated file name for proper mode of
processing.
Global Science & Technology, Inc.
30
Summary
• Established a procedure for managing attribute keywords
or Valids.
• Evaluate system Valids for consistency, scientific validity;
ensure compatibility, maintain interoperability, and
populate keyword definitions.
• Valids are submitted via the EDG Valids Processing (EVP)
tool.
• Data provider is responsible for ensuring proper ODL
syntax, semantics, and content.
• Final promotion to operational status (visible to world) is
performed by the Science & Operations staff.
Global Science & Technology, Inc.
31
Finally
• We would like to suggest that INFEO tap into
ECHO
Global Science & Technology, Inc.
32