Transcript Cluster Active Archive

Cluster Active Archive
Harri Laakso
Contents:
1.
2.
3.
4.
5.
6.
7.
8.
Rationale for CAA
Quick-look plots
CAA Database at present
Graphical user interface
Inventory plots
CAA tools
Raw data distribution
Summary and next steps
Rationale for the Cluster Active Archive
“Maximise the scientific return from the mission by making all
Cluster data available to the world-wide scientific community.”
“Ensure that the unique data set returned by the Cluster
mission is preserved in a stable, long-term archive for scientific
analysis beyond the end of the mission.”
“Provide this archive as a major contribution by ESA and the
Cluster science community to the International Living With a
Star programme.”
Implementation of the CAA
What does this mean?
The CAA should contain (all) the Cluster high resolution data.
The data should be of the best (achievable) quality.
The data should be suitable for detailed science investigations.
Need to start now while expertise still available.
ESA providing support to instrument teams to ensure this.
In addition the CAA will hold ancillary products and support information
including
• Auxiliary data (such as orbit and attitude)
• Survey data and plots
• CSDS data including JSOC parameters
• Documentation
CAA Login web-site
• For browsing
the CAA
database, e.g.
viewing quicklook plots or
downloading
data, you need
to login first
Quick-Look Plots
Quick-Look Plots of STAFF
Data products search
CAA Database: ASPOC
• Interval over which data
products exist; this does
not say anything about the
completeness of the
coverage
• Details of the dataset
Ion emission current,
0.5 sec resolution
He++
O+
RPA
He+
protons
CAA Database: CIS-CODIF
CAA Database: CIS-HIA
•
EFW: the E-field products
are still in spacecraft
coordinate system,
including vB effects. The
final transformation will be
done in CAA after the
calibrated FGM data have
been ingested into the
system
Level 2
EDI: drift velocities and
electric field files have not
yet been transferred into
CEF and therefore not yet
ingested into the system
Level 2
•
Level 1
CAA Database: EDI & EFW
CAA Database: FGM & STAFF
3DR: reduced 3D dataset
•
These data are provided
in NM and in addition
similar data are
calculated in BM to
produce a complete
dataset
•
Many other products are
to be introduced in a few
months after the
definition of the formats
of the products is
completed
LEEA
•
HEEA
CAA Database: PEACE
electrons
CAA Database: RAPID
CAA Database: Whisper
•
no electron density files
have not yet been produced
•
delivery schedule for the
density calibration was
recently finished and first
products should arrive
before summer
Graphical User Interface
Features
GUI - downloading
GUI - downloading
GUI - searching & user profiles
GUI - searching & user profiles
GUI - searching & user profiles
GUI - searching & user profiles
GUI - searching & user profiles
CAA Database
Instrument Team Delivery Status
Team
Interval Provided
ASPOC
2001-01-01 to 2002-12-31
CIS
2001-01-01 to2002-12-31
DWP
2003-12-26 to 2004-03-25
EDI
2001-02-03 to 2003-12-31
EFW
2001-02-03 to 2004-06-20
FGM
2001-01-30 to 2002-05-02
PEACE
2001-01-01 to 2002-07-01
RAPID
2001-01-01 to 2002-12-31
STAFF
2001-01-03 to 2004-12-31
WHISPER
2001-02-02 to 2002-12-31
WBD
2001-01-01 to 2002-12-31
ESOC
2001-01-01 to2004-12-31
Inventory Plots
• give details of the time periods for which CEF files have been ingested and
indicate data gaps within these files
• enable CAA users to
 review status of products ingested in the CAA database
 check the availability of data for specific events
• help the CAA team keep track of the completeness of data products time
periods.
• Input data sources:
 CAA Gap Files (interval > 5 minutes)
 Telemetry Modes Files (OFF, NM[1-4], BM[1-4] )
 JSOC Event Files(bow shock, magnetopause, north tail lobe...)
• Generate postscript (converted to gif for ingestion)
Inventory Plots
Inventory Plots
Inventory Plots
Inventory plots provide data coverage information to assist location of intervals
of interest:
• Pre-generated inventory plots provided at year, month and day resolution.
• Make use of record level data to show true data availability
• Provided per-instrument, per-spacecraft.
• Accessed via quicklook type interface.
• Spacecraft telemetry and JSOC predicted science events also displayed
Inventory Plots: GUI
CAA Standards and tools
CAA - Standards
The CAA standards form the basis of a consistent approach to the formatting
and description of the CAA digital data products
• The major standards development work was undertaken during the
implementation phase by the data format and metadata working groups
• The Cluster Exchange Format (DS-QMW-TN-0003), has proved very stable
and there have been no changes to the specification since May 2005
• The data model and data dictionary (CAA-CDPP-TN-0002) define the
semantics used to describe the data
 The data dictionary has also proved very stable although it is continuing to
be developed and enhanced in support of the instrument team product
deliveries.
 Enhancements have been extensions to the existing set of enumerated
values or additions of optional new attributes in support of ongoing product
development work.
 No changes have, so far, been made, or required, that would affect products
that have already been delivered.
CAA - Support tools
The main CAA provided support tools currently provided are the CAAtools,
originally developed by Tobias Eriksson
• Tools to allow checking of products against the metadata dictionary and CEF
specification. Used by instrument teams and within CAA DMS.
• Updated for bugs and changes to the metadata dictionary.
• Modified to support direct operation on Gzip’d CEF allowing data to be stored on
the CAA system in compressed form.
• Support for indexing allowing more rapid access to records within the
compressed (or uncompressed) ASCII CEF files.
QTran and QSAS tools provided by IC & QMUL
• Updated to provide improved support for CAA CEF-2 products.
• Used to provide CDF export from the CAA web interface.
• QSAS provides many useful features for multi-instrument and multi spacecraft
analysis and visualisation.
CAA – QSAS support for CEF-2
C4
C2
C3
C1
1 minute
CAA RDM Delivery
RDM CD-ROM delivery from ESOC ceased on 1 Jan 2006
 CAA provides RDM (Raw Data Media) data accessible over the network
• DDS access will continue unchanged
• CD-ROM directory structure will be maintained
 A web based system for data retrieval has been put in place and is
currently undergoing testing:
• Requests can be made for just the portions of the RDM directory
structure that are required, thus saving network bandwidth
• Automated transfers provided via wget
• ICD available on CAA website
• Details of the interface and examples are available on the CAA
technical forum: http://www.cluster.rl.ac.uk/caa-bin/yabb/YaBB.pl
Summary
 The CAA has made significant progress in the past year. The key milestones
 A special session at the Cluster 5th anniversary workshop
 Availability of the CAA for Beta testing after Sept 2005
 Formal opening of the CAA in February 2006
 Work has continued on the development of the core functionality of the system
including:
 Data export in NASA Common Data Format
 Support for user profiles
 Inventory plots
 Access to quicklook plots
 Raw data delivery system
 Instrument team activities have concentrated on refinement of the product
specification and delivery of the data from 2001/2 and beyond.
 Instrument team delivery plans have been reviewed and revised.
 A CAA cross-calibration working group has been established.
Future Plans
The major activities before the 2nd Operations Review (May 2007) include:
• Instrument team production/delivery of 2003/2004 data
• Routine operations, maintenance, data delivery and access
• Support for production and development of Level-3 products
• Production of EFW Level-3 products, removal of VxB and use of E.B=0 to calculate
full vector electric field. (Data join capability)
• On going activities of related to the cross calibration working group
• Josef Höök will be joining CAA in July to work on these tasks
• CAA ingestion of products from ESOC, CSDS, EDI and other sources
•
•
Where necessary conversion to CEF format
Production of CAA metadata information
• Support for overlapping data in ingestion and export
• Improved metadata (and version) information concatenation on export
• Development of machine accessible interface
•
•
Provides web service for automated data requests to CAA
Will support SPASE protocols allowing link to new NASA S3C VO systems
Future Plans
• Improvements to web-based user interface based on user feedback, problem
reports and RIDs
• Enhancement to user profile set-up and management
• Enhanced data search capability
• On-demand inventory generation (including product selection and filter by
ingestion date, version and dataset number)
• Development of on-demand digital data graphical output (expected to be IDL
based).
• Production of standard pre-generated multi-instrument and multi-spacecraft
plots using CAA and accessed via quicklook browser.
Next Milestones
June
Completion of cosmetic RID items
June
Ingestion of CSDS data 2001/02/03
July
On-demand inventory plotting
July/August
EFW Level 3 S/W development
August
Preliminary machine accessible interface
September
Preliminary version of on-demand graphical plot
production
September
CAA Demonstration at Cluster Workshop
December
Enhanced data search capability
14-15th May(TBC)
2nd Operations review Summer 07
Baseline delivery of data 2 years/year (2003 & 2004)
If you have any questions, suggestions etc,
please send a message to
[email protected]
CAA Management Structure
CAA Project Scientist
(Harri Laakso)
Cluster Project Scientist
(Phillipe Escoubet)
Technical Manager
(Chris Perry)
Working Groups
(Chairs:
C. Harvey
A. Allen)
Archive Developer
(Sinead McCaffrey)
JSOC
System Engineer
(Harvey Bowen)
Archive Developer
(D. Herment)
Deputy
(Matt Taylor)
ESOC
Archive Developer
(Steve Esson)
Wave Consortium
ASPOC
CIS
PEACE
EDI
RAPID
FGM
DWP
EFW
STAFF
WHISPER
Harald Jeszensky
Edita Georgescu
Stefan Mühlbachler
Ian Bates
Laurent Mirioni
Klaus Torkar
Berndt Klecker
Patrick Daly
Hugo Alleyne
Nicole C-Wehrlin
Alain Barthe
Hina Khan
Jonny Gloag
Henri Reme
Andrew Fazakerley
Elizabeth Lucek
Yuri Khotyaintsev
Per-Arne Lindqvist
Mats Andre
WBD
Jolene Pickett
Alban Rochel
Pierrette Decreau