Transcript Document 7432461

HEASARC Status and Plans
Nick White
February 13th, 2004
HEASARC Users Group
Nick White
The HEASARC Charter
Established December 1990
• Maintain and disseminate data from previous and concurrent highenergy astrophysics missions
• Provide software and data analysis support for these data sets
• Maintain and provide the necessary scientific and technical expertise
for the processing and interpretation of the data holding
• Develop and maintain multi-mission analysis and support tools
• Provide catalogs of observations and ancillary information for the
data holdings
• Coordinate data, software and media standards with other
astrophysics sites
• Support education and outreach activities in high energy
astrophysics (not part of original charter)
February 13th, 2004
HEASARC Users Group
Nick White
The HEASARC and High Energy
Astronomy Missions
GLAST
Swift
HETE2 Integral
Chandra data@CXC
HEASARC provides the multi-mission infrastructure that is used by
parallel GOFs and science support centers in the LHEA and elsewhere
February 13th, 2004
HEASARC Users Group
Nick White
2
HEASARC Science Personnel
• Nicholas White: HEASARC Head
• Steven Murray: HEASARC Deputy Head (at CXC)
• Lorella Angelini: Data Restoration, Image and timing
analysis, BeppoSAX, Swift (inc. AKBAR), Astro-E2
• Keith Arnaud: Data selection software, spectral analysis
• Michael Corcoran: Calibration, GLAST, HETE-2, ROSAT
• Stephen Drake: Catalogs, website & archive quality
assurance, EUVE, RXTE, CGRO
• James Lochner: Education and public outreach
• Thomas McGlynn: Archives, Virtual Observatory,
SkyView
• Bill Pence: FITS, Software, HEASOFT, Hera
February 13th, 2004
HEASARC Users Group
Nick White
Recent HEASARC Highlights
•
Hardware:
– Acquisition of 8.4 TB (12 TB raw) TB network appliance (March 2003)
– Upgrade to Gigabit network (Oct 2003)
•
Science Archive:
– XMM downloads now 2nd in volume to RXTE
– HETE-2 archive reaches 150 GB and 51 bursts
– Specialized Swift Browse interface
•
Software:
– HEAsoft 5.3 release (November 2003): Improved fv integration with DS9
– Browse 7.0 release (January 2004): new upload & download features
– XSPEC 12 testing by Integral SPI team (ongoing)
•
Catalogs:
– GRBCAT master catalog of gamma-ray bursts (Feb 2004)
•
Education and Outreach
– SkyView SciAm Top 50 award (June 2003: 2nd year in a row)
– Teacher’s Hera beta implementation (July 2003)
•
Services:
– First VO Tool: Data Inventory Service (July 2003)
February 13th, 2004
HEASARC Users Group
Nick White
The Physical Archive
Past Missions
Ariel 5
ASCA
BBXRT
CGRO
Copernicus
COS B
DXS
Einstein
EUVE
EXOSAT
Ginga
BeppoSAX
HEAO 1
HEAO 3
OSO 8
ROSAT
SAS 2
SAS 3
Vela 5B
Active Missions
RXTE (1995- )
Chandra (1999- ) [data at CXC]
HETE-2 (2000- )
Integral (2001-)
XMM-Newton (1999- )
Upcoming Missions
Swift (2004 Launch)
Astro-E2 (2005 Launch)
GLAST (2007 Launch)
• Data from 24
missions currently in
the archive
• 355 astronomical
catalogs & mission
tables
• The archive
volume was
3250 Gigabytes as of
the end of 2003
February 13th, 2004
HEASARC Users Group
Nick White
Usage & Data Statistics
Gigabytes transferred per year
1998
1999
2000
2001
2002
2003
Datasets by
ftp & http
844
1406
1880
2270
2917
3943
Science Web
Pages & Images
198
234
371
399
580
872
Main E&PO Web
Pages & Images
203
358
678
1010
3108
7581
February 13th, 2004
HEASARC Users Group
Nick White
Data Transfers by Mission over Time
February 13th, 2004
HEASARC Users Group
Nick White
Current Web Service Statistics
•
Archive volume: 3.3 TB (mostly compressed)
– Current ingest rate: 400 GB/yr
– End of 2004 anticipated rate: 1.4 TB/year, with increased or new flows from
Swift, Integral, and XMM
•
Missions in archive:
Ariel-5, ASCA, BBXRT, BeppoSAX, Chandra, Compton, Copernicus, COS-B, DXS, Einstein, EUVE, EXOSAT,
Ginga, HEAO-1, HEAO-3, HETE-2, OSO-8, ROSAT, RXTE, SAS2, SAS3, Vela 5B, XMM-Newton
•
Downloads (annual): 3.9 TB of data (by FTP and HTTP), 0.9 TB of science
web pages (by HTTP) and 7.6 TB of E&PO web pages
•
•
Browse: 355 local tables (+ ~3000 VizieR); 2,150,000 queries
SkyView: >60 surveys; >1,000,000 images generated
•
26,000 bibliography entries linking ADS to HEASARC datasets (ROSAT,
ASCA, XMM-Newton, and some RXTE datasets)
February 13th, 2004
HEASARC Users Group
Nick White
HEASARC Hardware Configuration
12 TB
Network
Appliance
Web servers
HEASARC1
GSFC Portal
Archive
2 GHz Dual Processor
HEASARC
DLT
Jukebox
HEASARC2
(load balancer)
2 GHz Dual Processor
Backup archive on 150 GB DLT
tapes (copy to NSSDC)
750 GB
NetApp
More slots available
Static web site and
shared disks
100 Mb
DBMS1 on line
Hera (x5)
1 GHz
1 GHz
150 GB
Surveys (x2)
240 GB
External
Survey Cache
February 13th, 2004
HEASARC Users Group
Database (Sybase)
2 GHZ dual Processors
DBMS2 backup
Gigabit
SkyView (x2)
400 GB
User
Cache
Safehouse
Internal
Machines
Nick White
Online
6 TB
Assorted archive
Storage backup
Linux, Solaris, Mac,
Windows, …
Web site maintenance
• Main HEASARC web site has 15,000 static pages and
hundreds of dynamic pages (excluding the FTP area and
data archive) comprising 7 GB in total
• Ensuring valid and up-to-date links pages requires
constant attention
– In the past, invalid links left unattended grew to ~500.
– Concentrated effort has now eliminated all but a few broken or
out of date links (Newly broken links are fixed weekly)
• HEASARC is now 99% Section 508 compliant
– Assuring appropriate access to HEASARC Web pages involved
changes to most Web pages
– Compliance changes were often beneficial for all users.
– Maintenance cost of 508 compliance is <=0.05 FTE effort
February 13th, 2004
HEASARC Users Group
Nick White
Browse Catalog and Archive Interface
•
•
•
•
Capable of querying all local and VizieR catalogs
Local and remote archives (e.g., CXC)
HTML, text, VOTable, FITS and Excel ouputs
New capabilities:
–
–
–
–
–
–
Expressions allowed in search criteria ( fluxX/fluxY > 2)
Better integration with SkyView
Upload files of targets (in Browse 7.0)
Download queries for later re-use (7.0)
Dataset ID info added when available (7.0)
Links between tables
February 13th, 2004
HEASARC Users Group
Nick White
Current Active Missions
•
Chandra
– Browse service fully linked to CXC archive
•
RXTE
– Dominates current archive downloads in terms of GB
– Simple XTEIndex table developed and deployed (more simplification needs to be
done)
•
HETE2
– Some HEASARC-written FITS products for GRBs
– Difficult to use most data delivered to archive
• Similarities to data recovery efforts of early HEASARC
•
XMM-Newton
– In 2003, second biggest HEASARC mission in archive volume & downloads
– Archive contains both US & non-US Guest Observer data
•
Integral
– Use modified version of HEASARC archive software at ISDC
• HEASARC Browse can link to ISDC Browse tables
– Archive at HEASARC will open in July 2004
February 13th, 2004
HEASARC Users Group
Nick White
Preparations for Swift
•
•
•
•
New HEASOFT package
MOU delineating HEASARC responsibilities
End to end testing including archive ingest.
Customized mission interfaces:
– Individualized interface to Browse
• HTML wrappers for standard Browse scripts
– Links between Browse tables
– AKBAR: new service to import external information from
observers
• HEASARC responsible for some high-level data
products – e.g., overall burst evolution
• Mission requires rapid dissemination of data
February 13th, 2004
HEASARC Users Group
Nick White
Swift Browse Interface
February 13th, 2004
HEASARC Users Group
Nick White
Preparations for GLAST
• PDMP review
• Consultations on archive and catalog
design
• Review proposed FITS formats
February 13th, 2004
HEASARC Users Group
Nick White
Standards
• FITS
– Subsumed GSFC FITS Office Web site
– Develop and coordinate domain conventions for high energy data
(OFWG))
• Organize periodic HEADCC meetings
– FITS Libraries Support
• CFITSIO is essentially reference implementation of FITS
• HEASARC also supports widely used FITS libraries in Java and IDL.
• New C++ object implementation based on CFITSIO
• Standard Dataset IDs
• VO Standards
–
–
–
–
Browse support for VO Cone search
SkyView support for VO Simple Image Access Protocol
VOTable (and FITS) outputs available through Browse
VO Service Metadata descriptions for HEASARC resources
Publication of metadata using Open Archives Initiative (OAI) protocol
February 13th, 2004
HEASARC Users Group
Nick White
Software Tools and Services
• HEASARC software is frequently downloaded
– fv (FITS browser and editor): 500 downloads/month
– CFITSIO library: 500/month
– HEASOFT: 200-300/month
• HEASOFT
– New packages for Swift and Astro E2 missions under
active development
• Use HEADAS in order to provide a simple platformindependent procedure for new software builds and updates
– XSPEC New major release (12.0) in alpha test
• Complete re-write of Xspec internals in object oriented
fashion
• Can support coded aperture mask data sets
February 13th, 2004
HEASARC Users Group
Nick White
Hera
• Provides a full, on-line analysis environment to users.
– Directly connected to archive, no downloads needed
– Up to date versions of HEASOFT tools automatically available
– Easily upload user data and download results.
• Uploaded data screened to ensure service is not used inappropriately.
– Uses standard HEASARC FV FITS browser tool as user interface.
• Why?
– Large volume of data (e.g., Integral, RXTE) downloads and/or software
installation may be infeasible for some users
– With Hera, users can start using HEAsoft and other software without the
need for local installation
• Status
– Initial system operational and being used actively.
– Proposal for continued funding has been submitted
– Both teacher and student interfaces in beta testing
February 13th, 2004
HEASARC Users Group
Nick White
New and Updated Catalogs
• ~24 new catalogs or major catalog updates in
last year.
– Major correlative catalogs: VERONCAT, GB6, etc
– High energy source catalogs: XMMSSC,
BMWHRICAT, INTREFCAT
– Master catalogs updated regularly
– GRBCAT: Compilation of all gamma-ray bursts
• VizieR connection for transparent access to
massive catalogs (2MASS, USNO B, GSC 2)
February 13th, 2004
HEASARC Users Group
Nick White
Education and Outreach
• Education Web Sites:
– Imagine the Universe (Grades 6-12): 500K
hits/month.
– StarChild (Grades 1-8) [in collaboration with L.
Whitlock]: 1.25 M hits/month.
• New poster on origin of the elements
• X-ray analysis school
– Highly successful and oversubscribed
• Host for APOD Web site
• StarChild/Imagine/APOD CDs
February 13th, 2004
HEASARC Users Group
Nick White
Other Projects
•
•
SkyView
– Java redesign
– New surveys
(DSS2, Halpha,
SFD) & SDSS
soon.
– Request for
continued funding
submitted
ClassX
– Automated
classification of
entire RASS and
WGACAT samples
of X-ray sources
using supervised
learning algorithm
Density of source requests in SkyView
February 13th, 2004
HEASARC Users Group
Nick White
Virtual Observatory Involvement
• Data Inventory Service
• HEASARC metadata inventory
– Standardized metadata describing all HEASARC
tables,data file types, surveys and services.
– Registered with VO to enable effective discovery of
HEASARC resources.
– Table metadata
• HEASARC support for VO standards
February 13th, 2004
HEASARC Users Group
Nick White
Data Inventory Service
• Access to distributed services that use VO
protocols.
– Many sites contributing data
• Dynamic service lookup using VO
registries
• Catalog, observations, Images
http://heasarc.gsfc.nasa.gov/vo/
February 13th, 2004
HEASARC Users Group
Nick White
Near Term Missions and Deadlines
• Launches
– Swift: Fall 2004
– Astro-E2: Early 2005
– GLAST: Early 2007
• HEASARC Senior Review Proposal Due:
March 17, 2004
February 13th, 2004
HEASARC Users Group
Nick White
Current and Upcoming Issues
Science
• May need greater emphasis on supporting
gamma-ray astronomers as Integral and GLAST
Archives come online and become an increasing
fraction of the HEASARC’s archive
• Must deal with ever increasing complexity of
data and calibration
– Coded aperture masks (Swift, HETE2, Integral)
– Complex source dependent response (“ + GLAST)
– Multi-satellite observations (Con-X, LISA)
February 13th, 2004
HEASARC Users Group
Nick White
Current and Upcoming Issues
Hardware
• CPU and disk space needs achievable, but bandwidth is
a potential limitation
– Datasets may be growing faster than bandwidth
– Demand for data will spike after GRB detections
– Higher speed linkage to Internet possible but costly
• … and doesn’t help users who have slower connections
– Move towards more user processing at HEASARC
• Hera, grid computing
• On-line backup for archive is presently in same building
as primary HEASARC archive.
– Planned move of backup archive to another building when space
and network is ready (Summer 2004)
February 13th, 2004
HEASARC Users Group
Nick White
Current and Upcoming Issues
Local Software and Services
• Integration of tools and services
– Strengthen SkyView/Browse, Browse/NVODIS,
Hera/Browse, … cross-links.
– Enable coupling archive/web services and HEASOFT
tools.
• Systematic attention to internal documentation of
HEASARC systems
February 13th, 2004
HEASARC Users Group
Nick White