Transcript SXDS database and Japanese Virtual Observatory

SXDS database and Japanese
Virtual Observatory
Yuji Shirasaki and JVO collaborations
National Astronomical Observatory of Japan
What is Virtual Observatory ?
• Virtual Observatory (VO) is an collection of
the astronomical DBs which are accessible
with a standard protocol over the internet.
Real Observatory
Sky
Virtual Observatory
Catalog DB, Data Archive
Telescope
Detector
Internet
Computer
Data Reduction
Reduced data will be provided or
auto reduction service
Weather sensitive
Long observation time
Small portion of Sky
any time OK (need network connection)
Short observation time
Whole Sky
Why do we need a Virtual Observatory ?
• Avalanche of the observational data
ALMA HST XMM Astro-F
– Nobeyama
~ 1TB/year
Astro-E2
– Subaru
~ 20TB/year
– ALMA
~ 1PB/year
Subaru
– XMM, Chandra, Astro-E2, Astro-F, ...
SDSS
2MASS
– The data growth rate (T2 < 1 year) is higher than the
improvement of computational power (T2 ~ 18 months) and
network bandwidth (T2 ~ 20 months) the newest
informational technology, Data Grid and Computational Grid.
• A multiwavelength study is crucial for understanding the
nature of astronomical objects, however...
– Calibration and analysis methods are highly non-uniform
across archives  manual data reduction is required. It is a
physical and mental barrier to the multiwavelength study .
What is enabled in the VO era ?
• Easy and seamless access to a large
number of databases.  You don’t need
to access to tens or hundreds of web
sites.
• Automated data reduction.  You are
free from calibration issues.
• Federation of databases and analysis
tools.  Efficient data analysis
environment.
• Discovery of rare and exotic objects
based on multiwavelength observations
of billions of objects instead of
hundreds.
Digital Universe is on
your desktop !
VO
Request
result
federation of data and
analysis services by VO
standards
Subaru ALMA Astro-E2 Astro-F
XMM HST
SDSS 2MASS
• Discovery of transient phenomena
exploring the time-domain.
• Discovery of ...
SExtractor HyperZ
IRAF
...
Key components
realizing the VO
VO Query Language: Multi-Purposes
Standard Query Language for VO
Registry
Open Sky
Server
VLA
DB
Subaru
DB
Registry: Data & Analysis
Service Discovery Service.
VO Query
Language
Source
Extractor
XMM
DB
VOTable
FITS
Hyper-Z
IRAF
VOTable: Tabular Data
Transfer Format (XML)
OpenSkyServer: VO
Compliant Data Service.
Resource Meta Data in Registry
Resource metadata describes what data and computational
facilities are available where, and once identified, how to use
them.
Identity
General Content
Title
Subject
ShortName
Description
Identifier
Source
Curation
Publisher
PublisherID
Creater
Contributor
Date
Version
Contact
ReferenceURL
Type
ContentLevel
Relationship
RelationshipID
Collection and
Service Content
data quality
Facility
Uncertainty
Instrument
Coverage
Resolution
UCD
Format
Rights
DataQuality
Capabilities
StandardURI
StandardURL
MaxSearchRadius
MaxReturnRecords
Interface
InterfaceURL
BaseURL
HTTPResultsMIMEType
VO Query Language
VO Query Language defines a syntax for searching
astronomical data.
• Astronomical Data Query Language (SQL)
Select a.*
from Tab a
where Region('Circle Cartesian 1.2 2.4 3.6 0.2')
• Simple Image Access Protocol (URL based query)
http://myimages.org/cgi-bin/VOimq?POS=180.,-30.&SIZE=0.0125...
• Simple Spectrum Access Protocol (URL based query)
http://myspectrum.org/findSpectrum?POS=180.,-30.&SIZE=0.0125...
Unified Query Language based on JVO Query Language
VOTable
VOTable is an XML format for exchanging tabular data
<?xml version="1.0"?>
<VOTABLE version="1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://vizier.u-strasbg.fr/xml/VOTable-1.1.xsd">
<DEFINITIONS>
<COOSYS ID="J2000" equinox="2000." epoch="2000." system="eq_FK5"/>
</DEFINITIONS>
<RESOURCE name="myFavouriteGalaxies">
<TABLE name="results">
<DESCRIPTION>Velocities and Distance estimations</DESCRIPTION>
<PARAM name="Telescope" datatype="float" ucd="TEL_SIZE" unit="m“ value="3.6"/>
<FIELD name="RA" ID="col1" ucd="POS_EQ_RA_MAIN" ref="J2000" datatype="float“ width="6" precision="2" unit="deg"/>
<FIELD name="Dec" ID="col2" "POS_EQ_DEC_MAIN" ref="J2000" datatype="float“ width="6" precision="2" unit="deg"/>
<FIELD name="Name" ID="col3" ucd="ID_MAIN" datatype="char" arraysize="8*"/>
<FIELD name="RVel" ID="col4" ucd="VELOC_HC" datatype="int“ width="5" unit="km/s"/>
<FIELD name="e_RVel" ID="col5" ucd="ERROR" datatype="int“ width="3" unit="km/s"/>
<FIELD name="R" ID="col6" ucd="PHYS_DISTANCE_TRUE" datatype="float“ width="4" precision="1" unit="Mpc">
<DESCRIPTION>Distance of Galaxy, assuming H=75km/s/Mpc</DESCRIPTION>
</FIELD>
<DATA><TABLEDATA>
<TR><TD>010.68</TD><TD>+41.27</TD><TD>N 224</TD><TD>-297</TD><TD>5</TD><TD>0.7</TD></TR>
<TR><TD>287.43</TD><TD>-63.85</TD><TD>N 6744</TD><TD>839</TD><TD>6</TD><TD>10.4</TD></TR>
<TR><TD>023.48</TD><TD>+30.66</TD><TD>N 598</TD><TD>-182</TD><TD>3</TD><TD>0.7</TD></TR>
</TABLEDATA>
</DATA></TABLE>
</RESOURCE>
</VOTABLE>
International Virtual Observatory
Alliance (IVOA) http://www.ivoa.net/
14 projects, ~$25 million in R&D
an alliance of existing and future national and international projects to
define standards on access to any kind of astronomical resources
(database, analysis tools and so on...)
Working Group in the IVOA
• Registry : standardization of meta data
• VOTable : define XML format for the exchange
of tabular data
• VO Query Language : standard query language for
astronomical data base
• Data Access Layer : define and formulate VO
standards for remote data access.
• UCD : defining and standardizing meta data
(Unified Content Descriptors)
• Data Modeling : the IVOA data modeling
standardization
• Grid & Web Service : Use of Grid technologies
and Web Services in the VO context
IVOA Interest Groups
• Application: intended to support developers and
users of Virtual Observatory applications
• VO Theory: formed with the goal of ensuring that
theoretical data and services are taken into
account in the IVOA standards process
Mailing List discussions:
http://www.ivoa.net/forum/
Interoperability Meeting
aimed at making significant progress in generating
new standards powering the development of the
world wide Virtual Observatory initiatives.
•
•
•
•
•
2003-05
2003-10
2004-05
2004-09
2005-05?
Cambridge, UK
Strasbourg, France
Boston, USA
Pune, India
Japan
Japanese Virtual Observatory
• Takes the initiative of DB standardization in the
Japanese astronomical community.
– Nobeyama, Subaru and ALMA DB
– Databases managed by ISAS and other institutes.
• Contributes to standardization of VO query
language.
• Provides a VO portal where one can seamlessly
access to the VO compliant data and analysis
services.
JVO Collaborators
Project Scientists
NAOJ
• Mizumoto
• Ohishi
• Shirasaki
• Tanaka
• Honda
ICRC
• Yasuda
Ochanomizu U.
• Masunaga
System Engineers
Fujitsu Ltd.
• Monzen
• Kawarai
• Ishihara
• Yanaka
• Yamaguchi
• Ishida
• Saito
• Abe
• Tsutsumi
SEC Ltd.
•
•
•
•
Morita
Nakamoto
Kobayashi
Yoshida
JVO Prototype
• Experiment for DB federation
• Functionality test of JVO Query Language
• Globus Toolkits 3
– Remote processing and file transfer.
• Web based User Interface
– Tomcat and Struts
• Distributed Databases
–
–
–
–
–
DB1  SXDS Suprime-Cam (Catalog & Image)
DB2  SXDS XMM (Catalog & Image)
DB3  SDF
DB4  SDSS Spectrum
DB5  2MASS
Architecture of JVO Proto 2
JVO
Server
Grid Service
• Data Search
JVOQL
• Analysis
Parameters
Registry
(XMLDB)
XML
Resource Metadata
XPath
1. User input
2. Create an
observation
procedure
3. Resolve data
service location
using registry
4. Execute data
search and/or
data analysis
5. Save results in
the user DB
User
Interface
•
•
•
•
•
JVOQL Editor
VOTable Viewer
DB Search
Plotter
etc
Controller
• Parser
• Scheduler
• Executer
User DB
JVOQL
(VOTable)
VOTable
FITS
JVOQL
(VOTable)
Grid Service
Meta
DB
Catalog FITS
DB
Server 2
Grid Service
Meta
DB
VOTable Catalog
FITS
FITS
DB
Parameter
List
XML etc.
JVO Portal Server
Server 1
Server 3
Grid Service
Demo 1: Cross match & Image request
Cross match of the optical and X-ray catalogs of
SXDS and image retrievals.
Registry Server
Data Server 1
(SXDS Subaru)
Grid Service
Grid Service
resolve data
service URL
Meta
DB
Registry
(XMLDB)
cross match
Catalog FITS
result (5
DB
)
(2
)
(7) image (4 xmatch
)
request
(8) result JVO Portal Server
image Data Server 2
(SXDS XMM)
(6)
(1
Search) Request
(JVOQL)
(3
)
catalog
search
Grid Service
Meta
DB
Catalog FITS
DB
Sample Query
select opt.POS_EQ_RA_MAIN as ra,
opt.POS_EQ_DEC_MAIN as dec,
opt.N18APMAGB as mag_B,
optical catalog
opt.N18APMAGR as mag_R,
opt.N18APMAGi as mag_i,
opt.N18APMAGz as mag_z,
x.POS_EQ_RA_MAIN as ra_x,
x.POS_EQ_DEC_MAIN as dec_x,
x.flux0,
X-ray catalog
x.flux1,
x.flux2,
x.flux3,
optical image
x.flux4,
img_opt.BOX(POINT(ra, dec), 20 arcsec, 20 arcsec) as image_opt,
img_x.BOX(POINT(ra, dec), 20 arcsec, 20 arcsec) as image_x
from naoj.sxds.sxdsR1 opt,
naoj.xmm.xmm_epic_sxds x,
naoj.sxds.sxds_image img_opt,
naoj.xmm.xmm_image img_x
where XMATCH(opt, x) < 5 arcsec
and
opt.N18APMAGR < 24
and
BOX(POINT(34.5, -5.0), 0.1, 0.1)
X-ray image
Precision for the cross
identification of the optical and
X-ray objects.
region selection
Demo 2: Cosmic String Search
Data request to the SXDS optical catalog, GL candidate
selection, String search by pattern recognition.
Registry Server
Analysis Server
Grid Service
Grid Service
resolve data
service URL in
each request
Registry
(XMLDB)
(8) result
String search
(7
)
(5) GL cand. selection
JVO Portal Server
(1)(4)(6
) request
Data
GL candidate selection
String search
result
(3)
(2
)
catalog
search
Data Server 1
(SXDS Subaru)
Grid Service
Meta
DB
Catalog FITS
DB
Cosmic String
http://www.damtp.cam.ac.uk/user/gr/public/cs_evol.html
• Prediction by Unified theory
– super heavy cosmic strings
with linear mass density of
1022 g/cm in the early universe.
• The lens effect by a long
cosmic string
– undistorted lensed image
– co-aligned in a direction of
string network
– distributed in a very large
scale.
• Because of its large scale
nature, wide fied deep survey is
crucial for its discovery.  Data
mining from Subaru SuprimeCam image data
User Authentification
JVO QL Editor
Create SQL
Search
Search Status
Search Result
Cosmic String Search Demo using SXDS Data base
1. Catalog Search
2. GL Candidate Search
Select pair objects of similar color.
3. Cosmic String Search
Pattern recognition
Cosmic String Search Result
Summary
Road map of the JVO project
2004 Prototype 3
–
–
–
–
development of components for operational system
Subaru Suprime-Cam reduced data DB
ISAS DARTS ?
Test connection to the international VO
2005 start to develop operational system
– late 2005 ?? trial use
2006-2007 trial use & upgrade
2007 prepare for partial operation of ALMA
Nobeyama
Subaru
ALMA
Astro-F
YOHKO
Astro-E
HALCA
JVO Query Language
1.
Select objects located in a circle region centered at ra=270 and dec=1.5 with 0.2 deg radius from optical catalog.
2.
Identify X-ray counter part for each selected object with 5 arcsec
precision.
3.
Get FITS images of 10” radius size from optical and X-ray image data
service.
select
optCat.ra, optCat.dec, xCat.ra, xCat.dec,
optImage.Image, xImage.Image
from
optCat, xCat, optImage, xImage
where
((‘ICRS’, 270.0 deg, -1.5 deg), 0.2 deg) ~ (optCat.ra, optCat.dec)
and Distance((optCat.ra, optCat.dec), (xCat.ra, xCat.dec)) < 5 arcsec
and optImage.regionSky = ((optCat.ra, optCat.dec), 10 arcsec)
and xImage.regionSky = ((optCat.ra, optCat.dec), 10 arcsec)
and optImage.FORMAT = “FITS”
and xImage.FORMAT = “FITS”
DB Search