Transcript The German Astrophysical Virtual Observatory proposal GAVO

Virtual Observatories
Wolfgang Voges
Max-Planck-Institut für extraterrestrische Physik
Garching
Workshop ‘‘Astronomie mit Großgeräten‘‘
Am 17.Oktober 2003 in Potsdam
Wolfgang Voges
1
Virtual Observatories
Overview:
Historical roots *
What’s happening in the world: IVOA
European VO-activities
German VO-activities (GAVO)
Federation of local data-sets
Next-generation search engine
Grid
Theory in GAVO
Outlook
* Viewgraphs partly copied from other presentations
Workshop ‘‘Astronomie mit Großgeräten‘‘
Am 17.Oktober 2003 in Potsdam
Wolfgang Voges
2
Virtual Observatories
Historical remarks
1. VO meeting at Caltec in Pasadena (June 2000)
Astronomers mostly from the US
Very enthusiastic talks
Big vision of the future
Foundation of the NVO (US)
Since then similar meeting in Europe (Garching)
Foundation of national European and later
Other VOs
Wolfgang Voges
3
Data  Knowledge ?
The exponential growth
of data volume (and
complexity, quality) driven
by the exponential growth
in information technology
…
1000
100
10
1
0.1
1970
1975
1980
1985
1990
1995
2000
CCDs
Glass
… But our understanding of the universe increases
much more slowly -- Why?
 Methodological bottleneck  VO is the answer
 Human wetware limitations …
 AI-assisted discovery  NGVO?
Wolfgang Voges
4
How and Where are Discoveries Made?
• Conceptual Discoveries:
e.g., Relativity, QM, Brane World,
Inflation … Theoretical, may be inspired by observations
• Phenomenological Discoveries:
e.g., Dark Matter, QSOs,
GRBs, CMBR, Extrasolar Planets, Obscured Universe …
Empirical, inspire theories, can be motivated by them
New Technical
Capabilities
IT/VO
Observational
Discoveries
Phenomenological Discoveries:
 Pushing along some parameter space axis
 Making new connections (e.g., multi-)
Theory
(VO)
VO useful
VO critical!
Understanding of complex astrophysical phenomena requires
complex, information-rich data (and simulations?)
Wolfgang Voges
5
Why is VO a Good Scientific Prospect?
• Technological revolutions as the drivers/enablers
of the bursts of scientific growth
• Historical examples in astronomy:
– 1960’s: the advent of electronics and access to space
Quasars, CMBR, x-ray astronomy, pulsars, GRBs, …
– 1980’s - 1990’s: computers, digital detectors (CCDs etc.)
Galaxy formation and evolution, extrasolar planets,
CMBR fluctuations, dark matter and energy, GRBs, …
– 2000’s and beyond: information technology
The next golden age of discovery in astronomy?
VO is the mechanism to effect this process
Wolfgang Voges
6
A Schematic Illustration of the VO-Based Science
Primary Data Providers
Surveys
Observatories
Missions
Survey
and
Mission
Archives
Secondary
Data
Providers
VO
Data Services
--------------Data Mining
and Analysis,
Target Selection
Follow-Up
Telescopes
and
Missions
Results
Digital libraries
Wolfgang Voges
VO as an integral part
of the whole system …
7
The Changing Style of Observational Astronomy
The Old Way:
Now:
Future:
Pointed,
heterogeneous
observations
(~ MB - GB)
Large, homogeneous
sky surveys
(multi-TB,
~ 106 - 109 sources)
Multiple, federated
sky surveys and
archives (~ PB)
Small samples of
objects (~ 100 - 103)
Archives of pointed
observations (~ TB)
Wolfgang Voges
Virtual
Observatory
8
This quantitative change in the information
volume and complexity will enable the
Science of a Qualitatively Different Nature:
• Statistical astronomy done right
– Precision cosmology, Galactic structure, stellar astrophysics …
– Discovery of significant patterns and multivariate correlations
– Poissonian errors unimportant
• Systematic exploration of the observable
parameter spaces (NB: Energy content = Information content)
– Searches for rare or unknown types of objects and phenomena
– Low surface brightness universe, the time domain …
• Confronting massive numerical simulations
with massive data sets
Wolfgang Voges
9
Panchromatic Views of the Universe:
A More Complete, Less Biased Picture
Radio
Far-Infrared
Visible
Dust Map
Visible + X-ray
Wolfgang Voges
Galaxy Density10Map
Examples of Possible VO Projects:
• A Panchromatic View of AGN and Their Evolution
– Cross-matching of surveys, radio to x-ray
– Understanding of the selection effects
– Obscuration, Type-2 AGN, a complete census
Evolution and net energetics, diffuse backgrounds
• A Phase-Space Portrait of Our Galaxy
– Matching surveys: visible to NIR (stars), FIR to radio (ISM)
– A 3-D picture of stars, gas, and dust, SFR …
– Proper motions and gas velocities: a 6-D phase-space picture
Structure, dynamics, and formation of the Galaxy
• Galaxy Clusters as Probes of the LSS and its Evolution
– Cluster selection using a variety of methods: galaxy overdensity,
x-rays, S-Z effect …
– Understanding of the selection effects
Probing the evolution of the LSS, cosmology
Wolfgang Voges
11
Exploration of new domains of the observable
parameter space: the Time Domain
Faint, Fast Transients (Tyson et al.)
Existing and Forthcoming surveys:
Microlensing experiments (OGLE, MACHO …)
 Solar System patrols, GRB patrols …
 DPOSS plate overlaps (Mahabal et al.)
 QUEST-2 and NEAT at Palomar
… and many, many others …


The future: LSST ?
Wolfgang Voges
Megaflares on normal
12
MS stars (DPOSS)
Data Mining in the Image Domain: Can We Discover
New Types of Phenomena Using Automated Pattern Recognition?
(Every object detection algorithm has its biases and limitations)
– Effective parametrization of source morphologies and environments
– Multiscale analysis
(Also: in the time/lightcurve domain)
Wolfgang Voges
13
Exploration of observable parameter spaces and
searches for rare or new types of objects
Wolfgang Voges
14
Advantages of a Virtual Observatory
 new, more, better, faster, and easier science
 comparative analysis of multi-instrument data,
permit new approaches to research and multi-wavelength exploration,
opening discovery capabilities not otherwise possible
This is clearly the primary mandate of all VO efforts
 minimise redundancy:
data collected by a single telescope /
instrument can be re-used multiple times by different teams and
for different scientific purposes
 data integrity:
data are archived and documented in a
controlled and uniform fashion, ensuring long-term scientific
usage
 improving calibrations and creating more higher-level data
products to make data more science-ready
Wolfgang Voges
15
Advantages of a Virtual Observatory
 interoperability of archives:
-
strengthening connections to other archives, catalogues and abstract
services for broader research parameter space and links to the literature
 advancing technologies for computers, networks, data
compression, and storage media:
-
to retrieve and analyse more information more readily at lower cost
 efficient serving of data to the public:
- there will be different levels of end-user from professional astronomers to
interested (high-school) students and enthusiastic amateurs – many of
whom may undertake projects which are simply unrealisable by large
institutes
 data-mining with new software tools and new catalogues of object
properties:
- to enable higher-order research based on questions posed in scientific
terms
Wolfgang Voges
16
Advantages of a Virtual Observatory
 improving the preparation, development, building of new groundbased and space-based projects
 improving new observation proposals
 comparison of real data with simulated data – to provide feedback
to new insights, new models, new physics
Wolfgang Voges
17
International Virtual Observatory Alliance
Korea, Japan, China, Australia, India, Russia, Hungary, Italy, France, Germany,
Europe (ESO++), Canada, USA
Wolfgang Voges
18
Virtual Observatories
What’s happening in the world: IVOA
International STANDARDS are needed
Registry
Data-Models
VO-Table
VO-Query
Uniform Content Descriptors (UCD)
Simple Image Access (SID)
GRID-standards
Tools e.g. data-mining
Wolfgang Voges
19
Virtual Observatories
European VO-activities
In the AVO (euro-vo.org) under the leadership
of ESO/ESA the following institutes/groups
are collaborating:
ESO
ESA/STECF
University of Edinburgh
CDS Strasbourg
University Louis Pasteur
Centre National de la Reserche Scientifique Delegation Paris
The Victoria University of Manchester
GAVO (RDS:MPE,AIP,HS,MPA)
Wolfgang Voges
20
German Astrophysical Virtual
Observatory
GAVO-Team:
Wolfgang Voges (PI)
Hans-Martin Adorf, Gerard Lemson, Achim
Bohnet, Joachim Paul
Max-Planck-Institut für extraterrestrische Physik, Garching
Matthias Steinmetz (Co-I)
Harry Enke, Detlef Elstner
Astrophysikalisches Institut Potsdam
Dieter Reimers, (Co-I)
Dieter Engels, Peter Hauschildt
Hamburger Sternwarte
Simon White, Anthony Banday, Volker Springel
Max-Planck-Institut für Astrophysik, Garching
Other institutes are most welcome to join
>>>www.g-vo.org<<<
Wolfgang Voges
21
Why do we need a German AVO?
 to remain internationally competitive (proposals, data utilisation,
quality of science output)
 to make available VO services to everyone and provide support
for the science community and public in Germany
 to prepare and maintain datasets obtained from German facilities
for GAVO and IVO
 to establish a network, within which the needs of the German
science community and public are coordinated
 to obtain financial support from German agencies for such a
national task
Wolfgang Voges
22
Activities and responsibilities of partners
Wolfgang Voges
23
Activities and responsibilities of partners
 Main goal is science driven, but it will drive science, too
- fast access to all kinds of astronomical and related data
- capability to use highly sophisticated software tools for new
studies
- GAVO will provide interoperability of distributed archives over a
high speed network through a set of interface/infrastructure tools
- GAVO ultimately will be incorporated into larger IVO federation
- Astronomical institutes will require expert data centres of
different local character
e.g. for providing key data archives, documentations, “simple” analysis-, correlation- and
visualisation tools
-
computer science groups will develop data handling and novel
analysis tools and are responsible for their maintenance
Wolfgang Voges
24
Activities and responsibilities of partners
-
-
university institutes will be able to use GAVO for teaching and
will provide a simple gateway to the public and to schools
the “service community” will be responsible for designing and
developing the interface/infrastructure tools necessary for
communication between the users
Wolfgang Voges
25
Archive publication through
GAVO
•
•
•
•
•
ROSAT source catalogs published in IVOA compliant manner:
– simple cone search
– webservices
RASS Photons stored in PostgreSQL database
– Spatial index using HEALPix
– Cone search, webservices
Federation: fast match between ROSAT source catalogues and RASS photons.
Published first proposal for unified datamodel to serve as an ontology for the
IVOA.
Plans:
– Extend query capabilities
– Publish ROSAT fields and pointed observations
– Federate with SDSS mirror at MPA
– Federate ROSAT catalogues with external catalogues for classification of
X-ray sources (in collaboration with ClassX team).
Wolfgang Voges
26
The local GAVO activities
Top priority during initial stage of development
 federation of local key datasets and provision of key applications
 ROSAT, SDSS, Planck, RAVE
 development of meta-data standards, especially for simulations
 development of common query tools for the local archives
 need ability to query/compare both real sky and simulated data
 post-processing tools - must be platform-independent
 installation of visualisation packages
existing software provides a strong foundation to allow
extension to different types of data and archives
Wolfgang Voges
27
The local GAVO activities
Technical challenges and requirements
 archive standards: rules for ingestion, data quality, associated
meta-data schema, data attributes
 archive maintenance/evolution: migration of data with new
technology and enhancements in data attributes
 meta-data requirements/standards for different data-sets
(observations, simulation, calibration)
 federation of archives, interoperability
 high-speed networking, streaming formats for data
 distributed processing power – GRID concept
 seeking active cooperation with industry in many of these areas
Wolfgang Voges
28
Next Generation Search Engine
• Download Manager
– Retrieves data from multiple distributed databases
• Matcher
– Matches sources based on sky-position (astronomical
sources have no unique identifier)
• Classifier
– Uses multi-wavelength data for identification purposes
Wolfgang Voges
29
NextGen Search Engine (cntd.)
Simple Cone
Search Service #1
VOTable
One or more
SCS Queries
BaseURLs
Simple Cone
Search Service #2
VOTable
Multi-Catalogue MultiCone Search
"Download Manager"
Simple Cone
Search Service #3
VOTable
VOTables
VOTable Processor
Matcher
DataSets
Probabilistic Matcher
Table
Internet
BaseURLs
Service
Registry
Table on
Local Disk
VOTables
VOTables
Local Disk
Wolfgang Voges
Table
Local Disk
30
Download Manager
• Features
– Tool …
• … accesses registry at JHU
– User …
• … selects distributed catalogues
• … specifies one or more sky-locations
– Tool …
• … queries remote catalogues
• … retrieves datasets for further processing
Wolfgang Voges
31
Download Manager (cntd.)
Wolfgang Voges
32
Matcher
• Essential for data mining
• Prototype features
• Performs “fuzzy” match between pairs of source
lists from different catalogues
• Computes probability of real match
• Moving matcher into production use
– Collaboration with Canadian Virtual
Observatory (CVO)
– Feeding ROSAT source matches to classifier
Wolfgang Voges
33
Matcher (cntd.)
Wolfgang Voges
34
Classification
of ROSAT X-ray sources
• ClassX: in collaboration with US-VO
• Requires data from several large skysurveys
•
•
•
•
X-ray: ROSAT (BSC + FSC)
Optical: SDSS, USNO B1.0
Infrared: 2MASS
Radio: FIRST, NVSS, SUMSS
• True multi-wavelength VO-application
Wolfgang Voges
35
Correlation of ROSAT and SDSS data
Probing the large-scale structure of the universe
with clusters of galaxies
 Project outline:
-
(ideally on single photon/galaxy basis) (Schuecker, Boehringer,Voges)
identify a sample of galaxy clusters using X-ray/optical correlation
>>>>>> see next 3 viewgraphs
utilise optical multi-colour images (u,g,r,i,z) to derive photometric redshifts
quantify completeness and selection limits by comparison to simulated cluster data
search for IR correlation and quantify galaxy evolution in clusters
determine correlation with radio surveys to identify the frequency of radio galaxies and
AGN in clusters, search for radio halos
optical correlation to identify AGN in clusters
identify correlations with microwave/sub-mm data to search for the Sunyaev-Zeldovich
(SZ) effect (distance measurements, velocity determination)
Wolfgang Voges
36
Search for clusters of galaxies
Maximum likelihood contours based on RASS-3 X-ray photons (upper panel, 1, 2 .. contours), SDSS galaxies
(middle panel, >10), and the combined maximum likelihood contours of RASS-3 and SDSS data (lower panel,
>10). Crosses mark the position of the deepest X-ray clustersamples available sofar (REFLEX-2, X-ray flux
limit 1.8 .10-12 erg s-1cm-2). Squares mark the position of the X-ray clusters of the final sample.
Wolfgang Voges
37
Search for clusters of galaxies
Cumulative X-ray cluster number counts of the RASS/SDSSclusters (histograms) for a log-likelihood minimum of 15 applied to
SDSS data (continuous line), for 25 (lower dashed line), and for 5 (upper dashed line). The RASS/SDSS cluster counts are
compared with results obtained with other surveys (squares: RDCS, REFLEX,REFLEX-2). No corrections for variations of the
angular survey-sensitivity (effective survey area) are applied to the RASS/SDSS and REFLEX-2 data. The figure shows that with
the combination of RASS and SDSS data a 10 times deeper X-ray flux limit can be obtained compared to traditional X-ray cluster
surveys like REFLEX.
Wolfgang Voges
38
Search for clusters of galaxies
General remarks: Our first results are quite
important as a guideline for future X-ray missions
like ROSITA and DUO. For the latter, about 8,000
X-ray clusters are expected to be detectable with
standard methods. The application of the matchedfilter technique allows the extraction of about
30,000 X-ray clusters with DUO. Such large
numbers of X-ray clusters are needed for precise
tests of the dark energy and alternative gravitational
theories.
Wolfgang Voges
39
Correlation of ROSAT and SDSS data
 VO functionality required:
-
-
-
federation of relevant datasets including interchange/merging of meta-data
identification of candidate cluster members by appropriate query applications to
optical and/or X-ray catalogues
acquire multi-colour information to determine photometric redshifts
identification of candidate radio galaxy cluster members by querying radio
catalogues with search criteria (e.g. location) tailored to the derived cluster
sample
identification of associated SZ by specific queries in existing catalogues; if no
candidate SZ cluster can be identified apply suitable search algorithms to
Cosmic Microwave Background (CMB) sky surveys to determine effect or
limits thereon
visualisation of multi-wavelength cluster data
deprojection algorithms to allow study of morphology in survey data
conversion of simulation data to the space of observable parameters
3D-interface for visualisation (to schools)
Wolfgang Voges
40
Query example
Correlation between radio, IR, optical and X-ray sources
Search for SDSS QSO´s with 1 < z < 2, which are variable in one of
the 4 wavelength-bands

-
search-engine:
which datasets do exist and in which archive?
multiple availability?  parallel handling on different servers
data available for different epochs?  comparison of fluxes, light
curves, period-search
Source catalogues available?
-
radio: FIRST, NVSS, …
IR: IRAS, 2MASS, …
Optical: SDSS, Tycho-2, HST-GSC, USNO-2…
X-ray: ROSAT, ASCA, XMM-Newton, Chandra, …
Wolfgang Voges
41
Query example
-
if no catalogue entry exists
 postage-stamp (pixel-image with/without contour-lines)
 creation of light curves, fluxes, spectra, etc.
by using original-data
-
high demand of CPU?
 GRID implementation
-
search for publications on derived variable SDSS QSO objects
Wolfgang Voges
42
Grid Technology for GAVO
GAVO grid :
integration of all GAVO-workstations
at MPE and AIP into a cluster
Basic services on GAVO-grid:
CertificationAuthority provides
single-sign-on/access-all facility via proxy-ca
Resource discovery and runtime information,
network-weather for the grid
Running distributed applications
Running MPI-based applications on the GAVO-cluster
Wolfgang Voges
43
Virtual Observatories
Theory and GAVO
Simulations
Comparison of Simulations and observations
Wolfgang Voges
44
Simulations in the Virtual Observatory



Merging of the Milky Way with the Andromeda galaxy (M31)
(3 Mio particles, cluster of 16 CPU’s, 1 week of CPU time)
(30 k particles would need 25 minutes of cluster-CPU time)
Wolfgang Voges
45
Simulations in the Virtual Observatory
COMA type cluster of galaxies (>1000 galaxies, 10^15Msolar, 7 Mio particles)
8 CPUs, runtime:2 days; Gravitation, Hydrodynamics, not included: cooling, star formation
(Volker Springel, MPA)
Wolfgang Voges
46
The Role of Datasets from
Theoretical Astrophysics
• Direct Comparisons with Observations
–
Verification (or not) of Models
• Data Mining for Both Observations and
Theory
–
New Applications
– Buried Physics
• Resource for Education and Outreach
Wolfgang Voges
47
Theory and the Virtual
Observatory
• Size of Datasets Appropriate to VO
–
Large Scale Simulations, Parameter Space
Libraries Imply 10GB – 10 TB Datasets
• Rich Complement to Observational Side
• Same/Similar Tools as for Obs. Datasets
• Use of VO Infrastructure
–
Grid Technology, Portals, etc.
Wolfgang Voges
48
Conclusions
• Theoretical Astrophysics is an Essential Part
of the Virtual Observatory Concept
–
Provides Benefits to Theorists
– Provides Benefits to Observers
– Provides Benefits to Education/Outreach
• Drives New Science
Wolfgang Voges
49
GAVO efforts on the TVO
•
•
•
•
Published IVOA whitepaper on “Theory in the VO”
Leading theory subgroup in IVOA data modeling effort.
Chair in IVOA special interest group on theory
Plans:
– Publish simulation archives at AIP, LMU-Obs., andMPA
– Collaborate with UPitt on publishing services on theoretical
datasets (NSF grant proposal)
– Collaboration with Technion Haifa to publish observed and
simulated Ly- forest spectra (GIF proposal)
– Collaboration in RTN proposal for comparison of simulated and
observed X-Ray clusters (Boehringer et al@MPE)
Wolfgang Voges
50
Virtual Observatories
Outlook
Great enthusiasm among astronomical community
that VO will work and will make life easier
BUT still a lot to be done
IVOA is combining all available forces
to attack the manyfold problems
There is still the need to incorporate other fields like mathematics,
informatics,computer science, networks
etc. since there is parallel work in progress
GRID paradigm, fast data-links, super-computer access, etc.
MORE MANPOWER NEEDED
Wolfgang Voges
51
The end
Wolfgang Voges
52