Transcript eVLBI Progress

NEW USE of An old
correlatoR
Arpad Szomoru
Joint Institute for VLBI in Europe
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Mark IV EVN Correlator
• Developed by international
consortium: EVN institutes, MIT
• Officially inaugurated October 1998
• Comparable correlators in use at
Haystack Obs., US Naval Obs.,
MPIfR, JIVE
• Correlator boards in use at WSRT,
SMA
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Basic specifications
Input:
•
•
•
•
•
No. of telescopes (N) 16
Array Observing frequencies: 329 MHz – 22 GHz
Data bandwidth: 128 MHz per polarization (Right and
Left-hand circular), divided into 8 bands.
Channel bandwidths 0.5 MHz to 16 MHz.
Data input rate: 1 Gbps per telescope
•
•
•
Integration time 1/4s (will become 1/32s with PCInt)
2048 spectral channels per baseline/band/polarization
Data Output rate 6 MB/s (will become 80 MB/s with PCInt)
Output:
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Field of View limitation
 VLBI FoV x 100
• Limited by tint
• Time smearing
• Shorter integrations
• Enable wide field surveys
• Study μJy sources
• Discriminate AGNs
• But, enormous increase of
output data volume..
6 arcmin [FWHP] Effelsberg beam
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
EVN MkIV Correlator limits
• Integration time
• Cycle time of 0.015s (actually, 1/64th of a second)
• Spectral resolution
• 131072 complex lags per readout = 65536 spectral points
per readout
• Divided over 32 products leads to 2048 spectral channels
per product
PCINT:
• Short for Post Correlator Integrator
• Capture the full output of the EVN MkIV correlator to
disk
• Need to replace output datapath
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
The PCInt project
• High speed readout of the correlator was already prepared
• Via DSP powered serial port
• Need hardware and software to enable this
• Receiving end of serial port
• Gbit ethernet for transfer from correlator rack to data collection
host
• Fast disk subsystem in order to support 160MByte/s (parallel RAID
arrays in a Storage Area Network)
Harro Verkouter
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
RT System
VME
High Speed Serial
C40 COMM
PCI
Ethernet Card
SBC (x2)
Correlator rack (x4)
100TX
1000FX
(x8)
Switch
CCC
DDD (xn)
Fibre Channel
EEE (xk)
FC Switch
Raid
Array (xm)
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Current situation
Correlator Board (x8)
RT System
Correlator Board (x8)
C40 COMM
PCI
Ethernet Card
SBC (x2)
Correlator rack (x4)
100TX
1000FX
(x8)
Switch
CCC
DDD (xn)
Fibre Channel
EEE (xk)
FC Switch
Raid
Array (xm)
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Phase 0
High Speed Serial
VME
RT System
High Speed Serial
VME
C40 COMM
PCI
Ethernet Card
SBC (x2)
Correlator rack (x4)
100TX
1000FX
(x8)
Switch
CCC
DDD (xn)
Fibre Channel
EEE (xk)
FC Switch
Raid
Array (xm)
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Phase 1
Correlator Board (x8)
RT System
Correlator Board (x8)
C40 COMM
PCI
Ethernet Card
SBC (x2)
Correlator rack (x4)
100TX
1000FX
(x8)
Switch
CCC
DDD (xn)
1000TX
EEE (xk)
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Raid
Array (xn)
Phase 2
High Speed Serial
VME
Post-processing issues
• Huge data volumes
1hour @160MByte/s equals
560GBytes of data
• Use a cluster of nodes
• Need automated processing
• Use a processing pipeline
Achieved 1/16s sampling, at 24 MB/s
data output
Users seem to be ready for 800 GB
data-sets…
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Price recording media ($/GB)
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Data Acquisition
• Move from tape to disk
recording
Disk based recording
•Reliability
•Cost
•Bandwidth
•Efficiency
• e-VLBI: the next step
•No consumables
•Higher bandwidth
•Fast turn-around
•ToO support
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
e-VLBI using fiber
Why e-VLBI ?
• Reliability – real-time feedback to the telescopes
• Logistics – No media management
• Sensitivity – sustained data rates >> 1 Gbps possible…
• Rapid science results:
• Geodesy (Earth rotation rate)
• Precision spacecraft navigation
• Transient phenomena… GRBs, SNe etc.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Why e-VLBI (cont) ?
• Target of Opportunity (ToO) capability:
• Dominated by VLBA currently…
• Reliability & Logistics  e-VLBI
• Sensitivity  e-VLBI
• Rapid science  e-VLBI
• Rapid publication  e-VLBI
• Optimal observing strategy (obs. freq., calibrators, telescope array)
• LOFAR Transients etc.  ToOs may become much more common 
e-VLBI.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
e-VLBI Proof-of-Concept Project
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
DANTE/GÉANT
SURFnet
GARR
UKERNA
PSNC
DFN
KTHNOC/NORDUnet
Manchester University
JIVE
Westerbork telescope
Onsala Space Observatory
MRO
MPIfR
Jodrell Bank
TCfA
CNR IRA
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Pan-European Network
Dutch NREN
Italian NREN
UK NREN
Polish NREN
German NREN
Nordic NREN
Network application software
EVN Correlator
Netherlands
Sweden
Finland
Germany
UK
Poland
Italy
GÉANT:
Access of NRENs to GÉANT
NL
DE
SE
CH
HU
IT
FR
GR
CZ
BE
AT
UK
PT
0
GEANT
ES
SI
PL
IE
HR
LU
RO
EVN
telescope
LV
BG
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
CY
LT
IL
SK
EE
POC results
• Demonstration of feasibility
• Identification of problems
• Has led to closer ties with networking community and
generated political interest
• Has laid the foundation for the next step forward (EXPReS):
• I3 proposal to the EC (Communication & Network
Development Call)
• Ranked first out of 43 proposals; nearly fully funded to an
amount of 3.9 MEuro.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
EXPReS – major aims:
• Making e-VLBI an operational astronomical instrument:
•
•
•
•
16 telescopes connected to JIVE at 1 Gbps
Robust real-time e-VLBI operations
Transparent inclusion of e-MERLIN antennas within e-EVN
Target of Opportunity Capability
• Future developments in e-VLBI
• >> 1 Gbps data transfer rates,
• extended LOFAR etc.
• distributed software correlation.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Expanding the e-VLBI Network
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Network testing
400
Dw-Bo
Bo-Dw
Dw-Bo*
Bo-Dw*
100
is
k
em
0
• Sensitive to congestion
2d
•Lot of fine-tuning necessary
•And possible
et
-m
•Not really required
200
2n
em
•TCP maximal reliability
300
di
sk
•UDP connectionless
m
•Use existing protocols on currently
available hardware
• Unaccountable
•Tailor made protocols?
•Lambda switching
•Hard to quantify
•Hard to pinpoint bottlenecks
600
Mbps
•Internet weather
800
Dw-On
400
Dw-Tr
200
Dw-Bo
0
m
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
ax
m
in
Network testing (2)
Onsala-Jive
1200000000
1000000000
800000000
TCP
600000000
UDP
400000000
200000000
•
February 2005: network
transfer test (BWCTL)
employing various network
monitoring tools involving
Jb, Cm, On, Tr, Bologna
and JIVE
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
25
23
21
19
17
15
13
11
9
7
5
3
1
0
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
First real-time eVLBI Image, 3
telescope observation of
gravitational lens, May 2004
First eVLBI science observation, OH masers
around IRC10420, Richards et al, Oct 2004
First broadband eVLBI science, detection of the
Hypernova SN2001em, Paragi et al, astro-ph/0505468
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
• First open e-EVN Call for Proposals (March 2006)
• First Target of Opportunity Observations (May
2006, Cygnus X-3), Tudose et al. (in prep)…
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Current status
• Regular science runs at
128 Mbps with 6 European
stations (24 hours)
• Arecibo sometimes
participates at 32 Mbps
• Fringes from all European
stations at 256 Mbps have
been demonstrated, and,
• on single baselines, 512
Mbps
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Issues, Developments
Convincing a correlator designed for tape
technology to become real-time..
Operational improvements:
• Robustness
• Reliability
• Speed
• Ease
• Bandwidth
• Station feedback
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Ongoing
New control computers (Solaris AMD servers)
• Cut down on (re-)start time
• Powerful code development platform
• Tightening up of existing code
Other hardware upgrades:
• Upgrade existing connectivity from 6*1 Gbps to
16*1 Gbps (lightpaths)
• SX optics (fibres + NICs)
• Replacement of SU functionality: Mark5A→B:
motherboards, memory, power supplies, serial
links.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE
Conclusions
• e-VLBI is changing the nature of VLBI
• Fast response, ToO capability
• better quality control, rapid data delivery
• New science, higher bandwidths
• Large fields of view
• Will allow the study of μJy sources
• Or many masers over a large star formation region
• Data archive will contain millions of weak sources
• EXPReS will realize an operational e-VLBI network
distributed across 1000’s km – a true pathfinder for
SKA.
Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE