Transcript EXPReS progress, Mark5 and e-VLBI Arpad Szomoru, JIVE TOG, Yebes, November 2007, A.

EXPReS progress,
Mark5 and e-VLBI
Arpad Szomoru, JIVE
TOG, Yebes, November 2007, A. Szomoru, JIVE
Correlator upgrades
• New data output platform
• Solaris AMD server with large raid array
• Fully interchangeable with control
computers
• Needs endian-safe re-work of correlator
code
• First fringes 2/10/07!
• Re-circulation
• Has been demonstrated to work
• Will be implemented on new platform
• Network upgrade
• All internal JIVE and e-VLBI traffic now
through own router
• Has removed dependencies on ASTRON
network
TOG, Yebes, November 2007, A. Szomoru, JIVE
JIVE upgrade Mk5A+ and Mk5B
• Mark5A
• Upgrade of hardware (motherboards, memory, power supplies)
nearly completed (12 out of 16)
• Mark5A+
• Upgrade of Mark5A, enables play-back of Mark5B data on
Mark5A units
• Used at JIVE with geodetic data sets
• And on converted Australian LBADR data
• Being used to help debug Westerbork TADUmax
• e-VLBI still an issue..
TOG, Yebes, November 2007, A. Szomoru, JIVE
JIVE upgrade to Mk5B
• Two JIVE Mark5As converted to
Mark5B
• One integrated in correlator
• One to Wb for TADUmax
• New control code:
• Dimino (at stations):
• Needs additional work to enable eVLBI
• Let alone UDP transfer
• Domino (at correlator):
• Re-write of correlator control code
progressing well
• Haystack engineers helping with
modifications of Mark5B code
• First autocorrelations, though still no
fringes
TOG, Yebes, November 2007, A. Szomoru, JIVE
EXPReS network upgrade
TOG, Yebes, November 2007, A. Szomoru, JIVE
New modes, feedback and streamlining
• Currently under development:
• Adaptive observing (to be tested soon)
• On-the-fly fringe fitting
• Real-time station-log processing, display
of Tsys, status information
• Use of Westerbork synthesis data to
improve e-VLBI calibration and target
selection
• Recently completed:
• General speedup of correlator control code
• Real-time Mark5 debug info feedback to
stations
• Re-connecting of individual stations
without job re-start
• Space-craft tracking mode
TOG, Yebes, November 2007, A. Szomoru, JIVE
Pilot for e-shipping
• We have the infrastructure, let’s use it!
• In principle data can be shipped electronically
• This is NOT e-VLBI, mostly driven by economic arguments
• Use the “free” bandwidth available now
• Concerns:
•
•
•
•
Requires extra work at correlator and stations
May require additional disk packs at first
Does not fit well with intensive sessions and 1 Mk5/station
Needs good logistic procedures
• Pilot project
• Benchmark the burden on staff, work out safe logistics
• Develop procedures and software using infrastructure available from
EXPReS
• But this is not an EXPReS deliverable, formally no resources
• For stations with high connectivity at first
• because of smaller impact on operations
• Maybe do some of the last NME’s in a session
TOG, Yebes, November 2007, A. Szomoru, JIVE
e-VLBI science
• 17 e-VLBI science projects accepted since 2006:
•
•
•
•
2 failed (in early 2006)
8 active binary systems
Algol-type, X-ray or gamma-ray binaries);
4 of these were ToO projects
• 3 part of multi-wavelength campaign
• 1 adaptive observation of 16 X-ray binaries
• 4 after compactness of calibrator or target
• 1 spectral line run
• 1 supernova ToO
TOG, Yebes, November 2007, A. Szomoru, JIVE
e-VLBI: data rate improvements
Number of telescopes @ data rate
1024
896
Data rate (Mbps)
768
1
640
512
1
2
3
6
5
384
256
6
4
128
Aggregate data rate
0
Jun-03 Jan-04
-128
Aug-04 Feb-05 Sep-05 Mar-06
Oct-06
Apr-07
3500
Nov-07 Jun-08
Data rate (Mbps)
3000
2500
2000
1500
1000
500
0
Apr-04 Sep-04 May-05 Sep-05 Oct-05 Nov-05 May-06 Aug-06 Dec-06 Feb-07 Jun-07
TOG, Yebes, November 2007, A. Szomoru, JIVE
EXPReS: connectivity progress
• Connectivity in Europe
• Dedicated 1-Gbps lightpaths to 5/6 stations
• Upgraded Mark5 hardware at 5/6 tels.
• 512 Mbps from all stations possible
• but not reliably with TCP
• Move to UDP works great
• green leds at ~1024 Mbps from Torun...
• And beyond...
•
•
•
•
Shanghai, Australian telescopes at 512 Mbps
Arecibo, TIGO at 64 Mbps
Thanks to large development effort
Arecibo to higher bandwidth soon
• Still waiting for big dishes
• Progress at Effelsberg
TOG, Yebes, November 2007, A. Szomoru, JIVE
EXPReS: connectivity progress
• Long-haul high-bandwidth data transport
• TCP on old linux kernels clearly inadequate
• Circuit TCP perfoms well (TCP with UDP-like behaviour, without
congestion control)
• UDP better, but needs modification of Mark5A control code
• And can be hostile to other users on open network
• Stability of original code a serious issue
• Has led to complete re-write of subset of Mark5 control code
TOG, Yebes, November 2007, A. Szomoru, JIVE
Trans-siberian demo at Xi’An
TOG, Yebes, November 2007, A. Szomoru, JIVE
e-VLBI results
• The APAN demo:
• Opened up Siberia connection
• Excellent PR
• Longest (e) baseline ever
TOG, Yebes, November 2007, A. Szomoru, JIVE
e-VLBI results
• The Australia-JIVE EXPReS demo:
• Three Australian telescopes (Mopra, ATCA, Parkes)
• 12 hours at 512 Mbps without a hitch
• Beautiful detection of SN1987a
TOG, Yebes, November 2007, A. Szomoru, JIVE
SCARIe FABRIC
• Software correlator developments
• First level of parallelization completed
• multiple subbands and time slices on multiple CPUs/cluster nodes
• New version was used to process recent FTP fringe tests.
• Web services to interface with workflow manager
• VLBI grid broker will be tested soon (with PSNC).
• Start doing benchmarking soon
• Spacecraft tracking mode versus performance
• Next step is to deliver AIPS compatible output data
TOG, Yebes, November 2007, A. Szomoru, JIVE
Software correlation (2)
SFXC Operations
250
EVLA-like antennas (4 subb/4 pols at 4
Gsamp/s, 96 Gb/s)
EVN antennas (8 subb/4 pols at 32 Msamp/s, 1
Gb/s)
Tflops
200
150
100
50
0
1
3
5
7
9
11 13 15 17 19 21 23 25 27 29 31
N station
TOG, Yebes, November 2007, A. Szomoru, JIVE
UDP development
• Initial work by Harro Verkouter, (re-)enable UDP on Mark5A
• Collaboration with Richard Hughes-Jones and Simon Casey,
integrate vlbi-udp in Mark5A code (some serious hacking)
• Packet dropping, re-ordering, spacing
• Quick-and-dirty solution by Harro, used for APAN and
EXPReS-Oz demos
• Simon version improved, green leds at 1024 Mbps from Tr
• Harro despairs of stability and racing conditions in original
Mark5A code, completely re-writes sub-set of code (in2net2out,
disk2net2disk)
• Casey features incorporated, rigorous thread management, very
stable
• Will be used in next e-VLBI run
TOG, Yebes, November 2007, A. Szomoru, JIVE
Dim and dom, to A or not to B
• Mark5A+: works fine with MarkB e-VLBI data
• Demonstrated in EXPReS-OZ demo
• On-the-fly conversion from LBADR to Mark5B format
• Playback on A+ at JIVE
• Mark5B:
• Dimino: was thought to do e-VLBI, but...
• Domino: no e-VLBI at all
• Effort ongoing at Haystack to enable e-VLBI
• First tests end of November
• Need to ensure reliable high data-rate transfer
• Message to stations: do NOT change over to B just yet
TOG, Yebes, November 2007, A. Szomoru, JIVE