Transcript SA1: Production e-VLBI service Arpad Szomoru JIVE Outline • • • • 2008 May 30 e-VLBI science/test runs Operational improvements Global e-VLBI: the demonstrations Hard- and software developments EXPReS Annual Review: SA1 Slide.

SA1: Production e-VLBI service
Arpad Szomoru
JIVE
Outline
•
•
•
•
2008 May 30
e-VLBI science/test runs
Operational improvements
Global e-VLBI: the demonstrations
Hard- and software developments
EXPReS Annual Review: SA1
Slide #2
e-VLBI science/test runs
• 1 Target of Opportunity project (target: nearby supernova)
• 1 response to a triggered proposal (target: accreting X-ray binary
in a special X-ray state)
• 2 exploratory projects (both targets were candidate radio
counterparts to so far unidentified gamma-ray source)
• 1 Spectral line project (targets: candidate MASER sources)
• October deadline only had one triggered proposal
• Lack of triggers meant no science runs until March 2009
• ...but, available time was used very well
• Different proposal/selection mechanism clearly needed
• 3rd year already much more productive
2008 May 30
EXPReS Annual Review: SA1
Slide #3
e-VLBI: operational improvements
• Overall improvement in second year:
• Huge increase in robustness, reliability
and user-friendliness
• Not limited to e-operations
• Increase of operational data rate from
256 to 512 Mbps
• First experiments at ~1 Gbps (with
European telescopes)
• Demonstrations of global e-VLBI with
Chinese and Australian telescopes
• Solution of long-haul problems
• Inclusion of Effelsberg, Metsahovi,
Shanghai, Arecibo (again), TIGO
2008 May 30
EXPReS Annual Review: SA1
Slide #4
e-VLBI: data rate improvements
Number of telescopes @ data rate
1280
1152
1024
5
6
768
640
Aggregate data rate
512
6
1 2
7000
384
256
6000
4
128
0
Sep/02
-128
7
Jan/04
May/05
Oct/06
Feb/08
Data rate (Mbps)
Data rate (Mbps)
896
5000
4000
Jul/09
3000
2000
1000
Ap
r/0
Se 4
p/
0
M 4
ay
/0
Se 5
p/
05
O
ct
/0
N 5
ov
/0
M 5
ay
/0
Au 6
g/
0
D 6
ec
/0
Fe 6
b/
0
M 7
ar
/0
7
Ju
n/
07
O
ct
/0
D 7
ec
/0
7
0
2008 May 30
EXPReS Annual Review: SA1
Slide #5
Towards global e-VLBI
• Long-haul high-bandwidth data transport:
• TCP Reno on old linux kernels clearly inadequate
• Modern TCP stacks much better, UDP even better, but
impossible without modification of Mark5A control code
• And can be hostile to other users on open network
• Stability of original Mark5 code a serious issue
2008 May 30
EXPReS Annual Review: SA1
Slide #6
Global e-VLBI: hacking time
• With several high-profile demos coming up: time to take matters in our
own hands
• First step: re-enable UDP in Mark5A code
• abysmal performance...
• Incorporate VLBI-UDP in Mark5 code, in collaboration with
colleagues from UMan
• Simplified version developed at JIVE, used in both APAN and
EXPReS-Oz demo
• Complete re-write of Mark5 control software, with rigorous thread
control, enabling packet dropping, re-ordering, padding
• Used successfully to transfer data at near- line-speed from European
telescopes, from Arecibo and, recently, from South Africa and South
America
• Large development effort, but has paid off in a big way
2008 May 30
EXPReS Annual Review: SA1
Slide #7
The road to China: the demo at Xi’An
2008 May 30
EXPReS Annual Review: SA1
Slide #8
The APAN demo
• Needed:
• Reliable transport from standard Mark5
at Shanghai (ancient Linux kernel)
• Physical connection between two noncommunicating Chinese NRENs
• Some way to bypass the Great Chinese
Firewall
• Interface between correlator and
Australian telescope control systems
• On-the-fly translation from LBA to
Mark5B format
• What could possibly go wrong......
2008 May 30
EXPReS Annual Review: SA1
Slide #9
APAN results
• Opened up Siberia
connection
• Excellent PR
• Longest (e) baseline ever
2008 May 30
EXPReS Annual Review: SA1
Slide #10
The EXPReS-Oz demo
• Three Australian telescopes (Mopra, ATCA, Parkes)
• 12 hours at 512 Mbps without a hitch
• Beautiful detection of SN1987a
2008 May 30
EXPReS Annual Review: SA1
Slide #11
Correlator upgrades
• Network upgrade:
• All internal JIVE and e-VLBI traffic now through own
routers (HP5412zl and HP3500yl-48 ProCurve)
• Found quite a few of (serious) bugs in HP equipment
• So far, one bug fix out of 5 active cases
• Multiple 1-Gbps UDP streams not foreseen by HP
engineers?
• Dedicated network monitoring platform alongside routers
• Removal of dependencies on ASTRON network
• Upgrade of Mark5 hardware (motherboards, memory, power
supplies) completed
• New data acquisition platform (Solaris AMD server, fully
interchangeable with redundant control computers
2008 May 30
EXPReS Annual Review: SA1
Slide #12
New modes, feedback and streamlining
• Completed:
• General speedup of correlator control
code
• Real-time Mark5 debug info feedback to
stations
• Re-connecting of individual stations
without job re-start
• Space-craft tracking mode
• Use of Westerbork synthesis data to
improve e-VLBI calibration and target
selection
• Network monitoring tools
• Currently under development:
• Adaptive observing
• On-the-fly fringe fitting
• Real-time station-log processing, display
of Tsys, status information
• Mixed-mode e-VLBI
• Integration of Mark5B
• 1024 Mbps e-VLBI (and more..)
2008 May 30
EXPReS Annual Review: SA1
Slide #13
New network monitoring tools...
2008 May 30
EXPReS Annual Review: SA1
Slide #14
And new telescopes...
2008 May 30
EXPReS Annual Review: SA1
Slide #15
In summary
• Guaranteed high bandwidth through use of an aggressive
protocol *and* reliable software
• Lightpath technology offers clear advantages; routed
connections need negotiations
• Convincingly demonstrated that it is possible to
overcome long-haul connectivity problems
• Simplicity paid off: trivial to connect new EVN stations
once they come online
2008 May 30
EXPReS Annual Review: SA1
Slide #16
2008 May 30
EXPReS Annual Review: SA1
Slide #17
2008 May 30
EXPReS Annual Review: SA1
Slide #18
SA1: state of deliverables
• A total of 15 deliverables were due by month 24, 11 at JIVE and 4 at the
University of Manchester
• 8 out of 11 JIVE deliverables were completed
• Most of the work on the remaining 3 was finished
• 3 deliverables that were not due until the last year of EXPReS
completed well ahead of time
were
• The 4 UniMan deliverables were delayed because of the delay in the
delivery of the e-Merlin correlator. Most of the IBOB design work has been
completed, but tests await the arrival of the first I/O boards, in the summer of
2008
2008 May 30
EXPReS Annual Review: SA1
Slide #19
Outsourcing
• Three workpackages were outsourced to MIT:
• SA1.24: Mark5A code modifications
• Working version available (though no “official” release
yet)
• SA1.25: Mark5B code modifications
• Dimino (sending) side largely done in-house
• Domino (receiving) side developed at MIT, debugging in
progress
• Extensive engineering support in integration of Mark5B
• SA1.20: Improved network applications
•
2008 May 30
mainly done in-house, with MIT support
EXPReS Annual Review: SA1
Slide #20
SA1: the next 18 months
• The focus of SA1 for the next 18 months will be on
• Adaptive scheduling
• Further integration of Mark5B in EVN operations
• Streamlining operational data rate adjustments
• Further improving the robustness and flexibility of
the correlation process
• Mixed-mode operations
• Increasing the data rate beyond 1 Gbps
2008 May 30
EXPReS Annual Review: SA1
Slide #21
Specific tasks
• Fast/adaptive (re-)scheduling
• Transform EVN into a truly flexible instrument
• Based on automated download and execution of modified
observing schedules - possibly on-the-fly
• Reactions from stations vary from “excellent idea” to “over
our dead bodies”
• But will further improve reliability and responsiveness of
EVN
• All bits tested and in place, full operational test is planned
2008 May 30
EXPReS Annual Review: SA1
Slide #22
Specific tasks (2)
• Data rate adjustments
• Packet dropping:
• Drop packets at sending side (while preserving headers!)
• Pad datastream at receiving side with dummy data
• Channel dropping:
• Selectively drop frequency channels at receiving side
• High demand on cpu, need hyperthreading, right kernels,
dual cpu’s
• Mixed configurations:
• Use different configurations at different stations (limited to
32, 64, 128 etc Mbps)
2008 May 30
EXPReS Annual Review: SA1
Slide #23
Specific tasks (3)
• Mixed-mode operations:
• Enable all possible combinations of recording (at stations
and/or at correlator), real-time and delayed data transfer
• Make best use of complete EVN network, including stations
that have less-than optimal connectivity
2008 May 30
EXPReS Annual Review: SA1
Slide #24
Specific tasks (4)
• Beyond 1 Gbps
• 1024 Mbps will simply not fit on 1 Gbps...
• Westerbork: use of DWDM equipment, dual 1-Gbps
connection, channel bonding
• Could be used to Jodrell Bank as well (dual 1-Gbps
lightpaths to JIVE)
• Onsala, Effelsberg: share 10-Gbps connections with
eLOFAR, via VLAN (nearly in place)
• Next, Torun?
• Most sensitive telescopes in Europe soon at a full 1024 Mbps
• And tests ongoing at 4 Gbps, from Onsala
2008 May 30
EXPReS Annual Review: SA1
Slide #25
And soon to come...
• Yebes first light (any moment now)
• Hartebeesthoek: demo last month at 64 Mbps, S-African
universities expect to be connected to Europe at 10 Gbps in 2009
• Arecibo: recent tests at ~512 Mbps
• St Croix?
• Urumqi: 622 Mbps connection (still in rumor phase)
• Japan: talks ongoing about combining VERA telescopes with
EVN in e-VLBI pilot project
• Australia-China-Japan e-VLBI demo upcoming (at e-VLBI
workshop in Shanghai)
2008 May 30
EXPReS Annual Review: SA1
Slide #26
SA1- Past 12 month overview
SA1 Deliverables
D#
A
AD#
Descrip
Plan
Actual
11
SA
1.02
Job preparation utilities
16
27
SA
1.04
eMERLIN VSI interfaces design 13)
22
28
SA
1.05
Network Protocol decision:
16
45
SA
1.09
Tests using local Jodrell Bank home e-MERLIN telescope 14)
22
51
SA
1.13
Tested software for operational improvements
18
52
SA
1.14
Test using remote e-MERLIN telescope 17)
24
54
SA
1.15
VSI Interfaces 16)
23
16
4
71
SA
1.17
Flexible local GE network 4)
18
19
4
84
SA
1.18
Network monitoring tools 20)
30
24
4
85
SA
1.19
Multiple e-MERLIN telescope tests
24
?
95
SA
1.2
Improved network applications 3)
17
2
96
SA
1.21
Monitoring user interfaces
30
7
4
124
SA
1.22
Mark5 hardware upgrade
16
20
4
126
SA
1.23
Use of WSRT synthesis data for e-VLBI calibration
17
22
4
127
SA
1.24
Mark5A code modifications
17
2
128
SA
1.25
Mark5B code modifications
17
2
136
SA
1.26
Investigating a 1024M sub-array
20
23
4
148
SA
1.27
Space craft tracking correlator mode
27
20
4
161
SA
1.31
Removing/adding stations from the correlation process on the fly
34
20
4
2008 May 30
EXPReS Annual Review: SA1
Slide #28
16
Status
4
2
20
4
?
18
4
?