Transcript Software Correlators (for VLBI and more)

DiFX: a general purpose software
correlator
Adam Deller
Swinburne University/CSIRO Australia Telescope National Facility
Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes
(Swinburne), Dr John Reynolds (ATNF)
Outline
• DiFX
 Reason for existence
 Structure
 Capabilities
 Performance and verification
• Science results
• Software correlators as part of the
bigger picture
26/06/2006
NextGen Correlator Conference
Why DiFX?
• Development of LBA disk-based system
demanded new correlator (>128 Mbps)
• First correlator (XF: West 2004) too slow
• Cross compatibility between formats
(separation of unpack from process)
• Allow niche projects that couldn’t be
done any other way (eg pulsar
scintillation: Brisken et al. 2006)
26/06/2006
NextGen Correlator Conference
Guiding principles
• Don’t exclude anything!
• Modular/heirachical: aid cross-platform
development/acceleration (hence C++)
• Separation of correlator from
extraneous tasks (eg delay modeling)
• Optimise for cluster environment
(Swinburne) and LBA, but don’t exclude
more specific optimisations
26/06/2006
NextGen Correlator Conference
DiFX structure
DataStream 1
Baseband data
Core 1
DataStream 2
Source data …
Core 2
…
DataStream N
Core M
Timerange, destination
Visibilities
Master Node
Uses MPI for inter-process comms
26/06/2006
NextGen Correlator Conference
DiFX structure
• Configuration is presented as ascii file
(keyword/value pairs)
• Delay/uvw information generated via
CALC and presented as ascii file
• MPI, thread and pulsar info supplied in
separate ascii files
• Output written as RPFITS (for now)
26/06/2006
NextGen Correlator Conference
Capabilities: Now
•
•
•
•
•
•
•
Compatible with LBA, MkV, K5
a priori tsys calibration
Arbitrary time/frequency resolution
Arbitrary pulsar binning from polyco file
All calculations in 32 bit floating point
Partially configured from vex file
Real-time fringe checking (LBA): < 1 min
26/06/2006
NextGen Correlator Conference
Capabilities: Soon
• Coming shortly:
 GUI frontend for configuration
 Real-time operation on streaming data
• Side project: alternative platforms (Cray
XD-1, FPGA acceleration/hybrid), Cell
26/06/2006
NextGen Correlator Conference
Performance
• Real-time operation for the LBA (6
telescopes, “continuum”) @ 1 Gbps
(4x64 MHz bands): <200 CPUs (P4)
• Scales linearly with aggregate
bandwidth, near-linear with number of
telescopes (to ~10-15)
• Small (< x2) penalty for pulsar binning,
polyphase filterbank, overlapped FFTs
26/06/2006
NextGen Correlator Conference
Performance benchmarks
• Recently carried out on the Swinburne
cluster for NRAO on VLBA data (1
blade: 28 x 3.0 GHz P4)
• Tests had separate nodes for the
Datastreams thus Nprocs (Cores) does
not include these
• Full polarisation, with autocorrs, pre-F
quadratic interpolation fringe rotation
(worst for both network and CPU)
26/06/2006
NextGen Correlator Conference
Scalability: Antennas
• 256 Mbps, 256 chans, 1s int, 8 Core
Cores required for real-time
300
250
Nodes
200
150
100
50
0
0
5
10
15
20
A ntennas
26/06/2006
NextGen Correlator Conference
Scalability: Channels
• 10 antennas, otherwise as before
Cores required for real-time
300
250
Nodes
200
150
100
50
0
10
100
1000
10000
100000
Num channels
26/06/2006
NextGen Correlator Conference
Early science results
• NZ tests (first trans-Tasman
fringe - to 6m antenna!!)
• 3mm VLBI (ATCA to Mopra)
• Detection of double pulsar
• Wide-field VLBI (Lenc & Tingay)
• Pulsar scintillation (Brisken)
26/06/2006
NextGen Correlator Conference
Correlator comparison
• 10 station VLBA experiment, 2x8 MHz
bands x 2 pols (128 Mbps)
• 43 GHz
• Strong (~4 Jy source) for 8 minutes
• Different version of CALC and different
earth orientation parameters (EOP) so
absolute phase not expected to track
26/06/2006
NextGen Correlator Conference
Amplitude/phase comparison
26/06/2006
NextGen Correlator Conference
Still to do
• More detailed comparisons (require
model to be included - need UVFITS
output with IM table)
• More exhaustive comparisons (not just
one combination of parameters)
• Testing of special modes (pulsar
binning, polyphase etc)
26/06/2006
NextGen Correlator Conference
Correlator parameter space
• Three (blurry) types - sw, fpga, hw
Software
FPGA
Hardware
Development time & cost, power per final dollar spent, break-even point
• Moores Law increases capabilities of all
three - as instruments demand more
• Software development cost spent once
26/06/2006
NextGen Correlator Conference
VLBI Correlators
• VLBI has small numbers of antennas
and they’re not growing - VLBI is falling
farther into software dominated space
• Cost of software correlator cpus <<
cost of disks << cost of eVLBI fibres
• Eg: eEVN - 20x8Gbps (2GHz total bw)
= ~6000 P4s (circa 2005)
26/06/2006
NextGen Correlator Conference
Piggybacking
• Small (zero??) development cost means
software could be used for special
purpose experiments on arrays that
have other dedicated correlators
• Eg: GMRT low frequency high time/freq
resolution (pulsars, extrasolar planets)
• Can do slower than real time with
existing computers on-site
26/06/2006
NextGen Correlator Conference
Conclusions
• DiFX is still under development but
basically ready for use (has been used)
• Software offers the opportunity to do
science that can’t be done elsewhere
• A large and growing area of correlator
parameter space is perfect for software
correlators - should we concentrate on
one general purpose code? Clusters are
plentiful and can be shared.
26/06/2006
NextGen Correlator Conference
26/06/2006
NextGen Correlator Conference
Disclaimer
• 10 second run - could be affected by
start-up/finalise costs (worse with more
nodes - more synchro)
• Initially everyone sends at once - this
smooths out. Causes an effective slowdown - worse with more Cores
• VLBA data less efficient (headers)
• Only reading from standard disk
26/06/2006
NextGen Correlator Conference
Scalability with # Cores
• 10 antennas, 256 channels, 1 sec
Cores required for real-time
100
99
98
Nodes
97
96
95
94
93
92
91
90
0
5
10
15
Num cores
26/06/2006
NextGen Correlator Conference
More amplitude comparisons
26/06/2006
NextGen Correlator Conference
Comparison discussion
• Agreement is good, but clearly rms in
amplitude is higher in swc
• Averaging from 128 to 16 chans?
• Tried to match my CALC delays to VLBA
- apparent 2ns offset
• EOP is certainly different, therefore
unsurprising that absolute phase
doesn’t match
26/06/2006
NextGen Correlator Conference
Further tests
• Need to account for the model
differences - ie need IM table in
software correlator output
• Ideally this would happen at same time
as implementation of uvfits
• Obviously before adoption detailed and
exhaustive acceptance testing will be
required - this just a start
26/06/2006
NextGen Correlator Conference
Discussion of benchmarking
• As the disclaimer said - indicative!
• Could be further optimized for MkV
• A very flexible cluster that can average
>256 Mbps x 10 antennas would be 37
dual dual-cores + 10 antenna machines
+ 1 master (48 port switch) - <$200k?
• Doesn’t include disk costs
26/06/2006
NextGen Correlator Conference
Outline
• Interferometry, the role of a correlator,
and why we choose software
• Software correlator science results
• Software correlating and VLBI in
Australia
• Performance and capabilities of DiFX
(current FX correlator)
• Future drivers and opportunities
26/06/2006
NextGen Correlator Conference
Interferometry & Correlators
• Interferometer samples
visibility: Fourier transform
of sky brightness
u ()
RA
v ()
Dec
• XF correlator accumulates
lags, then FFTs
• FX correlator FFTs segments
of baseband data, and crossmultiply/accumulates
26/06/2006
NextGen Correlator Conference
Interferometry & Correlators
• Major operations performed by
FX correlator:






Delay
Unpack quantized data to float
Fringe rotate
FFT
Correct fractional sample error
Cross multiply and accumulate
26/06/2006
A
B
Datastream
NextGen Correlator Conference
Hardware vs software
• Unclocked, could be faster or slower
than real-time
• No channel/integration time restrictions
• Floating pt vs int calculations
• Software naturally lends to FX since
greater precision is required at FFT
output
26/06/2006
NextGen Correlator Conference
Why software?
• Flexibility - you can do things that are
impossible with a hardware correlator
• Compatibility
• Expandability
• Rapid (and cheap) development
• Add-ons MUCH easier in software
• In general, less approximations since
everything is done in floating point
26/06/2006
NextGen Correlator Conference
Why software?
Software FPGA ASIC
Development
cost
Low
Med
High
Development
speed
Fast
Med
Slow
Cost per unit
compute power
??
Med
Low
26/06/2006
NextGen Correlator Conference
Software correlating down under
• 2004: Began with Craig West’s masters
thesis, (XF correlator, fringe-finding)
• 2005: MkI FX correlator, limited offering
• 2006: MkII FX correlator (DiFX),
integrated, user-friendly, total switch
from tape-based S2 system (128 Mbps
max) by end of year
26/06/2006
NextGen Correlator Conference
The LBA antennas
26/06/2006
NextGen Correlator Conference
LBA disk system
• MRO PC-EVN based
• Apple Xserve RAID
for raw data storage
(currently procuring
additional disk space)
• Max 1 Gb/s, per station (2 x DAS)
• 1 Gb/s fibre links active to 3 ATNF
antennas
26/06/2006
NextGen Correlator Conference
DiFX (Distributed FX)
DataStream 1
Baseband data
Core 1
DataStream 2
…
Core 2
…
DataStream N
Core M
Timerange, destination
Visibilities
Master Node
26/06/2006
NextGen Correlator Conference
Performance continued
• High spectral resolution (>1024
channels/band) begins to carry a small
performance penalty due to larger FFT
• Other things which can increase
compute load by a factor of up to 2:
 Pulsar binning
 Replacing FFT with polyphase filterbank
 Overlapping FFTs (better for spectral line
observations)
26/06/2006
NextGen Correlator Conference
Present applications
• Masers
• More wide-field VLBI (radio
counterparts in Chandra deep field)
• RRATs (Parkes - Tidbinbilla)
• Southern Hemisphere pulsar parallax
program (double pulsar)
26/06/2006
NextGen Correlator Conference
Future applications
•
•
•
•
NZ interferometer
VLBA
Geodesy (MPIfR, Bonn)
Piggy-back on existing interferometers
for special purpose (eg GMRT)
• Spacecraft tracking??
• Anything you can record and generate
a delay model for
26/06/2006
NextGen Correlator Conference
Thinking outside the square
• Instant feed-back offers whole new
dimension to VLBI!!
• Adaptive scheduling (transients,
variables, scintillating pulsars etc)
• Egalitarian: No need for custom
hardware makes it easier to do parttime interferometry!
26/06/2006
NextGen Correlator Conference
Conclusions
• DiFX is a general purpose software
correlator
• Already used with success in Australia
• Perfect for filling niche/part time
applications with minimal effort
• Will become more streamlined as LBA is
converted to wholly disk + software
26/06/2006
NextGen Correlator Conference
26/06/2006
NextGen Correlator Conference