Distributed FX software correlation

Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF)

Outline

• Software vs hardware correlators • Why software (what it does best, and the price you pay) • Current applications • The DiFX architecture • Performance and current status • Conclusions 28/09/2006 Torun, Poland

Software vs Hardware

• “Software correlator”: program running on a supercomputer/cluster • Software is unclocked, could be faster or slower than real-time • No channel/integration time restrictions • Floating pt vs int calculations 28/09/2006 Torun, Poland

Why software

• Flexibility - can do science that is impossible with hardware correlators • Rapid (and cheap) development • Add-ons MUCH easier in software • Compatibility and expandability • In general, less approximations due to the use of floating point 28/09/2006 Torun, Poland

The price you pay

• Because software & hardware not specifically tuned, less computation per $$ hardware • Therefore not useful for EVLA/SKA scale correlators • Running costs (electricity, aircon) may be higher 28/09/2006 Torun, Poland

Current applications/results

• New science the software correlator has enabled: – Disk-based LBA (greater bandwidth) – Adding global disk-based antennas to LBA experiments – High time and frequency resolution allowing wide-field imaging – VERY high frequency resolution for pulsar scintillation studies 28/09/2006 Torun, Poland

The DiFX architecture

DataStream 1 DataStream 2 Source data … DataStream N Timerange, destination Baseband data Core 1 Core 2 … Core M Visibilities Master Node MPI is used for inter-process communications 28/09/2006 Torun, Poland

Usage in the LBA

• PC-EVN recorders give up to 512 Mbps per DAS (x2 at ATNF antennas) • Data is recorded directly to Linux formatted storage (Apple Xraid) • Disks shipped to Swinburne • Correlated at the Swinburne supercomputer (~300 P4 machines ) • Ftp fringe tests and 256 Mbps eVLBI 28/09/2006 Torun, Poland

Performance

Cores required for real-time, 256 Mbps, 256 channels, 1 second integrations

300 250 200 150 100 50 0 0 5 10

Antennas

15 20 Real-time LBA @ 1 Gbps (6 stations x 4x64 MHz bands): 100-200 CPUs (circa 2004 P4s) 28/09/2006 Torun, Poland

Current status

• Verification via correlator comparison with VLBA completed • Used in production capacity with LBA - complete switch as soon as sufficient disks available • Further development (graphical frontend, FITS-IDI etc) ongoing • Exploring potential for geodesy 28/09/2006 Torun, Poland

Conclusions

• Software correlators can be rapidly developed and enable science that cannot be done elsewhere • Perfect for niche/part-time experiments and feasibility tests • Used successfully with the LBA • Early science results encouraging!

28/09/2006 Torun, Poland

28/09/2006 Torun, Poland