Transcript SKA Correlator

xNTD/SKAMP/LFD Correlator
4th RadioNet Engineering Forum Workshop
Next Generation Correlators for Radio Astronomy and Geodesy
27-29 June 2006, Groningen, The Netherlands,
John Bunton
CSIRO ICT Centre, Sydney
Common Design
www.ict.csiro.au
 Developing a set of technologies that will be used in
xNTD, SKAMP3 and LFD
 Common correlator, VHDL code and hardware elements
 This talk mainly the correlator and xNTD beamformer
 Team (SKAMP3, xNTD and LFD)
 Uni Sydney/ ATNF - Ludi de Souza,
 Uni Sydney Duncan Campbell-Wilson, John Russel, Chris
Weimann, Adrian Blake
 ATNF/ ICT Centre John Bunton, Jaysri Joseph
 MIT Roger Cappallo, Brian Fanous
 ANU Frank Briggs
 and growing
New Generation Correlators
Groningen 26-29 June 2006
xNTD specification
www.ict.csiro.au
 Connected element telescope with 20-30 antennas
within ~10km area
 Each antenna generates 32 dual pol. beams giving a
field of view of ~30 square degrees – frequency
independent
 Dynamic range ~50dB (8-bit A/D) aided by low RFI
environment at Mileura WA, >60dB filterbanks
 Frequency range below 1.7 GHz with 300 MHz BW
New Generation Correlators
Groningen 26-29 June 2006
xNTD Correlator
www.ict.csiro.au
 FX correlator, FPGA based using polyphase filterbank
 Large design with 20-40 antennas x ~32 beams
 Input to beamformers 1.5 THz, to correlator 500 GHz (16bits/Hz)
 6,000-15,000 baselines, each full Stokes
 ~1000 Tops/sec (mainly beamformer/filterbanks)
 Output data rate 1-2 Gcorrelation/visability set
 Full 300 MHz processed to full frequency resolution
 ~50,000 channel – 6 kHz resolution (no options)
 BUT
 Low resolution and continuum by averaging channels
 Daisy chain data for special processing
 Later can trade bandwidth for
higher resolution or time sampling
New Generation Correlators
Groningen 26-29 June 2006
Design
www.ict.csiro.au
 Beamformers and filterbank
 ~200 inputs generating 32 dual pol. beams
 Signal transport from feeds optical analogue (TBD)
 Beamformer location TBD
 Design similar to SKAMP3 filterbank (Ludi’s talk)
 Correlator based on correlation cell
 Highly configurable as to number inputs
 If the beamformer is at the antenna crossconnect
in fibre network and O/E interface
 Needs data re-ordering (Ludi’s talk)
New Generation Correlators
Groningen 26-29 June 2006
xNTD Beamformer
www.ict.csiro.au
 Beamformer input
 96 Dual polarisation inputs
 300 MHz bandwidth
 About 1 Terabit/sec input data rate per antenna
 Each of 32 output beams is the sum of 20 to 96 inputs
depending on frequency
 Would like to have data from all inputs to each unit that generates
a beam
 Instead “Divide and Conquer”
 Use the same topology as SKAMP3 filterbanks (Ludi’s
talk)
New Generation Correlators
Groningen 26-29 June 2006
Filterbank to Beamformer
www.ict.csiro.au
SKAMP Filterbanks (one card cage)
~200 MS/s Real
24 channel
OS Polyphase
12x10 MHz
complex
192 inputs
~200 MS/s Real
24 channel
OS Polyphase
12x10 MHz
complex
Cross
Connect
Outputs
10 MHz
for all
signals to
each fine
filterbank
192 inputs
10MHz band
2k Channel
Polyphase
192 feeds
4MHz 800 channels
Dual 6.1 Gbits/s
12 sets
192 inputs
10MHz band
2k Channel
Polyphase
xNTD Beamformer
~660 MS/s Real
Inputs from 96
dual polarisation
feeds
(192 signals)
32 channel
OS Polyphase
Output 16x23MHz
complex
32 channel
~660 MS/s Real
OS Polyphase
Output 16x23MHz
complex
Cross
Connect
Outputs
23MHz
for all
signals to
each BF
Beamformer
33x2 beam
23 MHz
Beamformer
33x2 beam
23 MHz
20.6MHz
4k Channel
Polyphase
20.6MHz
4k Channel
Polyphase
32x2 beams
309MHz
309 MS/s
(4R,4I) complex
163 Gbits/s
to central
correlator
New Generation Correlators
Groningen 26-29 June 2006
Correlator
www.ict.csiro.au
 The challenge was to develop a correlator concept the
could be used in xNTD, SKAMP3 and LFD - 20-30, 192,
and 512 antennas
 Systolic array too inflexible
 xNTD 760 correlations/beam
 LFD 500,000 correlations
 Approach developed – Correlation Cell




Combination of multiply-accumulate and storage
Each cell handles 256 correlations at a time
49,000 correlations per FPGA simultaneously
512 time sample short integration on chip
New Generation Correlators
Groningen 26-29 June 2006
Correlation Cell
 Input 16+ pairs of data
 4bit complex multiply in 18-bit
multiplier
 Accumulation to block RAM
 Calculate 256 correlation,
512 successive time samples
 Data reordering in filterbank
 30-70 MHz BW per FPGA
 All baselines LFD 12,
SKAMP3 1.5 FPGAs
Serial input data
8-bit shift register
8-bit shift register
Data loading
Dual Port
Memory
Two
4+4 bit outputs
Dual Port
Memory
MUX
MUX
Latch
Latch
Complex multiplier
Resources one 18-bit multipliers
Dual 4+4 bit inputs - 9+9 bit output
18+18 bit outputs
+
Single Dual Port Memory 2 x 256 36-bit words
36-bit shift register
 xNTD 4-7 cells for all
correlations
www.ict.csiro.au
Serial output data
 1.2-1.5 MHz of bandwidth
New Generation Correlators
Groningen 26-29 June 2006
22 antenna xNTD configuration
 Store 22 or 22 pairs per
correlation cells
Correlation
Cell
Correlations
within a
group of 22
www.ict.csiro.au
Correlation
Cell
Correlations
within a
group of 22
First group of 22 inputs
 Within group 231 correlation
 Between groups 484 correlations
 Need two cells
Second group of 22 inputs
Correlation
Cell
Half the
correlations
between two
groups of 22
Correlation
Cell
Half the
correlations
between two
groups of 22
 48 sets of 22-antenna correlators per FPGA (Virtex4 SX35)
 Clock cycles 242 for 44 inputs = 5.5 clocks per input
 Input data rate 48/5.5 = 8.7 bytes/clock
 As systolic array 192 multipliers gives 13x13 array
 Input data rate 3 times higher at 26 bytes/clock
New Generation Correlators
Groningen 26-29 June 2006
Board Manufacture Simplification (1)
www.ict.csiro.au
 Manufacture of correlator board a major task
 Examples SKAMP1, EVLA …
 Correlation cell reduces input data rate into correlation
chip
 For xNTD correlation cell data rate 3 times less compared
to systolic array
 For individual correlation cell 2 sets of 16 inputs requires
256 clock cycles to process.
 Data rate reduction up to a factor of 16
 This value approached for SKAMP3 and LFD
 For xNTD need ~5 byte wide busses into each correlator
FPGA. (input data rate higher than correlator clock)
New Generation Correlators
Groningen 26-29 June 2006
Board Manufacture Simplification (2)
www.ict.csiro.au
 Correlation cell also reduces data duplication
 SKAMP1 4x4 systolic array, EVLA 8x8 systolic array
 Data duplication 8 in EVLA, higher in SKAMP1 due to array reuse
 Each Correlation Cell process 256 correlations at once
 Can reduce size of systolic array by sqrt(256)=16
 No data duplication on board for up to 150 antennas
 Data duplication none for xNTD, 1.5 for SKAMP3, and 3.5
for LFD, LFD 12 FPGAs data input 1/sqrt(12) of total
 Correlation cell leads to a large simplification of correlator
board
New Generation Correlators
Groningen 26-29 June 2006
Putting it Together – The SKAMP protoype
Routing
Correlations
www.ict.csiro.au
Correlator interface
Long Term Accumulations
New Generation Correlators
Autocorrelations
Groningen 26-29 June 2006
Input, Daisy Chain, Route, & Autocorrelate
www.ict.csiro.au
 Two FX20
 Interconnection for high
antenna number designs
 Input 16 rocket I/O on
unidirectional Infinband
 Output 16 Rocket I/O
unidirectional Infiniband
 Can daisy chain modules
for reuse of data in
further processing
modules or conform to
standards, Infiniband …
Two FX20
New Generation Correlators
Groningen 26-29 June 2006
Compute Engine
www.ict.csiro.au
 Six SX35 FPGAs
 Input 24, Output 18 LVDS
per FPGA
 1152 Correlation cells total
 Up to 294k correlations on
board at a time (256 per cell)
 Data re-ordering in filterbank
to achieve
 Process of 512 time values for
each frequency channel
 Then dump to LTA
New Generation Correlators
Groningen 26-29 June 2006
Long Term Accumulator
www.ict.csiro.au
 Number of Correlation require DRAM for storage
 Data rate requires two DIMM modules for three SX35
Three SX35
Two LX15 + DDR2 DIMM
Half of Correlation and LTA hardware
New Generation Correlators
Groningen 26-29 June 2006
Estimated Performance
www.ict.csiro.au
 Correlator board clock rate 330MHz, 192
cells/FPGA, 6 FPGAs
 Board processing rate 400GCMACs/s = 2.8Tops/s
 Power consumption ~100W
 Power efficiency 0.25W/GCMAC (4bit FX)
 Filterbank board 3.2Gsample/s, two polyphase
filterbanks ~32 operations per sample
 Board processing rate 100Gops/s (18 bit)
 Power consumption ~60W
 Power efficiency 0.6W/Gop
New Generation Correlators
Groningen 26-29 June 2006
Conclusion
www.ict.csiro.au
 Common hardware, hardware modules and VHDL
for xNTD, SKAMP3 and LFD
 SKAMP3 in the lead with filterbank and correlator
hardware well on the way
 Initial Manufacture this year
 Correlator common to all using correlation cell to
gain required flexibility
 xNTD beamformer hardware starts July 06
 LFD receiver will piggy back on this
 Developing international project - distributed
design team
New Generation Correlators
Groningen 26-29 June 2006