Transcript BEE3 Update - University of California, Berkeley

BEE3 Update
Chuck Thacker
Technical Fellow
Microsoft Research
11 January, 2007
January 2007 RAMP Retreat
Outline
•
•
•
•
What is BEE3?
BEE2-BEE3 Differences
Project participants
Engineering plan, schedule
January 2007 RAMP Retreat
What is BEE3?
• Follow-on to BEE2 (BWRC, 2004)
• Board with several highly-connected FPGAs
• Vehicle for computer architecture research
– Microsoft’s primary interest
• Potential platform for high performance DSP applications
– Astronomers, and perhaps others.
• Allows large scale architectural experiments
– Although perhaps not as large as originally hoped
– And certainly not at the speed of a real implementation
• Can scale smoothly from a single board to 64 boards
(256 FPGAs)
January 2007 RAMP Retreat
BEE2
January 2007 RAMP Retreat
BEE2 – BEE3 Differences
• 4 Xilinx Virtex 5 vs 5 Virtex 2 Pro FPGAs
– We use XC5VLX110T-ff1136
– V2Pro is now obsolete (130nm)
– V5 is a major improvement (65nm)
•
•
•
•
6-input LUT (64 bit DP RAM)
Better Block RAMs
Improved interconnect
Better signal integrity
• 8 Infiniband/CX4 channels vs 18
• 4 x8 PCI Express Low Profile slots
January 2007 RAMP Retreat
BEE3 – BEE2 Differences (2)
• 2 Banks DDR2 x 2 vs 4 Banks DDR2 x 1
– Same capacity (64 GB likely)
– Lower bandwidth
– Mandated by fewer signal pins on V5
• 4 10/100/1000 Ethernet channels
• No SATA
– BEE2 SATA didn’t work anyway 
– iSCSI instead (?)
• No PowerPCs
– This version has not yet been released by Xilinx
January 2007 RAMP Retreat
BEE2 – BEE3 Differences (3)
• Divided the system into two boards, Main and Control
– Main board has FPGAs, all high speed logic
– Control board handles downloading, monitoring
– Simplifies main board engineering – can design control board in parallel
• Smaller main board
– 168 vs 374 in2
– Fewer layers for lower cost
• Much more “PC-like”
• Can use PC power supplies, peripherals
• Several layouts are being considered
– All fit in 2U enclosure
– Much more attention is being given to thermal design
– Must pass UL, FCC
January 2007 RAMP Retreat
BEE3 Main Board
DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3
DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3
133 133
133 133
CX4
QSH-DP040
40x2
CX4
User1
5VLXT
CX4
108
PCI-E
8X
108
40x2
40x2
QSH-DP040
CX4
108
PCI-E
8X
CX4
QSH-DP040
User2
5VLXT
PCI-E
8X
User3
5VLXT
PCI-E
8X
User4
108
5VLXT
CX4
CX4
40x2
CX4
133 133
DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3
133 133
DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3
January 2007 RAMP Retreat
QSH-DP040
Bandwidths (per-FPGA)
• Memory
– 400 MT/s * 8B/T * 2 channels: 6.4GB/s
• Ring
– 400 MT/s * 12 B/T: 4.8 GB/s
• QSH
– 400 MT/s * 10 B/T: 4 GB/s
• Ethernet
– 125 MB/s
• CX4
– 1.25 GB/s * 2 directions * 2 channels: 5GB/s
• PCI Express
– Same as CX4
January 2007 RAMP Retreat
BEE3 Clocking, JTAG
SMA
125
MHz
SMA
200
MHz
333
MHz
100
MHz
125
MHz
156.25
MHz
Sel3,En3
Sel2,En2
Clock
Buf
1:4
Clock
Buf
1:4
Clock
Buf
1:8
Clock
Buf
1:8
GTP 8x
GTP 8x
Gclk
DDR2
GTP 8x
GTP 8x
Gclk
DDR2
GTP 8x
GTP 8x
Gclk
DDR2
Sel0,En0
GTP 8x
GTP 8x
Gclk
DDR2
Sel1,En1
User1
5VLXT
User2
5VLXT
User3
5VLXT
User4
5VLXT
TDO
TDI
JTAG
SelectMAP{Data[15:0], Cclk, RDWR_B, Busy, Prog_B, Init_B, Done} + JTAG{TMS, TCK}
January 2007 RAMP Retreat
PCI-Expres 8x Slot#4
CS_B[3:0]
PCI-Expres 8x Slot#3
PWR_OK
PCI-Expres 8x Slot#2
64pin 0.1" Header Connector
PS_ON#
PCI-Expres 8x Slot#1
5Vsb x4
GND x20
BEE3 Control Board
USB/H
JTAG
USB/D
RJ45
USB
Ctrl
Ethernet
PHY
16x4
Char LCD
PROM
32
FLASH
16
Spartan3
FT256
64pin 0.1" Header Connector
DRAM
LED x4
PushBtn x4
50
MHz
GPIO x40
5Vsb x4
GND x20
January 2007 RAMP Retreat
BEE3 System (v1)
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
PCI-Express 8x
PCI-Express 8x
PCI-Express 8x
PCI-Express 8x
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
FF1738
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
12V ATX
PWR
64pin 0.1" Header Connector
12V
ATX
PWR
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
ATX PWR
2.5V
Control Board
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
QSH-DP-040
QSH-DP-040
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
QSH-DP-040
QSH-DP-040
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
FF1738
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
FF1738
Main cable
harness exits
here
1.0V
1.0V
5VLXT
FF1136
5VLXT
FF1136
GE
Switch
FF1738
1.8V
1.8V
5VLXT
FF1136
5VLXT
FF1136
1.0V
1.0V
S
M
A
S
M
A
Fujitsu 2x2
CX4
Fujitsu 2x2
CX4
RJ45
64pin 0.1" Header Connector
Available for Fans
Power Supply
January 2007 RAMP Retreat
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
5VLXT
FF1136
12V
PWR
24 pin ATX PWR
5VLXT
FF1136
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
Fujitsu 2x2
CX4
Fujitsu 2x2
CX4
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
1.8V
2.5V
1.8V
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
GE
Switch
1.0V
1.0V
PCIe 1x
PCI-Express 8x
QSH-DP-040
QSH-DP-040
QSH-DP-040
QSH-DP-040
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
64-pin 0.1" Header Connector
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
1.0V
1.0V
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
PCI-Express 8x
PCI-Express 8x
PCI-Express 8x
5VLXT
FF1136
5VLXT
FF1136
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
BEE3 System (v2)
I/O Panel
Control Board
January 2007 RAMP Retreat
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
64-pin 0.1" Header Connector
QSH-DP-040
QSH-DP-040
QSH-DP-040
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
5VLXT
FF1136
QSH-DP-040
January 2007 RAMP Retreat
1.0V
1.8V
1.0V
24 pin ATX PWR
1.0V
12V
PWR
1.8V
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
5VLXT
FF1136
PCI-Express 8x
PCI-Express 8x
PCI-Express 8x
PCI-Express 8x
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
5VLXT
FF1136
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM
5VLXT
FF1136
Fujitsu 2x2
CX4
RJ45
RJ45
Fujitsu 2x2
CX4
2.5V
1.0V
BEE3 Main Board (v3)
Remaining Issues
• Precise EATX compatibility, or not?
– Affects layout complexity, thermal design
• Power supply sizing
– We don’t want to leave the overclockers in the lurch
• Standard power supplies (?)
– “2U” supplies aren’t as efficient, have fewer vendors
– Prefer Intel/Google “12V only” supplies (minimum loading issue),
if available in time and at reasonable cost
• PCI Express is nonstandard
– Xilinx hard macro is “device only”, not host
– Need an intrepid graduate student
– Can still use it for additional Infiniband/CX4 channels
January 2007 RAMP Retreat
Project Participants and Roles
• Microsoft Research (Silicon Valley)
– Funds and manages system engineering
• Celestica (Ottawa and elsewhere)
– Does main board engineering, produces final systems
– Microsoft has a very deep relationship with Celestica
• Function Engineering (Palo Alto)
– Does thermal and mechanical engineering
• Xilinx (San Jose)
– Provides FPGAs for academic machines
– Provides FPGA application expertise
• Ramp Group (BWRC)
– Control board, basic software
• Ramp Community
– Uses the systems for research
January 2007 RAMP Retreat
Why is Microsoft interested?
• We believe the overall RAMP effort will have significant impact, and
want to support it in the most effective way we can.
– Simply paying for grad students seems suboptimal
• We observe that universities aren’t very good at this sort of system
engineering and production
– Grad students are great for many things, but doing things like board
layout aren’t among them.
– Requires deep understanding of tools and production processes. Pros
have this.
– We can open doors that academia can’t
– We have experience in managing this sort of program
• We want the systems ourselves
– As infrastructure for our new effort in computer architecture (yes, this is
a recruiting pitch).
• We also want systems to be available to other industrial users
– This might be more difficult if the systems came from academia.
– But we don’t want to be in the hardware business.
January 2007 RAMP Retreat
Plan, schedule
• Generate design spec: 6 weeks
– Scope layout problems and layer count
• Layout and signal integrity: 12 weeks
– Parts procurement proceeds in parallel
– Will probably do 4-5 prototypes.
• Board fab, test and assembly: 3 weeks
• Design verification testing:5 weeks
– This happens at Microsoft or BWRC
• Production can start in Summer ‘07
January 2007 RAMP Retreat
Discussion?
January 2007 RAMP Retreat