[Sample Course Title Slide Insert Presentation Title]

Transcript [Sample Course Title Slide Insert Presentation Title]

RAMP Blue Status

Andrew Schultz, John Wawrzynek June 21, 2006 RAMP MIT Summer Workshop

• •

Contributors

Andrew Schultz

Dave Patterson, and the Spring 2006 CS252 (grad computer architecture) class: Mitch Harwell David Tylman Xiaofen Jiang Neelima Balakrishnan Khang Tran Matt Brockmeyer Marghoob Mohiyuddin Jue Sun Zhangxi Tan Wei Xu Gary Voronel Luke Beamer © 2006 Regents University of California. All Rights Reserved

MIT Workshop - RAMP Blue Status 3

•

Outline

Review of project goal and requirements

•

RAMP Blue Architecture

– – – – –

Design principles Processor infrastructure Network interface and on-chip switch Double precision floating point Software support

•

Implementation experience

•

Future work

MIT Workshop - RAMP Blue Status 4

•

Project Goal and Requirements Goal:

1000 node cluster of MicroBlaze cores running uClinux and real MPI benchmarks

•

Requirements:

– – – – Infrastructure to boot uClinux on MicroBlaze cores situated on BEE2 user FPGAs Double precision floating point unit for real MPI benchmarks On-chip switch capable of routing packets between FPGAs on and off module Port of message passing framework (MPI, UPC, etc.) © 2006 Regents University of California. All Rights Reserved

MIT Workshop - RAMP Blue Status 5

• •

BEE2 Module Design

Per-module: – – – 5 Virtex-IIPro70 FPGAs 20GB DRAM 20 10Gbps connections • Supports 10GigE/Infinibnd • • System I/O Inter-mod connections RAMP-blue – maps target MBs to four “user” FPGAs, and hard PowerPC on “control” FPGA as host maintenance processor.

2VP70 FPGA 2VP70 FPGA 2VP70 FPGA 2VP70 FPGA 2VP70 FPGA MIT Workshop - RAMP Blue Status 6

•

Andrew’s Design Principles

KISS: We tried to keep everything simple. Don’t over-engineer the network, FPU, or infrastructure until we have a working design.

• Share the wealth: Resources are tight and MicroBlazes are wimpy. Share infrastructure such as interchip pins, memory controllers, and even FPUs.

• Cut the fat: Wherever possible take care to remove unnecessary logic and interfaces not required by MicroBlaze in this context.

• FSL everywhere: FSL is simply FIFO based communication (very similar to very basic RAMP channel). Ease routing and provide easy migration to RDL © 2006 Regents University of California. All Rights Reserved

MIT Workshop - RAMP Blue Status 7

Processor Interfaces

MIT Workshop - RAMP Blue Status 8

• • •

Console Network

Console network serves several purposes – – – Download application/kernel from control FPGA Provide terminal to booted uClinux Network conduit to route packets from MB to control FPGA (or even off board via 10/100 Ethernet) Simple, general purpose, FSL based network with OPB FIFO attachment at PPC Linux driver for TTY, char device, and Ethernet abstraction

MIT Workshop - RAMP Blue Status 9

MB/MB Network Interface

• • Current network interface is raw FSL connected directly to a on chip switch – – – Interrupt driven, programmed I/O approach Simple Linux driver provides Ethernet interface so applications can utilize network via tradition socket interface Very inefficient, yet very simple for first network implementation Discussion and paper design of second generation network interface – – Direct memory access through direct port to memory controller Possible RDMA support for UPC as well © 2006 Regents University of California. All Rights Reserved

MIT Workshop - RAMP Blue Status 10

On-Chip Switch

• • • • Switch provides drop-free transmission of variable length packets from MB to MB Composed of two units: buffer unit and switch – – Buffer unit provides buffering at each hop and address lookup logic Switch provides cross-bar connectivity between input ports and output ports and arbitration for each port Packets are source routed (currently encapsulated Ethernet packets) CRCs are end-to-end, so end-points must manage retransmits or fail-stop

MIT Workshop - RAMP Blue Status 11

Double Precision FPU

• • • FPU is treated as a co-processor – Investigation into integrating FPU with RF as SP FPU does was too complicated and didn’t facilitate sharing Operands are transferred via FSL in four instructions, and MB blocks for result FPU is highly pipelined so to better utilize it makes sense to share (and saves loads of resources) © 2006 Regents University of California. All Rights Reserved

MIT Workshop - RAMP Blue Status 12

Initial FPU Performance

2D FFT (ffbench) Execution Times MicroBlaze FP emulation

Mitch Harwell & David Tylman

MIT Workshop - RAMP Blue Status 13

Main Memory

• • Clusters of MBs share a single physical DIMM (1GB) Memory is partitioned so each core has its own physical address space

MIT Workshop - RAMP Blue Status 14

•

Other Infrastructure

Bootstrapping: Reduced boot-strap block RAM from four to one and fit simple boot-loader and cache-invalidation code in single, read-only BRAM.

• Peripherals: Remove the OPB bus and port interrupt controller and timer to LMB to save logic. Pending. • Debugging: Using existing opb_mdm core and JTAG we can use existing debugging infrastructure (i.e. XMD/GDB) to debug up to 8 cores. Group of students also worked on ideas for real time instruction tracing.

MIT Workshop - RAMP Blue Status 15

•

Software Support

MBs boots relatively unmodified version of uClinux and runs stably • MPICH2 has been successfully compiled and run on an XUP test system with a pair MicroBlaze cores • UPC has also been built and run on a XUP test system • GCC has been modified to emit instructions that utilize double precision FPU co-processor (when SOFTFPU flag turned on) • Currently finishing up final modifications to first network driver to allow proper source routing of packets between FPGAs and to other BEE2 boards © 2006 Regents University of California. All Rights Reserved

MIT Workshop - RAMP Blue Status 16

RAMP Blue FPGA Floor-plan

XAUI 1 LVCMOS Left LVCMOS Up XAUI 2 MicroBlaze MicroBlaze D I M M Memory Arbiter 4 MicroBlaze MicroBlaze MicroBlaze MicroBlaze Memory Arbiter D I M M MicroBlaze MicroBlaze 1 Local Switch Global Switch Local Switch D I M M MicroBlaze MicroBlaze Memory Arbiter 3 MicroBlaze MicroBlaze XAUI 3 LVCMOS Right MicroBlaze MicroBlaze D I M M Memory Arbiter 2 MicroBlaze MicroBlaze LVCMOS Up XAUI 4

MIT Workshop - RAMP Blue Status 17

Implementation Experience

• •

System with 8 MicroBlaze cores per user FPGA running on the BEE2

– This system has the integrated SP FPU per core, we haven’t yet integrated DP FPU core into this base system, although we expect fewer resources with sharing (each SP FPU is ~1300 slices and the DP FPU is ~2000 slices)

Early attempts to implement a 16 MicroBlaze system have failed in placement, although there are enough raw resources

– We expect that with some simple floor planning we should be able to reach a 16 core system

MIT Workshop - RAMP Blue Status 18

8 MicroBlaze Cores (SP FPU each) 4 Memory Controllers 4 XAUI Controllers

4-LUTs:

40,625 out of 66,176

61% FFs:

27,085 out of 66,176

40% BRAMs and MULTs:

116 out of 328

35%

56 out of 328

17% MIT Workshop - RAMP Blue Status 19

Near-term Work

• • •

Improve density to get 16 core system

– – Analysis of data paths and floor planning should allow us to increase the density of cores since current design does very little deliberate area optimization Integrate shared DP FP core

Convert design to RDL

– – Present design is XPS only (however it is fully parametrized with embedded TCL to allow fast changes to topology) Have version of network switch in RDL, need to wrap the rest in RDL

Improve performance of known bottlenecks

MIT Workshop - RAMP Blue Status 20

Spares

MIT Workshop - RAMP Blue Status 21

Processor Infrastructure

• •

Key components are required for each processor

– – – – – – – MicroBlaze core Console interface Network interface Floating point unit Memory interface Debug port Miscellaneous infrastructure (timer, interrupt controller)

Build one and then replicate, connect with on-chip switch MIT Workshop - RAMP Blue Status 22

Farther Future Work

• •

System level possibilities?

– – – Hardware paging of memory (ala VMWare ESX) to better utilize memory capacity and allow content based sharing Coherent shared memory between MicroBlazes More exploration of tracing and system level debugging

Networking possibilities?

– – Highly FPGA optimized switch More complicated routing mechanisms

MIT Workshop - RAMP Blue Status 23

•

Conclusion

–

Close to functional multi-MB system:

Successfully provided infrastructure to boot multiple MicroBlaze cores on single FPGA with full uClinux support – Determined ease of porting and running MPI and UPC on uClinux – Areas targeted for both performance increase (network interface, FPU integration) and on-chip density

MIT Workshop - RAMP Blue Status 24