Transcript Why Parallel Computer Architecture

Convergence of Parallel Architectures
History
Historically, parallel architectures tied to programming models
• Divergent architectures, with no predictable pattern of growth.
Application Software
Systolic
Arrays
System
Software
Architecture
SIMD
Message Passing
Dataflow
Shared Memory
• Uncertainty of direction paralyzed parallel software development!
2
Parallel Architecture

Extension of “computer architecture” to support
communication and cooperation



Architecture defines



OLD: Instruction Set Architecture
NEW: Instruction Set Architecture + Communication
Architecture (Comm. And Sync.)
Critical abstractions, boundaries, and primitives (interfaces)
Organizational structures that implement interfaces (hw or sw)
Layers of abstraction in parallel architecture
3
Layers of abstraction in parallel architecture
CAD
Database
Multiprogramming
Shared
address
Scientific modeling
Message
passing
Data
parallel
Compilation
or library
Operating systems support
Communication hardware
Parallel applications
Programming models
Communication abstraction
User/system boundary
Hardware/software boundary
Physical communication medium

Compilers, libraries and OS are important bridges today
4
Programming Model

Conceptualization of the machine that programmer uses in
coding applications


Specifies communication and synchronization
Examples:

Multiprogramming:


Shared address space:


like bulletin board
Message passing:


no communication or synch. at program level
like letters or phone calls, explicit point to point
Data parallel:


more regimented, global actions on data
Implemented with shared address space or message passing
5
Evolution of Architectural Models

Historically machines tailored to programming models





Prog. model, comm. abstraction, and machine organization lumped
together as the “architecture”
Shared Address Space
Message Passing
Data Parallel
Others:


Dataflow
Systolic Arrays
6
Shared Address Space Architectures

Any processor can directly reference any memory location


Convenient:



Location transparency
Similar programming model to time-sharing on uniprocessors
Naturally provided on wide range of platforms



Communication occurs implicitly as result of loads and stores
History dates at least to precursors of mainframes in early 60s
Wide range of scale: few to hundreds of processors
Popularly known as shared memory machines or model
7
Shared Address Space Model


Process: virtual address space plus one or more threads of control
Portions of address spaces of processes are shared
Virtual address spaces for a
collection of processes communicating
via shared addresses
Load
P1
Machine physical address space
Pn pr i v at e
Pn
P2
Common physical
addresses
P0
St or e
Shared portion
of address space
Private portion
of address space
P2 pr i vat e
P1 pr i vat e
P0 pr i vat e
•
Writes to shared address visible to other threads (in other processes too)
• Natural extension of uniprocessors model: conventional memory operations
for comm.; special atomic operations for synchronization
•
OS uses shared memory to coordinate processes
8
Shared Memory Multiprocessor


Also natural extension of uniprocessor
Already have processor, one or more memory modules and I/O controllers
connected by hardware interconnect of some sort
I/O
devices
Mem
Mem
Mem
Interconnect
Processor
Mem
I/O ctrl
I/O ctrl
Interconnect
Processor
Memory capacity increased by adding modules, I/O by controllers
• Add processors for processing!
• For higher-throughput multiprogramming, or parallel programs
9
Historical Development

“Mainframe” approach



Motivated by multiprogramming
Extends crossbar used for mem bw and I/O
Originally processor cost limited to small




later, cost of crossbar
P
P
I/ O
C
I/ O
C
M
Bandwidth scales with p
High incremental cost; use multistage instead
M
M
M
“Minicomputer” approach





Almost all microprocessor systems have bus
Motivated by multiprogramming, TP
Used heavily for parallel computing
Called symmetric multiprocessor (SMP)
Bus is bandwidth bottleneck


caching is key: coherence problem
Low incremental cost
I/ O
I/ O
C
C
M
M
$
$
P
P
10
Example: Intel Pentium Pro Quad
CPU
P-Pro
module
P-Pro
module
PCI
I/O
cards
PCI
bridge
PCI
bridge
Memory
controller
PCI bus
P-Pro bus (64-bit data, 36-bit addr ess, 66 MHz)
PCI bus
P4
400MHz
P-Pro
module
256-KB
Interrupt
L2 $
controller
Bus interface
MIU
1-, 2-, or 4-way
interleaved
DRAM

All coherence and
multiprocessing glue in
processor module

Highly integrated, targeted at
high volume

Low latency and bandwidth
11
Example: SUN Enterprise
P
$
P
$
$2
$2
CPU/mem
cards
Mem ctrl
Bus interface/switch
Gigaplane bus (256 data, 41 addr ess, 83 MHz)
2.5GB/s
I/O cards



2 FiberChannel
SBUS
SBUS
SBUS
100bT, SCSI
Bus interface
16 cards of either type: processors + memory, or I/O
All memory accessed over bus, so symmetric
Higher bandwidth, higher latency bus
12
Scaling Up
M
M

M
Net work
Net work
$
$
 $
M $
M $
P
P
P
P
P
“Dance hall”


P
Distributed memory
latencies to memory uniform, but uniformly large
Distributed memory or non-uniform memory access (NUMA)


M $
Problem is interconnect: cost (crossbar) or bandwidth (bus)
Dance-hall: bandwidth still scalable, but lower cost than crossbar



Construct shared address space out of simple message transactions across a
general-purpose network (e.g. read-request, read-response)
Caching shared (particularly nonlocal) data?
13
Example: Cray T3E
External I/O
P
$
Mem
Mem
ctrl
and NI
XY
Switch
Z



Scale up to 1024 processors, 480MB/s links
Memory controller generates comm. request for nonlocal references
No hardware mechanism for coherence

SGI Origin etc. provide this
14
Message-Passing Abstraction
Match
ReceiveY, P, t
AddressY
Send X, Q, t
AddressX


Local process
address space
Local process
address space
ProcessP
Process Q
Send specifies buffer to be transmitted and receiving process
Recv specifies sending process and application storage to receive into





Memory to memory copy, but need to name processes
Optional tag on send and matching rule on receive
User process names local data and entities in process/tag space
In simplest form, the send/recv match achieves pairwise synch event
Many overheads: copying, buffer management, protection
15
Message Passing Architectures

Complete computer as building block, including I/O


Programming model:



directly access only private address space (local memory)
communications via explicit messages (send/receive)
High-level block diagram similar to distributed-memory SAS




Communication via explicit I/O operations
Communication integrated at IO level, needn’t be into memory
system
Like networks of workstations (clusters), but tighter integration
Easier to build than scalable SAS
Programming model more removed from basic hardware
operations

Library or OS intervention
16
Evolution of Message-Passing Machines

Early machines: FIFO on each link



HW close to prog. Model;
synchronous ops
topology central (hypercube algorithms)
101
001
100
000
111
011
110
010
CalTech Cosmic Cube (Seitz, CACM Jan 95)
17
Diminishing Role of Topology

Shift to general links

DMA, enabling non-blocking ops



Buffered by system at destination until recv
Store&forward routing
Diminishing role of topology


Any-to-any pipelined routing
node-network interface dominates
communication time
Intel iPSC/1 -> iPSC/2 -> iPSC/860
H x (T0 + n/B)
vs
T0 + HD + n/B


Simplifies programming
Allows richer design space

grids vs hypercubes
18
Example Intel Paragon
i860
i860
L1 $
L1 $
Intel
Paragon
node
Memory bus (64-bit, 50 MHz)
Mem
ctrl
DMA
Driver
Sandia’ s Intel Paragon XP/S-based Super computer
2D grid network
with processing node
attached to every switch
NI
4-way
interleaved
DRAM
8 bits,
175 MHz,
bidirectional
19
Building on the mainstream: IBM SP-2

Made out of
essentially
complete RS6000
workstations
Network interface
integrated in I/O
bus (bw limited by
I/O bus)
Power 2
CPU
IBM SP-2 node
L2 $
Memory bus
General interconnection
network formed fr om
8-port switches
4-way
interleaved
DRAM
Memory
controller
MicroChannel bus
NIC
I/O
DMA
i860
NI
DRAM

20
Berkeley NOW


100 Sun Ultra2
workstations
Intelligent network
interface


proc + mem
Myrinet Network


160 MB/s per link
300 ns per hop
21
Toward Architectural Convergence

Evolution and role of software have blurred boundary




Hardware organization converging too



Tighter NI integration even for MP (low-latency, high-bandwidth)
At lower level, even hardware SAS passes hardware messages
Even clusters of workstations/SMPs are parallel systems


Send/recv supported on SAS machines via buffers
Can construct global address space on MP (GA->P|LA)
Page-based (or finer-grained) shared virtual memory
Emergence of fast system area networks (SAN)
Programming models distinct, but organizations converging


Nodes connected by general network and communication assists
Implementations also converging, at least in high-end machines
22
Convergence: Generic Parallel
Architecture
Network

Communication
assist (CA)
Mem
$
P
Systolic
Arrays
Dataflow
Generic
Architecture
SIMD
Message Passing
Shared Memory
23
Data Parallel Systems

Programming model




Operations performed in parallel on each element of data structure
Logically single thread of control, performs sequential or parallel steps
Conceptually, a processor associated with each data element
Architectural model

Array of many simple, cheap processors with little memory each




Processors don’t sequence through instructions
Attached to a control processor that issues instructions
Specialized and general communication, cheap global synchronization
Original motivations


Matches simple differential equation solvers
Centralize high cost of instruction fetch/sequencing
Control
processor
PE
PE

PE
PE
PE

PE


PE
PE


PE
24
Application of Data Parallelism

Example:




Other examples:



Each PE contains an employee record with his/her salary
If salary > 100K then
salary = salary *1.05
else
salary = salary *1.10
Logically, the whole operation is a single step
Some processors enabled for arithmetic operation, others disabled
Finite differences, linear algebra, ...
Document searching, graphics, image processing, ...
Some recent machines:


Thinking Machines CM-1, CM-2 (and CM-5)
Maspar MP-1 and MP-2,
25
Connection Machine
(Tucker, IEEE Computer, Aug. 1988)
26
Evolution and Convergence

Rigid control structure (SIMD in Flynn taxonomy)


SISD = uniprocessor, MIMD = multiprocessor
Popular when cost savings of centralized sequencer
high


60s when CPU was a cabinet
Replaced by vectors in mid-70s




More flexible w.r.t. memory layout and easier to manage
Revived in mid-80s when 32-bit datapath slices just fit on chip
No longer true with modern microprocessors
Structured global address space, implemented with either SAS
or MP
27
Evolution and Convergence

Other reasons for demise


Simple, regular applications have good locality, can do well
anyway
Loss of applicability due to hardwiring data parallelism


MIMD machines as effective for data parallelism and more general
Prog. model converges with SPMD (single program
multiple data)

need for fast global synchronization
28
CM-5

Repackaged
SparcStation



4 per board
Fat-Tree network
Control network
for global
synchronization
29
Dataflow Architectures

Represent computation
as a graph of essential
dependences
1
a = (b +1)  (b  c)
d=ce
f=ad



Logical processor at each
node, activated by
availability of operands
Message (tokens)
carrying tag of next
instruction sent to next
processor
Tag compared with
others in matching store;
match fires execution
b
c
e

+

d

Dataflow graph
a

Network
f
Token
store
Program
store
Waiting
Matching
Instruction
fetch
Execute
Form
token
Network
Token queue
Network
30
Evolution and Convergence

Key characteristics


Problems





Operations have locality across them, useful to group together
Handling complex data structures like arrays
Complexity of matching store and memory units
Expose too much parallelism (?)
Converged to use conventional processors and memory




Ability to name operations, synchronization, dynamic scheduling
Support for large, dynamic set of threads to map to processors
Typically shared address space as well
But separation of progr. model from hardware (like data-parallel)
Lasting contributions:



Integration of communication with thread (handler) generation
Tightly integrated communication and fine-grained synchronization
Remained useful concept for software (compilers etc.)
31
Systolic Architectures

VLSI enables inexpensive special-purpose chips



Represent algorithms directly by chips connected in regular pattern
Replace single processor with array of regular processing elements
Orchestrate data flow for high throughput with less memory access
M
M
PE
PE
PE
PE
Different from pipelining
Nonlinear array structure, multidirection data flow, each PE may have (small)
local instruction and data memory
SIMD? : each PE may do something different
32
Systolic Arrays (contd.)
Example: Systolic array for 1-D convolution
y(i) = w1 ´ x(i) + w2 ´ x(i + 1) + w3 ´ x(i + 2) + w4 ´ x(i + 3)
x8
x7
x6
x5
x4
x3
x2
x1
w4
y3
y2
xin
yin

w1
x
w
xout
xout = x
x = xin
yout = yin + w ´ xin
yout
Enable variety of algorithms on same hardware
But dedicated interconnect channels


w2
Practical realizations (e.g. iWARP) use quite general processors


w3
y1
Data transfer directly from register to register across channel
Specialized, and same problems as SIMD

General purpose systems work well for same algorithms (locality etc.)
33
Convergence: Generic Parallel
Architecture
Network

Communication
assist (CA)
Mem
$
P



Node: processor(s), memory system, plus communication assist
 Network interface and communication controller
Scalable network
Convergence allows lots of innovation, within framework
 Integration of assist with node, what operations, how efficiently...
34