EENG 449b/CPSC 439b Computer Systems Lecture 4 Intro to Pipelining 1/25/05 January 25 and 25, 2005 Prof. Andreas Savvides Spring 2005 http://www.eng.yale.edu/courses/2005s/een g449b EENG449b/Savvides Lec 4.1

1/25/05

Announcements

• Reading for this week – Appendix A • Homework #1 (Due Feb 10) – Chapter 2: Problems 2.5, 2.11, 2.12, 2.19 – Appendix A: Problems A.1, A.5, A.6, A.7, A.11 • Next week overview – Complete pipelining discussion – Different processor architectures – Requirements for simulator discussion

EENG449b/Savvides Lec 4.2

1/25/05

Fast, Pipelined Instruction Interpretation

Next Instruction Instruction Address Instruction Fetch Instruction Register Decode & Operand Fetch Operand Registers NI NI IF NI IF D NI IF D E NI IF D E W IF D E W D E W E W W Time Execute Result Registers Store Results Registers or Mem

EENG449b/Savvides Lec 4.3

6 PM 7

Sequential Laundry

8 9

Time

10 11 Midnight 30 40 20 30 40 20 30 40 20 30 40 20

T a s k

A

O r d e r

B C D

• Sequential laundry takes 6 hours for 4 loads • If they learned pipelining, how long would laundry take?

1/25/05 EENG449b/Savvides Lec 4.4

6 PM 7

Pipelined Laundry Start work ASAP

8 9

Time

10 11 Midnight 30 40 40 40 40 20

T a s k

A 1/25/05

O r d e r

B C D

• Pipelined laundry takes 3.5 hours for 4 loads

EENG449b/Savvides Lec 4.5

T a s k O r d e r

1/25/05

Pipelining Lessons

B C D A 6 PM 7 8 9

Time

30 40 40 40 40 20

• Pipelining doesn’t help

latency of single task, it entire workload

• Pipeline rate limited by

slowest pipeline stage

• Multiple

tasks operating simultaneously

• Potential speedup =

Number pipe stages

• Unbalanced lengths of

pipe stages reduces speedup

• Time to “

fill ” pipeline and time to “ drain ” it reduces speedup EENG449b/Savvides Lec 4.6

1/25/05

Instruction Pipelining

• Execute billions of instructions, so

throughput

is what matters

– except when? • What is desirable in instruction sets for pipelining? – Variable length instructions vs.

all instructions same length?

– Memory operands part of any operation vs.

memory operands only in loads or stores?

– Register operand many places in instruction

format vs. registers located in same place?

EENG449b/Savvides Lec 4.7

Requirements for Pipelining

1/25/05 Goal: Start a new instruction at every cycle What are the hardware implications?

• Two different tasks should not attempt to use the same datapath resource on the same clock cycle.

• Instructions should not interfere with each other • Need to have separate data and instruction memories • Need increased memory bandwidth – A 5-stage pipeline operating at the same clock rate as pipelined version requires 5 times the bandwidth • Need to introduce pipeline registers • Register file used in two places in the ID and WB stages – Perform reads in the first half and writes in the second half.

EENG449b/Savvides Lec 4.8

1/25/05

Pipelining Limitations

• The CPU clock cannot run faster than the

time needed for the slowest pipeline stage

• Imbalance of pipelining overhead • Time needed to fill up the pipeline

Pipelining speedup=avg. instruction time w/o pipelining/avg. instruction time w pipelining EENG449b/Savvides Lec 4.9

1/25/05

Recap: 5 Steps of MIPS Datapath

Instruction Fetch

Next PC

Instr. Decode Reg. Fetch Execute Addr. Calc Memory Access Write Back 4

Next SEQ PC RS1 RS2 RD Zero?

L M D Imm

Sign Extend

WB Data

EENG449b/Savvides Lec 4.10

Pipeline Requirements…

Register file Read in the first half, write in the second half cycle Need separate instruction and Data memories: Structural Hazard

1/25/05 EENG449b/Savvides Lec 4.11

1/25/05

Add registers between pipeline stages

• Prevent interference between 2 instructions • Carry data from one stage to the next • Edge triggered

EENG449b/Savvides Lec 4.12

1/25/05

Pipelining Hazards

Hazards: circumstances that would cause incorrect execution if next instruction where launched Structural Hazards: time Data Hazards: instruction still in the pipeline Control Hazards: of instructions and decisions about changes in control flow (branches and jumps) is resolved by inserting one or more pipeline Attempting to use the same Instruction depends on result of prior Caused by delay between the fetching “Stall” “bubbles” in the EENG449b/Savvides Lec 4.13

Data Hazards

1/25/05 Occurs when the relative timing of instructions is altered because of pipelining Consider the following code: DADD R1, R2, R3 DSUB R4, R1, R5 AND R6, R1, R7 OR R8, R1, R9 XOR R10, R1, R11 What is the problem here?

EENG449b/Savvides Lec 4.14

Data Hazard

1/25/05 EENG449b/Savvides Lec 4.15

1/25/05

Data Hazards: Data Forwarding

EENG449b/Savvides Lec 4.16

1/25/05

Data Hazards Requiring Stalls

LD R1,0(R2) DSUB R4,R1,R5 AND R6,R1,R7 OR R8,R1,R9

HAVE to stall for 1 cycle…

EENG449b/Savvides Lec 4.17

1/25/05

Implementing Stalls for Data Hazard

• Pipeline Interlock – Hardware: detects a hazard and stall the pipeline until

the hazard is detected

• Interlock function – Stall the pipeline at the instruction that wants to use

the result of a hazard

– Stall time: from beginning of instruction until the end

of the stall EENG449b/Savvides Lec 4.18

1/25/05

Stall Introduces Bubbles in the Pipeline

EENG449b/Savvides Lec 4.19

1/25/05

Branch Hazards

• A branch hazard is a

control hazard

• Branch execution – PC=PC+4 (branch not taken) or Something else (branch

taken)

• Simple solution – Redo an instruction fetch – BUT even a 1 cycle stall may cause 10 – 30% program

execution overhead

• Some branch prediction can easily help with

performance EENG449b/Savvides Lec 4.20

1/25/05

Four Branch Hazard Alternatives

#1: Stall until branch direction is clear #2: Predict Branch Not Taken

– Execute successor instructions in sequence – “Squash” instructions in pipeline if branch actually taken – Advantage of late pipeline state update – 47% MIPS branches not taken on average – PC+4 already calculated, so use it to get next instruction

#3: Predict Branch Taken

– 53% MIPS branches taken on average – But haven’t calculated branch target address in MIPS » MIPS still incurs 1 cycle branch penalty » Other machines: branch target known before outcome

EENG449b/Savvides Lec 4.21

Four Branch Hazard Alternatives

1/25/05 #4: Delayed Branch

– Define branch to take place

AFTER a following instruction branch instruction sequential successor 1 sequential successor 2 ........

sequential successor n ........

branch target if taken Branch delay of length n

– 1 slot delay allows proper decision and branch target

address in 5 stage pipeline

– MIPS uses this

EENG449b/Savvides Lec 4.22

1/25/05

Delayed Branch

• Where to get instructions to fill branch delay slot? – Before branch instruction – From the target address: only valuable when branch taken – From fall through: only valuable when branch not taken – Canceling branches allow more slots to be filled • Compiler effectiveness for single branch delay slot: – Fills about 60% of branch delay slots – About 80% of instructions executed in branch delay slots useful in

computation

– About 50% (60% x 80%) of slots usefully filled • Delayed Branch downside: 7-8 stage pipelines, multiple

instructions issued per clock (superscalar) EENG449b/Savvides Lec 4.23

Independent instruction

Scheduling Branch Delay

Cannot be used 1/25/05 Preferred when branch taken w/ high prob EENG449b/Savvides Lec 4.24

Pipelining Performance Issues

1/25/05 Consider an unpipelined processor 1ns/instruction cycle 4 cycles for ALU operations 40% 4 cycles for branches 5 cycles for memory operations 40% Pipelining overhead 0.2ns

For the unpipelined processor Frequency



Clock 20% cycle



average CPI



1 ns



((40%



20%) x 4



40%



5



4.4

ns EENG449b/Savvides Lec 4.25

1/25/05

Speedup from Pipelining

Now if we had a pipelined processor, we assume that each instruction takes 1 cycle BUT we also have overhead so instructions take 1ns + 0.2 ns = 1.2ns

Speedup from pipelining



Average instructio Average n time unpipeline instructio n time pipelined d



4.4

ns 1.2

ns



3.7 times EENG449b/Savvides Lec 4.26

Considering the stall overhead

1/25/05 Speedup from pipelining



Average instructio Average n instructio time n unpipeline time pipelined d



CPI unpipeline d



Clock cycle unpiplined CPI pipelined



Clock cycle pipelined CPI pipelined



Ideal CPI



Average Stall cycles per Inst Speedup



CPI unpipeline d 1



Pipeline stall cycles per instructio n Speedup



1 Pipeline depth



Pipeline stall CPI



Cycle Time unpipeline Cycle Time pipelined d EENG449b/Savvides Lec 4.27

1/25/05

Performance of Branch Schemes

Assuming an ideal CPI of 1: Pipeline speedup

 1 

Pipeline depth Pipeline stall cycles from branches Pipeline stall cycles from branches



Branch frequency



Branch penalty Pipeline speedup



1



Pipeline Branch frequency depth



Branch penalty EENG449b/Savvides Lec 4.28

1/25/05

Instruction Formats Review

EENG449b/Savvides Lec 4.29

Implementing a MIPS Pipeline

1/25/05 We are developing a subset of the MIPS pipeline supporting

– Load store word – Branch equal zero – Integer ALU Operations • Remember MIPS has

register-register ALU instructions (e.g Add R1, R2, R3)

• Attention: In the homework you will have to

redesign the pipeline for R1,R2,(R3)!!!

register-memory instructions for ALU operations (e.g Add EENG449b/Savvides Lec 4.30

1/25/05

IR  NPC Mem[PC];  PC  4;

MIPS Datapath Review

EENG449b/Savvides Lec 4.31

MIPS Datapath Review

1/25/05 A B



Regs[rs];



Regs[rt]; Imm



sing extended immediate field of IR; EENG449b/Savvides Lec 4.32

MIPS Datapath Review

1/25/05 Memory Ref : ALUOutput



A



Imm; or Reg Reg ALU : ALUOutput



A func B; or Reg Imm : ALUOutput



A op Imm; or Branch : ALUOutput



NPC



(Imm



2) Cond



(A



0) EENG449b/Savvides Lec 4.33

MIPS Datapath Review

1/25/05 Mem ref : LMD



Mem[ALUOut put]; Mem[ALUOut put]



B; or Branch : if(cond) PC



ALUOutput EENG449b/Savvides Lec 4.34

1/25/05

MIPS Basic Pipeline

Depend on instruction type Load or ALU operation ALUOut LMD

Data needs to be written in the registers at the end of each cycle

EENG449b/Savvides Lec 4.35

1/25/05

Events at every pipe stage

EENG449b/Savvides Lec 4.36

1/25/05

Events at every pipe stage

EENG449b/Savvides Lec 4.37

1/25/05

Controlling the Pipeline

EENG449b/Savvides Lec 4.38

1/25/05

Hazards Review

From previous lecture we know the situations that would cause incorrect execution

• Structural Hazards - • Data Hazards - • Control Hazards -

EENG449b/Savvides Lec 4.39

1/25/05

Three Generic Data Hazards

• Read After Write (RAW)

Instr J tries to read operand before Instr I writes it I: add r1 ,r2,r3 J: sub r4, r1 ,r3

• Caused by a “Data

Dependence need for communication.

” (in compiler nomenclature). This hazard results from an actual EENG449b/Savvides Lec 4.40

Three Generic Data Hazards

• Write After Read (WAR)

Instr J writes operand

before

Instr I reads it I: sub r4, r1 ,r3 J: add r1 ,r2,r3 K: mul r6,r1,r7

• Called an “

anti-dependence ” by compiler writers.

This results from reuse of the name “ r1 ”.

1/25/05

• Can’t happen in MIPS 5 stage pipeline because: – All instructions take 5 stages, and – Reads are always in stage 2, and – Writes are always in stage 5

EENG449b/Savvides Lec 4.41

Three Generic Data Hazards

• Write After Write (WAW)

Instr J writes operand

before

Instr I writes it.

I: sub r1 ,r4,r3 J: add r1 ,r2,r3 K: mul r6,r1,r7

• Called an “

output dependence ” by compiler writers

• Can’t happen in MIPS 5 stage pipeline because: – All instructions take 5 stages, and – Writes are always in stage 5

r1 ”.

• Will see WAR and WAW in later more complicated pipes

1/25/05 EENG449b/Savvides Lec 4.42

IF

MIPS Basic Pipeline

ID EX IF WB 1/25/05

Data Hazards can be detected here Instruction issued

EENG449b/Savvides Lec 4.43

1/25/05

Hardware Hazard Detection

EENG449b/Savvides Lec 4.44

1/25/05

Logic to Detect Load Interlocks

EENG449b/Savvides Lec 4.45

1/25/05

Forwarding of Results to the ALU

Mem output ALU output EENG449b/Savvides Lec 4.46

Control Hazards Revisited

A branch causes a 3-cycle stall in the 5-stage pipeline Branch Instruction IF ID EX MEM WB Branch Successor+1 IF stall stall IF ID EX MEM WB Branch Successor+2 IF ID EX MEM WB Branch Successor+3 IF ID EX MEM WB Higher overhead than data hazards… Can HW changes improve that? YES!

• Try to make an early decision whether a branch is taken or not.

1/25/05 EENG449b/Savvides Lec 4.47

Improved Pipeline – Dealing with Branches

Write the PC faster Additional adder in ID stage 1/25/05 Can detect branch hazard 2 cycles earlier EENG449b/Savvides Lec 4.48

Improved Pipeline – Dealing with Branches

Write the PC faster Additional adder in ID stage Note change of order in text!

Figure A.11 says a branch hazard would stall for 1 cycle. This is after the optimization in Figure A.24!!!

Note the change of order… 1/25/05 EENG449b/Savvides Lec 4.49

1/25/05

Challenges in Pipeline Implementation

Exceptions:

• Breakpoint

Situations that can disrupt the in-order execution of instructions (interrupt, fault, exception)

• I/O device request • Invoking an OS service from a user program • Integer arithmetic overflow or FP arithmetic

anomaly

• Page fault (not in main memory) • Misaligned memory access • Hardware failure • Power loss etc…

EENG449b/Savvides Lec 4.50

1/25/05

Exceptions Requirements

• Synchronous vs. Asynchronous – Synchronous: Always at the same memory and time – What is asynchronous? • User requested vs. coerced – User requested – not really exceptions – Coerced: not under the control of user program • User maskable vs. user non-maskable • With vs. between instructions • Resume vs. terminate

EENG449b/Savvides Lec 4.51

1/25/05

Exception Types & Classification

EENG449b/Savvides Lec 4.52

1/25/05

Exception Challenges

• Exceptions happening within instructions • Exceptions that need to be restarted – as in

the case of a page fault

• Need to make sure that the occurrence of

an exception does not influence the correct execution of a program

• Pipeline design should ensure that the

processor is in a predictable state after the occurrence of an exception

– Block writing to pipeline registers

EENG449b/Savvides Lec 4.53

Exceptions in MIPS Pipeline

Pipeline State Problem Exceptions IF ID EX MEM 1/25/05 WB Page fault on instruction fetch misaligned memory access memory protection violation Undefined or illegal opcode Arithmetic exception Page fault on data fetch; misaligned memory access; memory protection violation None EENG449b/Savvides Lec 4.54

Floating-Point Support in Pipelines

1/25/05

• Floating point operations will take more than

1 or 2 cycles to complete

– Structural hazards – Data hazards • Multiple functional units required – Loads, stores and integer ALUs – FP and integer multiplier – FP adder that handles FP add, subtract and conversion – FP and integer divider • Initiation interval – number of cycles that

must elapse before issuing two operations of EENG449b/Savvides Lec 4.55

1/25/05

Multiple FUs and Latencies

Functional Unit Late ncy Initiation Interval Integer ALU Data memory (integer and FP Loads) FP add FP Multiply FP Divide 0 1 3 6 24 1 1 1 1 25 EENG449b/Savvides Lec 4.56

Support for Multiple Outstanding Operations

Additional pipeline registers needed 1/25/05 EENG449b/Savvides Lec 4.57

Hazards in Longer Pipelines

1/25/05 1. Divide unit is not fully pipelined - structural hazards can occur 2. Instructions have varying running times so the number of register writes required in a cycle can be larger than 1.

3. WAW hazards are possible, since instructions don’t reach WB in order 4. Instructions can complete in different order than the one they were issued causing problems with exceptions 5. Because of longer latency of operations, stalls for RAW hazards will be more frequent EENG449b/Savvides Lec 4.58

Figure A.34

FP Pipeline Hazards Example

Simultaneous writeback 1/25/05

•

Stall an instruction in the ID stage

• Stall the instruction when it tries to enter WB

EENG449b/Savvides Lec 4.59

Checks for Detecting Hazards

1/25/05 Three checks to be performed before a multicycle instruction can issue in the ID stage:

• Check for structural hazards – A structural unit is not busy and a write register port

is available when needed

• Check for a RAW data hazard – Wait until the source registers are not listed as

pending destinations

• Check for WAW data hazard – Determine an instruction that already issued has the

same destination as this instruction. If so stall the instruction issue in ID.

EENG449b/Savvides Lec 4.60

1/25/05

MIPS R4000 Pipeline

• Decompose the 5-stage pipeline to a deeper

8-stage pipeline(superpipeline)

– achieve higher clock rates => better performance • Extra stages come from decomposing

memory accesses

• Longer pipelines increase the amount of

forwarding and branch delays EENG449b/Savvides Lec 4.61

Branch Delay Cycles

Branch outcome needs 3 cycles 1/25/05 EENG449b/Savvides Lec 4.62

Dynamic Scheduled Pipelines

1/25/05

• Simple pipelines result in hazards that require

stalling.

• Static scheduling

– compilers rearrange instructions to avoid stalls.

• Dynamic scheduling

into stages:

– Issue – Read operands

– processor executes instructions out-of-order to minimize stalls

• Dynamic scheduling requires splitting the ID stage

– Decode instructions, check for structural hazards – Wait until there are no data hazards, then read operands

– Also

need to know when each instruction begins and ends

• Requires a lot more bookkeeping! More when we

discuss Tomasulo’s algorithm in chapter 3… EENG449b/Savvides Lec 4.63

Scoreboarding

1/25/05 Scoreboarding mid 60s.

– a technique that allows out of-order execution when resources are available and there are no data dependencies – originated in CDC6600 in the

• Scoreboard fully responsible

operations execution for instruction execution and hazard detection

– Requires changes in # of functional units and latency of – Needs to keep track of status of all instructions in

EENG449b/Savvides Lec 4.64

1/25/05

Scoreboarding II

EENG449b/Savvides Lec 4.65

More Hazards

1/25/05

• WAR and WAW hazards are now possible!

DIV.D F0, F2, F4 ADD.D F10, F0, F8 SUB.D F8, F8, F14 WAR! If SUB.D

Executes first DIV.D F0, F2, F4 ADD.D F10, F0, F8 SUB.D F10, F8, F14 WAW! If SUB.D

Executes first EENG449b/Savvides Lec 4.66

1/25/05 Refer to figures A.52 – A.54 for example scoreboard tables Scoreboarding is limited by:

• Amount of parallelism among instructions • The number of scoreboard entries • The number and types of functional units • Presence of antidependencies and output

dependencies EENG449b/Savvides Lec 4.67