Transcript Re-Reference Interval - Computation Structures Group
High Performance Cache Replacement Using Re-Reference Interval Prediction (RRIP)
Aamer Jaleel
, Kevin Theobald, Simon Steely Jr., Joel Emer Intel Corporation, VSSAD
International Symposium on Computer Architecture ( ISCA – 2010 )
Motivation
• • •
Factors making caching important
Increasing ratio of CPU speed to memory speed Multi-core poses challenges on better shared cache management • •
LRU has been the standard replacement policy at LLC
However LRU has problems!
2
Problems with LRU Replacement
LLC size W size Working set larger than the cache causes thrashing miss miss miss miss miss References to non-temporal data (scans) discards frequently referenced working set hit hit W size hit
scan
miss hit
scan
miss hit
scan
miss
Our studies show that scans occur frequently in many commercial workloads 3
Desired Behavior from Cache Replacement
Working set larger than the cache Preserve some of working set in the cache hit hit hit hit hit W size LLC size Recurring scans Preserve frequently referenced working set in the cache hit hit hit
scan
hit hit
scan
hit hit
scan
hit
4
Prior Solutions to Enhance Cache Replacement
Working set larger than the cache Preserve some of working set in the cache Dynamic Insertion Policy (DIP) Thrash-resistance with minimal changes to HW Recurring scans Preserve frequently referenced working set in the cache Least Frequently Used (LFU) addresses scans LFU adds complexity and also performs bad for recency friendly workloads
GOAL: Design a High Performing Scan-Resistant Policy that Requires Minimum Changes to HW 5
Belady’s Optimal (OPT) Replacement Policy
• • Replacement decisions using
perfect knowledge
of future reference order •
Victim Selection Policy:
Replaces block that
will
be re-referenced furthest in future Physical Way # Cache Tag “Time” when block will be referenced next victim block 0 a 4 1 c 2 b 13 11 3 h 5 4 f 3 5 d 6 6 g 9 7 e 1
6
Practical Cache Replacement Policies
• • • Replacement decisions made by
predicting
the future reference order •
Victim Selection Policy:
Replace block
predicted
to be re-referenced furthest in future • Continually
update
predictions on the future reference order Natural update opportunities are on cache fills and cache hits victim block Physical Way # Cache Tag “Predicted Time” when block will be referenced next 0 a 1 c 2 b 3 h 4 f 5 d 6 g 7 e
7
LRU Replacement in Prediction Framework
• • • The “LRU chain” maintains the re-reference prediction • • Head of chain (i.e. MRU position) predicted to be re-referenced soon Tail of chain (i.e. LRU position) predicted to re-referenced far in the future LRU predicts that blocks are re-referenced in reverse order of reference Rename “LRU Chain” to the “
Re-Reference Prediction (RRP) Chain
” • Rename “MRU position” RRP Head and “LRU position” RRP Tail LRU chain position stored with each cache block RRP head MRU position h 0 g 1 f 2 e 3 d 4 c 5 RRP tail LRU position b 6 a 7
8
Practicality of Chain Based Replacement
RRP Head RRP Tail RRPV (n=2): Qualitative Prediction: h g 0 ‘near immediate’ f e d 1 ‘intermediate’ c 2 ‘far’ b a 3 ‘distant’ • • •
Problem:
Chain based replacement is too expensive!
log 2 (associativity) bits required per cache block (16-way requires 4-bits/block) • • •
Solution:
LRU chain positions can be
quantized
into different buckets Each bucket corresponds to a predicted
Re-Reference Interval
Value of bucket is called the
Re-Reference Prediction Value (RRPV)
Hardware Cost: ‘n’ bits per block [ ideally you would like
n < log 2 A
]
9
Representation of Quantized Replacement (n = 2)
RRP Head RRP Tail RRPV: Qualitative Prediction: h g 0 ‘near immediate’ f e d 1 ‘intermediate’ c 2 ‘far’ b a 3 ‘distant’ Physical Way # Cache Tag RRPV 0 a 3 1 c 2 2 b 3 3 h 0 4 f 1 5 d 1 6 g 0 7 e 1
10
Emulating LRU with Quantized Buckets (n=2)
• • • • •
Victim Selection Policy:
Evict block with distant RRPV (i.e. 2 n -1 = ‘3’) If no distant RRPV (i.e. ‘3’) found, increment all RRPVs and repeat the search If multiple found, need tie breaker. Let us always start search from physical way ‘0’
Insertion Policy:
Insert new block with RRPV=‘0’
Update Policy:
Cache hits update the block’s RRPV=‘0’ victim block hit Physical Way # Cache Tag 0 1 c 2 b 3 h 4 f 5 d 6 g 7 e RRPV 2 0 1 1 0 1
But We Want to do BETTER than LRU!!!
11
Re-Reference Interval Prediction (RRIP)
Physical Way # Cache Tag RRPV 0 a 3 1 c 2 2 b 3 3 h 0 4 f 1 5 d 1 6 g 0 7 e 1 • • • Framework enables re-reference predictions to be tuned at insertion/update Unlike LRU, can use non-zero RRPV on insertion Unlike LRU, can use a non-zero RRPV on cache hits • • •
Static Re-Reference Interval Prediction (SRRIP)
Determine best insertion/update prediction using profiling [and apply to all apps] •
Dynamic Re-Reference Interval Prediction (DRRIP)
Dynamically determine best re-reference prediction at insertion
12
Static RRIP Insertion Policy – Learn Block’s Re-reference Interval
• • • •
Key Idea
: Do not give new blocks too much (or too little) time in the cache Predict new cache block will not be re-referenced soon Insert new block with some RRPV other than ‘0’ • Similar to inserting in the “middle” of the RRP chain However it is NOT identical to a fixed insertion position on RRP chain (see paper) Physical Way # Cache Tag RRPV victim block 0 1 c 2 2 b 3 3 h 0 4 f 1 5 d 1 6 g 0 7 e 1
13
Static RRIP Update Policy on Cache Hits
• • • Hit Priority (HP) Like LRU, Always update RRPV=0 on cache hits. Intuition: Predicts that blocks receiving hits after insertion will be re-referenced soon hit Physical Way # Cache Tag RRPV 0 s 2 1 c 2 b 3 3 h 0 4 f 1 5 d 1 6 g 0 7 e 1
An Alternative Update Scheme Also Described in Paper 14
SRRIP Hit Priority Sensitivity to Cache Insertion Prediction at LLC Averaged Across PC Games, Multimedia, Server, and SPEC06 Workloads on 16-way 2MB LLC
10,00 7,50 5,00 2,50 0,00 -2,50 -5,00 -7,50
n=1
-10,00 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Re-Reference Interval Prediction (RRIP) Value At Insertion n=1 is in fact the NRU replacement policy commonly used in commercial processors 15
SRRIP Hit Priority Sensitivity to Cache Insertion Prediction at LLC Averaged Across PC Games, Multimedia, Server, and SPEC06 Workloads on 16-way 2MB LLC
10,00 7,50 5,00 2,50 0,00 -2,50 -5,00 -7,50
n=1 n=2
n=3
n=4
n=5
-10,00 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Re-Reference Interval Prediction (RRIP) Value At Insertion Regardless of ‘n’ Static RRIP Performs Best When RRPV insertion Regardless of ‘n’ Static RRIP Performs Worst When RRPV insertion is 2 n -2 is 2 n -1 16
Why Does RRPV
insertion
W size S len
of 2
n
-2 Work Best for SRRIP?
hit hit hit
scan
?
hit
scan
?
hit
scan
?
• • • • Before scan, re-reference prediction of active working set is ‘0’ • Recall, NRU (n=1) is not scan-resistant For scan resistance RRPV insertion MUST be different from RRPV of working set blocks • Larger insertion RRPV tolerates larger scans Maximum insertion prediction (i.e. 2 n -2) works best!
In general, re-references after scan hit IF S len < ( RRPV insertion – Starting-RRPV workingset ) * (LLC size – W size ) SRRIP is Scan Resistant for S len < ( RRPV insertion ) * (LLC size – W size )
For n > 1 Static RRIP is Scan Resistant! What about Thrash Resistance?
17
DRRIP: Extending Scan-Resistant SRRIP to Be Thrash-Resistant
SRRIP DRRIP miss miss miss hit miss miss hit miss miss hit miss miss hit miss • • Always using same prediction for all insertions will thrashes the cache • Like DIP, need to preserve some fraction of working set in cache Extend DIP to SRRIP to provide thrash resistance • • •
Dynamic Re-Reference Interval Prediction:
Dynamically select between inserting blocks with 2 n -1 and 2 n -2 using Set Dueling Inserting blocks with 2 n -1 is same as “no update on insertion”
DRRIP Provides Both Scan-Resistance and Thrash-Resistance 18
5 0 -5
Performance Comparison of Replacement Policies
20 NRU DIP SRRIP DRRIP
16-way 2MB LLC
15 10 GAMES MULTIMEDIA SERVER SPEC06 ALL
Static RRIP Always Outperforms LRU Replacement Dynamic RRIP Further Improves Performance of Static RRIP 19
1 0 -1 3 2 5 4 8 7 6
Cache Replacement Competition (CRC) Results
NRU 3-bit SRRIP 3-bit DRRIP Dueling Segmented-LRU (CRC winner) Averaged Across PC Games, Multimedia, Enterprise Server, SPEC CPU2006 Workloads
DRRIP D R R I P Un-tuned DRRIP Would Be Ranked 2 nd and is within 1% of CRC Winner Unlike CRC Winner, DRRIP Does Not Require Any Changes to Cache Structure 20
Total Storage Overhead (16-way Set Associative Cache)
• • • •
LRU: NRU DRRIP-3: CRC Winner: 4-bits / cache block 1-bit / cache block 3-bits / cache block ~8-bits / cache block DRRIP Outperforms LRU With Less Storage Than LRU NRU Can Be Easily Extended to Realize DRRIP!
21
Summary
• • • • • Scan-resistance is an important problem in commercial workloads State-of-the art policies do not address scan-resistance • • We Propose a Simple and Practical Replacement Policy Static RRIP (SRRIP) for scan-resistance Dynamic RRIP (DRRIP) for thrash-resistance and scan-resistance • DRRIP requires ONLY 3-bits per block In fact it incurs less storage than LRU • Un-tuned DRRIP would be 2 nd place in CRC Championship DRRIP requires significantly less storage than CRC winner
22
Q&A 23
Q&A 24
Q&A 25
Static RRIP with n=1
• • • • Static RRIP with n = 1 is the commonly used NRU policy (polarity inverted)
Victim Selection Policy:
Evict block with RRPV=‘1’
Insertion Policy: Update Policy:
Insert new block with RRPV=‘0’ Cache hits update the block’s RRPV=‘0’ victim block hit Physical Way # Cache Tag RRPV 0 1 c 1 2 b 3 h 0 4 f 1 5 d 1 6 g 0
But NRU Is Not Scan-Resistant
7 e 1
26
SRRIP Update Policy on Cache Hits
• • • Frequency Priority (FP): Improve re-reference prediction to be shorter than before on hits (i.e. RRPV--). Intuition: Like LFU, predicts that frequently referenced blocks should have higher priority to stay in cache Physical Way # Cache Tag RRPV 0 s 2 1 c 2 b 3 3 h 0 4 f 1 5 d 1 6 g 0 7 e 1
27
SRRIP-HP and SRRIP-FP Cache Performance
20 15 10 5 0 SRRIP-Frequency Priority • • • GAMES MULTIMEDIA SERVER n=1 n=2 SPEC06 n=3 n=4 ALL SRRIP-Hit Priority 15 10 -5 -10 5 0 n=1 n=2 n=3 n=4 n=5 n=5 GAMES MULTIMEDIA SERVER SPEC06 ALL
28
Common Access Patterns in Workloads
• • • • •
Stack Access Pattern: (a 1 , a 2 ,…a k ,…a 2 , a 1 ) A Solution:
For any ‘k’, LRU performs well for such access patterns •
Streaming Access Pattern: (a 1 , a 2 ,… a k )
for k >> assoc
No Solution:
Cache replacement can not solve this problem • •
Thrashing Access Pattern: (a 1 , a 2 ,… a k ) A
, for k > assoc LRU receives no cache hits due to cache thrashing •
Solution:
preserve some fraction of working set in cache (e.g. Use BIP) BIP does NOT update replacement state for the majority of cache insertions • • • •
Mixed Access Pattern: [(a 1 , a 2 ,…a k ,…a 2 , a 1 ) A (b 1 , b 2 ,… b m )]
LRU always misses on frequently referenced: (a 1 , a 2 , … a k , … a 2 , a 1 ) A (b 1 , b 2 , … b m ) commonly referenced to as a
scan
in literature In absence of scan, LRU performs well for such access patterns
N
, m > assoc-k •
Solution:
preserve frequently referenced working set in cache (e.g. use LFU) LFU replaces infrequently referenced blocks in the presence of frequently referenced blocks
29
Performance of Hybrid Replacement Policies at LLC
20 10 0 -10 70 60 50 40 30 PC Games / multimedia server SPEC CPU2006 Average DIP HYB(LRU, LFU) 4-way OoO Processor, 32KB L1, 256KB L2, 2MB LLC • DIP addresses SPEC workloads but NOT PC games & multimedia workloads • Real world workloads prefer scan-resistance instead of thrash-resistance
30
Understanding LRU Enhancements in the Prediction Framework
RRP Head RRP Tail h g f e d c b a • • • • Recent policies, e.g., DIP, say “Insert new blocks at the ‘LRU position’”
What does it mean to insert an MRU line in the LRU position?
Prediction that new block will be re-referenced later than existing blocks in the cache
What DIP really means is “Insert new blocks at the `RRIP Tail’ ”
• • Other policies, e.g., PIPP, say “Insert new blocks in ‘middle of the LRU chain’” Prediction that new block will be re-referenced at an intermediate time The Re-Reference Prediction Framework Helps Describe the Intuitions Behind Existing Replacement Policy Enhancements
31
5 0 -5
Performance Comparison of Replacement Policies
20 NRU DIP SRRIP Best RRP Chain Insertion DRRIP
16-way 2MB LLC
15 10 GAMES MULTIMEDIA SERVER SPEC06 ALL
Static RRIP Always Outperforms LRU Replacement Dynamic RRIP Further Improves Performance of Static RRIP 32