Transcript PPT

Evaluating memory compression and
deduplication
Yuhui Deng, Liangshan Song, Xinyu Huang
Department of Computer Science
Jinan University
Agenda






Motivation
Memory compression
Memory deduplication
Characteristics of memory data
Evaluation
Conclusions
Motivation




Many programs require more RAM (e.g. memory
database).
For example, the maximal number of virtual machines
that can run on a physical machine is in most cases
limited by the amount of RAM space on that physical
machine.
Furthermore, the runtime of programs that page or
swap is likely to be dominated by the disk access time
when the amount of physical RAM is less than what the
programs require.
Therefore, we need more RAM!!!
Data compression


Data compression transforms a string of
characters into a new string that contains the
same information but whose length is as small as
possible.
The compression algorithms are categorized as
lossless compression and lossy compression.
Memory compression




Memory compression reserves some memory space
that would normally be used directly by programs,
It compresses relatively unused memory pages, and
stores the compressed pages in the reserved space.
This method enlarges effective memory space.
We employ six lossless compression algorithms
(Arithmetic algorithm, Huffman algorithm, LZ77,
LZ78, LZW, and RLE) to compress memory data.
Data deduplication




Data deduplication involves chunking and deduplication
detection.
The chunking phase splits data into non-overlapping
data blocks (chunks).
The duplication detection phase detects if another
chunk with exactly the same content has already been
stored by using hash algorithms.
Chunking phase can be classified into four categories:
Whole file chunking (WFC), Fixed-size partition (FSP),
Content-defined Chunking (CDC), Sliding Block (SB).
Memory deduplication



Memory deduplication periodically calculates a unique
hash number for every physical memory page by using
hash algorithms such as MD5 and SHA-1.
The calculated hash number is then compared against
other existing hash numbers in a database that
dedicates for storing page hash number.
If the hash number is already in the database, the
memory page does not need to be stored again, a
pointer to the first instance is inserted in place of
the duplicated memory page. Otherwise, the new hash
number is inserted into the database and the new
memory page is stored.
Evaluation environment
Trace name
1
2
3
4
espresso
gcc-2.7.2
gnuplot
grobner
Description
A circuit simulator
A GNU C/C++ compiler
A GNU plotting utility
Calculated Grobner basis functions
5
6
7
lindsay
p2c
rscheme
A hypercube simulator
A Pascal->C transformer
An implementation of Scheme
Table 1. Features of memory page traces
Size (GByte)
1.47
1.13
1.47
3.28
1.11
1.36
0.25
Components
Description
CPU
Intel Core2 T6400 (2M Cache, 2.00
GHz, 800 MHz FSB)
Memory
2G, DDR2 800MHz
Hard disk
WDC WD2500BEVT-60ZCT1（250GB
/5400RPM）
Chipset
Intel 4 Series - ICH9M
Table 2. Configuration of the experimental platform
Statistic results
Tracename
Zero(%)
Continuous
zeros (%)
Bound(%)
Low(%)
Power(2,n)(%)
Entropy
espresso
gcc2.7.2
gnuplot
grobner
lindsay
p2c
rscheme
45.94
56.09
80.72
58.34
86.00
59.62
14.61
29.88
44.53
30.91
49.96
13.87
10.34
6.44
44.50
19.78
25.77
7.63
16.25
11.13
3.29
18.62
6.40
6.30
16.46
12.25
4.66
15.35
6.72
7.89
4.19
3.85
2.01
3.30
1.33
3.60
35.98
10.75
6.39
17.86
17.85
4.94
Table 3. Statistic results of the page image traces



Entropy is normally employed as a measure of redundancy.
The entropy of a source means the average number of bits
required to encode each symbol present in the source.
The compressibility grows with the decrease of the entropy value.





The Zero column indicates that a large volume of memory data are
zero bytes across seven traces.
The Continuous zero column summarizes the percentage of pages
that contain continuous zeros longer than 32 bytes.
The Bound column includes the percentage of memory data which is
continuous zeroes longer than 32 bytes, and the continuous zeroes
start or end at a page boundary.
The Low column shows percentage of memory data that are low
values ranging between 1 and 9.
The Power(2,n) column implies the integral power-of-two values. It
indicates the values of 2, 4, 8, 16, 32, 64, 128, 255 using decimal.
Compression ratio
0.9
Compression ratio
0.8
espresso
grobner
rscheme
gcc-2.7.2
lindsay

gnuplot
p2c
0.9
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
ArithmeticHuffman LZ77
LZ78
LZW
RLE

The compression ratio
is defined as the size
of compressed memory
data divided by the
size of uncompressed
memory data.
The block size of this
evaluation is 8KB that
is equal to two memory
pages.
Algorithms


It shows that the LZ algorithms (LZ77, LZ78, LZW) obtain
the significant compression ratios (around 0.4),
gunplot trace achieves the best compression ratio across the
six algorithms.
110
100
90
80
70
60
50
40
7
6
5
4
3
2
1
0
110
100
90
80
70
60
50
40
7
6
5
4
3
2
1
0
espresso
gcc-2.7.2
gnuplot
grobner
lindsay
p2c
rscheme
ArithmeticHuffman LZ77
LZ78
LZW
Algorithms
Compression time




RLE
55
Decompression time (millisecond)
Compression time (millisecond)
Compression/decompression time
50
45
40
espresso
gcc-2.7.2
gnuplot
grobner
lindsay
p2c
rscheme
55
50
45
40
5
5
0
0
ArithmeticHuffman LZ77
LZ78
LZW
RLE
Algorithms
decompression time
the compression time of LZ77 and decompression time of LZW are
over 70 milliseconds and 50 milliseconds, respectively.
Not acceptable!
the latest Hitachi Ultrastar 15K------2 milliseconds
LZ78 strikes a good balance between compression ratio,
compression time, and decompression time.
Impact of block size (LZ78 )
gnuplot
p2c
200
0.55
0.55
180
0.50
0.50
160
0.45
0.45
0.40
0.40
0.35
0.35
0.30
0.30
0.25
0.25
0.20
0.20
4
8
16
32
64
128
Block size(KB)
(a)compression ratio



200
espresso
gcc-2.7.2
gnuplot
grobner
lindsay
p2c
rscheme
140
120
100
160
140
120
100
80
80
60
60
40
40
20
20
0
0
4
8
140
140
180
16
32
64
128
Block size(KB)
(b)compression time
Decompression time(millisecond)
gcc-2.7.2
lindsay
Compression time(millisecond)
Compression ratio
espresso
grobner
rscheme
espresso
gcc-2.7.2
gnuplot
grobner
lindsay
p2c
rscheme
120
100
80
120
100
80
60
60
40
40
20
20
0
0
4
8
16
32
64
128
Block size(KB)
(c) decompression time
It shows that the compression ratio decreases with the increase of
block size (from 4Kbyte to 128Kbyte) across the seven traces.
Fig. (b) and (c) reveal that the bigger the block size is, the higher
the compression and decompression time are.
This pattern is reasonable, because larger data block is more
compressible and requires more time to compress and decompress.
However, the performance decrease is not linearly proportional to
the block size.
Deduplication ratio
Deduplication ratio
espresso
grobner
rscheme
gcc-2.7.2
lindsay
gnuplot
p2c
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
This evaluation adopts
three schemes including
FSP, CDC, and SB.
0.0
FSP-4K FSP-8K FSP-16KFSP-32K
CDC
SB-4K
Algorithms



It shows that FSP-4K and SB-4K achieve the best deduplication
ratio across the seven traces.
When the chunking size of FSP is increased from 4Kbyte to
32Kbyte, the deduplication ratio is significantly increased.
The deduplication ratio of CDC is close to 1.
1800
1600
1400
1200
espresso
gcc-2.7.2
gnuplot
grobner
lindsay
p2c
rscheme
1800
1600
1400
1200
200
200
100
100
0
0
FSP-4K FSP-8K FSP-16KFSP-32K CDC
Algorithms




SB-4K
80
Restore time(microseconds)
Deduplication time(microseconds)
Deduplication/restore time
70
60
50
espresso
gcc-2.7.2
gnuplot
grobner
lindsay
p2c
rscheme
80
70
60
50
40
40
30
30
20
20
10
10
0
0
FSP-4K FSP-8K FSP-16KFSP-32K CDC
SB-4K
Algorithms
From a compression ratio standpoint, FSP-4K and SB-4K are the
best candidates to perform memory deduplication.
The deduplication time of SB-4K is about 40 times higher than that
of the FSP-4K, although the restore time is comparable.
FSP-4K is the best candidate policy for memory deduplication.
Please note that the Y axis of the two figures is in microseconds.


The chunking size has an opposite impact on
the performance of compression and
deduplication.
This is because the probability of those
identical characters contained in a chunk grows
with the increase of the chunk size, while the
probability of two chunks that are exactly the
same is decreased with the growth of the
chunk size.
Conclusion




Memory deduplication greatly outperforms
memory block compression.
Fixed-size partition (FSP) achieves the best
performance in contrast to Content-defined
Chunking (CDC) and Sliding Block (SB).
The optimal chunking size of FSP is equal to the
size of a memory page.
We believe, the analysis results in this paper should be
able to provide useful insights for designing or
implementing systems that require abundant memory
resources to enhance the system performance.
Thanks!