Transcript ECE-C490 Winter 1999 Image Processing Architecture

ECE-C490 Winter 2004
Image Processing Architecture
Lecture 2, 1/8/2004
Lossless Coding (More)
Oleh Tretiak
Drexel University
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 1
Review: Need for Compression
•
Example: Fax document
-
-
-
•
8.5x11”, 200 dpi, 8.5x11x(200)2 = 3.74 Mbits
@28.8 kbits/sec, 3740000/28800 = 130 sec
Typical compression = 15, with compression 8.65 sec.
Example: Video
-
-
-
Lecture 2
640x480 pictures, 24 bits/sample, 30 frames/sec
640x480x24x30 = 2.21E+08 bits/sec = 2.76E+07bytes/sec
A CD-ROM stores 650 Mbytes -> playing time = 650/27.6 = 23.5 sec
With compression, 74 minutes of low (VHS) quality video can be stored on a
CD-ROM
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 2
Review: How is Compression Possible?
•
Statistical Redundancy
-
-
-
•
Perceptual Redundancy
-
•
Adjacent pixels are similar (spatial correlation)
Color components are similar (spectral correlation)
Successive frames are similar (temporal correlation)
Data can be eliminated from signal with no visible changes.
Lossy compression:
-
Lecture 2
Send (inferior) picture that requires fever bits.
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 3
Example: Compression
•
•
•
We have 1000 symbols from alphabet {a b c d}
Coding: 2 bits per symbol, total bits = 2x1000 = 2000
Variable length coding, some symbols are more frequent
Symbol
a
b
c
d
Number Code
900
0
50
10
25
110
25
111
Bits
900
100
75
75
Total bits = 900 + 100 + 75 + 75 = 1150
Average bits/symbol = 1150/1000 = 1.15 < 2
Compression = source bits/code bits = 2/1.15 =1.74
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 4
Review: Typical Encoder System
Signal
Data
Preprocessor
•
Entropy Coder
Issues
-
Constant Bit Rate vs. Variable Bit Rate
o In lossless encoder, bit rate depends on compression efficiency
o Variable bit rate is undesirable in real-time applications
o In lossy encoder, bit rate can be kept constant by varying quality
-
Single or Multiple Sample encoding
o Multiple sample encoding is usually more efficient but also is more
complex.
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 5
Review: Information Theory
•
Let the i-th symbol have probability pi. The information of this
symbol is defined as log2(1/ pi) bits. The average information of
this symbol is pi log2(1/ pi) bits. The entropy of this symbol set is
defined as
N
H   pi log2(1/ pi )
•
•
•
bits
i 1
Shannon Coding Theorem: It is possible to encode a (long)
sequence of symbols with H +e bits per symbol
How to do it?
If the symbols are statistically independent, it is impossible to
encode these with fewer than H bits per symbol.
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 6
Review: Entropy and Huffman Codes
•
Theory
-
-
-
•
Information
Entropy
Shannon Compression Theorem
Practice
-
-
Deriving symbols from signals
Huffman encoding
o Coder construction
o Encoders
o Decoders
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 7
Review: Variable Length Codes
•
•
•
•
•
Symbol set: si, i = 1 … N, pi — symbol probability
Code: ci, li, where ci is a sequence of 1’s and 0’s of length li.
The code words must be decodable: the transmitted bit stream
is just a set of 1’s and 0’s, code word boundaries are not
indicated. For a decodable code, the code word boundaries can
be found uniquely from the transmitted sequence.
Sufficient condition for decodability: A code is decodable if no
code word is at the beginning (prefix) of another code word. This
is called the prefix condition
N
Average code word length: Lave   pi l i
i 0
For a decodable code, Lave ≥ H
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 8
This Lecture
•
Huffman codes
-
-
-
•
•
How to construct codes
Encoding and decoding algorithms
Huffman codes with constrained length
Golomb and Rice coding
Arithmetic Coding
-
Lecture 2
Coding when H < 1
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 9
Construction of a Huffman Code
•
Tree construction
-
-
Order the symbols according to probabilities
Apply contraction process to the two symbols with lowest
probability
o assign a new (hypothetical) symbol to these two,with probabilities equal
to sum of the code word probabilities
-
•
Repeat until one symbol is left
Code construction
-
-
For the two initial branches of the tree, attach bit ‘0’ and ‘1’ at end
of code word.
For both branches
o If (branch is a source symbol) done
o else repeat above process
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 10
Code construction example
Symbol Probability
?
0.10
e
0.30
k
0.05
l
0.20
r
0.20
u
0.10
w
0.05
H = 2.55
ST 1
e 0.30
l 0.20
r 0.20
? 0.10
u 0.10
k 0.05
w 0.05
ST 2
e 0.30
l 0.20
r 0.20
? 0.10
u 0.10
A 0.10
ST 4
e 0.30
C 0.30
l 0.20
r 0.20
ST 6
ST 5
D 0.40 D 0.40
e 0.30 E 0.60
C 0.30
E=e,C
D=l,r
C=B,?
B=u,A
A=k,w
00
01
10
1100
11010
11011
111
Lecture 2
ST 3
e 0.30
l 0.20
r 0.20
B 0.20
? 0.10
r
l
e
u
k
w
?
?
e
k
l
r
u
w
111
10
11010
01
00
1100
11011
3
2 Lave
5
2
2
4
5
Image Processing Architecture, © 2001-2004 Oleh Tretiak

N
 pi l i
i 0
 0.10  3  0.30  2 
0.05  5  ...  0.05  5
 2.6
Page 11
Variable Length Code: Encoding
Source sequence: werule?
•
?
e
k
l
r
u
w
111
10
11010
01
00
1100
11011
Lecture 2
3
2
5
2
2
4
5
11011100011000110111
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 12
Prefix code: bit-serial decoding
11011100011000110111
•
Algorithm steps: bold denotes output symbols
-
Lecture 2
ECBAwEeDrECBuDl...
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 13
Prefix code: table decoding
•
Let k be the maximum code symbol length. Construct table with
2k entries. Each table location contains input symbol and code
word length. Order code by binary value. A code symbol of
length l will have 2k-l entries.
00
01
10
111
1100
11010
11011
Lecture 2
r
l
e
?
u
k
w
Since k = 5, we use 32 table entries. Code symbol ‘00’ will
use 25-2=8 entries. Each entry will have output symbol ‘r’ and
length 2. The next 8 entries will be for ‘l’. The following 8
entries will be for ‘e’. The following 4 entries will be for ‘?’, etc.
Decoding:Take k = 5 bits from encoded sequence. Decode it
by table lookup. From table, find the symbol length, discard
these many bits from the code word used for lookup, take
additional bits from encoded sequence.
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 14
Lookup table decoding example
11011100011000110111
00
01
10
111
1100
11010
11011
r
l
e
?
u
k
w
•
•
•
•
•
Lecture 2
First lookup code: 11011. Output = ‘w’, l = 5, read 5
bits from input stream.
Lookup code: 10001. Output = ‘e’, l = 2, discard the
initial ‘10’ from lookup code, read two more bits
Lookup code 00110. Output = ‘r’, l = 2, discard the
initial ‘00’ from lookup code, read two more bits.
Lookup code 11000. Output = ‘u’, l = 4, discard initial
‘1100’, read 4 more bits
Lookup code 01101. Output = ‘l’, ...
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 15
Huffman Codes With Constrained Length
•
•
•
Some codes used in practice can have longest code words 20
bits long: huge lookup tables for lookup sequences
Solution concept 1: ‘Escape’ sequences
Solution concept 2: limit max code word length
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 16
Code construction - Escape
•
Escape sequence approach
-
-
-
-
•
High probability code symbols decoded with lookup table, table is
modest in size.
All low probability codes are lumped together: they are assigned
one symbol (escape symbol) in the high probability table.
To encode low probability symbol, send escape symbol plus lowprobability sequence (this will be another Huffman code).
Decoding: use high probability table: if escape symbol is
encountered, switch to low probability table.
This approach uses a hierarchical set of lookup tables for
decoding
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 17
Rule of Thumb
•
•
If symbol has probability p, then the length of a VLC code word
should be about
• l = -log2p
Examples
-
-
-
Lecture 2
p = 0.5, l = 1 bit
p = 0.1, l = 3 bits
p = 0.01, l = 7 bit
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 18
Two level hierarchy: method
•
•
•
•
•
•
S is the source, L be the maximum code word length
Sort the symbols by probability, so that p1 ≥ p2 … ≥ pN
Split the source into two sets:
1
S1  {si | pi  L }  {s1,s2 , ,st 1}
2
1
S 2  {si |p i  L }  {st ,st 1, ,sN }
2
1
P (S 2 )  L
N
Create a special symbol Q with
2 probability
q  pi
Augment S1 by Q to form new set W. Design Huffmani
code for this this
t
set
Encoding: For symbols is S1, output code word. For symbols in S2,
send Q, then symbol without encoding (this requires
bits).
log2 NS 2 
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 19
Example
Symbol
a
b
c
d
e
f
g
h
I
j
k
l
m
n
o
p
pi
l i Codeword
0.2820 2 11
0.2786 2 10
0.1419 3 011
0.1389 3 010
0.0514 4 0011
0.0513 4 0010
0.0153 5 00011
0.0153 5 00010
0.0072 6 000011
0.0068 6 000010
0.0038 7 0000011
0.0032 7 0000010
0.0019 7 00000011
0.0013 8 00000010
0.0007 9 000000011
0.0004 9 000000010
Lecture 2
•
•
•
•
•
•
•
H = 2.65, Lave = 2.69
Max l is 9, decoding table requires 512
entries
Target max length 5, min p > 1/2^5 = 1/32
= 0.03125
S1 = (a, b, c, d, e), S2 = (f … p)
P(S2) = 0.0559, min p in S1 = 0.0513
Both are greater than 1/32
Expect longest code word should be 5 bits
or less
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 20
Example - Continued
•
•
•
Build Huffman code for a-f + S2
If a-f needs encoding, send code word
If g-p needs encoding, send code for S2, followed by
binary number (one of 10) -> 4 bits
Symbol
a
b
c
d
e
f
S2
pi
l i Codeword
0.2820 2 00
0.2786 2 10
0.1419 3 010
0.1389 3 011
0.0514 4 1100
0.0513 4 1101
0.0559 3 111
Example: encode “cam”
010 00 111 0110
Lecture 2
Symbol
g
h
I
j
k
l
m
n
o
p
pi
0.0153
0.0153
0.0072
0.0068
0.0038
0.0032
0.0019
0.0013
0.0007
0.0004
Image Processing Architecture, © 2001-2004 Oleh Tretiak
li
4
4
4
4
4
4
4
4
4
4
Codeword
111 + 0000
111 + 0001
111 + 0010
111 + 0011
111 + 0100
111 + 0101
111 + 0110
111 + 0111
111 + 1000
111 + 1111
Page 21
Performance Analysis
•
•
•
•
•
What is Lave?
Lave = Avg no of bits to send VLC + p(S2)*4
= 2.5421 + 0.0559*4 = 2.766
How does this compare with Huffman code?
For Huffman code, Lave = 2.694
Other ideas?
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 22
Constrained Length - 2nd method
•
Long code words produced by low probability symbols
-
-
•
•
Idea: modify probabilities, increasing the low probability values
Design Huffman code for this
This approach produces codes with lower maximum length, but
larger average length
Leads to simpler encoder and decoder structure than escape
(hierarchical) approach, but performance may not be as good.
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 23
Huffman vs. Arithmetic Code
•
Lowest Lave for Huffman codes is 1. Suppose H << 1?
-
-
•
One option: use one code symbol for several source symbols
Another option: Arithmetic code.
Idea behind arithmetic code:
-
Lecture 2
Represent the probability of a sequence by a binary number.
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 25
Arithmetic Encoding
•
•
Assume source alphabet has values 0 and 1, p0 = p , p1 = 1 – p.
A sequence of symbols s1, s2, … sm is represented by a
probability interval found as follows:
-
-
Initialize, lo = 0; range = 1
For i = 0 to m
o if si = 0
§ range = range*p
o else // si = 1
§ lo = lo + range*p
§ range = range*(1-p)
o end
-
•
end
Send binary fraction x such that lo ≤ x < hi. This will require
log 2 range  bits
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 27
Arithmetic coding: example
•
p0 = 0.2, source sequence is 1101
symbol
low
1
1
0
1
0
0.2000
0.3600
0.3600
0.3856
range
1
0.8000
0.6400
0.1280
0.1024
Number of bits = ceiling(-log2(0.1024)) = 4
low2 =.01100010, (low+range)2 = .01111100
Bits sent: 0111
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 29
Arithmetic Decoding
•
•
•
We receive x, a binary fraction
lo = 0; hi = 1
for i = 1 to m
-
if (x - lo) < p*(hi-lo)
o si = 0
o hi = lo + (hi-lo)*p
-
else
o si = 1
o lo = lo + (hi-lo)*p
-
•
end
end
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 30
Arithmetic Decoding
•
•
•
We receive x, a binary fraction
lo = 0; range = 1;
for i = 1 to m
-
if (x - lo) < p*range
o si = 0
o range = p*range
-
else
o si = 1
o lo = lo +range*p
o range = range*(1 - p)
-
•
end
end
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 31
Arithmetic Decoding
•
•
We receive x, a binary fraction
for i = 1 to m
-
if x < p
o si = 0
o x = x/p
-
Receive x = 0111 = 0.4375
p = 0.2
Else // x > p
o si = 1
o x = (x - p)/(1 - p)
-
•
end
end
Lecture 2
symbol
1
1
0
1
Image Processing Architecture, © 2001-2004 Oleh Tretiak
x
0.4375
0.2969
0.1211
0.6055
0.5068
range
1
0.8000
0.6400
0.1280
0.1024
Page 32
Arithmetic decoding example
•
Receive 0111 (0.4375), decode 4 bits, p0 = 0.2
x= 0.4375
low
high
0
1
0.2
1
0.36
1
0.36
0.488
Lecture 2
bit
1
1
0
1
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 33
Arithmetic decoding example
•
Receive 0111 (0.4375), decode 4 bits, p0 = 0.2
Lecture 2
symbol
low
1
1
0
1
0
0.2000
0.3600
0.3600
0.3856
range
1
0.8000
0.6400
0.1280
0.1024
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 34
Magic Features of Arithmetic
Coding
•
Remember I (information) = - log2p
-
-
-
•
•
p = 0.5, I = 1
p = 0.125, I = 3
p = 0.99, I = 0,0145 (wow!)
High p symbol, less than 1 code bit per symbol!
In encoder, hi - lo = ∑ I(symbols)
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 35
Summary: Arithmetic Coding
•
•
•
Complexity: requires arithmetic (multiplications, divisions),
rather than just table lookups
Algorithms are complex, accuracy (significant bits) is tricky
Can be made to operate incrementally
-
•
•
Both encoder and decoder can output symbols with limited internal
memory
Provides important compression savings in certain settings
Part of standards
Lecture 2
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 36
Summary: VLC
•
•
Lowest level (basis) of image compression
We talked about
-
-
-
•
Two phases
-
-
•
Huffman
Golomb/Rice
Arithmetic
Code design
Encoding/decoding
More about VLC
-
-
Lecture 2
Adaptive coding: estimate probabilities
There are universal coders (good non-adaptive coding) such as
Lempel-Ziv (zip)
Image Processing Architecture, © 2001-2004 Oleh Tretiak
Page 37