Transcript No Slide Title

Entropy Coding
Heejune AHN
Embedded Communications Laboratory
Seoul National Univ. of Technology
Fall 2013
Last updated 2013. 10. 28
Agenda





[Midterm exam solution]
Entropy Coding in Video Codec
Basic Information Theory on Entropy coding
Huffman Coding and Implementation
Arithmetic Coding and Implementation
Heejune AHN: Image and Video Compression
p. 2
1. Entropy Coding in Video Coding

Video Encoder Components

A source model + an Entropy Coder
Source Model
Block
Motion
Transfor
m
Entropy coder
Texture Info
Quantiz
er
Motion Info
Symbol
mapper
Entropy
coders
Control/Mode info

Design Goals of entropy coder (lossless)



Compression efficiency : smaller output bit count
Computational efficiency: Implementation suitability
Error robustness and access efficiency: easy to synchronize
Heejune AHN: Image and Video Compression
p. 3
Data symbol Mapping

Symbols




Input to (General) Entropy Coder
Well-defined Information units from real-raw data (numbers)
e.g.) Coefficients in blocks, Motion Vectors for MBs, Coding Modes
for Video Coding
(DCT) Coefficient symbol mapping

Properties
• Sparse
• non-zeros Near DC (0, 0)
• vertical and horizontal : even or not

Solution
• Zig-zag scanning
• (Run, Length) symbol mapping
Heejune AHN: Image and Video Compression
102 -33
21 -2
-3
-3 -4 -2 -1
-3
-1
-1
2
1
1
-2
p. 4

Zigzag Scanning

Frame picture
FieldX Picture (horizontal-first scan)
Y

Run-Length





(run, length) : 2-D symbol mapping in H.261 and MPEG-1/2
EOB (end of block) : extra symbol for all remaining zeros
(run, length, last) : 3-D in H.263 and MPEG-4
Intra DC is differential coded form previous block
Do the text example (p 166)
Heejune AHN: Image and Video Compression
p. 5

Motion Vector Symbols

Differential coded
• Other
H.261
MV2
MV3
MV1
MV1
MV
MV
pi = Median (mv1i, mv2i, mv3i)
mvdi = mvi -pi

Quaitzation Parameters


Differential coded : DQ = Qcurrent -Qprev
No abrupt quant step change (-> abrupt quality change)
Heejune AHN: Image and Video Compression
p. 6
Change MQUANT
Coded
No Change to MQUANT
Inter

Not Coded
Coded Mode

Perform motion
compensation
Coding Mode
Change MQUANT
Intra
• Motion compensated or not, Coded or not etc

block has data or not
No Change to MQUANT
P picture
Change MQUANT
• No texture info
Coded
16
Not Coded
8
1
No Change to MQUANT
Inter
2
16
5
6
8
Set motion vector
to zero
(Skipped)
Change MQUANT
Intra
No Change to MQUANT
3
4
Cr
Y


휘도신호
Cb
색차신호
Synchronization Headers
High-Level Header

Global parameters (coding options): resolution, pixel aspect ratio,
quantization options, etc
Heejune AHN: Image and Video Compression
p. 7
2. Entropy Coding Background

Definition of Entropy (by. C. Shannon)
H ( S )  [ I ( S )]
1
  pm I ( sm )   pm log2 ( ) (bits)
pm
m
m

Input symbols space & occurrence prob. :
{a 0 , a1 ,  , a M 1 }

Information




pm  Pr (s  am )
Uncertainty = information (when revealed)
1
I ( sm )  log( )
length, amount of info for a symbol
m
Rare Events (p = 0.0000001, e.g. ..) => big news ! => morepinfo
Interpretation (the implication)

The amount of Information of S (Qualitatively)
• The less predictable, the more information

Shannon, Information theory (Quantitatively)
• H(S) is the theoretical limit (upper-bound) bits for representing S.
Heejune AHN: Image and Video Compression
p. 8

Binary Signal Example (0,1)

Entropy

Entropy is maximum at p = ½ , ie. equi-probable.
H 2 ( p)   p log p  (1  p) log(1  p)
1.0
1  H 2 ( p)
0.5
H 2 ( p)
p
0.0
0.0
0.5
Heejune AHN: Image and Video Compression
1.0
p. 9
Entropy coding

Concept




Assign shorter length code for more frequent symbol, Average codelength is reduced
So, VLC (variable-length-coding)
Lossless coding algorithm: used in File compression
An example



Let code a file with “squeeze”
{s, q, u, e, z} with prob. { 1/7, 1/7, 1/7, 3/7, 1/7}
Fixed length coding
• 3 bits (5 symbol)* 7 symbol = 21 bits

VLC coding
• {u, s, u, q, z} => {‘1’, ‘000’, ‘001’, ‘010’, ‘011’}
• 1bit*3symbols + 3bits*4symbols = 3 + 12 = 15 bits
Heejune AHN: Image and Video Compression
( << 21 bits)
p. 10
Coding algorithm

How to code?



Huffman Coding




Still “Open” Problem
Welcome you if you dare
Integer length Code Book
Used in H.261, H.263, MPEG-2
Pioneered David Huffman (in 1952)
David Huffman
Arithmetic Coding




Factional length Coding
file compression (zip)
Used in H.263, MPEG-4, H.264
Pioneered by Jorma Rissanen
Jorma Rissanen
Heejune AHN: Image and Video Compression
p. 11
Prefix-free coding

Prefix-free coding




A code is not same as the prefix part of any other code
So, uniquely decodable by scanning from the start
Huffman Code is a kind of prefix code
Compare two codes
symbol
codeword
Symbol
codeword
a1
a2
a3
a4
0
10
101
0101
a1
a2
a3
a4
0
10
110
1110
Heejune AHN: Image and Video Compression
p. 12
Huffman Coding

Mapping a symbol to a variable length code



variable & integer length code (an optimal integer length code)
uniquely decodable (prefix-free)
Huffman Code Generation




Step 1 : Order the symbols in increasing order of probability
Step 2 : Take two smallest, assign ‘1’ to the larger probability symbol and ‘0’ to the
smaller.
Step 3 : Combine the two smallest-probability symbols into a node (group), and
assign the combined probability to the node
Step 4 : IF only one node remains, STOP, ELSE reorder the resultant symbols
(considering the combined node as one symbol)
Heejune AHN: Image and Video Compression
p. 13
Huffman Encoding & Decoding

Encoding



Table lookup
Symbol => code (code value, length)
Decoding


Comparing codes bit by bit
Bit stream => boundary (code value and length) => symbol
0
0001
01
…
1
11
11
1
2
10
10
2
3
000001
001
N-3
4
00001
0001
0
…
01
00001
4
N-3
001
000001
3
N-1
000000
000000
N-1
Encode Table
Heejune AHN: Image and Video Compression
Decode Table
p. 14
An Example

Motion Vector


Code book Generation (See Fig 8.5)
Encoding example
• 3 motion values, 1, 0, 0.5 => 01011/1/011

Decoding example
• 01011/1/011 => 1, 0, 0.5
mvx
Code
bit length
(Huffman)
Bit length
(Ideal)
0
1
1
0.63
-0.5
00
2
3.10
+0.5
011
3
3.31
-1.5
01000
5
5.21
3.31
1.5
01001
5
5.38
0.027
5.21
-1
01010
5
5.97
0.016
5.97
1
01011
5
6.16
mvx
Prob.
log2(1/P)
-1.5
0.014
6.16
-1
0.024
5.38
-0.5
0.117
3.10
0
0.646
0.63
0.5
0.101
1
1.5
Heejune AHN: Image and Video Compression
p. 15
Code Table Design

Ideal Huffman Code


Input statistics adaptive; performance
Disadvantages
• Statistics calculation burden (big data)
• Transmission overhead from an encoder to decoders
• Implementation difficulty

Generic Huffman code


based on large set data (various video sequences)
Disadvantages
• Lower performance possible

Video coding standards

Provides generic code with optional extension
Heejune AHN: Image and Video Compression
p. 16
Code table design example

H.263/MPEG-4 TCOEF

3 Dimensional (last, run, level)
• Ideal table size = 2 x 63 x 128 x 2 (sign). Huge!!

Only 102 most frequent values has specific code (upto 13 bits)
• Others : ESCAPE Code (0000011) + 13 bit (last, run, level)
• Note : Fixed-length code is better for very small probability codes

No successive 8 0’s allowed
• Used for synchronization start code
Heejune AHN: Image and Video Compression
p. 17

26 partial list
‘s’ : sign bit

Last
Run
LEVEL
Code
Last
Run
LEVEL
Code
0
0
1
10s
1
1
1
001111s
0
1
1
110s
1
2
1
001110s
0
2
1
1110s
1
3
1
001101s
0
0
2
1111s
1
4
1
001100s
1
0
1
0111s
0
0
4
0010111s
0
3
1
01101s
0
10
1
0010110s
0
4
1
01100s
0
11
1
0010101s
0
5
1
01011s
0
12
1
0010100s
0
0
3
010101s
1
5
1
0010011s
0
1
2
010100s
1
6
1
0010010s
0
6
1
010011s
1
7
1
0010001s
0
7
1
010010s
1
8
1
0010000s
0
8
1
010001s
ESCAPE
0
9
1
010000s
.etc
Heejune AHN: Image and Video Compression
0000011s
p. 18

H.263/MPEG-4 MVD (motion vector difference)




dmvx and dmvy
See table 8.6 and Figure 8.8
MVD = 0 is 1 bit (‘1’)
H.264 Universal VLC (UVLC)



Systemic (Universal) table generation (just need ordering)
1 1-bit code, 2 3-bit codes, 4 5-bits codes….
Easy to implementation and statistics adaptive
1
=> most frequent 1 symbol
0 x0 1
=> the next 2 symbols
0 x1 0 x0 1
=> the next 4 symbols
0 x2 0 x1 0 x0 1
=> …
0 X3 0 x2 0 x0 1
…
Heejune AHN: Image and Video Compression
p. 19
Example

Textbook

Coeff matrix
4
-1
1
-3
2
-1


(0,0,4), (0,0,-1), (0,1,2), (0,0,-3), (0,5, -1), (1,3,1)
Final bit stream
• 00101110/101/0101000/010111/0011010
Heejune AHN: Image and Video Compression
p. 20
Implementation Issues

Encoder

Length required (because of Variable length)
Symbol
Table select

Code
Lookup
Or Calculate VLC
Length
Output bitstream
Packing Register
(Shifter)
Decoder
Symbol
bitstream
Input bitstream
Lookup
Or Calculate VLC
Packing Register
(Shifter)
Error
Length
Heejune AHN: Image and Video Compression
p. 21
Table Based Method

Difficulties

Table Size is huge
•
•
•
•

num of last X num of run X num level
H.263/MPEG-4 TCOEF => 16002 entries
over 64 MB TLB for Encoder
Over 16K Comparator for Decoder
Complexity Reduction in Encoder
If((last, run, level) in most frequent range){
output TLB1[last][run][level];
}else if((last, run, level) in 2nd frequent
range){
If(|level| < 13 && run < 39){
output TLB[last][run][level];
}else{
output ESCAPE CODE;
output last;
output run;
output level;
}
Code Based Method
Switch(last){
case ..
switch(run){
case ..
switch(level){
}
…
} else ….
….
Heejune AHN: Image and Video Compression
p. 22

Complexity reduction in decoder
do{
Bi = take first Li bits from stream;
symbol = TLB1[Bi];
}while( symbol != continue);
01
…
01
…
11
1
11
1
10
2
10
2
001
N-3
001
NA
0001
0
001
N-3
00001
4
0001
0
000001
3
00001
4
000001
3
000000
N-1
000000
N-1
Heejune AHN: Image and Video Compression
p. 23

Error Handling

Cause of error
• Transmission error, storage error, system errors
• packet loss, bit-reversal, bit-erosion

Error detection
• Undefined code (in syntax), visual corruption (in semantics)

Error recovery
• Even after Forward error correction/Feedback Recovery
• Synchronization Code (slice or picture level)
• Reversible VLC code (in MPEG-4)
Error detection
Real Error
Heejune AHN: Image and Video Compression
Sync. code
p. 24
Arithmetic Coding

Motivating example



Table 8.1: 0.5 (p = 0.101)  ideally 3.31 bits but 5 bits assigned
Why ideal and why 5 bit assigned ?
Arithmetic coding

Assign code a range not a value (fractional bit code)
Pr (ai )
I (ai )  [ Li , Hi )  [CDF (ai 1), CDF (ai ))
0.0

(-2)
0.1
(-1)
0.3
(0)
0.7 (+1) 0.9 (+2) 1.0
Better compression than Integer Coding
• Especially one big symbol

Patent
• IBM: Q-Coder
Heejune AHN: Image and Video Compression
p. 25
Arithmetic Encoding Algorithm


Step 1: Ro  1  0  1, L0  0, H0  1
Step 2: Calculate Range for n-th symbol
 I ( x n )  Ln1  Rn1  I o ( xn )
 [ Ln , H n )
symbolinterval[ Li , H i ), i.e., [CDF (ai 1 ), CDF (ai
low end
of I ( x n 1 )
by theinitialor updated Pr (ai )
range (length)of I ( x n1 )
 range; Rn1  H n  Ln


Repeat remaining symbols or Stop
Send any value in the range

Normally the shortest representation (e.g. binary)
Heejune AHN: Image and Video Compression
p. 26
Example: Encoding and Decoding

Input symbols

0, -1, 0, 2
0.1
0.3
0.3
0.34
0.42
0.34
0.348
0.364
0.364
0.3672
0.3736
0.0
(-1)
(0)
(0)
0.7
0.9
0.58
0.66
0.7
0.396
0.412
0.42
0.3864
0.3928(2)
0.396
1.0
0.394531 => check !!
Heejune AHN: Image and Video Compression
p. 27
More Issues on Arithmetic Coding

Performance



Termination method




0.394531 =0.011001010b => 9 bits
Theoretic min = 0.828 bits
Why I stopped decoding with 4 symbols?
I knew that 4 symbols required (predefined length)
Extra “End of Sequence” code
Fixed point arithmetic



no symbolic computing CPU yet. All floating point is not exact.
H.263/4 14 bits accuracy, i.e. smallest probability is 1/16384
Shift the determined MSB bits and contains only LSB bits in buffer
01011111010100101010010101
Unchangeable after new symbol range calculation
Heejune AHN: Image and Video Compression
p. 28
Conclusion

Entropy coding





Symbol to bit stream mapping and reverse
Data compression using statistical redundancy approaching to the
theoretical limit (entropy)
Integer length code like Huffman code
Fractional length codes like Arithmetic code
Entropy coding in video coding





Texture (DCT coefficients)
Motion Vectors
Mode and side information
Huffman code is traditional tool for entropy coding
Arithmetic code is new stronger tool but not widely used yet.
Heejune AHN: Image and Video Compression
p. 29