Ch. 6- H.264/AVC Part I (pp.160~199)

Sheng-kai Lin Shengkai Lin@CMLAB, NTUCSIE 2004 1

Outline

• Introduction • Baseline profile – Reference Picture Management – Slice – Inter-prediction – Intra-prediction – Deblocking filter – Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 2

H.264/AVC Introduction (I)

• 4:2:0 • Define VCL and NAL – VCL: Video Coding Layer – NAL: Network abstraction • Variable block size • Multiple reference frame • ¼ sample motion compensation Shengkai Lin@CMLAB, NTUCSIE 2004 3

H.264/AVC Introduction (II)

• CAVLC • CABAC Shengkai Lin@CMLAB, NTUCSIE 2004 4

Terms (I)

• Picture (frame) – one or more slices • Slice – group of macroblock (1~ ) – minimal inter-dependency between coded slices can limit the error propagation • Macroblock Shengkai Lin@CMLAB, NTUCSIE 2004 5

Terms (II)

• I slice – I macroblock only • P slice – P, I macroblock • B slice – B,I macroblock • SI slice • SP slice Shengkai Lin@CMLAB, NTUCSIE 2004 6

Terms (III)

• I macroblock – intra prediction from decoded samples in the current slice • for entire macroblock • for each 4*4 block • P macroblock – inter prediction using past references • 16*16, 16*8, 8*16, 8*8, 4*8, 8*4, 4*4 Shengkai Lin@CMLAB, NTUCSIE 2004 7

Terms (IV)

• B macroblock – inter prediction using past/ future references • list 0 – past reference frame buffer • list 1 – future reference frame buffer • RBSP (Raw Byte Sequence Payload) – NAL payload Shengkai Lin@CMLAB, NTUCSIE 2004 8

H.264/AVC profiles

Shengkai Lin@CMLAB, NTUCSIE 2004 9

H.264/AVC Encoder

Shengkai Lin@CMLAB, NTUCSIE 2004 10

H.264/AVC Decoder

Shengkai Lin@CMLAB, NTUCSIE 2004 11

Outline

• Introduction • Baseline profile – Reference Picture Management – Slice – Inter-prediction – Intra-prediction – Deblocking filter – Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 12

Reference Picture Management (I) • Reference pictures are stored in decoded picture buffer (DPB) • short/long term reference picture • A decoded frame may be marked as – unused for reference – short term picture – long term picture Shengkai Lin@CMLAB, NTUCSIE 2004 13

Reference Picture Management (II) • “ Sliding Window – clear ref buffer – I frame ” – issued by encoder memory management – Keep #(long_term_pic+ short_term_pic) – Remove short term picture if lack of space • Adaptive memory control – change the type of the ref frame • IDR (Instantaneous Decoder Refresh) Shengkai Lin@CMLAB, NTUCSIE 2004 14

Outline

• Introduction • Baseline profile – Reference Picture Management – Slice – Inter-prediction – Intra-prediction – Deblocking filter – Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 15

Redundant coded picture

• Send the duplicated part or all of a coded picture • Error resilience Shengkai Lin@CMLAB, NTUCSIE 2004 16

Arbitrary Slice Order (ASO)

• The decoding order of the slices could be arbitrary • Application example – reduce end-end transmission delay in RT app Shengkai Lin@CMLAB, NTUCSIE 2004 17

Slice Group (I)

• former called “ FMO • Application example – Error resilience ” – Flexible Macroblock Ordering • a subset of the macroblocks and may contain one or more slices Shengkai Lin@CMLAB, NTUCSIE 2004 18

Slice Group (II)

Shengkai Lin@CMLAB, NTUCSIE 2004 19

Outline

• Introduction • Baseline profile – Reference Picture Management – Slice – Inter-prediction – Intra-prediction – Deblocking filter – Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 20

Inter Prediction

• Variable block size • ¼ pixel motion compensation Shengkai Lin@CMLAB, NTUCSIE 2004 21

Interpolation Samples (I)

1. Generate ½ samples by six tap FIR • b, h, m, s • FIR para :(1/32, -5/32, 5/8, 5/8, 5/32, 1/32) 2. Generate center • j ½ sample by FIR 3. Generate ¼ interpolation sample by linear Shengkai Lin@CMLAB, NTUCSIE 2004 22

Interpolation Samples (II)

Shengkai Lin@CMLAB, NTUCSIE 2004 23

Motion vector prediction (I)

• Encoding MV can cost a lot of bits – especially small partition sizes • Neighboring MV is highly correlated – predictable – encode the diff MVD and transmit it Shengkai Lin@CMLAB, NTUCSIE 2004 24

Motion vector prediction (II)

• Partitions excluding 16 MVp = (MV A + MV B × 8 and 8 × + MV C ) /3 • For 16 × 8 partitions, MVp of the upper 16 × ;MVp for the lower 16 × 8 = MV 8 = MV A B 16, Shengkai Lin@CMLAB, NTUCSIE 2004 25

Motion vector prediction (III)

• For 8 × left 8 8 × × 16 partitions, MVp for the 16 = MV 16 Inter mode A ;MVp for the right 16 = MV C • For skipped macroblocks, do as 16 × Shengkai Lin@CMLAB, NTUCSIE 2004 26

Motion vector prediction (IV)

Shengkai Lin@CMLAB, NTUCSIE 2004 27

Outline

• Introduction • Baseline profile – Reference Picture Management – Slice – Inter-prediction – Intra-prediction – Deblocking filter – Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 28

Intra Prediction

• For Luma samples – 4*4 block • 9 prediction modes – 16*16 block • 4 modes – I_PCM • transmit the encoded samples w/o pred. & trans Shengkai Lin@CMLAB, NTUCSIE 2004 29

4*4 luma prediction modes

Shengkai Lin@CMLAB, NTUCSIE 2004 30

16*16 prediction modes

Shengkai Lin@CMLAB, NTUCSIE 2004 31

8*8 Chroma Prediction Modes

• Similar to 16*16 luma intra prediction except the different numbering.

– DC(mode 0) – horizontal (mode 1) – vertical(mode 2) – plane (mode 3).

Shengkai Lin@CMLAB, NTUCSIE 2004 32

Signaling Intra Prediction Modes (I) • The choices of the mode should be signaled to the decoder.

– might cost lots of bits.

• Intra modes for neighboring 4 × 4 blocks are often correlated – predict the parameters !

• The prediction mode for luma coded in Intra 16 × 16 mode or chroma coded in Intra mode is signalled in the macroblock header and the following method is not used.

Shengkai Lin@CMLAB, NTUCSIE 2004 33

Signaling Intra Prediction Modes (II) B A C • If A and B are available, C = min (A,B) • else if (neither A nor B are available) C = 2 (DC) • else C = available (A,B) Shengkai Lin@CMLAB, NTUCSIE 2004 34

Signaling Intra Prediction Modes (III) • Flag prev_intra4 × – 0: Flag rem_intra4 4_pred_mode – 1: most probable prediction mode is used.

× 4_pred_mode indicates the change of the mode Shengkai Lin@CMLAB, NTUCSIE 2004 35

Signaling Intra Prediction Modes (IV)

• If rem_intra4 × 4_pred_mode < most_probable_mode, the prediction mode = rem intra4 (rem_intra4 × × 4 pred mode • else the prediction mode = 4_pred_mode+1) Shengkai Lin@CMLAB, NTUCSIE 2004 36

Outline

• Introduction • Baseline profile – Reference Picture Management – Slice – Inter-prediction – Intra-prediction – Deblocking filter – Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 37

Deblocking filter

• Filter 4 vertical/horizontal boundaries of luma • Filter 2 vertical/horizontal boundaries of chroma • Affect up to 3 samples on the either side.

• The filter is stronger at places where there is likely to be significant blocking distortion – e.g.: such as the boundary of an intra coded macroblock or a boundary between blocks that contain coded coefficients.

Shengkai Lin@CMLAB, NTUCSIE 2004 38

Outline

• Introduction • Baseline profile – Reference Picture Management – Slice – Inter-prediction – Intra-prediction – Deblocking filter – Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004 39

Transform and Quantisation

• 3 transforms – DCT-base transform for all 4*4 residual block – Hadamard transform for 4*4 luma DC coefficient (in 16*16 intra) – Hadamard transform for 2*2 chroma DC coefficient Shengkai Lin@CMLAB, NTUCSIE 2004 40

DCT-base transform (I)

• Integer transform • Might be zero mismatch • The code part of the algo need add and shift only • Scale multiplication could be integrated into quantiser.

• Inverse operation could be down by 16-bit arithmetic.

Shengkai Lin@CMLAB, NTUCSIE 2004 41

DCT-base transform (II)

• a=1/2, b = (2/5) 1/2 , d = 1/2 Shengkai Lin@CMLAB, NTUCSIE 2004 42

Quantisation

• Combine with the scale of the DCT • Skip the rest details Shengkai Lin@CMLAB, NTUCSIE 2004 43

4*4 DC luma Transform and Quantisatoion

• |Z D(i, j) | = (|Y D(i, j) | MF (0,0) • sign (Z D(i, j) ) = sign (Y D(i, j) ) + 2f ) >> (qbits +1) Shengkai Lin@CMLAB, NTUCSIE 2004 44

2*2 DC chroma Transform and Quantisatoion • |Z D(i, j) | = (|Y D(i, j) | MF (0,0) • sign (Z D(i, j) ) = sign (Y D(i, j) ) + 2f ) >> (qbits +1) Shengkai Lin@CMLAB, NTUCSIE 2004 45

• Thank you • Next time, we ’ ll start at CAVLC • Bye … Shengkai Lin@CMLAB, NTUCSIE 2004 46