Transcript Project proposal Multi-stream and multi

Low-latency streaming of liveencoded and pre-stored video
HPL Low-latency Video
Streaming Project Meeting
Feb. 20, 02
Outline






Latency in video streaming
Long-term memory prediction and error-resilience
Delivery of live-encoded video
Delivery of pre-encoded video
Experimental results
Open issues and future work
HP Low-latency Video Streaming Project
Challenges for Low-latency Video Streaming

Undesirable latency in today’s video streaming - typical streaming
system: large receiver buffer and retransmission (10-15 second
latency)



Today’s Internet provides best-effort services with no QoS guarantee.
Hybrid video codec: Inter frames predicted from a reference frame
with MC; decoding depends on the reference
Goal of this work: better management of packet dependency to
achieve higher error-resilience and eliminate the need for
retransmission
HP Low-latency Video Streaming Project
LTM Prediction and Packet Dependency

Long-term Memory (LTM) prediction on Macroblock level



Higher coding efficiency [Wiegand, Zhang, Girod ‘99]
Higher Error-resilience [Wiegand, Färber, Girod ‘00]
Reference Picture Selection (RPS) in Annex N of H.263+
NACK
In this work:
 Extended RPS
 Dynamically manage packet dependency
 LTM prediction on the frame level
 Packetize one frame into one IP packet for transmission
HP Low-latency Video Streaming Project
Error Resilience vs. Coding Efficiency
Extension of picture types:




INTER frame: P -> P1
Extended INTER: P2, P3, … PV
INTRA: I
P1
P2
P5
230 frames of Foreman coded using
H.26L TML8.5. Average PSNR=33.4dB
I

Different types of pictures (or prediction structure) provide different
error-resilience, at the cost of coding efficiency.
HP Low-latency Video Streaming Project
Optimal Reference Picture Selection
Dv 
L( n )
p
l 1
vl
Dvl
J v  D v  Rv
vopt (n )  arg minv 1, 2,...V , J v (n )
  5e
0.1Q
5Q
34  Q
Max number of outcomes: 2

d fb 1
Optimal reference picture is selected within a ratedistortion (RD) framework – minimal cost.
HP Low-latency Video Streaming Project
Live-encoding – Results (1)

Rate-distortion performance:
V  5, d fb  7, p  0.10.
HP Low-latency Video Streaming Project
Live-encoding – Results (2)
Foreman, distortion vs.
Foreman, distortion vs.
channel loss rate.
length of LTM.
V  5, d fb  7.
d fb  7, p  0.10.
HP Low-latency Video Streaming Project
Cost of Error-resilience (1)

Error-resilience / low-latency is not free
Distortion at the encoder
V  5, d fb  7.
HP Low-latency Video Streaming Project
PSNR (dB)
Bitrate
increase for
5% loss
Bitrate
increase for
10% loss
33.4
17%
39%
35.9
20%
43%
37.8
14%
35%
Cost of Error-resilience (2)
PSNR (dB)
Distortion at the encoder
V  5, d fb  7.
HP Low-latency Video Streaming Project
Bitrate
Bitrate
increase for increase for
5% loss
10% loss
35.0
20%
52%
36.4
17%
45%
39.3
22%
46%
40.0
16%
40%
Dynamic Bit-stream Assembly of Preencoded Video

Motivation



Low complexity of the server – bit-stream assembly can be done at
real-time
Pre-encoded and pre-stored copies of video streams benefit large
number of users (at the cost of higher disk storage)
Challenges: mismatch between encoder and decoder
ENCODED
S0
S1
I I

I
I …
I P P P P P …
TRANSMITTED I
DECODED
I I
P P I
P P …
I P P I
P P …
Previous work to solve the mismatch problem: S-frame [Färber,
Girod ICIP’97 ]; SP-frame[H. 26L]
- Both at the cost of higher bitrate
HP Low-latency Video Streaming Project
Layered Prediction Structure (1)
LAYER I
TGOP=25
I
I
1) I frames define GOPs, with max length TGOP;
LAYER II
I
P5
I
(need TGOP/V versions)
P5
P5
I
V=5
P5
P5
P5
I
P5
P5
P5
P5
I
I
P5
P5
P5
P5
P5
P5
P5
2) Defines SGOP. Frames in Layer II only have two types: PV (predicted from
previous PV or I) and I.
SYNC-frame: Layer I and II frames, positioned at kV , where switching allowed.
LAYER III
P5
P5
3) Restriction: can only use previous frames in the same SGOP as a reference.
HP Low-latency Video Streaming Project
…
…
Layered Prediction Structure (2)

SYNC-frames:


Pre-encode: TGOP/V versions
encoded offline with (R,D)
values saved;
Transmit: assembly determined
within an R-D framework, with
feedback considered; requiring
V  d fb

Layer III:


HP Low-latency Video Streaming Project
Pre-encode: frames are
encoded offline with restricted
OPTS, using binary tree
structure;
Transmit: the right version used
according to the selected SYNC
frame.
Schemes Compared



Proposed pre-encoding/dynamic assembly scheme
Live-encoding with ORPS (baseline)
Simple P-I with multiple versions of bit-stream, and with feedback
I
P P P P P P P
I P P P P P
I P P P
I P
HP Low-latency Video Streaming Project
P
P
P
P
I
P
P
P
P
P
I
P
P
P
P
P
P
P
P
P
…
I…
P P I…
P P P P I…
P P P P P P I…
Pre-encoded – Results (1)

Rate-distortion performance:
V  5, d fb  5, p  0.10.
HP Low-latency Video Streaming Project
Pre-encoded – Results (2)

Only one version of Layer III pictures stored, predicted
from the leading I-frame.
V  5, d fb  5, p  0.10.
HP Low-latency Video Streaming Project
Cost of Layered Coding Structure (1)

Lossless channel
23%
30%
25%
32%
V  5, d fb  5, p  0.
HP Low-latency Video Streaming Project
Cost of Layered Coding Structure (2)

Channel loss rate=5%
V  5, d fb  5, p  0.05.
HP Low-latency Video Streaming Project
Results – Video Sequence
Pre-encoded
Mother-Daughter 100kbps
33.72dB – OPTS
31.89dB – P/I
HP Low-latency Video Streaming Project
Live-encoded
Foreman 132kbps
32.20dB – ORPS 29.73dB – P/I
Conclusions





With ORPS and dynamic management of packet
dependency, error-resilience is increased
The need for retransmission is eliminated, which
reduces latency from 10-15 second to several hundreds
of milliseconds
For pre-stored video, mismatch can be solved by storing
multiple versions of the pictures and the restricted
prediction structure
Restricted coding structure does not compromise RD
performance in lossy channels
Improved RD performance by using OPTS
HP Low-latency Video Streaming Project
Future Work (1)
Study and tradeoff between latency and RD performance

Considering retransmission, LTM prediction, and FEC




Retransmission: highest RD efficiency at the cost of high delay
LTM: lower RD efficiency, lowest delay
FEC: lower RD efficiency, medium delay
Quantify and jointly optimize delay, rate and distortion
HP Low-latency Video Streaming Project
Future Work (2)



Extend the work using path diversity
The problem: given the bandwidth, loss probability (Gilbert model)
of the multiple channels, find out the optimal picture type and the
path to use
Past related work:

Apostolopoulos et al., VCIP ‘01; INFOCOM ‘02

Lin et al., ICME ’01 (RPS on multiple paths)
HP Low-latency Video Streaming Project