Transcript ShanDefense - Shivkumar Kalyanaraman

Scalable Joint SourceNetwork Coding of Video
Yufeng Shan
Supervisors: Prof. Shivkumar Kalyanaraman
Prof. John W. Woods
April 2007
Outline
 Problem statement and motivations
 Our proposed techniques
 Fine grain adaptive FEC (FGA-FEC)
 Generalized FGA-FEC for wireless networks
 Overly multi-hop FEC scheme
 Distributed FGA-FEC over multihop network
 Conclusion and future work
2
Problem statement and challenges
 Problem:
 Simultaneously streaming video to diverse users, such as
powerful PCs, laptops and handset devices, over heterogeneous
networks.
 Challenges:
 Different users may have different video frame rate /resolution
/quality preferences.
 May be in different networks, such as high speed wired network,
multi-hop ad hoc wireless network or cellular network.
 Neither the network nor the video application can provide quality
assurances working independent of each other.
3
Conventional Approaches
 Windows media, Real player
 Send separate copies of the same bitstream to different users,
bandwidth utilization is inefficient (unicast to each user).
 Layered multicast (by Steven McCanne)
 Send different video layer to different multicast groups,
limited by number of video layers, no efficient error
correction approach.
 Proxy-based streaming
 Caches video content at local proxy disk and transcodes it for
different users, delay and computation inefficiency.
4
Our Techniques
Scalable joint source-network coding:
 Scalable video + Scalable FEC + in network adaptation
 FGA-FEC
 Generalized FGAFEC
Video
Performance
Encoding
G-FGAFEC
FGA-FEC
Adaptation
De/re-code
DSN
Video
server
DSN
 OM-FEC
 Distributed FGAFEC
 Header CRC/FEC
 Cross-layer FEC
DSN
Router
G-FGAFEC
Router
G-FGAFEC
DSN
Adaptatio
DSN Router
DSN n
Video
Performance
Router
Router
Router
FGA-FEC
De/re-code
5
Outline
 Problem statement and motivations
 Our proposed techniques
 Fine grain adaptive FEC (FGA-FEC);
 Generalized FGA-FEC for wireless networks;
 Overly multi-hop FEC scheme;
 Distributed FGA-FEC over multihop network
 Conclusion and future work
6
Background -- scalable video
We treat three types of scalability:
 SNR (aka quality, bitrate) scalability
 Spatial (resolution) scalability
 Temporal (frame-rate) scalability
7
SNR (quality) scalability
Embedded compressed file
Decoding more data corresponds to better quality
8
Resolution scalability
Embedded compressed file
Decoding more data corresponds to larger picture size
9
Frame-rate scalability
Original video
One GOP
- 16 frames
Half frame rate – 8 frames
¼ frame rate – 4 frames
1/8 frame rate – 2 frames
Begin
Embedded compressed file
End
Decoding more data corresponds to displaying frames at
10
higher rate
3-D representation of bitstream
Frame Rate
Resolution
A(4,0,2)
A(4,1,2)
A(4,2,2)
A(4,3,2)
A(4,4,2)
A(4,0.1)
A(0,0,0) A(4,1,1)
A(0,0,0) A(4,2,1)
A(0,0,0) A(4,3,1)
A(0,0,0) A(4,4,1)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(4,0,0)
A(4,1,0)
A(4,2,0)
A(4,3,0)
A(4,4,0)
A(0,0,0)
A(0,0,0) A(0,0,0)
A(0,0,0) A(0,0,0)
A(0,0,0) A(0,0,0)
A(0,0,0) A(0,0,0)
A(0,0,0)
A(3,0,0)
A(3,1,0)
A(3,2,0)
A(3,3,0)
A(3,4,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(2,0,0)
A(2,1,0)
A(2,2,0)
A(2,3,0)
A(2,4,0)
A(0,0,0)
A(0,0,0) A(0,0,0)
A(0,0,0) A(0,0,0)
A(0,0,0) A(0,0,0)
A(0,0,0) A(0,0,0)
A(0,0,0)
A(1,0,0)
A(1,1,0)
A(1,2,0)
A(1,3,0)
A(1,4,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,0,0)
A(0,1,0)
A(0,2,0)
A(0,3,0)
A(0,4,0)
Quality
Digital items in view of three forms of scalability (quality / resolution / frame-rate).
Adaptation of the bitstream is achieved by choosing a subset
of these items along user’s preferred adaptation order:
For example: SNR -> Temporal -> Spatial
11
Next, we need scalable FEC
To protect the scalable video bitstream, we have
these basic requirements for channel coding:
 High adaptivity: If part of the video bitstream is
actively dropped (scaled), FEC protecting that part
of data should also be removed
 Accuracy: near that of robust non-scalable video
 Efficiency: Low computational cost
12
Fine Grain Adaptive FEC
Goal:
 To encode a video to facilitate efficient and
precise adaptation of the encoded bitstream at
intermediate overlay nodes for diverse users.
Main idea:
Extend existing approaches (PET, MD-FEC) to work with
scalable video and perform the adaptation in the network.
13
Multiple Description FEC (MD-FEC)
MSB
LSB
by Puri and
Ramachantran
If any i out
of N packets
are received,
the decoder
can decode
up to Ri.
Our work extends this concept to provide fine-grained scalability
of FEC, applied to scalable video
14
FGA-FEC overview
FGA-FEC uses overlay infrastructure to assist video streaming to
heterogeneous users simultaneously by providing light weight
support at intermediate overlay nodes.
Adaptatio
n
30/C/2M A
C
30/C/1M
E
30/C/3M
Server
Encoding
DSN
DSN
30/C/3M B
DSN
D 15/Q/1M
30/C/1M
Adaptatio
n
15/Q/384k
G
F 15/C/384k
Adaptatio
n
A-G : users and their requirements (frame rate/resolution/bitrate)
DSN : Data service node, which aggregates video requirements to server
and performs data adaptation for users, without transcoding.
15
FGA-FEC Encoding
MSB
S1
Ri
Ri-1
RN-1
one GOP video bitstream
...
Si
...
Description 1
B1
Bi+1
B2i+1
SN(2)
Description 2
B2
Bi+2
...
...
SN(3)
B3
Bi+3
...
...
...
...
Description 3
...
...
...
Si(i)
FEC
...
...
Bi
B2i
B3i
FEC
FEC
FEC
...
...
...
FEC
FEC
FEC
...
Sect. i
S1(1)
S2(1)
...
Si(1)
...
SN(1)
FEC
S2(2)
...
Si(2)
...
FEC
FEC
...
Si(3)
...
...
...
FEC
FEC
FEC
FEC
...
...
...
...
SN
view of Section S i
...
Sect. 2
FEC
LSBEnlarged
B1
...
Sect. 1
FEC
RN
...
FEC
...
Bi
...
B2i
...
...
Bmi
...
Si(1)
...
Si(2)
...
Si(3)
Sect. N
...
SN (i) Description i
SN(i+1) Description i+1
...
SN(N) Description N
RS encoding
R1
R0
...
...
...
...
...
...
...
...
...
Bmi
Si(i)
FEC
...
FEC
 Bitstream is divided into N sections from MSB to LSB




Each section is further split into small blocks
RS(N,i) codes are applied at block level to section Si, vertically
Each block column is independently accessible
one description = one network packet
16
FGA-FEC – optimal rate allocation
FEC assignment is optimized by minimizing the expected
distortion over the channel with packet loss probability p,
available bandwidth B.
N
E[ D( R)]   qi D( Ri )
i 0
Subject to:
0  R1  R2   RN

 Rtotal  B
 R  R  r  i, i  [1, N ]
i 1
i
 i
The result of the
optimization is the
video source-rate
breakpoints Ri
where qi is the probability of receiving i out of N packets, Rtotal is the total
bandwidth occupied by both FEC and video data. Also ri is the source rate of
17
section Si. N is the number of descriptions encoded for each GOP
FGA-FEC adaptation at intermediate nodes
One description
At DSN
Adapted description
one GOP
Adapted
Adapted
GOP
GOP
One
description
One
description
One
One
GOP
GOP
DSN
DSN
f f
Shorten
& drop
Two adaptation methods are used in FGA-FEC :
 Direct truncation: only shorten each packet in a GOP by removing
unwanted blocks.
 FGA-FEC adaptation: adapt the encoded GOP by a combined
shortening and/or dropping packets.
 An algorithm is proposed to near optimally adapt the encoded GOP for
available bandwidth and user preference
18
 No video or FEC transcoding, only a packet shorting/dropping
Simulation-- FGA-FEC adaptation
vs. optimal decoding/recoding
500 K
1
User 1
2
User 2
8
6
User 6
79
User7
600 K
S
D
Video
Server
DSN
1000 K
1100k
 Comparison of FGA-FEC, optimal decoding/recoding solution, and
direct truncation at different available bandwidths;
 In FGA-FEC, MC-EZBC encoded GOP #7, Foreman CIF is
adapted from 1100 Kbps to satisfy different users;
 FGA-FEC optimization is based on 15% loss rate at source, N=64.
19
ns-2 Simulation-- FGA-FEC vs.
MD-FEC transcoding
Source
1.64
MD-FEC transcoding:
1.77
2
1.32
Group2
FEC decode/recode at each
intermediate node
Group1
1
1.51
4
1.25
1.45
5
3
1.38
7
6
1.44
8
1.32
9
Group3
1.52
1.48
11
10
FGA-FEC adaptation:
0.66
12
0.66
Encode at server, adaptation in
the intermediate nodes
13
0.73
0.60
15
Group4
14
Want to see the received video quality at User 4 and User 12,
Link loss probability is 0.01, N=64, Foreman, CIF
20
ns-2 Simulation-- FGA-FEC vs.
MD-FEC transcoding
Receiver 4
Receiver 12
FGA-FEC is 0.01dB lower
FGA-FEC is 0.4 dB lower
The quality loss is due to the
adaptation precision.
21
ns-2 Simulation-- FGA-FEC vs.
Layered multicast
Source
Layered multicast:
Base layer bit rate 0.6, and
enhancement layers 0.32,
0.42, 0.32 Mbps,
respectively
FGA-FEC adaptation:
Encode at server, adaptation in
the intermediate nodes
1.64
Group1
1
1.77
2
1.32
Group2
1.51
4
1.25
1.45
5
3
1.38
7
6
1.44
8
1.32
9
Group3
1.52
1.48
11
10
0.66
12
0.66
13
0.73
0.60
15
Group4
14
Want to see the received video quality at User 5 and User 7,
Link loss probability is 0.01, N=64, Foreman, CIF
22
ns-2 Simulation-- FGA-FEC vs.
Layered multicast
Receiver 5
Receiver 7
Layered multicast can subscribe
up to two layers
Layered multicast can subscribe
up to three layers
The quality loss of layered multicast is due to
coarser available layers.
23
Outline
 Problem statement and motivations
 Our proposed techniques
 Fine grain adaptive FEC (FGA-FEC);
 Generalized FGA-FEC for wireless networks;
 Overly multi-hop FEC scheme;
 Distributed FGA-FEC over multihop network
 Conclusion and future work
24
Generalized FGA-FEC for wireless
In addition to congestion related packet losses,
in wireless must deal with:




channel bit errors due to channel fading and noise;
large bandwidth fluctuations, and
intermediate node computational capability constraint
limited maximum transmission unit (MTU) size.
25
Generalized FGA-FEC for wireless
R0
R1
S1
RS encoded in block level
Sect. 1
S1(1)
FEC
FEC
Rk
Rk-1
...
Sk
...
Sect. k
...
Sect. N
S2(1)
S2(2)
FEC
...
...
...
Sk(1)
Sk(2)
Sk(3)
...
...
...
SN (1)
SN (2)
SN (3)
FEC
FEC
...
...
...
Sk(k)
FEC
...
FEC
...
Sect. 2
...
FEC
FEC
R N-1
FEC
...
...
...
...
FEC
...
RN
SN
Embedded bitstream
BCH
Description 1
CRC
FGA-FEC + BCH/CRC Encoded Packet1
BCH
Description 2
CRC
FGA-FEC + BCH/CRC Encoded Packet2
BCH
Description 3
CRC
...
SN (k) Description k
SN(k+1) Description k+1
FGA-FEC + BCH/CRC Encoded Packet3
…
BCH
CRC
FGA-FEC + BCH/CRC Encoded Packet k
BCH
CRC
FGA-FEC + BCH/CRC Encode Packet k+1
...
SN(N) Description N
…
BCH
CRC
Generalized FGA-FEC = FGA-FEC
FGA-FEC + BCH/CRC Encoded Packet N
+
Bit level protection
A kind of Product codes (extend Sherwood and Zeger’s codes):
 Column codes: Reed Solomon for packet loss
 Row codes: CRC + BCH for bit errors
A BCH code is represented as: BCH(n,k,t): n=2m-1, n-k<=mt
26
Optimal product code assignment
Find a concatenated column RS code assignment cc and row
BCH code assignment cr from a set of RS codes CRS and
BCH codes CBCH BCH(n,k,t), such that the end-to-end video
distortion over a lossy channel is minimized.
cc , cr  arg min E[ D | CRS , CBCH , CCRC ]
cc C RS ,cr C BCH
Subject to channel
rate constraint:
RS  RRS  RCRC  RBCH  B
where Rs , RRS , RCRC and RBCH are rates allocated to the video
source, RS parity bits, CRC and BCH check bits, respectively.
Here B denotes the maximum available channel bandwidth
27
Fast optimization method
n=8191, m=13
Given a BCH code BCH(n,k,t)
 n j
PBCH (C )     pb (1  pb ) n j
j 0  j 
t
Probability of correctly
decoding BCH code
Near optimal points,
search starts here,
Exhaustive search results
it can find the optimal
solution within a few
iterations
GOP #7 of Foreman, CIF, N= 64,
pdrop=0.05, BER pb varies
28
Multi-cluster descriptions coding
 Single-cluster description coding: one GOP  N descriptions
 Multi-cluster descriptions coding: one GOP  Multiple of N
descriptions due to:
 Limited network packet size;
 Easier adaptation, such as encode each temporal layer as
one cluster as below.
One GOP of scalable bitstream
Low frame
rate
Mid frame
rate
High frame
rate
Example of encoding one GOP into 3 clusters, each
cluster is coded with product codes
29
Simulation – FGA-FEC vs. MD-FEC in wireless
Channel changes over time between node 2 and 3
Consider two adaptation orders:
SNR->Temporal, SNR-> Spatial
Frame rate scaling
Resolution scaling
pdrop=0.05 at node2
30
Outline
 Problem statement and motivations
 Our proposed techniques
 Fine grain adaptive FEC (FGA-FEC);
 Generalized FGA-FEC for wireless networks;
 Overly multi-hop FEC scheme;
 Distributed FGA-FEC over multihop network
 Conclusion and future work
31
Motivations -- an example
Given an overlay path:
(B6 , P6 , RTT6 )
(B1, P1, RTT1 )
N2
N1
Sender
N3
Receiver
N5
N4
OM-FEC vs. Traditional FEC
The parameter of each hop
Hop
1
2
3
4
5
6
FEC
method
End
to end
Hopby-hop
OM-FEC
B(i)
660K
625K
615K
700K
900K
1100K
data rate
(goodput)
529K
594 K
594K
P(i)
2.5%
3%
3.5%
2.5%
1.5%
1%
Path
loss rate
14%
14%
14%
Segment 1
Segment 2
Segment 3
32
OM-FEC algorithm
OM-FEC partitions a given overlay path into segments:
N1
…
Ni
i
…
Nh
…
Nj
L
…
Nn
M
By solving the following: minimize(Nsegement)
Subject to:
E[ DOM FEC ]  E[ Dhop by hop ]
The i-th node is boundary node, if the two inequalities hold:
min{Bstart , …, Bi}  (Bdata + BFEC(start>i))
min{Bstart, …, Bi+1}< (Bdata +BFEC(start>(i+1))
FEC that should be added to the Segment is BFEC(start ->i),
33
Video Simulations
Topology: loss rate on each hop (L1–L10) random within [0.4%-1.2%]
 H.263+ encoded Foreman
CIF, at bitrate 1530 Kbps;
 Available bandwidth for
each hop: 1656 Kbps
Average number of segments
partitioned is only 2 vs. 10 in
hop-by-hop FEC
34
PlanetLab Experiments
video.testbed.ecse.rpi.edu
planetlab1.flux.utah.edu
planetlab1.cs.cornell.edu
planetlab-1.cmcl.cs.cmu.edu
nima.eecs.berkeley.edu
 Video: Foreman QCIF, H.263+, 512 Kbps, 30 fps, I frame every second
 Packet-loss rate from Utah to CMU is set to 5%, other links are set to 1%.
 The available bandwidth from Utah to CMU is also upper bounded to
550 Kbps.
 Two paths: 1): with nodes shown at map
35
2): Add Node 4: planet1.ecse.rpi.edu at last hop with 1% loss rate
45
45
40
40
35
35
30
30
PSNR (db)
PSNR(db)
OM-FEC experimental results
25
20
25
20
15
15
10
10
our OM-FEC Scheme
5
End-to-End Scheme
our OM-FEC Scheme
5
End-to-End Scheme
0
0
0
50
100
150
200
250
300
Frame Number
Video PSNR of OM-FEC scheme vs.
e2e scheme (4 virtual links)
350
0
50
100
150
200
250
300
350
Frame Number
Video PSNR of OM-FEC scheme vs.
e2e scheme (5 virtual links)
Results show:
• OM-FEC outperforms e2e FEC scheme
• As more congested links are encountered, gain due to OM-FEC increases
Large PSNR loss of the e2e scheme is due to the dependency of bitstream,
one packet drop could cause later packets of the same GOP useless .
36
Outline
 Problem statement and motivations
 Our proposed techniques
 Fine grain adaptive FEC (FGA-FEC);
 Generalized FGA-FEC for wireless networks;
 Overly multi-hop FEC scheme;
 Distributed FGA-FEC over multihop network
 Conclusion and future work
37
Problem statement and our solution
 In FGA-FEC
We assume no congestion between DSNs
 Remaining problem:
– What if there is congestion between DSNs? For
example, in a rate constrained multihop ad hoc
wireless network.
 Our solution – distributed FGA-FEC:
- FEC coding and optimization algorithm will be run
at DSNs in a distributed way to serve the diverse
users.
38
The optimization algorithm
Problem:
Solution:
Minimize the e2e distortion for each user:
N
E[ D( Ri )]   qi D( Ri )
i 0
Subject to:
 0  R1  R2    RN

 Rtotal  B
 R  R  r  i r  0,  i  [1, N ]
i 1
i
i
 i
where
N
qi   (1  p) i p N i
i
N
Rtotal
N
N

Ri  i Ri
i 1 i (i  1)
i 1
Use Lagrange multiplier method:
N
N
i 1
i 1
F ( R1 , R2 ,...,RN ,  )   qi D( Ri )   ( i Ri  B)

 D ( R1 )
 1
 R
q1
1




D
(
R
)

i

 i 
 Ri
qi


 D ( R )

N

 N 
qN
 RN
N

   i Ri  B
i 0

Given one value of λ,
D( Ri )

 i 
Ri
qi
corresponds to the point on the
D(R) curve with slope equal to  
i
qi

Solution can be found by searching over λ
39
The Distributed FGA-FEC Algorithm
To reduce the overall computation while maintain the best possible
video quality, we propose two approaches:
 A coordination method between optimization processes running
at adjacent nodes to reduce the optimization computation.
 Full search, search with previous GOP, search with neighbor
 Applied the idea of OM-FEC to reduce the number of FGA-FEC
decode/recode nodes, do FEC computation at selected DSNs
40
Simulation - reduce optimization computation
Test number of λ iterations need to reach optimization stop points,
Foreman, CIF.
The proposed coordination between adjacent nodes can reduce the
optimization computation requirement.
41
Simulation– distributed FGA-FEC
vs. Hop-by-hop FEC decode/recode
Source
Hop-by-hop FGA-FEC
decode/recode:
1.15
Group1
1
1.25
1.32
Group2
1.51
4
1.25
1.15
5
FGA-FEC decode/recode
at each intermediate node
2
3
1.38
7
6
1.44
8
1.32
9
Group3
1.32
1.48
11
10
Distributed FGA-FEC:
Identify the congested links and
select appropriate nodes to do
FGA-FEC decode/recode.
0.66
12
0.66
13
0.73
0.60
15
Group4
14
Want to see the received video quality at User 5 and User 12
Link loss probability is 0.01, N=64, Foreman, CIF
42
Simulation– distributed FGA-FEC vs. Hopby-hop FGA-FEC decode/recode
Receiver 5
Receiver 12
The two schemes deliver similar video quality, but
distributed FGA-FEC uses fewer FEC computation
43
nodes 3 vs. 6 at FEC decode/recode
Measured CPU time
We measure the CPU time using FGA-FEC adaptation and FGA-FEC
decode/recode in a Dell PC with 1.6 GHZ CPU, 256 M memory, running Red
Hat linux 8.2, N=64.
Scheme performed at DSN
CPU time (ms)
FGA-FEC decode/recode
FGA-FEC adaptation
52.6
2.9
FGA-FEC direct truncation
10-2
Distributed FGA-FEC can greatly reduce the computation
44
burden, while can deliver near optimal video quality
Conclusions
 Fine grain adaptive FEC (FGA-FEC)
 to encode and adapt a scalable video
 Generalized FGA-FEC for wireless networks
 to do encoding and adaptation over wireless networks
 Overly multi-hop FEC scheme
 to efficiently utilize one congested path
 Distributed FGA-FEC over multihop network
 to stream video over a congested heterogeneous networks
 Header error correction
 To improve the effective throughput of wireless networks
 Cross-layer FEC
 To joint optimize protocol with application layer FEC
45
Future work
 Evaluate FGA-FEC over DCT-based
standard coders such as H.264/AVC and
SVC
 Extend the distributed FGA-FEC to work
with Spatial and Temporal scalability
 Extend FGA-FEC idea to multi-point
video conferencing
46
Publication and submissions
[1] Yufeng Shan, Ivan Bajic, John W. Woods and Shivkumar Kalyanaranman “ Scalable video streaming
with fine grain adaptive forward error correction” submitted to IEEE trans. CSVT, 2006
[2] Su Yi, Yufeng Shan, Shivkumar Kalyanaraman and Babak Azimi-Sadjadi, "Video streaming over 802.11
Ad Hoc wireless ntworks with header error protection", submitted to Ad Hoc Networks, 2006
[3] Yufeng Shan, Ivan V. Bajic, Shivkumar Kalyanaraman and John W. Woods, “Overlay multi-hop FEC
scheme for video streaming,” Signal Processing: Image Communications Vol. 20/8, 2005
[4] Yufeng Shan, John. W. Woods and Shivkumar Kalyanaraman “Fine grain adaptive FEC over wireless
networks”, submitted to ICIP 2007
[5] Su Yi, Yufeng Shan, Shivkumar Kalyanaraman and Babak Azimi-Sadjadi, "Header error protection for
multimedia data transmission in wireless AdHoc networks", ICIP, 2006
[6] Yufeng Shan, Ivan Bajic, Shivkumar Kalyanaraman, and John W. Woods, "Joint source-network error
control coding for ccalable overlay streaming," ICIP, 2005
[7] Yufeng Shan, Su Yi, Shivkumar Kalyanaraman and John.W. Woods, "Two-Stage FEC scheme for
scalable video transmission over wireless networks" SPIE Communications/ITCom, Multimedia Systems
and Applications, Oct. 2005
[8] Yufeng Shan, Ivan Bajic, Shivkumar Kalyanaraman, and John W. Woods, "Overlay multi-hop FEC
scheme for video streaming over peer-to-peer networks," ICIP, 2004
[9] Yufeng Shan and Shivkumar Kalyanaraman “Hybrid video downloading/streaming over peer-to-peer
networks,” ICME, 2003
47
Thanks
48