Transcript Combat Inter-Symbol Interference with Equalization

Motivation
• Video Communication over
Heterogeneous Networks
– Diverse client devices
– Various network connection
bandwidths
• Limitations of Scalable Video
Coding Schemes
– Limited layers supported
– No video format changes
• Video Transcoding Provides
Dynamic Solutions
– Channel bandwidth adaptation
– Video coding format adaptation
1
Challenges in Video Transcoding
Video Transcoder
... 01010111
Input Compressed
Video Stream
Decoding
(Partially)
Video
Manipulation
Entropy
Encoding
...1011011
Output Compressed
Video Stream
• Improve Efficiency of Video Transcoding
– Large data volume
– High computational complexity
• Optimize Visual Quality for a Given Bit Rate
– Human vision system (HVS) based video transcoding is desirable
2
Proposed Solutions
• Exploit Foveation Property of the HVS in Video Transcoding
Foveation Embedded Video Transcoder
... 01010111
Uniform Resolution
Compressed Video
Decoding
(Partially)
Video
Manipulation
& Foveation
Video
Re-encoding
...1011011
Foveated Video
Stream
• Develop Fast Algorithms for Video Transcoding
– DCT-domain foveation filtering technique
– Fast algorithms for DCT-domain inverse motion compensation
• Local bandwidth constrained DCT-domain inverse motion compensation
• Look-up-table based DCT-domain inverse motion compensation
3
Foveation
• The Human Eye Samples Visual Field Non-uniformly
– The highest sampling resolution is at Fovea
– The sampling resolution decreases rapidly as away from Fovea
Cells per degree
• Retinal Images are Inherently Non-uniform in Spatial
Resolution
Eccentricity (deg)
Eccentricity (left eye)
4
Foveation Modelling
• Foveated Contrast Threshold [Geisler & Perry 98]
e  e2
)
e2
• f: Spatial frequency (cyc/degree)
• e: Retinal eccentricity(degree)
• a: Spatial frequency decay constant
• Foveated Cut-off Frequency fc
fc 
e2
1
ln
a(e  e2 )
CT0
• Spatial Frequencies Beyond
the Cut-off Frequency is
Invisible (Foveated Image)
a  0.106 , e2  2.3,
1
1
 CT 0 
.
76
64
•
e2: Half-resolution eccentricity
• CT0: Minimum contrast threshold
• CT: Contrast threshold
Local cut-off frequency (cyc/deg)
CT ( f , e)  CT0 exp (a f
Image size: 512 x 512
Unit of v: image height
Pixel position relative to foveation point (unit: pixel)
5
Foveated Images
JPEG-coded Uniform Image (168KB)
JPEG-coded Foveated Image (136KB)
Foveation point is marked by ‘X’
6
Foveated Contrast Sensitivity Function (FCSF)
• Foveated Contrast Sensitivity Function (FCSF)

e  e2 
1
1

FCSF( f , e) 

exp  a f
CT CT0
e2 

Normalized contrast sensitivity of human eye
• Shape the Compression Distortion According to FCSF
Image size: 512 x 512
Viewing distance: 3 times the image height
Distance from foveation point (unit: pixel)
7
Video Transcoding Architecture
• Open-Loop Video Transcoding
– Simple and fast
– Error drift
VLD
Rin
I
P
P
Delay
Bit Allocation
Analysis
VLD
VLC: Variable Length Coding
VLD: Variable Length Decoding
Requantization
VLC
Rout
P
Transcoding Error Propagation
8
Drift Free Video Transcoders
• Cascaded Pixel Domain Video Transcoding
– Low efficiency
– Long delay
Rin
Complete
Decoding (Q1)
Rout
Encoding
(Q2)
• Fast Pixel Domain Video Transcoding
– Save motion estimation, one frame memory and one IDCT operation
• Fast DCT-Domain Video Transcoding
– No IDCT-DCT operations; Lower data volume
– DCT-domain inverse motion compensation is complex (Research topic)
Rin
IQ1
+
Rout
Q2
+
IQ2
DCT
+
MC
FM
-
Rin
IQ1
+
+
DCT-Domain
Inverse Motion
Compensation
Rout
Q2
IQ2
+
-
IDCT
Fast Pixel Domain Video Transcoder
Fast DCT Domain Video Transcoder
9
Foveation Embedded DCT Domain Video Transcoding
DECODER
ENCODER
Intra-coded frame
MVs
R1(n)
VLD
IQ1
+
DCT-Domain
Foveation
+
+
DCT-Domain
MVR
Q2
MVs
MVs
FM
DCT-Domain
IMC
R2(n)
Foveation
Point
Selection
VLC
IQ2
+
DCT-Domain
IMC
+
+
FM
IQ :
Q :
FM :
MVs :
Inverse Quantization
Quantization
Frame Memory
Motion Vectors
VLD : Variable Length Decoding
VLC : Variable Length Coding
MVR : Motion Vector Refinement
IMC : Inverse Motion Compensation
10
Foveation Filtering
• Pixel Domain Foveation Filtering Technique [Lee, 99]
– High computational complexity
11
DCT-Domain Foveation Filtering
• DCT-Domain Block Mirror Filtering [Rao, 90]
• Pros
1-D signal
0
Block
mirroring
– Significantly simplified
– Combine with inverse quantization
– Easy to parallelize
f
• Cons
fˆ
– Blocking artifacts
0
~ ~
ˆ
h f  h  f
DCT
~
h
h
Filter Kernel
H. R. Sheikh, S. Liu, B. L. Evans and A. C. Bovik,
“Real-Time Foveation Techniques for H.263 Video
Encoding in Software”, ICASSP 2001.
~
f
DCT of f
12
Multipoint Video Conferencing
Internet
Multipoint Control Unit
(MCU)
from conferee #1
VLD
BUF
from conferee #2
VLD
...
...
BUF
from conferee #N
BUF
Video
combiner
Foveation
Embedded DCT
domain video
transcoder
to conferee #2
BUF
...
BUF
to conferee #1
to conferee #N
VLD
BUF
H. R. Sheikh, S. Liu, Z. Wang and A. C. Bovik,“Foveated Multipoint Videoconferencing at Low Bit Rates”,
ICASSP 2002, accepted.
13
Simulation Results
Uniform resolution video at 256 kb/s
Foveated video at 256 kb/s
Foveation point is at the center of the upper-left quadrant
14
Foveation Point Selection
• Interactive Methods
– Mouse, eye tracker
– Reverse channel is assumed
– End to end delay is assumed short enough
• Automatic Methods
– Fixation points analysis (Very challenging)
– Application oriented methods
• DCT-Domain Human Face Detection [Wang & Chang, 97]
– Skin color region segmentation
– Face template constraint
– Spatial Verification
15