Transcript MINMAX Optimal Video Summarization with Frame Skip Constraint Zhu Li

MINMAX Optimal Video
Summarization with Frame
Skip Constraint
1,2Zhu
Li
3Guido Schuster
1Aggelos K. Katsaggelos
2Bhavan Gandhi
1Department
of ECE, Northwestern
University, Evanston, USA
2Motorola Labs, Schaumburg, USA
3Hochschule fur Technik Rapperswil
(HSR), Switzerland
Outline
Introduction
Definitions and Assumptions
Formulations
Optimal and Heuristic Solutions
Solution to the Dual Problem
Frame Distortion Metric
Simulation Results
2
Introduction
Why Video Summary
• View time constraint, a shorter version is more desirable in
some applications, for eg, security, mil and entertainment apps
• Storage, Bandwidth and Energy Constraint, a shorter version
with better SNR quality conveys more useful information.
Solution:
• Video Shot Segmentation,
• Key frames selection within a video shot
• Previous works: clustering on visual features, curve simplification,
utility maximization, etc.
3
An application scenario
2G/2.5G data channel
6kpbs ~ 15kpbs
•Operating at voice rate
•Reasonable visual quality
•Synchronized with audio if so wish
4
Definitions and assumptions
Video Sequence (n-frame): V= { f 0 , f1 , f n1}
Video Summary (m-frame): S  { f l0 , f l1 , f lm1 }
Reconstructed Sequence: VS '  { f 0 ' , f1 ' , f n1 ' }
Summary Distortion: D ( S )  max d ( f k , f k ' )
k
Summary Rate: R ( S ) 
5
m
n
Rate-Distortion Formulation
Summarization as a rate-distortion optimization
problem
•MDOS formulation:
S *  arg min D( S ), s.t. R ( S )  Rmax
S
•MROS formulation:
S *  arg min R ( S ), s.t. D( S )  Dmax
S
•Frame skip constrained:
S *  min D( S ), s.t. R( S )  Rmax , and lk  lk 1  Kmax  1,  k
S
S *  arg min R( S ), s.t. D( S )  Dmax , and lk  lk 1  K max  1, k
S
6
Dynamic Programming Solution
Dynamic Programming
• Exhaustive search not practical.
• Segment distortion state and rate definitions:
Dlltt 1  max d ( f lt , f j )
j[ lt , lt 1 1]
Rlltt 1

r ( f lt )  1,


,
if Dlltt 1  Dmax 



otherwise

MROS summary frame selections: {l0, l1, … lm-1}, l0=0.
min {R0l1  Rll12    Rlnm1 }
l1 ,l2 ,lm 1
7
The Algorithm
MROS algorithm:
• The recursion:
J lt 1  min {R0l1  Rll12    Rlltt1  Rlltt 1 }
l1 ,l2 ,lt
 min {J lt  Rlltt 1 }
lt
• The initial condition:
1,
J l1  
,
8
if D0l1  Dmax 

else

The DP Trellis
MROS algorithm:
MinMax DP trellis: n=6
6
5
5
4
4
frame k
frame k
MinMax DP trellis: n=6
6
3
2
2
1
1
0
0
1
2
3
4
5
epoch t
n=6, no skip constraint
9
3
6
0
0
1
2
3
4
5
epoch t
n=6, max skip=3
6
An Example Solution
•Start from R10
•Dmax and Kmax Constrained
state transition
•Stop when the final virtual
frame fn is reached.
•Multiple optimal solutions
S*={f0, f4, f7}, { f0, f4, f6} … {
f0, f2, f5}
minmax summary: n=8 Dmax=2.40 Kmax=6
8
7
6
5
frame k
f
r
a
m
e
4
3
2
k
1
0
1
2
3
4
5
epoch t
epoch t
10
6
7
8
Frame Distortion Metric
• An elusive problem
• Application specific
• PCA analysis to find the subspace spanned by a large set
of video frames
• Weighted Euclidean distance in PCA space as frame
distortion metric
• Works well with subjective perception.
11
Simulation results :
summary frames
10
d(f k, fk-1)
8
6
4
2
0
0
50
100
150
100
150
summary distortion
7
6
d(f k, fk)
5
4
3
2
1
0
0
50
MROS: “foreman” sequence, frames 150~299
n=150, Dmax = 6.4, Kmax= no constraint
Results: m=25
12
Simulation results: skip constrained:
summary frames
10
d(f k, fk-1)
8
6
4
2
0
0
50
100
150
100
150
summary distortion
7
6
d(f k, fk)
5
4
3
2
1
0
0
50
MROS: “foreman” sequence, frames 150~299
n=150, Dmax = 6.4, Kmax= 10
Results: m=32.
13
A Heuristic Solution
Distortion Constrained Skip (DCS) algorithm:
Set L=0, add fL to the summary S
FOR k=1 TO n
IF d(fL, fk) > Dmax
L=k,
add fL to the summary S
END
END
DCS is the optimal solution if:
Dkj p  Dkj , for j  p  k
14
Solution to the MDOS formulation
Bi-Section searching on the operational R-D function:
1
• The ORD is non-increasing
(lemma 1.)
0.9
0.8
0.7
•Bi-section search on the
distortion, and solve for each
distortion with MROS solution.
Rate m/n
0.6
Rmax
0.5
0.4
0.3
•Will converge to the optimal
solution D*.
0.2
0.1
0
0
5
10
15
20
Dmax
D*
15
25
30
35
40
Conclusion and Future Work
• The solution is rate-distortion optimal
• The heuristic DCS algorithm is near optimal most of time
and quite efficient
• Summaries operates at voice rate suitable for 2G and
2.5G deployment (demo):
• “bond” sequence at 13.2kpbs, Dmax=6.0
• “bond” sequence at 11.7kpbs, Dmax=8.0
• “bond” sequence at 9.7kpbs, Dmax=12.0
• “foreman” sequence at 10.8kpbs, Dmax= 6.0
• “foreman” sequence at 9.4kpbs, Dmax=8.0
• “foreman” sequence at 8.4kpbs, Dmax=12.0
•Future work: bit constrained MINMAX summarization.
16
Questions ?
?
…….
brigado !
17