Sparsity-based Image Interpolation With Nonlocal Autoregressive Modeling

Weisheng Dong (

董

伟生 )

School of Electronic Engineering, Xidian University, Xi’an, China Sample codes available: http://see.xidian.edu.cn/faculty/wsdong W. Dong, L. Zhang, R. Lukac, and G. Shi, “Sparse representation based image interpolation with nonlocal autoregressive modeling,”

IEEE Trans. Image Process.

, vol. 22, no. 4, Apr. 2013 5/1/2020 1

Outline

 Background: image interpolation  Edge-directed methods  Sparsity-based methods  Sparse image interpolation with nonlocal autoregressive modeling  The nonlocal autoregressive modeling  The clustering-based sparsity regularization  Experimental results  Conclusions 2

Image Upconversion is Highly Needed

Small images SD videos Related applications: deinterlacing: Display small images / video in HD devices 5/1/2020 3

Challenge

Jagged artifacts Original HR image 2X upscaling LR image Challenge: How can we upconvert the image without annoying jagged artifacts?

5/1/2020 4

Previous work: edge-based methods

 Linear interpolators: bilinear, bicubic  Blurring edges, annoying jagged artifacts  Edge directed interpolators (EDI) [TIP’95]  Interpolate along edge directions  Difficult to estimate the edge directions from LR image Smooth variations  New EDI [TIP’01], Soft-decision adaptive inter. [TIP’08]  Local autoregressive (AR) model of LR image  Ghost artifacts due to Wrong AR models 5/1/2020 5

Artifacts of Edge-based methods

Artifacts Original 5/1/2020 Bicubic NEDI [TIP’01] SAI [TIP’08] 6

Previous works: sparsity-based methods

 Dictionary learning based method, [TIP’10] ; Nonlocal method, [ICCV’09];  Failed when the LR image is directly downsampled Bicubic ( 32.07 dB ) ICCV09 ( 29.87 dB ) Bicubic ( 23.48 dB ) ICCV09 ( 21.88 dB ) 5/1/2020 7

Previous works: sparsity-based methods

 Upscaling via solving a sparse coding problem:  ˆ 

y



D

  2 || + 2   ||  || }, 1

x

   ˆ  Failed too, ringing and zippers artifacts 5/1/2020 Original DCT ( 32.49 dB ) PCA ( 32.50 dB ) 8

Coherence issue of sparsity method

 Two fundamental premises of sparsity recovery method: 

Incoherence sampling

: sampling operator

D Φ

and the dictionary should be

incoherent

. The coherence value:  

n

 max 1   | 

d

k

, 

j

 ，

n



Sparsity

: the original image should have sparse representation over

Φ



Problem:

The downsampling matrix

D

is often coherent with common dictionaries, e.g., wavelets, K-SVD dictionary D. Donoho and M. Elad, PNAS, 2003. E. Candes, et al., “An introduction to Compressive sensing,” IEEE SPM, 2008 5/1/2020 9

Contributions

 Propose a new image upscaling framework: combining edge-based method and sparsity-based method  Suppress the artifacts regularization of edge-based method using sparse  Overcome the coherence issue modeling by nonlocal autoregressive  Introducing a structured sparse regularization model  Improve the sparse regularization performance 5/1/2020 10

Local autoregressive (AR) modeling

 Local AR image modeling

x

ˆ

i



t

4   1

x



t

 Compute the AR model: least-square min 

i n

  1 ||

x i



t

4   1

x



t

2 || 2  min ||   || 2 2  Exploiting the local image structure  AR-based image interpolation methods:  NEDI, [TIP 2001] (Over 1300+ citation)  SAI, [TIP 2008] (Previous state-of-the-art method) 5/1/2020 11

Nonlocal self-similarity

5/1/2020 12

Nonlocal NAR modeling (NARM)

 Nonlocal regression:

x i

 

j j w x i i j

 Patch matching: nonlocal similar neighbor selection

G i



x

i



x

j

|| 2 2 

T

}  Regularized least-square:

i

 arg min 

i

x

i



Xw

i

2 2  

w

i

2 2 

i

 (  

I

)  1

i

 Nonlocal AR modeling:

x

=

S

x

+

e x



y

DS

x



e

5/1/2020 13

NARM based image interpolation

 Improved objective function 

y



DS

  2 2 +  

R



s t

y



D

  

S

: impose structural constraint on

Φ

 The benefit of the NARM  The coherence value between new sampling matrix

DS

and

Φ

is significantly decreased  Coherence values (8x8 patch): wavelets: 1.20~4.77

; Local PCA: 1.05~4.08

5/1/2020 14

NARM based image interpolation

 Local PCA bases: (a) (b) (c) (d) (e) (a)  1 =3.78 vs.  2 = 1.60

; (b)  1 =3.28 vs.  2 = 1.80

; (c)  1 =3.22 vs.  2 = 2.08

; (d)  1 =6.41 vs.  2 = 1.47

; (e)  1 =7.50 vs.  2 = 1.69

. μ1 -- coherence values between

D

μ2 -- coherence values between

DS

and

Φ

and

Φ

5/1/2020 15

Structural sparse regularization

 Conventional sparsity regularization 

y



DS

  2 2 +  

N i

 1 

i

1 }, . .

y



D

   Cannot exploit structural dependencies between nonzero coefficients  Clustering-based sparse representation (CSR) [Dong, Zhang, et al., TIP 2013]  Unify structural clustering and sparse representation into a variational framework 5/1/2020 16

Clustering-based Sparse representation

 Motivation: Nonzero sparse coefficients are NOT randomly distributed 5/1/2020 The distribution of the sparse coefficients associated with the 3 rd atoms in the K-SVD approach 17

Clustering-based Sparse representation

 The clustering-based regularization  Exploiting the self-similarity: min 

k k K

  

k

Φα

i

 

k

2 2  min 

k k K

  

k

Φα

i



Φ



k

2 2  Unifying the clustering-based sparsity and the learned local sparsity  arg min 

y



DS

Φ

 2 2 + 

α

1  

k K

   

k

Φα

i



Φ



k

2 2

local sparsity Clustering-based regularization

5/1/2020 18

Clustering-based Sparse representation

 Final CSR objective function  The unitary property of the dictionary  arg min 

y



DS

Φ

 2 2 + 

α

1  

k K

   

k

α

i

 

k

 What does the CSR model mean?

 Encode the sparse codes with the learned exemplars 2 2  Unify dictionary learning and structure clustering into a variational framework 5/1/2020 19

Bayesian interpretation of CSR

 The connection between the sparse representation and the Bayesian denoising , e.g., wavelet soft-thresholding   arg max log ( 

y

|

 )  

Gaussian likelihood term Laplacian Prior

 The connection between CSR and the MAP estimator 

y

|

  

The joint prior term !

5/1/2020 20

Bayesian interpretation of CSR

 The factorization of the joint prior term

P

   

P

 

P P

 

P

 

  

prediction residual / noise 

P

  

P



P



Gaussian prior

5/1/2020

Laplacian prior

    

i K



k



k

1 2   exp(    2 

i

)  1 2   exp(  2   2 

i

2 ) 21

Iterative reweighted CSR

 The final MAP estimator  arg min  

y



DS

Φ

 2 2 +   1  

K

 

k



k

 

i



k

2 2   where   2 2 

n

  2 ,   2 2 

n

  2 2  The iterative reweighted CSR  Adaptively estimate

λ

and

η

coefficients for each local sparse  Update

λ

and

η

iteratively 5/1/2020 22

The proposed objective function

 Adaptive selection of the dictionary: local PCA  Variable splitting:

x

i k

)  arg min{

x

,{ 

i

} 

N i

 1 

i



i

y



DS

x

2 2 +  

N i

 1

R

i

x

 

i



i

1 +  

k



k

|| 

k

(  

i



k

2 ) || }, . . 2

s t

y

2 2  

D

x



β

k

update: 

k

 1 |

C k

|  

k



i

5/1/2020 23

Alternative optimization algorithm

 α -subproblem: for each

i

 ˆ

i

 arg min{ 

i



R x

i

 

k



i

2 2  

i



i

1 + 

i

(  

i



i

2 ) } 2  Closed-form solution: bi-variate shrinkage operator   ˆ  )  0,   

v

 ) 2   2, 1,

j j

 1

v

 2  , otherwise  2, 1,

j j

 1   /(2  2,

j

 1), 

i

 2 

i



i

* /   

k T

R x

i i

5/1/2020 24

Alternative optimization algorithm

 X-subproblem: 

x x y



DS

x

2 2 + 

k K



i S k

R

i

x

 

k



i

2 2

y



D

x

Alternative direction method of multiplier (ADMM)

L



DS

x

|| 2 2  

k K

  

k

||

R

i

x

 

k



i

|| 2 2  

D

x

  ||

y



D

x

|| 2 2 

x Z

 

x

(

l

 1) 

Z

(

l

 1) 

Z

 (

l

 1)

x

L

x Z

  ,  (

y



D

x

(

l

 1) ) ) 5/1/2020 25

Alternative optimization algorithm

(

l

 1)  [  

T

DS

 

i N

  1

i T

R R

i

   

T

y

 

i N

  1

i T

D D

]  1

i

 

k



i

 

D

T

Z

2  

D

T

y

] Solved by a conjugate gradient method 5/1/2020 26

Overall interpolation algorithm

5/1/2020 27

Exp. Results (scaling factor = 2)

Original NEDI (29.36 dB) SAI (30.76 dB) Proposed ( 31.72 dB ) Original

5/1/2020

NEDI (22.97 dB) SAI (23.78 dB) Proposed ( 24.79 dB )

28

Exp. Results (scaling factor = 2)

Original NEDI (27.36 dB) SAI (29.17 dB) Proposed ( 30.30 dB ) Original

5/1/2020

NEDI (33.85 dB) SAI (34.13 dB) Proposed ( 34.46 dB )

29

Exp. Results (scaling factor = 3)

Original bicubic (23.48 dB) ScSR (23.84 dB) Proposed ( 25.57 dB ) Original

5/1/2020

bicubic (30.14 dB) ScSR (30.00 dB) Proposed ( 31.16 dB )

30

Exp. Results (scaling factor = 3)

Original bicubic (21.85 dB) ScSR (21.93 dB) Proposed ( 23.33 dB ) Original

5/1/2020

bicubic (32.07 dB) ScSR (32.29 dB) Proposed ( 34.80 dB )

31

Conclusions

 A new image upconversion framework combining edge based interpolator with sparse regularization  A nonlocal AR model is proposed for edge-based interpolation  The nonlocal AR model can increase the stability of the sparse reconstruction  Clustering-based sparsity regularization is adopted to exploit the structural dependencies 5/1/2020 32

References

• • • • • • • W. Dong, L. Zhang, et al., “Sparse representation based image interpolation with nonlocal autoregressive modeling,”

IEEE Trans. Image Process.

, vol. 22, Apr. 2013.

Y. Romano, M. Protter, and M. Elad, “Single image interpolation via adaptive non local sparsity-based modeling,”

IEEE Trans. Image Processing

, vol. 23, July 2014. X. Zhang and X. Wu, “Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation”,

IEEE Trans. Image Processing

, vol. 17, no. 6, 2008.

X. Li and M. Orchard, “New edge-directed interpolation”,

IEEE Trans. Image Processing

, vol. 10, no. 10 2001.

J. Yang, J. Wright, et al., “Image super-resolution via sparse representation,”

IEEE Trans. Image Processing

, 2010.

W. Dong, X. Li, et al., “Sparsity-based image denoising via dictionary learning and structural clustering,”

IEEE CVPR

, 2011.

G. Shi, W. Dong, X. Wu and L. Zhang, “Context-based adaptive image resolution upconversion”,

Journal of Electronic imaging

, vol. 19, 2010. 5/1/2020 33

Thanks for your attention!

Questions?

5/1/2020 34