Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View Morphing

by Steven M. Seitz Charles R. Dyer Irwin Chiu Hau Computer Science McGill University Winter 2004

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Overview

Mona Lisa view morphs Source: http://www.cs.washingto

n.edu/homes/seitz/vmorp h/vmorph.htm

• What is view morphing?

• How to do view morphing?

• Results • Conclusion

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View Morphing

Virtual Cameras Source: View Morphing; Steven M. Seitz, Charles R. Dyer

What is view morphing?

– What is it?

– Why do we care about it?

– Where do we see them?

– Image Morphing vs View interpolation vs View Morphing

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View Morphing: Key Idea Beier-Neely morph is NOT shape-preserving!

– distortions – un-natural

A Shape-Distorting Morph Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View Morphing: Key Idea

View morphing uses 3D shape preserving morph!

– no distortions – natural A morph is

3D shape preserving

if the results of two different views represent new views of the same object View morphing fixes these intermediate steps!

A Shape-Distorting Morph Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Why do we care?

View morphing is efficient Produces new views without – 3D modelling – Taking additional photos

trueSpace Source: www.caligari.com

View morphing creates impressive effects – Camera motion – Image morphing

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Image Morphing vs View interpolation vs View Morphing

View Morphing is an extension to Image Morphing (Beier and Neely, 1992) – Produces physically plausible new views of a scene View Morphing (Seitz and Dyer, 1996) is an improvement over View Interpolation (Chen and Williams, 1993) – Addresses non-rigid transformations problems – Does not require depth values – Creates realistic image transitions

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Overview

Mona Lisa view morphs Source: http://www.cs.washingto

n.edu/homes/seitz/vmorp h/vmorph.htm

• What is view morphing?

• How to do view morphing?

• Results • Conclusion

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

How to do View Morphing?

View morphing in three steps 1. Prewarp two images 2. Compute a morph between the prewarped images 3. Postwarp each in between images produced by the morph

View Morphing Procedure Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

How to do View Morphing?

View morphing in three steps 1. Prewarp two images 2. Compute a morph between the prewarped images 3. Postwarp each in between images produced by the morph View morphing in 1 steps: Assume parallel views!

1. Compute a morph between the parallel images

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Parallel Views

Basic Theory

Morphing Parallel Views Source: Steven M. Seitz, Charles R. Dyer

image point p 0 = (x 0 ,y 0 ) scene point P = (X,Y,Z) p 0 = Π 0 P where Π 0 is a projection matrix image point p 1 = (x 1 ,y 1 ) scene point P = (X,Y,Z) p 1 = Π 1 P where Π 1 is a projection matrix

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Parallel Views

Basic Theory

Morphing Parallel Views Source: Steven M. Seitz, Charles R. Dyer

image point p = (x,y) scene point P = (X,Y,Z) p = Π P where Π is a homogenous projection matrix Π = [ H | - HC] H: position and orientation of image plane C: euclidean position of the camera

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Parallel Views

Mathematics for View Interpolation Πs: Linear interpolation of Π 0 and Π 1 Πs = (1 – s) Π 0 s = [0,1] + s Π 1

f : focal lengh C : center of a camera

Cs = (sCx, sCy, 0) fs = (1 – f) f 0 + s f 1

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View Interpolation Recap

View Interpolation vs View Morphing

View Interpolation Source: 3D Games by Alan Watt and Fabio Policarpo Morphing Parallel Views Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

How to do View Morphing?

View morphing in three steps 1. Prewarp two images 2. Compute a morph between the prewarped images 3. Postwarp each in between images produced by the morph View morphing in 1 steps: Assume parallel views!

1. Compute a morph between the parallel images

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Non-Parallel Views

The General Case This is where the 3-Step Algorithm comes into play 1. Prewarping: I 0 2. Morphing: Î 0 to Î 0 and I 1 and Î 1 into Î s 3. Postwarping: Î s into I s to Î 1

Morphing in Three Steps Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Non-Parallel Views

Mathematics for Image Reprojection ĤH -1 p = p H and Ĥ are 3x3 matrices that represent the position and the orientation of their image planes The resulting 3x3 matrix, ĤH -1 is a projective transformation that reprojects the image plane I into Î

Morphing in Three Steps Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Limitations

• Singular Views cannot be reprojected to form parallel views • Singular configurations are settings where one of the camera resides in the field of view of another camera • Still works, just kind of conceptually hacky

Parallel view Source: Steven M. Seitz, Charles R. Dyer Singular view Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Traditional Problems Change in visibility creates

1. Folds – Occurs when a visible area becomes occluded 2. Holes – Occurs when an occluded area becomes visible

Area Point Fold Source: Irwin Chiu Hau Penumbra, umbra and hole regions Source: Chen and Williams

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Producing The Morph We have talked about theory behind algorithm

1. How to project/unproject images to parallel planes 2. How to warp between parallel planes 3. Theoretical problems Now, let’s do an actual View Dependent Morph!

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Producing The Morph

We need: 1. Two images I 0 and I 1 2. Two perspective projection matrices Π 0 and Π 1 3. Correspondence between pixels Note that a sequence of projection matrices Π s is required to control the entire animation, but Π s can be computed automatically if we know H s .

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Controlling The Morph

H s can be obtained indirectly by establishing constraints – Recall H s is position and orientation of image plane – One way is to specify four control points Note: Control points implicitly determine the postwarping transformation

Four control points form the red bounding box to determine the postwarping stage Yellow lines are set of features to determine the prewarping stage View Morphing Procedure Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View Morphing sans Prewarping

Prewarping is not necessary for: – Objects that aren’t closely related • Prewarping is less effective • Computation is unstable – Images that are approx. orthographic (eg. telephoto) However, postwarping should not be left out to: – Reduce image plane distorsions

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Overview

Mona Lisa view morphs Source: http://www.cs.washingto

n.edu/homes/seitz/vmorp h/vmorph.htm

• What is view morphing?

• How to do view morphing?

• Results • Conclusion

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Results

Facial view morphs Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Results

Facial view morphs Source: http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Results

Mona Lisa view morphs Source: http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Results

Image Morphing vs View Morphing

Image Morphing vs View Morphing Source: Steven M. Seitz, Charles R. Dyer

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Conclusions

Things to remember View Morphing • Powerful extension to image morphing • Produces new views of a scene • 3D shape preserving

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

References

• View Morphing - Seitz and Dyer, 1996 • View Interpolation - Chen and Williams, 1993 • Image Morphing - Beier and Neely, 1992 • 3D Games: Realtime rendering and Sofware Technology - Alan Watt and Fabio Policarpo

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Questions?