Transcript Seamless image-based texture atlases using multi-band blending Cédric Allène Jean-Philippe Pons Renaud Keriven Université Paris-Est – Ecole des Ponts – CERTIS Seamless image-based 3D textures Outline 1) Context:

Seamless image-based texture
atlases using multi-band blending
Cédric Allène
Jean-Philippe Pons
Renaud Keriven
Université Paris-Est – Ecole des Ponts – CERTIS
Seamless image-based 3D textures
Outline
1) Context: 3D reconstruction
2) Image regions choice
3) Multi-view blending
4) Atlas building and results
Seamless image-based 3D textures
- 1 Context:
3D reconstruction
Seamless image-based 3D textures
1 – Context: 3D reconstruction
3D reconstruction
from multi-view
First step:
Get a set of pictures or the model to reconstruct.
Seamless image-based 3D textures
1 – Context: 3D reconstruction
3D reconstruction
from multi-view
Second step:
Autocalibration from your set of pictures.
Seamless image-based 3D textures
1 – Context: 3D reconstruction
3D reconstruction
from multi-view
Third step:
3D mesh building.
Seamless image-based 3D textures
1 – Context: 3D reconstruction
3D reconstruction
from multi-view
Problems:
 Which image should
the texture be from?
 How to minimize color
discontinuities at the
frontiers of regions
from different images?
This is here our method
comes into play!
Fourth step:
Add colored textures on the mesh from multi-view images.
Seamless image-based 3D textures
- 2 Image regions choice
Seamless image-based 3D textures
Surface partition
2 – Image regions choice
We note:


the input calibrated images, and
the projection from 3D space to image
We have to assign each face of the mesh in
to one of the input views in which it is visible.
We obtain a labeling vector
What do we want?
 good visual detail, and
 color continuity at region boundaries
Seamless image-based 3D textures
Energy terms
2 – Image regions choice
Good visual detail:
Color continuity at region boundaries:
where
neighbour faces
is a dissimilarity term between the
and
with labels
and
.
Seamless image-based 3D textures
Energy minimization
2 – Image regions choice
Global energy to minimize:
Since
is a sum of regular functions, we can use
the MRF/min-cut minimization method to solve our problem.
Problem:
Even if minimized, color discontinuities remain between
regions of different labels…
Seamless image-based 3D textures
- 3 Multi-view blending
Seamless image-based 3D textures
3 – Multi-view blending
Multi-frequencies blending
Method proposed in:
P. J. Burt and E. H. Adelson. A multiresolution spline with
application to image mosaics. ACM Trans. on Graphics, 2(4), 1983.
Two main functions:
 Reduce function (RED):



Input: image I
Output: image I’ which dimensions are half of those of input image I
reduced through a gaussian kernel
Expand function (EXP):


Input: image I
Output: image I’ which dimensions are twice those of input image I
expanded through the same gaussian kernel
Seamless image-based 3D textures
Gaussian & laplacian pyramids
3 – Multi-view blending
 Gaussian
pyramid (left),
obtained by successive
reductions of input image.
 Laplacian
pyramid (right),
obtained by substractions
of two gaussian pyramid
levels.
As a consequence, the
laplacian pyramid is a
multi-frequencies
decomposition.
Seamless image-based 3D textures
3 – Multi-view blending
Summing laplacian pyramid
Important remark:
Summing through
successive expansions
the laplacian pyramid
allows to get back the
gaussian pyramid and,
so, the original image.
Seamless image-based 3D textures
3 – Multi-view blending
2D blending
?
2D blending:
Merging
in a “natural” way the pictures following their respective
Third
step:
masks.
Get
the final blended image by summing the newly created laplacian
pyramid.
First step:
Build the laplacian pyramid for each image, and
Build the gaussian pyramid for each mask.
Second step:
Create a new laplacian pyramid from the image ones through the
gaussian pyramids of their masks.
Seamless image-based 3D textures
Multi-view blending
3 – Multi-view blending
Straightforward from 2D blending using the projection
functions
for each image .
So we obtain the final color at point x of the surface:
with
Seamless image-based 3D textures
- 4 Atlas building and results
Seamless image-based 3D textures
4 – Atlas building and results
Results – Aiguille du Midi (France)
"Naive" approach
Optimized patchwork
Optimized patchwork + blending
Successive approaches
Copyright Bernard Vallet
(www.bvallet.com)
Seamless image-based 3D textures
4 – Atlas building and results
Atlas – Aiguille du Midi (France)
Seamless image-based 3D textures
4 – Atlas building and results
Results – Ettlingen castle (Germany)
"Naive" approach
Optimized patchwork
Optimized patchwork + blending
Successive approaches
Courtesy Christoph Strecha, EPFL
(http://cvlab.epfl.ch/strecha/multiview/)
Seamless image-based 3D textures
4 – Atlas building and results
Atlas – Ettlingen castle (Germany)
Seamless image-based 3D textures
4 – Atlas building and results
Conclusion
Seamless image-based 3D textures:
 Graph-based combinatorial optimization for texture partition
 Multi-view blending on the partition
Advantages:
 Good results
Drawbacks:
 Computing time dependant of the number of levels in the pyramids
Future work:
 GPU acceleration of the blending part
Seamless image-based 3D textures
Thank you for your attention!
Any question?