Transcript Image-Based Synthetic Aperture Rendering MIT9904-14 PIs: Prof. Leonard McMillan (MIT LCS), Prof.

Image-Based Synthetic Aperture Rendering
MIT9904-14 PIs: Prof. Leonard McMillan (MIT LCS), Prof. Julie Dorsey (MIT LCS), and Dr. Hiroshi Murase (NTT)
Project Goals
The primary focus of our
research effort is to develop
technology to create virtual
experiences that will approach
the fidelity of the real world. In
the future, such technologies
will have a dramatic impact on
the way we work and play.
They will enable new forms of
commerce, bring together
individuals separated by large
distances, and provide us with
new forms of entertainment.
Real-time Acquisition
Low Cost Acquisition
Ultimately we intend to
create a device for capturing
and processing dynamically
reparameterized light fields
in real-time. We call this
device a synthetic aperture
camera array. It is composed
of a two-dimensional array of
randomly accessible image
sensors that memory-mapped
in the address space of a host
processor. Such a system will
allow images to be synthesized
from a wide range of virtual camera positions in real-time. We plan to support
multiple simultaneous video streams to support stereoscopic display as well as
multiple viewers.
Approach
‘closest’ ray
(PCI)
FIFO
Interface
Solution
‘closest’ camera
Address
FIFO
Light Field Parameterization
Data
Sensor
Pod - A
B
A
B
C
D
C
D
A
B
A
B
C
D
C
D
We have also prototyped two lowcost devices for acquiring light fields.
We have developed two acquisition
systems for acquiring light fields of
static scenes. The first uses a robotic
XY-platform to move a digital camera.
This system allows us to explore the
trade-offs between camera spacing and
resolution in order to estimate the performance of our camera array. This
system uses a precision image sensor,
precision optics, and a motion platform with a travel distance of approximately one
meter squares. It can acquire a 16 by 16 image light field in under 20 minutes, and
it cost approximately $10,000 US to construct. Our second system is based on an
off-the-shelf flat bed scanner, and an array of plastic lenses. We have modified the
scanner to operate off of battery power so that this system can be taken out into the
field to acquire images. Additional processing is required to correct for shortcoming
in the image sensor and low cost optics. None the less, the system can acquire an 8
by 12 image light field in under 3 minutes, and cost under $100.
desired
ray
camera
surface
reference
images
focal
surface
We take a non-traditional approach to
computer graphics modeling and rendering, in
which a scene is represented by a collection of
images rather than the geometry and surface
properties used in typical computer graphics.
Essentially, we treat a collection of images as a
database of rays. New views can be constructed
from this database on a ray-by-ray basis by
selecting the closest ray to each desired ray.
Motherboard
A high-level block diagram of our proposed system is shown above. The camera’s
host interface will be an industry standard personal computer bus. The camera array
will be constructed from modular sensor units mounted on a common motherboard.
Address
Mux/
Logic
Sensor
Pod - A
Sensor
Pod -A
Sensor
Pod -A
Sensor
Pod -A
Sensor
Pod - B
Sensor
Pod - B
Sensor
Pod - B
Sensor
Pod - B
Mux/
Logic
32 bit
Motherboard Interleaving
Our dynamically reparameterized light field representation allows us to synthesize
images with photographic effects such as variable focus and depth-of-field. Depthof-field effects are created by varying the extent of the reconstruction filters used
on the camera surface.
A variable focal length can be simulated by varying the focal plane used in the
reconstruction process. In a synthetic aperture camera both the aperture and focallength settings can be varied from pixel to pixel. The allows effects that are
impossible with a traditional camera.
Data
The addressing of sensor modules will be interleaved in order to maximize
the communication bandwidth between the image sensors and the host
computer. Each sensor
pod contains a CMOS
Sensor Pod
Data
image sensor, buffer Address
memory, and glue
logic. The multiframe buffer
Control
CMOS Sensor
memory is used for
(on board A/D)
two functions. It
is used to store
information for
Data
noise cancellation,
and it allows the
Address
host to access
Address
FPGA
image rays asynchSDRAM
Data
Logic
ronous to the image
scanning process.
This modular
design approach
will allow us to upgrade to higher resolution sensors as they become available.
Display Technology
We have also developed techniques for direct autostereoscopic viewing of our light fields. These methods
are similar to various lenticular techniques for viewing
stereo images. Our synthetic aperture generation approach provides much greater flexibility than tradition optical approaches. In particular
it can overcome many limitations such a focus control and skewed frustums. We have
demonstrated viewers with true parallax (both horizontal and vertical), and variable
controlled focus. Our displays have nearly all of the desired properties of holograms,
yet they are true color and viewable
under normal lights. Furthermore,
the technology is easily adaptable
to the display of dynamic 3-D
images. Currently we are only
limited by the resolution of flat
panel displays.
The image on the left,
when viewed through a
hexagonal lens array, can
be seen as a threedimensional image of a
flower. It can be simultaneously seen by
multiple viewers. It was
computed from a dynamically reparameterized light
field, which allows us to
precisely control the
focus at all viewing
angles. The inset
provides a magnified
view of the image.