Transcript Real-Time Auralization of Sound in Virtual 3D Environments by Scott McDermott [email protected] Overview & Objective Develop an adaptive virtual environment that simulates real-time generation of 3D.

Real-Time Auralization
of Sound in Virtual 3D
Environments
by Scott McDermott
[email protected]
Overview & Objective
Develop an adaptive virtual environment that
simulates real-time generation of 3D sound.
Design algorithms to efficiently and effectively
compute realistic 3D sounds in this environment.
Apply these techniques to various applications,
including simulations, virtual reality, gaming, and
modeling.
Outline
Sound Perception
Digital Sound and Computers
3D Sound Approximations
“Surround” Sounds (Stereo Expansion) Approach
Head Response Transfer Function (HRTF) Approach
True 3D Sound
Beam Tracing Approach
The Graphics Analogy
Sound Perception
When we hear a sound, we automatically obtain
certain information about the source:
Direction
Distance
Elevation
Environmental conditions
Status of source
Sound Perception
8 types of cues for sound spatialization [1]:
Interaural
Head
Motion
Delay Time (direction)

delay between
move
head to re-evaluate
time arrivesthese
at each
filters
ear (0 to 0.63 ms)
Head Shadow (direction and distance)
Vision

difference
ignore
audio
in volume
cues if different
from onefrom
ear visual
to the other (up to 9 dB)
PinnaEcho
Early
Response
Response
(direction and
(distance
elevation)
and direction)

outer ear
echos
from
filters
environment
sound, compare
(50 to 100
between
ms) two ears
Shoulder Response
Reverberation
(distance and(elevation
direction)and direction)

reflections
late
dense off
echos
upper
from
body
environment
(1-3 kHz) (> 100 ms)
Sound Perception
An environment with true 3D sound will
need to take all of these into account.
It must also be able to perform
calculations and apply filters in real-time.
The result must be convincing to the
listener and enhance the virtual
experience.
Sound in the Digital World
Theanalog
Sound
An
sounds
in the
tocan
physical
digital
be convert
stored
worldinchanges
exists
various
as these
waves ofsignals
voltage
formats
and
pressure
qualities
to discrete
changes.
(suchdigital
as mono
signals.
or
stereo,
8 or 16
bit, 11
orpressure
44 kHz).and
A computer
microphone
stores,
converts
manipulates,
changes
to changes
retransmits
The
reversein
these
ofvoltage.
thisabstractions
process allows
of sound.
the
computer to re-generate the sound.
Blah blah
blah…
3D Sound, The Basics…
In a virtual 3D environment, sound can
originate from an infinite number of
locations relative to the observer.
Ideally, when the observer hears the
sound it should take into account the
environment.
Specifically:
Distance:
Causes sound to arrive at different times.
Reflection & Reverberation:
Causes “copies” of the sound to arrive at different times.
Diffraction & Refraction:
Causes sound to bend around objects or arrive at
different times.
Absorption & Attenuation:
Causes the sound to be weaker when it arrives.
3D Sound Approximations:
Surround Sound
Surround sound uses various filters to
simulate the effects of sound
spatialization.
These filters create effects such as
reverberation, localization, and
attenuation.
Sound paths are not calculated.
Used in most theaters and home
entertainment units.
Surround Sound
The user is situated with a set of speakers around him.
To simulate 3D localization, sound is played louder, out
of phase, and/or at slightly different times from each
speaker.
Comes in a variety speaker placement setups [8]:
Bark bark
bark!!
Dolby
Two
Quadraphonic
Headphones
Speaker
5.1
Stereo
3D Sound Approximations:
Head Related Transfer Functions
Used in conjunction with surround sound
to create better 3D approximations.
Microphones record sound from within the
ear of a person or a model.
Differences between original sound and
recordings are used to create filters.
These filters are applied to generated
sounds to create the illusion of
dimensionality.
Surround Sound &
Head Related Transfer Functions
Pros:
Relatively cheap.
Effective.
Makes sense.
Cons:
Many different approaches (non-standard).
Works only with limited speaker positions.
Not entirely generic.
Still not pure 3D sound.
True 3D Sound
3D graphical environments already exist.
Light paths traverse the scene and surface
intensities are calculated.
Currently, sound paths are at most
superficially computed.
Yet, programmers already have a wealth
of environmental data.
Various possible approaches…
True 3D Sound
Beam Tracing
Approach:
Divide the environment into
cells or regions.
Precompute and store beam
paths from various source
Source: Real-Time Acoustic Modeling for
locations.
Distributed Virtual Environments [4]
Lookup, in real-time, reverberation paths from the avatar
to the source.
Use these paths to calculate delay and attenuation from
the original, anechoic, audio signal for each of the
echoes.
Beam Tracing
Pros:
Quick and effective (with a good data structure).
Intuitive.
Scalable for large environments.
Cons:
Needs offline computations.
Assumes sources are stationary.
Assumes source locations are finite.
True 3D Sound
On a basic level, we can determine sound
propagation similar to how light travels through a
3D environment.
One simple, but computationally intensive
method would be similar to ray tracing.
Ray tracing algorithms are generally very
effective but also extremely slow and prone to
sampling errors.
Most real-time algorithms for graphical
computers make various assumptions:
True 3D Sound
The Graphics Analogy
The 3D Graphics Pipeline:
Objects are made from geometric primitives composed of points.
These vertices are transformed to be relative to the camera.
Objects outside of the viewing field are clipped.
Rays are sent from the camera, through each point on the projection
plane, and into the scene.
Corresponding pixel values in the viewport are calculated from these
rays.
True 3D Sound
The Graphics Analogy
Objects are made from geometric primitives (triangles,
rectangles) composed of points.
Light intensities are calculated based on surface normals
of these points.
These intensities are fed into the graphics pipeline.
True 3D Sound
The Graphics Analogy
Many of these computations are forwarded
to optimized 3D graphics cards.
Many of these same techniques could be
employed for generating realistic 3D
sounds.
We would need to develop and design 3D
sound cards and appropriate algorithms.
Conclusion
3D graphics and many other components
of today’s computer systems have been
almost thoroughly developed.
3D sound is still in the infancy stage.
This field has a great deal of research
potential.
References
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
Burgress, David, A. Techniques for Low Cost Spatial Audio. ACM UIST, pages 53-59, 1992.
Ellis, Sean. Towards More Realistic Sound in VRML. ACM Virtual Reality and Modeling, pages
95-100, 1998.
Flaherty, Nick. 3D audio: new directions in rendering realistic sound. Electronic Engineering,
pages 49, 52, 55, & 56, 1998.
Funkhouser, Thomas, A. , Patrick Min, and Ingrid Carlbom. Real-time Acoustic Modeling for
Distributed Virtual Environments. SIGGRAPH, pages 365-374, 1999.
Funkhouser, Thomas, A. , Ingrid Carlbom, Gary Elko, Gopal Pingali, and Mohan Sondhi. A
Beam Tracing Approach to Acoustic Modeling for Interactive Virtual Environments.
Funkhouser, Thomas, A. , Ingrid Carlbom, Gary Elko, Gopal Pingali, and Mohan Sondhi.
Interactive Acoustic Modeling of Complex Environments. Acoustical Society of America, 1999.
Min, Patrick, and Thomas A. Funkhouser. Priority-Driven Acoustic Modeling for Virtual
Environments. EUROGRAPHICS, 2000.
Tsingos, Nicolas, Thomas A. Funkhouser, Addy Ngan, and Ingrid Carlbom. Modeling Acoustics
in Virtual Environments Using the Uniform Theory of Diffraction.
Hull, Joseph. Surround Sound Past, Present, and Future. Dolby Laboratories Inc.
http://www.dolby.com/tech/.
Suen, An-Nan, Jhing-Fa Wang, and Jia-Ching Wang. VLSI Implementation of 3-D Sound
Generator. IEEE Transactions on Consumer Electronics, pages 679-688, 1997.
Real-Time Auralization
of Sound in Virtual 3D
Environments
by Scott McDermott
[email protected]