Transcript ppt
Interruptible Rendering
David Luebke
University of Virginia
But first…
Cat
– More evil use of computer graphics
Then, to cleanse the palette…
– Work in Progress
Industrial Light + Magic, 1999
– After You
4/30/2003
Ringling School of Art & Design, 2003
Interruptible Rendering
2
Interruptible Rendering
David Luebke
Department of Computer Science
University of Virginia
Cliff Woolley
Ben Watson
Abhinav Dayal
Presented at ACM SIGGRAPH 2003
Symposium on Interactive 3D Graphics
Problem: fidelity vs. performance
Interactive graphics requires trading off
detail vs. frame rate
Conventional wisdom dictates:
Maintain a high and constant frame rate
Thus, use a level of detail that can always be
rendered in time
This is simplistic! Can we do better?
July 3, 2003
Interruptible Rendering
David Luebke
4
Improve on traditional LOD
Is a high, constant frame rate good enough?
Is fidelity ever more important than frame
rate?
How can this be decided at runtime while
still guaranteeing interactivity?
July 3, 2003
Interruptible Rendering
David Luebke
5
Key observation
The fidelity/performance tradeoff can be seen in
terms of spatial versus temporal error…
July 3, 2003
Interruptible Rendering
David Luebke
6
Key observation
The fidelity/performance tradeoff can be seen in
terms of spatial versus temporal error…
…and these should be measured and compared directly!
July 3, 2003
Interruptible Rendering
David Luebke
7
Unified spatial/temporal error
We combine temporal error (“lateness”) and
spatial error (“coarseness”) into a unified
dynamic visual error metric
We use this to drive progressive refinement
July 3, 2003
Interruptible Rendering
David Luebke
8
Progressive refinement
The big question:
At what point does further refinement of the
current frame become pointless?
July 3, 2003
Interruptible Rendering
David Luebke
9
Progressive refinement
The big question:
At what point does further refinement of the
current frame become pointless?
Answer: When temporal error exceeds
spatial error
July 3, 2003
Interruptible Rendering
David Luebke
10
Methodology
Refine a stream of continuous LODs
Monitor input frequently
Minimize dynamic visual error
July 3, 2003
Interruptible Rendering
David Luebke
11
Refine a stream of continuous LODs
Render progressive refinements on top of each
other until “out of time”
Ensure that we can stop refining at any time and
move on to the next frame
July 3, 2003
Interruptible Rendering
David Luebke
12
Monitor input frequently
Ideally: input monitored continuously
Realistically: check every x ms
Allows quick reaction when sudden changes
in input occur
Allows system to be self-tuning
July 3, 2003
Interruptible Rendering
David Luebke
13
Minimize dynamic visual error
Always display image with least error
Sometimes:
Further refinement is pointless
Temporal error (lateness) exceeds spatial error
(coarseness) for the current frame
The front buffer is “out of date”
Dynamic visual error (spatial + temporal) in front
is greater than in the back
July 3, 2003
Interruptible Rendering
David Luebke
14
start
Clear the back buffer
Clear the
front buffer
Refine the back buffer
N
tf > sf?
N
t b > s b?
Compute
dynamic
visual error
Y
N
ef >= eb?
Y
Swap buffers
Y
Swap buffers
Rendering to Front Buffer
Rendering to Back Buffer
time
Refine the
front buffer
Refinement
Need a progressive rendering scheme
Subtlety: In depth-buffered rendering, you can’t
“unrender” pixels from a coarse model
Thus for depth-buffered rendering scheme should
satisfy containment
July 3, 2003
Interruptible Rendering
David Luebke
16
Refinement
Three refinement schemes implemented:
Splatting (Tech sketch, SIGGRAPH 2002)
Progressive polygonal hulls
Progressive ray casting
July 3, 2003
Interruptible Rendering
David Luebke
17
Progressive hulls
Simplification method by [Sander et al. 2000]
Record a sequence of constrained edge collapses and
play it back in reverse
Guarantees containment
V3
V2
Vn
V1
July 3, 2003
Interruptible Rendering
V4
David Luebke
18
Calculating dynamic visual error
We use a simple screen-space metric
Spatial error:
Maximum projected error of visible geometry
Temporal error:
Project points on object and/or bounding volume
Find the maximum screen-space distance any point
has moved since frame began
More elaborate/correct metrics possible!
July 3, 2003
Interruptible Rendering
David Luebke
19
Progressive ray casting
Progressive refinement
Coarse to fine sampling/reconstruction
Sampling
Where to shoot the ray
Adaptive
Non-adaptive
Reconstruction
Producing imagery from samples
July 3, 2003
Interruptible Rendering
David Luebke
20
Sampling
Coarse to fine - quadtree approach
Sample every quadtree node's center
Breadth first traversal
(Predetermined) random traversal per level
Ray casting – using OpenRT
Per ray API ~ 600,000 rays per sec
Simple shading, shadows and specular highlights
July 3, 2003
Interruptible Rendering
David Luebke
21
(Simplistic) reconstruction
Placement
Place splat at center of each
quadtree node
Shading
Flat shaded quad covering
the node's screen space
Alpha-textured quads
(smooth reconstruction)
Quad size = twice of node's
screen space
Texture: Gaussian blob
July 3, 2003
Transparent at edges
Opaque at center
Interruptible Rendering
David Luebke
22
Flat shaded
Alpha textured
Calculating dynamic visual error
Temporal error
Computed as in case of polygonal hulls
Spatial error
Diagonal length of the largest quadtree node displayed on
the screen
Spatial
error
July 3, 2003
Interruptible Rendering
David Luebke
24
Demo
July 3, 2003
Interruptible Rendering
David Luebke
25
Evaluation: the “Gold Standard”
Compares an ideal rendering to interactive
approximations
Ideal rendering: full detail, zero delay
Interactive approximations are
Unmanaged
Constant fidelity (in pixels)
Constant frame rate (fixed Hz)
Interruptible rendering
July 3, 2003
Interruptible Rendering
David Luebke
26
Frame generation
Record what the user sees and when
Generate each approximation offline
Record actual frames displayed over time
Account for:
Render time
Double buffering
Frame locking
Rendering into front buffer (interruptible)
July 3, 2003
Interruptible Rendering
David Luebke
27
Comparing frames
Difft = Idealt - Renderedt
Error measures
RMS – Root Mean Square error
Lindstrom's perceptually based error
July 3, 2003
Interruptible Rendering
David Luebke
28
Two test input streams
Rotation
Model-centered
Fixed angular velocity
User interaction
Includes both view translation and rotation
Includes both static and dynamic segments
Both input streams recorded to files, with
timestamps
July 3, 2003
Interruptible Rendering
David Luebke
29
Interaction sequence: ray caster
Frames:
Ideal
Interruptible
Constant
Fidelity
Unmanaged
33
36
39
42
45
48
51
54
57
60
Video
July 3, 2003
Interruptible Rendering
David Luebke
31
Rotation input stream
error(rms)
error(rms)
Progressive hulls
Interactive input stream
time(secs)
error(rms)
error(rms)
Ray casting
time(secs)
time(secs)
constant frame rate
interruptible
time(secs)
constant fidelity
unmanaged
Benefits
Accuracy
Balances spatial and temporal error
Interactivity
Even w/ slow renderers like our ray caster
Or large models
Self-tuning system
Adapts to hardware
July 3, 2003
Interruptible Rendering
David Luebke
33
Limitations
Overdraw in progressive renderer
Progressive ray caster better here
Cost of monitoring temporal error
Rendering immersive models
Requires reversing the containment criteria for
polygon rendering
Not a problem for ray caster
July 3, 2003
Interruptible Rendering
David Luebke
34
Ongoing and future work
Improve implementation:
Textures, normal maps, etc.
Reducing overdraw
Parallelization of ray tracing system (Ben & Joe)
Adaptive (view-dependent) refinement (Ben & Bob)
Ray tracing seems like the way to go…
Better estimates of temporal & spatial error
Use color comparisons, e.g. to an up-to-date but
low-resolution ray-traced image
Use contrast sensitivity function or related ideas
July 3, 2003
Interruptible Rendering
David Luebke
35
To think about
Ultimately this work is about:
A principled approach to the fidelity vs. performance
tradeoff
New ideas on temporal sampling & error
Can we take these ideas further?
Frames and framebuffers are overrated
Can we decouple temporal & spatial sampling?
When and where to sample?
How to reconstruct?
July 3, 2003
Interruptible Rendering
David Luebke
36
Next steps
Motivation: Interactive ray tracing
From bad joke to old hat in a couple of years
Lots of work on “how”
Supercomputer [Parker et al. 99]
PCs and PC clusters [Wald & Slussalek 2000]
GPUs [Purcell et al. 2002, Carr & Hart 2002]
We are interested in “why”
July 3, 2003
Interruptible Rendering
David Luebke
37
Interactive ray tracing
The big question:
What can you do with an interactive ray-based
renderer that you can’t do with a rasterizer?
July 3, 2003
Interruptible Rendering
David Luebke
38
Interactive ray tracing
The big question:
What can you do with an interactive ray-based
renderer that you can’t do with a rasterizer?
Can vary sampling rate across the image
Focus on sampling edges, high-frequency regions
Exploit eccentricity, velocity, etc.
July 3, 2003
Interruptible Rendering
David Luebke
39
Interactive ray tracing
The big question:
What can you do with an interactive ray-based
renderer that you can’t do with a rasterizer?
Can vary sampling rate across time
Sample more frequently when things are changing
Sample more frequently where things are changing
July 3, 2003
Interruptible Rendering
David Luebke
40
Rethinking rendering
Goal: rethink spatial & temporal strategies
for interactive sampling & reconstruction
Related work:
Frameless rendering
Just-in-time pixels
Sample reuse
Sparse sample reconstruction
July 3, 2003
Interruptible Rendering
David Luebke
41
Nuts and bolts
What we’re doing: spatio-temporally
adaptive sampling
Update samples with higher priority in regions of
higher variance
Spatial variance: edges
Temporal variance: motion
Reconstruction of resulting samples
The “deep buffer” stores samples in time & space
Reconstruct image at front edge of time: apply
filter kernel with varying width in space and time
July 3, 2003
Interruptible Rendering
David Luebke
42
Questions going forward
How best to vary sampling rate to respond
to variance?
What exactly do we mean by “variance”?
How best to generate an image from nonuniform samples when some are more
“stale” than others?
What shape should kernel be, especially in time?
Can we simulate frameless display
hardware?
July 3, 2003
Interruptible Rendering
David Luebke
43
Acknowledgements
Peter Lindstrom for ltdiff
OpenRT Interactive Raytracing Project
Stanford 3D Scanning Repository
National Science Foundation
Awards 0092973, 0093172, and 0112937
July 3, 2003
Interruptible Rendering
David Luebke
44