Transcript Progressive Transmission and Rendering of Foveated Volume Data

Progressive Transmission and
Rendering of Foveated
Volume Data
Chen Chen, Zhiyong Huang, Hang Yu, Ee-Chien Chang
School of Computing
National University of Singapore
Introduction
- motivation

Problem



Large volume data provided by modern scanners like
CT and MRI
Slow network speed comparing to the large size of
volume data
Solution


Wavelet foveation
Progressive transmission and rendering


2015/7/16
coarse-to-fine
region-based
grapp 2006
2
Introduction
- wavelet foveated volume


2015/7/16
Foveated volume is a non-uniform sampled volume
whose resolution is highest at the fovea (user
specified ROI) and falls off as the distance to the
fovea increases – priority selection
Wavelet foveated volume is constructed by select
ROI at each level of details in the volume’s wavelet
form - compression
grapp 2006
3
Proposed method
- overview

Transform to Wavelet Form






2015/7/16
Divide into 83 block, run-length encode (RLE) each
block
Client request = ROI {(x,y,z),s}
Generate Fw
Reorder data blocks of Fw
Progressive transmission
Progressive Rendering
grapp 2006
4
Proposed method
- overview

Client-server volume rendering system

Server Side:



Client Side:



2015/7/16
Input:
Cw & ROI{(x,y,z),s}
Output: Fw
Input:
Fw
Output: Iw1 -> Iw2 -> … -> Iwfinal
Two transmission and rendering schemes
grapp 2006
5
Proposed method
- overview
Server Site
Client Site
Wavelet Transform Cw
Generate Fw from
Cw and ROI
Reorder data blocks in Fw
according to the schemes
request
Progressive
transmission
User Input
ROI={(x,y,z),r}
Progressive Render
RB or CTF
Progressive
rendering
Display
2015/7/16
grapp 2006
6
Wavelet transform




2015/7/16
Divide the volume into blocks of size 2k3
Apply generic Haar wavelet transform on each block
to obtain 7 detail blocks and 1 average block
Group 8 adjacent blocks to again obtain a block of
size 2k3
Repeat this procedure until the average volume
reaches size m3
grapp 2006
7
Wavelet transform
- run-length encoding (RLE)

A more compact representation




2015/7/16
RLE is performed block by block
Compressed block will be a list of value(v) and
length(l) pairs
v= Integer value of a coefficient
= Number of successive v
l
Example:
 L=1,1,1,1,0,0,5,0,0,0,0,0,0,0,0
 R=(1,4),(0,2),(5,1),(0,8)
grapp 2006
8
Derive foveated volume




2015/7/16
Cw, ROI{(x,y,z),s} => Fw
Layer 0 image = average image of Cw
Layer i image = IWT(layer i-1 image, 7 layer i-1
detail images)
ROI in ith layer = ROI in Layer i image
grapp 2006
9
Derive foveated volume



2015/7/16
Center of ROI in ith layer is
Size of ROI in ith layer is s3
Boundary of ROI in ith layer can be defined as
grapp 2006
10
Derive foveated volume


2015/7/16
4th ROI = {(9,9),6} = {[6,11],[6,11]}
Layer 3 details (i,j) =([3,5],[3,5])
grapp 2006
11
Progressive transmission
- motivation





2015/7/16
Data size = 10243, Fovea size = 483, Average
image size = 163
Maxlayer=Log2(1024/16)=6
Total amount of data = 243*(5*7+1)=497,664
coefficients
Reorder data blocks
Progressive transfer
grapp 2006
12
Progressive transmission
- two schemes

Coarse-to-fine



Region-based


2015/7/16
Description: Transmit a rough average image first and
refine the fovea from outer ROIi to inner ROIi+1
Implementation: Send the wavelet foveated data from
inner layer to outer layer progressively
Description: Transmit full resolution ROImaxlayer first and
expand the image from ROImaxlayer-1 to ROI0 with
decreasing resolution
Implementation: Data blocks and parent data blocks of
ROImaxlayer-i+1 will be sent in ith iteration
grapp 2006
13
Progressive transmission
- illustration in 2D





2015/7/16
2D image size= 128*128, m=16, k=8,
ROI={(86,86),45}, maxlayer=3
ROI0={(10,10),45}=[0,15] blk_index=[0,1]
ROI1={(21,21),45}=[0,31] blk_index=[0,1]
ROI2={(43,43),45}=[20,63] blk_index=[1,3]
ROI3={(86,86),45}=[64,107] blk_index=[4,6]
grapp 2006
14
Progressive transmission
- illustration in 2D
2015/7/16
grapp 2006
15
Progressive transmission
- coarse-to-fine
2015/7/16
grapp 2006
16
Progressive transmission
- coarse-to-fine
2015/7/16
grapp 2006
17
Progressive transmission
- region-based
2015/7/16
grapp 2006
18
Progressive rendering
- rendering equation



2015/7/16
For each voxel:

intensity value q i

opacity value i
The final intensity result reaches the viewer of each
pixel will be Eulerian Sum over accumulated opacity
Rendering equation to render a subvolume with
same intensity and opacity value
grapp 2006
19
Progressive rendering
- rendering equation

2015/7/16
For 2 subvolumes Sa and Sb with size na and nb,
voxels in Sa and Sb have qa , qb , and  a ,  b
respectively. Sa is in front of Sb, that is, Sa is “over”
Sb. The intensity of Sa and Sb and Sfinal can be
defined as
grapp 2006
20
Region-based
- partial volume reconstruction

Starting coefficient:

Ending coefficient:

Wavelet coefficients at ith layer detail image



2015/7/16
begin:
end:
Reconstruction starts from layer 0, that is, the
average image and move out to higher level of
details until the desired resolution is reached
grapp 2006
21
Region-based



2015/7/16
Center fovea is reconstructed using partial volume
reconstruction and rendered at iteration 1
6 subvolumes are rendered: Top, Bottom, Left,
Right, Front, Back with resolution scale 2(1-i), i is the
number of iteration
6 sub volumes are rendered separately and combine
the rendered result to the previous rendering image
grapp 2006
22
Region-based
- image composition



Top, bottom, left and right are added to the previous
rendering result image at their right position
Front and back image needs image composition with
the previous rendering result image: (front) over
(inner) over (back)
“over” operator can be defined as: given image pixel
A(da, αa) and B(db, αb)



2015/7/16
C = A over B
dc = da + αa*db
αc = αa* αb
grapp 2006
23
Region-based
- image composition
2015/7/16
grapp 2006
24
Coarse-to-fine



2015/7/16
Start rendering from ROI0 to ROImaxlayer
Each ROIi except ROImaxlayer can be divided into 7
subvolumes: top, bottom, left, right, front, back and
center
Center subvolume is the average image of the inner
ROI and its rendering result, accumulated density
value and opacity value will be kept for next
iteration
grapp 2006
25
Coarse-to-fine





2015/7/16
ROIi is constructed by center part of ROIi-1 and level i
detail coefficients
Number of voxels each coefficient in ROIi represent is
2maxlayer-i
Rendering equation for each voxel in ROIi is
Rendering result of front and back subvolume RFi and
RBi will be take down for next iteration
Rendering result of iteration i+1 will be combined
with RFi and RBi in iteration i+1
grapp 2006
26
Results
- direct rendering
2015/7/16
grapp 2006
27
Results
- region-based
2015/7/16
grapp 2006
28
Results
- region-based
2015/7/16
grapp 2006
29
Results
- coarse-to-fine
2015/7/16
grapp 2006
30
Results
- coarse-to-fine
2015/7/16
grapp 2006
31
Results
- time
2015/7/16
grapp 2006
32
Results
- coarse-to-fine
2015/7/16
grapp 2006
33
Results
- coarse-to-fine
2015/7/16
grapp 2006
34
Results
- region-based
2015/7/16
grapp 2006
35
Results
- region-based
2015/7/16
grapp 2006
36
Results
- coarse-to-fine
2015/7/16
grapp 2006
37
Results
- coarse-to-fine
2015/7/16
grapp 2006
38
Results
- region-based
2015/7/16
grapp 2006
39
Results
- region-based
2015/7/16
grapp 2006
40
Conclusion and future work

Two progressive transmission and rendering
schemes, region-based and coarse-to-fine




Large memory is still needed


2015/7/16
Blockwise RLE used in wavelet compression
Wavelet foveation
Image composition
cut the data set exceeding a certain amount into
smaller data sets and build location map to indicate
actual position of each data segments
Cache can be used to store frequently used blocks,
for example, those high level details
grapp 2006
41