Augmented Virtual Environments (AVE): Dynamic Event Visualization Ulrich Neumann, Suya You
Download ReportTranscript Augmented Virtual Environments (AVE): Dynamic Event Visualization Ulrich Neumann, Suya You
Augmented Virtual Environments (AVE): Dynamic Event Visualization Ulrich Neumann, Suya You
Integrated Media Systems Center Computer Science Department University of Southern California September 2003 1
Problem Statement
Imagine dozens of video/data streams from people, UAVs, and robot sensors distributed and moving through a scene…
Problem
: visualization as separate streams/images provides no integration of information, no high-level scene comprehension, and obstructs collaboration 2
A Simple Example – USC Campus 1 3 streams provides no no high-level scene obstructs collaboration 2 3
AVE: Fusion of 2D Video & 3D Model
VE: captures only a snapshot of the real world, therefore lacks any representation of dynamic events and activities occurring in the scene
AVE Approach
: uses sensor models and 3D models of the scene to integrate dynamic video/image data from different sources Visualize
context all
data in a
single
to maximize collaboration and comprehension of the big picture Address dynamic visualization and change detection 4
Research Highlights & Progress
We address basic algorithm research and technology barriers inherent in AVE system
Integrated Modeling System
− whole campus, semi-automated − feature finding and extraction − linear/non-linear element fitting
Capture System
− real time DV streams (<4)
Rendering System
− real-time graphics HW produces ~28fps on dual 2G PC - 1280x1024 screen
Image Analysis System
− detection and tracking of moving objects (cars, vehicles) and pseudo models Range Sensors Data Acqui sition Model Reconstructi on Im age Sensors Model Refi nem ent Obj ect Detecti on and T racking T racking Sensors Moti on Tracking Dynami c Fusion Im agery Projecti on Augmented Vi rtual Envi ronment 5
Integrated Modeling System Approach Model reconstruction − Input: LiDAR point cloud − Output: 3D mesh model − Automated Building extraction − Vegetation remove − Building detection − Model fitting − Semi-automated
User interaction LiDAR point cloud Model reconstruction
(Re
-
sampling, hole-filling, tessellation)
Building extraction
(Segmentation, edge detection)
Modeling fitting
(Linear and non-linear fitting)
Modeling assembly
(Element relationship)
Vegetation & ground Building models
6
Model Reconstruction from LiDAR Model reconstruction Grid re-sampling (range image) Hole-filling (adaptive weighted interpolation) Tessellation (Delaunay triangulation, depth filter) 7
Reconstructed USC Campus Model Reconstructed range image Reconstructed 3D model 8
Model Needs to be Refined LiDAR is noisy and incomplete − Artifacts result in the model hard to visualize and map texture 9
Model Refinement and Extraction Produces complete models and improves texture visualization − Remove vegetation and ground − Extract and refine building models Semi-automated − Element based approach − Supports linear and nonlinear (high-order) surface fitting − Models irregular shapes 10
Model Extraction Segmentation & building extraction − Users define an interested area (two or three points) − Edge and surface points are then automatically segmented
y
kx
b
Edge points Surface points User selected Fitted surface 11
Model Fitting - linear Cylinder
F
(
x i
,
y i
) (
x i G
x
0 ) 2 (
y i
i n
1
F
(
x i
,
y i
) 2
y
0 ) 2
R
2 Sphere (
x
x
0 ) 2 (
y
y
0 ) 2 (
z
z
0 ) 2
R
2 0 Slope
ax
by
cz
d
0
ax
by
z
d
0 12
Model Fitting - nonlinear
r
( , )
a
1
a
2 cos 1 cos
a
3 1 sin cos 2 sin 1 2 / 2 / 2
F
(
x
,
y
,
z
)
x
(
a
1 ) 2 2
y
(
a
2 ) 2 2 2 1
z
(
a
3 ) 2 1 Superquadric Levenberg-Marquardt nonlinear fitting 13
LA Natural History Museum (before model fitting ) 14
LA Natural History Museum (after model fitting) 15
LA Natural History Museum (embedded) 16
USC Campus & University Park ( reconstructed ) 17
USC Campus & University Park ( ground removed ) 18
USC Campus & University Park ( model fitting ) 19
USC Campus & University Park ( embedded ) 20
USC Campus & University Park (with aerial photo texture) 21
USC Campus (close view) 22
AVE Sensor Models (Tracking) Portable tracking package DGPS ( Z-Sensor base/mobile from Ashtech ) INS ( IS300 from Intersense ) Stereo camera head ( MEGA-D from Videre Design ) Real-time data acquisition and AR display - GPS: ~1Hz - INS: ~150Hz - Video: ~30Hz Synchronize & fuse at 30Hz video rate 23
Tracking Needs to be Stabilized GPS/INS accuracy is not enough − Error is easily visible and undesirable − One degree of orientation error results in about 11-pixels of alignment error in the image plane 24
Camera Pose Stabilization Vision tracking is used for pose refinement Vision tracking is also essential to overcome GPS dropouts Complementary vision tracker Originally developed for feature auto-calibration (99 – 2002) - Pose and 3D structure estimated simultaneously Line (edge) and point features are used for tracking Model based approach 25
Model Based Tracking Combines geometric and intensity constraints to establish accurate 2D-3D correspondence Hybrid tracking strategy − GPS/INS data serve as an aid to the vision tracking by reducing search space and providing tolerance to interruptions − Vision corrects for drift and error accumulation − Extended Kalman Filter (EKF) framework 26
Dynamic Image/model Fusion Update sensor pose and image to “paint” the scene each frame Compute texture transformation during rendering of each frame Dynamic control during visualization session to reflect most recent information Supports 1-3 real-time video streams Real-time rendering - graphics HW produces ~28fps on dual 2G PC - 1280x1024 screen 27
Dynamic Event Analysis & Modeling Video analysis − Segmenting and tracking moving objects (people, vehicle) in the scene Event modeling − Creating pseudo-3D animated model − Improving visualization & situational awareness 28
Tracking and Modeling Approach Object detection − Background subtraction − A variable-length time average background model −Morphological Filtering Object tracking − SSD correlation matching Object modeling − Dynamic polygon model − 3D parameters (position, orientation and size) Input Video Sequence Background Subtraction Thresholding and Noise Cleaning Segmentation Background Estimation Pseudo-Tracking Output Object Ground A Image Plane B
v n
C Model 29
Tracking and Modeling Results 30
Integrated AVE Environment An integrated visualization environment built in the IMSC laboratory 8x10 foot acrylic back-projection screen (Panowall) with stereo glasses interface Christie Mirage 2000 stereo cinema projector with HD SDI 3rdTech ceiling tracker A dual 2G CPU Computer (DELL) with Nvidia Quadro FX 400 graphics card Supports multiple DV video sources (<4) in real-time (28pfs) 31
Integrated AVE Environment Video demonstration 32
Interactions
Collaboration with Northrup Grumman (TRW
) - install system (8/03) for demonstrations
Publications
− IEEE CG&A (accepted): “Approaches to Large-Scale Urban Modeling” − PRESENCE (accepted): “Visualizing Reality in an Augmented Virtual Environment” − IEEE CG&A (accepted): “Augmented Virtual Environments for Visualization of Dynamic Imagery” − CGGM’03: “Urban Site Modeling From LiDAR” − VR2003: “Augmented Virtual Environments (AVE): Dynamic Fusion of Imagery and 3D Models” − SIGMM’03 (accepted): “3D Video Surveillance with Augmented Virtual Environments”
Demos/proposals/talk
− NIMA, NRO, ICT, Northrup Grumman , Lockheed Martin, HRL/DARPA, Olympus, Airborne1 33
Future Plan Automate Modeling - automate segmentation, primitive selection, fitting, fusion of imagery data Real time tracking of moving cameras – model based tracking with fused gyro, GPS, vision Dynamic Modeling – classify and model-fitting for moving objects Texture Management - texture retention, progressive refinement System Architecture - scalable video streams and rendering capability, (PC clusters?) 34