Transcript Development of immersive real
Development of immersive real time applications
Miroslav Andel [email protected]
Lectures
• • Development of immersive real-time applications – – – Online/direct rendering Hardware Programming Video production – – Offline rendering Hybrid solutions
2014-09-03 Miroslav Andel - Development of immersive real-time applications 2
Outline
• • • Hardware – – – Understanding the technology Clusters and GPUs Warping, color matching & blending Immersion in a virtual environment (VE) Software design – – – – Desktop application vs. immersive cluster application Scalability & configuration of VEs Frameworks Scene graph
2014-09-03 Miroslav Andel - Development of immersive real-time applications 3
Hardware
4 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Projectors
• • •
6x Digital Projection Titan 1920 x 1200 @ 120 Hz Frame doubling
5 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Image Generators (IGs)
– – – – Workstation high-end graphics cards • • • • • Frame sync External house sync Genlock (video signal sync) Swaplock (swap buffer sync) 3D sync (for 3D glasses) Supports active stereoscopy using OpenGL Nvidia Quadro or AMD/ATI FirePro No gaming cards
2014-09-03 Miroslav Andel - Development of immersive real-time applications 6
Gaming vs. Workstation GPU
• • • • GPU is pretty much the same but different driver software and firmware Gaming cards are optimized for speed and DirectX – Hi frame-rate in games, highly optimized Workstation cards are optimized for precision, OpenGL & OpenCL – – – Designed for 3D modeling, CAD, video editing and computation Higher floating point and sub-pixel precision Better quality and larger memory (12 GB in Quadro K6000, 16 GB AMD FirePro 9100) • Ray tracing on the GPU Our problem: We use gaming techniques but need workstation features like synchronization and stereoscopy (less performance)
2014-09-03 Miroslav Andel - Development of immersive real-time applications 7
Cluster of IGs
• • • Often consists of workstations with the same specification Master – Slave In our case the master has an additional surround sound card (live encoding to Dolby Digital or DTS)
8 Miroslav Andel - Development of immersive real-time applications
Clusters
(at the visualization center) • • Dome – Alpha (primary real-time cluster used for shows) – – Beta (secondary development cluster) Delta (movie and presentations playback cluster) – Gamma (single node eyefinity machine) VR-Arena – – Theta Videowall (just a single node with 6 outputs) used for movie and presentations playback
Miroslav Andel - Development of immersive real-time applications 9
•
Color/gammut matching
What is white?
11 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Blending
• Digital blending – good bright levels – – Not a linear gradient (gamma) Done after color and brightness matching
15 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Blending
• Optical blending for better dark levels – Projected black is not black, still projected light
16 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Blending
• Optical blending for better dark levels – Projected black is not black, still projected light
17 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Blending
• Blending in the dome – – Blue = Projection borders Red = Blend zones
2014-09-03 Miroslav Andel - Development of immersive real-time applications 18
Warping/geometric correction
• • • Correct the rendered frame to match the geometry of the screen, lens and projector’s position Correct overlaps between adjacent channels Spatial 2D transform
19 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Warping and blending hardware
• Processing unit with input & outputs cards (FPGA) with minimal latency
20 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Warping and blending using OpenGL
• No latency but uses a bit of the GPU
21 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Calibration process
• • • Manual warping and blending calibration takes many hours – – Few reference points in the dome Moving mesh points by hand causes linearity problems Semi-automated calibration Automated camera-based systems – Current systems don’t handle optical blends
22 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Hardware (end)
23 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Immersion
• To give the audience the impression that they have stepped into a synthetic world
24 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Immersion
• • To give the audience the impression that they have stepped into a synthetic world Enhance immersion by: – – – – – – Large field of view High resolution Stereoscopy (can also reduce immersion) Surround sound Smooth navigation (based on a physical model) Realistic rendering (shadows etc., to improve depth cue)
2014-09-03 Miroslav Andel - Development of immersive real-time applications 25
Immersion
• Stereoscopy in the dome?
– – – The paradox Field of view wearing 3D glasses Use stereoscopy wisely
2014-09-03 Miroslav Andel - Development of immersive real-time applications 26
Software design
28 2014-09-03 Miroslav Andel - Development of immersive real-time applications
The difference
(from normal desktop application development) • • • • Several projectors/screens – – – A lot of pixels Alignment, blending & warping Color & Brightness matching Several computers (cluster) Synchronization – – – – Frame Buffers swap Data Video signal (genlock) Multi-threading
2014-09-03 Miroslav Andel - Development of immersive real-time applications 29
Stereoscopy • How does it work?
30 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Stereoscopy
31 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Stereoscopy
32 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Stereoscopy
33 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Stereoscopy
•
Co-location
– – Camera position = eye position Projection plane = real physical screen
34 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Stereoscopy
• • Don’t change eye separation, change the scale of your scene ~15 meters limit
35 2014-09-03 Miroslav Andel - Development of immersive real-time applications
2014-09-03 Miroslav Andel - Development of immersive real-time applications 36
2014-09-03 Miroslav Andel - Development of immersive real-time applications 37
2014-09-03 Miroslav Andel - Development of immersive real-time applications 38
•
Stereoscopy
Object in focus in negative parallax
39 2014-09-03 Miroslav Andel - Development of immersive real-time applications
•
Stereoscopy
Object in focus in positive parallax
40 2014-09-03 Miroslav Andel - Development of immersive real-time applications
•
Stereoscopy
Problems with a fixed head setup (no head tracking)
41 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Scalability
• One application can run in several virtual environments without the need of changing the code – – – – – In the dome In the VR-Arena In the VR-Lab At home using your computer At the bus using a laptop • If the platform is the same then there is no need for recompiling the application (just
42
Scalability
• • • • • • • •
The application uses a like: configuration file that describes the hardware in terms
Number of IGs Number of displays Number of viewports Physical location of displays/projection planes Physical location of the user Eye separation of the user Type if Stereoscopy Etc…
2014-09-03 Miroslav Andel - Development of immersive real-time applications 43
• “3d drawing”
Configuration
44 2014-09-03
Monoscopic projection
Configuration
Immersive applications Often uses non-symmetric frustums/projections, while normal OpenGL applications don’t (left FOV = right FOV, top FOV = bottom FOV)
2014-09-03 45
Stereoscopic projection
Configuration
46 2014-09-03
Configuration
47
All users except the target user will perceive the rendered object incorrectly.
Configuration
48
All users except the target user will perceive the rendered object incorrectly.
One solution would be using HMDs or a multi-view screen.
Configuration
49
Navigation
• In a desktop application a user navigates by moving the camera
50 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Navigation
• • In a desktop application a user navigates by moving the camera In a scalable application the number of cameras/frustums is not known
51 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Navigation
• • • In a desktop application a user navigates by moving the camera In a scalable application the number of cameras/frustums is not known Instead of moving the user and all “cameras” in the same direction the whole scene is moved in the opposite direction
52 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Frameworks
(the good stuff) • Will help you with – – – – Cluster setup & synchronization Frustums & viewports Input Tracking
2014-09-03 Miroslav Andel - Development of immersive real-time applications 53
• VR Juggler
Frameworks
• • CAVElib (commercial) Equalizer
2014-09-03 Miroslav Andel - Development of immersive real-time applications 54
Frameworks
(the bad stuff) • • • • Massive and have a lot of dependencies Big initial effort to get started Often in legacy OpenGL Takes usually several days to get started – – – – Getting all dependencies Compile all dependencies Compile the whole framework Read the documentation
55 2014-09-03 Miroslav Andel - Development of immersive real-time applications
SGCT Simple Graphics Cluster Toolkit
56 2014-09-03 Miroslav Andel - Development of immersive real-time applications
•
SGCT Simple Graphics Cluster Toolkit
Static library based on – – – – – – – – – – GLFW GLEW GLM Freetype TinyThread++ TinyXML PNG Z VRPN TurboJPEG
2014-09-03 Miroslav Andel - Development of immersive real-time applications 57
SGCT Simple Graphics Cluster Toolkit
• • • • • • Developed here and is free for everybody to use Designed to get started immediately (rapid development) Cross-platform C++ Modern and Legacy OpenGL support (OGL 2.2 – 4+) Single static library (no dll-hell) Using callbacks rather than inheritance/polymorphism similar
58
SGCT Simple Graphics Cluster Toolkit
• • Wiki at: c-student.itn.liu.se
– – – – – Documentation Tutorials Configuration files Downloads (installers) Support Forum (on it’s way)
2014-09-03 Miroslav Andel - Development of immersive real-time applications 59
SGCT – How it works
• • Has an engine which handles – – – – – – Initiation of OpenGL Rendering Synchronization Network communication Configuration handling Input The user can bind callbacks (function pointers) to the engine to customize the functionality
2014-09-03 Miroslav Andel - Development of immersive real-time applications 60
2014-09-03 61
SGCT – How it works
• • • Set only the callbacks you need A minimal cluster application contains: – – Draw callback: Render your stuff Pre Sync callback: generate the data to be shared – – Encode callback: Send the data to the slave(s) Decode callback: Receive the data from the master Other callbacks to handle input devices and external GUIs
2014-09-03 Miroslav Andel - Development of immersive real-time applications 62
SGCT - Features
• • Frame-, swapbuffer- and application data synchronization 3D/Stereoscopic rendering – – – – – – Active/Quadbuffer stereo Side-by-side stereo Top-bottom stereo DLP/Checkerboard stereo Line interleaved stereo Anaglyph stereo
2014-09-03 Miroslav Andel - Development of immersive real-time applications 63
SGCT - Features
• • • • • • • • • • • • Fisheye rendering PNG texture management GLSL shader management Freetype font management Error messaging across the cluster External TCP control interface Tracking support Joystick, gamepad, Kinect, steering wheel etc. input Antialiasing 3D screenshots in maximum texture resolution Overlays Statistics and performance graphs
2014-09-03 Miroslav Andel - Development of immersive real-time applications 64
SGCT - Configuration
• • XML configuration All nodes are using the same configuration and can automatically determine if they are a slave or a master
65 2014-09-03 Miroslav Andel - Development of immersive real-time applications
66
SGCT arguments
• • • Normal use: app.exe –config “config.xml” Running cluster application locally as master: app.exe –config “cluster_config.xml” –local 0 Running cluster application locally as slave: app.exe –config “cluster_config.xml” –local 0 - slave
67 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Running SGCT
• • • •
Command prompt/terminal Create a Batch file (.bat) Create a Shortcut From IDE
2014-09-03 Miroslav Andel - Development of immersive real-time applications 68
Running cluster applications
• • • The application folder needs to be accessible from all IGs/nodes 1. Copy the whole folder • Use Microsoft’s robocopy tool for mirroring folders across a cluster 2. Use a network share DemoManager tool for launching (and killing) processes across a cluster Or manually start the application on all nodes
2014-09-03 Miroslav Andel - Development of immersive real-time applications 69
SGCT – Example
• Basic spinning triangle synchronized across a cluster using less then 80 lines of code (including comments and spaces)
70 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Data synchronization
• – – – –
What to share?
– – Time or dt Run simulations only on master or on whole cluster?
Random numbers?
Minimize traffic Compression Bit-masking
2014-09-03 Miroslav Andel - Development of immersive real-time applications 71
Math
• • • • OpenGL Mathematics (GLM) is included in SGCT Computer graphics is linear algebra!
GLM makes life easy – – – Vectors Matrices Quaternions Similar syntax to glsl
72 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Testing anywhere
• •
Local cluster on a single computer View
73 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Testing anywhere
74 2014-09-03 Miroslav Andel - Development of immersive real-time applications
2014-09-03
SGCT + OSG
• • • • •
Open Scene Graph (OSG)
www.openscenegraph.org
Feature rich SGCT manages – – Cluster sharing & synchronization Window, frustum and viewport creation OSG manages – Content OSGMaxExporter (3Ds Max plugin)
Miroslav Andel - Development of immersive real-time applications 75
Scene graph
• • Tree structure of nodes Intelligent structure – Visibility & occlusion culling
2014-09-03 Miroslav Andel - Development of immersive real-time applications 76
OSG example
77 2014-09-03 Miroslav Andel - Development of immersive real-time applications
Audio
•
Positional 3D audio
– – OpenAL Fmod Ex
2014-09-03 Miroslav Andel - Development of immersive real-time applications 78
Next time
• • Video pipeline Offline rendering for the dome
79 2014-09-03 Miroslav Andel - Development of immersive real-time applications