Transcript MRS-LAB-04-Augmented_Reality
Mixed Reality Systems
-Lab IV – Augmented Reality Christoph Anthes
Overview
• ARToolKit • Combining ARToolKit with OpenSG • Initialisation • ARToolKit Loop • Helper Functions • Tracking Objects • Interaction with objects • Combining ARToolKit with inVRs Mixed Reality Systems 2 Lab
ARToolKit
• Current Version 2.72.1available at Sourceforge page • C and C++ API • Cross-platform API (Windows, Linux, MacOS, IRIX) • OpenGL used for rendering, GLUT used for event handling • Used video library depends on chosen platform • Architecture – General Picture • Build on GLUT with OpenGL, C and C++ • Makes use of the video device specific and graphics drivers • Often used only as an independent tracking library • Own application is designed to be build on top of OpenGL and ARToolKit
From http://www.hitl.washington.edu/artoolkit/documentation/
• Interfaces to larger tracking libraries exist (e.g. OpenTracker) Mixed Reality Systems 3 Lab
ARToolKit
• Architecture – ARToolKit Focus • AR module • Core module with marker tracking, calibration and parameter collection • Video module • Collection of video routines for
From http://www.hitl.washington.edu/artoolkit/documentation/
capturing the video input frames • Wrapper around the standard platform SDK video capture routines • Gsub module • Graphic routines based on the OpenGL and GLUT libraries • Gsub_lite module • Replaces GSub with a more efficient collection of graphics routines • Independent of any particular windowing toolkit Lab
ARToolKit
• Coordinate Systems • Several coordinate systems are to be used • Most important camera and marker coordinates • From camera to screen coordinates a transformation via a distortion function can be performed
From http://www.hitl.washington.edu/artoolkit/documentation/
• Z-axis of the marker is pointing upward • Z-axis of the camera is pointing in the scene • Top left corner of the screen is 0,0 • arGetTransMat() • Returns the coordinates of the marker in the Camera coordinate System • arMatrixInverse() • Returns the coordinates of the camera in the marker coordinate system Lab
• Basic Application
ARToolKit
From http://www.hitl.washington.edu/artoolkit/documentation/
Mixed Reality Systems 6 Lab
ARToolKit
• Corresponding Function Calls in plain ARToolKit API
From http://www.hitl.washington.edu/artoolkit/documentation/
Mixed Reality Systems 7 Lab
ARToolKit
• Connections to Scene Graphs in General • Three steps • Initialising the Camera • Transforming the Object • “Real Occluders” – slightly advanced • OpenSG • Creation of a new node with fscEdit • OpenSceneGraph • Extension of scene view class • Display of occluding geometry • Overwrite colour buffer with video image • Finally display of recognised objects • Two big approaches • OSGART ( http://www.artoolworks.com/community/osgart/index.html
) • OSGAR ( http://www.gvu.gatech.edu/ael/projects/ARSceneGraph.html
) Mixed Reality Systems 8 Lab
Combining ARToolKit and OpenSG
• OpenSG provides examples for interconnecting either ARToolKit or ARToolKit Plus • We are going to work with ARToolKit • So we first need the additional include files • gsub.h – contains main display functions used in ARToolkit • video.h – provides multi-platform video input support for ARToolKit • param.h – contains principal routines for loading, saving, and modify camera parameters • ar.h – provides image analysis and marker detection routines Mixed Reality Systems 9 Lab
Combining ARToolKit and OpenSG
• If we take a look at our example we start with a set of forward declarations some which we have not seen in the previous OpenSG or inVRs tutorial • Additionally more methods are used at the end of the code • We start with the setup and cleanup methods: • initARToolkit() is used for initialisation of the ARToolKit components of the example • initOpenSG() initialises the OpenSG setup of the example • initGlut() registers the GLUT callbacks as we have seen in previous examples • setupCamera () provides a setup for our interconnected webcam • cleanupARToolkit() stops the video capture of ARToolKit and closes the video stream processing • cleanupOpenSG() frees used variables, stops the binding to ARToolKit, and call osgExit() • Let’s have a detailed look at the setup methods Mixed Reality Systems 10 Lab
Combining ARToolKit and OpenSG
• initARToolkit() • Wraps and calls the different internal setup functions for ARToolKit • The cleanup method is registered as a callback at program termination Mixed Reality Systems 11 Lab
Combining ARToolKit and OpenSG
• setupCamera() • The camera setup is defined in this function • Camera parameters from calibration file are parsed and evaluated • Conversion has to take place in order to write data out in the right format • The ModelViewMatrix and the ProjectionMatrix are set • This setup is stored inside an OpenSG camera object • The camera parameters are stored in binary files • E.g. currentParams.mat
Mixed Reality Systems 12 Lab
Combining ARToolKit and OpenSG
• setupCamera() Mixed Reality Systems 13 Lab
Combining ARToolKit and OpenSG
• initOpenSG() • The standard OpenSG setup is performed • A GLUT window is created and initialised • A root node with an anonymous group core is created • The previously described camera setup is triggered • A background object is interconnected with a video texture • The SimpleSceneManager is initialised and interconnected with the window and the root node of the scene • The background image as well as the camera are attached to the viewport of the just generated window • All changes are committed to OpenSG • Finally the OpenSG cleanup function is registered to be triggered at the termination of the application Mixed Reality Systems 14 Lab
Combining ARToolKit and OpenSG
• initOpenSG() Mixed Reality Systems 15 Lab
Combining ARToolKit and OpenSG
• Then we have the ARToolKit processing methods • In the captureFrame() method the image is retrieved from the camera • The detectMarkers() function triggers marker detection and outputs the amount of found markers • applyMarkerTrans() applies the transformation from a given marker to an OpenSG transformation core • ARToolKit Loop • These steps represent our steps 2-4 from our ARToolKit loop • Step 1 was the initialisation given with the previous set of functions • Step 5 the rendering is performed by OpenSG • They have to be processed frame by frame, thus they are called inside the display loop of the application Mixed Reality Systems 16 Lab
Combining ARToolKit and OpenSG
• captureFrame() • This method retrieves an image which was captured by ARToolKit from the incoming video stream • The image data is set in an OpenSG image and an update notification is issued • This frame will later on be rendered as a background image • It is as well used by ARToolKit for image processing which is performed in the next step Mixed Reality Systems 17 Lab
Combining ARToolKit and OpenSG
• detectMarkers() • In this method the markers visible on the image are detected • The amount of detected markers as well as the markers themselves are returned by reference • A threshold parameter determines the binarisation of the image • Output describing the amount of detected markers as well as the ids of the markers is generated on the console Mixed Reality Systems 18 Lab
Combining ARToolKit and OpenSG
• applyMarkerTrans() • Extracts the transformation information from a marker and applies it to a given OpenSG model • By using an STL map a binding between • The marker transformation is requested from ARToolKit via arGetMarkerTrans • It is then converted into an OpenSG matrix • If a marker is found in the map it becomes activated again and the transformation matrix just retrieved is applied on an OpenSG object • The changes are finally committed Mixed Reality Systems 19 Lab
Combining ARToolKit and OpenSG
• applyMarkerTrans() Mixed Reality Systems 20 Lab
Combining ARToolKit and OpenSG
• Additional methods which are used as helper functions and appear in the code • The method arMatrixToOSGMatrix() performs a data conversion from ARToolKit to OpenSG • The argConvGLcpara() method is used for conversion of ARToolKit camera parameters to OpenGL parameters • getTranslation() returns the translation vector of a transformation matrix Mixed Reality Systems 21 Lab
Combining ARToolKit and OpenSG
• arMatrixToOSGMatrix() • Some very basic reformatting of an ARToolKit Matrix to an OpenSG Matrix is performed in this method Mixed Reality Systems 22 Lab
Combining ARToolKit and OpenSG
• argConvGLcpara() is used to transform ARToolKit intrinsic camera parameters matrix format to an OpenGL matrix format • More details on camera calibration and the parameters are given in the computer vision class in Winter semester Mixed Reality Systems 23 Lab
Combining ARToolKit and OpenSG
• Additional methods which are used as helper functions and appear in the code • createPattern() connects a marker with an OpenSG sub scene graph • createBackground() creates an image background based on the image gathererd from the ARToolKit Video stream • createModel() loads a sub scene graph from disk and equips it with an additional transformation code • Before we start coding let’s have a more detailed look at these methods Mixed Reality Systems 24 Lab
Combining ARToolKit and OpenSG
• createPattern() • This helper method interconnects a pattern from ARToolKit with a sub scene graph provided by OpenSG, it uses createModel() as helper • The pattern is then registered at an STL map Mixed Reality Systems 25 Lab
Combining ARToolKit and OpenSG
• createBackground() • This method creates the image background of OpenSG based on the data gathered from the ARToolKit video stream Mixed Reality Systems 26 Lab
Combining ARToolKit and OpenSG
• createModel() • This is a helper method which simply loads a model and attaches it to a node with a ComponentTransform core • Now we should know all necessary helper functions and will go on with the main function and the display function Mixed Reality Systems 27 Lab
Tracking Objects
• Our main function in this example is very simple since most of the processing is performed in the display loop • Initialisation of OpenSG and ARToolKit is performed • The video loop and the display loop are triggered • We now insert our first snippet into main in order to display an object on a marker
Snippet 1-1
• Compile and execute now Mixed Reality Systems 28 Lab
Tracking Objects
• If we now take a closer look at our display loop we can see at the beginning our 3 ARToolKit steps • We should now enhance the scene by adding two objects and removing the first one • If you compile and execute you code now you should be able to see two objects
Snippet 2-1
• Now we want to perform some interaction between the objects Mixed Reality Systems 29 Lab
Tracking Objects
• With the next snippet we check the proximity of the two objects and change the scale • First we retrieve the transformations of the objects and calculate the distance Mixed Reality Systems 30
Snippet 2-2 – First Part
Lab
Interaction with Objects
• We calculate the scale based on the distance and apply it on the transformation matrices of our two objects • Afterwards changes have to be committed Mixed Reality Systems 31
Snippet 2-2 – Second Part
Lab
Interaction with Objects
• If you execute your code now you should see something like this (most likely with a different user) Mixed Reality Systems 32 Lab
Interaction with Objects
• In the next step things will become more complicated an object should move from a marker to another marker • Actually we want to have a frog jump from one stone on a marker to another stone on a different marker • First we uncomment the scaling part of the object by inserting the following snippet Mixed Reality Systems 33
Snippet 3-1
Lab
Interaction with Objects
• We initialise a frog model and other nodes for our animation Mixed Reality Systems 34
Snippet 3-2
Lab
Interaction with Objects
• Now we trigger the initialisation in the main method
Snippet 3-3
• For the animation of motion between one and another stone we install a timer • Inside the Display loop delta values are calculated in order to determine the time since the last frame • An overall counter is incremented Mixed Reality Systems 35
Snippet 3-4
Lab
Interaction with Objects
• For animating a jumping frog from one stone to another stone, we implement a state machine with 5 states • SITTING – the frog sits for a given time before it can try to jump, afterwards it automatically switches to the WAITING state • WAITING – two markers have to be close enough to make the frog jump, if the proximity check returns positively it will jump • JUMPING – in the state a change in the scene graph is performed. The frog node is attached to a transformation node • FLYING – in this state the actual calculation for the trajectory of the frog is calculated and applied on the transformation node. Once the target is reached the state changes to LANDING • LANDING – as in the JUMPING state a change in the scene graph hierarchy is performed. The state machine switches to SITTING again.
• The whole state machine is implemented inside a single snippet through which we will go through now Mixed Reality Systems 36 Lab
Interaction with Objects
• State machine – SITTING – WAITING • We check the passed time in sitting state and switch after 2000ms to the WAITING state. The counter is reset afterwards.
• In the WAITING state we check the marker proximity and switch to the JUMPING state if successful.
Mixed Reality Systems 37
Snippet 3-5 Part 1
Lab
Interaction with Objects
• State machine – JUMPING • We detach the frog from the stone and attach it to a transformation core the transformation matrix of the core is initialised with the markers transformation matrix • The state changes to FLYING and the counter is reset.
Mixed Reality Systems 38
Snippet 3-5 Part 2
Lab
Interaction with Objects
• State machine – FLYING • Retrieve the target position and the current position and determine the path of the frog • In case the flying time becomes to high we switch back to the LANDING state • If the distance between the frog and the target becomes to big it is going back Mixed Reality Systems 39
Snippet 3-5 Part 3
Lab
Interaction with Objects
• State machine – FLYING continued • If everything went alright we determine the direction vector by normalising the path • We scale this normalised vector to move the frog at a constant speed based on the current deltaTime • No we calculate a transformation matrix based on this vector • Finally the generated transformation is applied on the frog transformation Mixed Reality Systems 40
Snippet 3-5 Part 4
Lab
Interaction with Objects
• State machine – LANDING • The frog is detached from the transformation node and attached to the new marker • The counter is reset and the state is set back to the initial SITTING state
Snippet 3-4 Part 5
• At the end of each iteration of the display loop the changes are committed 41 Lab
Interaction with Objects
• This is how it should look • The final step is fully up to you to implement • Have a frog jumping between three markers • You will have to load an additional marker • You will have to determine the distance between the three different markers and determine the closest • Let the frog hop from one marker to the closest marker Mixed Reality Systems 42 Lab
Combining ARToolKit with inVRs
• inVRs provides a simple binding to the ARToolKit which implements interaction with markers and the scene • The next lab will show some basic desktop interaction with the ARToolKit • To enable full AR support for the inVRs framework the input interface as well as the output interface can be enhanced • Things to do at home • Try to make two frogs switch stones when they look at each other • Take a look at multi-pattern calibration • Try to interconnect inVRs with the ARToolKit and write an ARToolKit marker input device • Take a look at ARToolKit Plus, what are the advantages?
• Can you manage to interconnect ARToolKit Plus with OpenSG or inVRs?
Mixed Reality Systems 43 Lab
Useful Links
• ARToolKit • • • http://artoolkit.sourceforge.net/ - Sourceforge Entry page http://www.hitl.washington.edu/artoolkit/ - GPL version http://www.artoolworks.com/ - commercial version • ARToolKit Plus Web Page • http://studierstube.icg.tu-graz.ac.at/handheld_ar/artoolkitplus.php
• OpenSG Web Page • http://www.opensg.org/ • inVRs Web Page • http://www.invrs.org/ Mixed Reality Systems 44 Lab