Transcript Part 2 - Stanford HCI Group

Advanced Kinect Tricks
Shapes Game
• Goal is to demonstrate how to create a simple game that
uses Kinect audio and skeletal tracking information
• The game displays the tracked skeleton of the players and
shapes (circles, triangles, stars, and so on) falling from the
sky.
• Players can
– move their limbs to make shapes change direction or even
explode,
– and speak commands such as "make bigger"/"make smaller" to
increase/decrease the size of the shapes
– or "show yellow stars" to change the color and type of falling
shapes.
Demo
How does it work?
• App.xaml
– Declaration of application level resources.
• App.xaml.cs
– Interaction logic behind app.xaml.
• FallingShapes.cs
– Shape rendering, physics, and hit-testing logic.
• MainWindow.xaml
– Declaration of layout within main application window.
• MainWindow.xaml.cs
– NUI initialization, player skeleton tracking, and main game logic.
• Recognizer.cs
– Speech verb definition and speech event recognition.
Step 1 (Register for Events)
Kinect.Initialize(RuntimeOptions.UseDepthAndPlayerIndex |
RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor )
Kinect.SkeletonFrameReady += new
EventHandler<SkeletonFrameReadyEventArgs>(SkeletonsReady);
speechRecognizer = SpeechRecognizer.Create();
speechRecognizer.Start(new KinectAudioSource());
speechRecognizer.SaidSomething += new
EventHandler<SpeechRecognizer.SaidSomethingEventArgs>(recognizer_SaidS
omething);
Process Events
• Process skeleton data
– MainWindow.xaml.cs
– Method: void SkeletonsReady(object sender,
SkeletonFrameReadyEventArgs e)
– Update the joint data positions
• Process speech
– MainWindow.xaml.cs
– Method: void recognizer_SaidSomething(object sender,
speechRecognizer.SaidSomethingEventArgs e)
– Match it with the speech vocabulary and get the command
Translate the speech command to a list
of operations for the falling objects
• For ex. the speech commands “reset” will get
translated to:
– SetPolies(PolyType.All);
– SetDropRate(dropRate);
– SetGravity(dropGravity);
– SetSize(dropSize);
– SetShapesColor(Color.FromRgb(0, 0, 0), true);
How do you find out a hit?
• Any guess?
How do you find out a hit?
• Convert joints data into bones/segments data (a
bone is a segment that connects two joints)
• Now, we can maintain a dictionary of {bone,
<bone-data>}
• Update the segment's position and compute a
smoothed velocity for the circle or the endpoints
of the segment based on the time it took to
move from the last position to the current one.
The velocity is in pixels per second.
How do you find out a hit?
• Let’s see
– FailingShapes.cs
– class FallingThings
– public bool Hit(Segment seg, ref
Point ptHitCenter, ref double
lineHitLocation)
– if hit, the center point on the
segment being hit is returned, along
with the spot on the line from 0 to 1
if a line segment was hit.
Bouncing off after a hit
• Let’s see
– FailingShapes.cs
– class FallingThings
– public void BounceOff(double x1,
double y1, double otherSize, double
fXv, double fYv)
The useful code that you can reuse
• SpeechRecognizer.cs
– We have already covered it in lab3
• Finding out a hit between the skeleton and an
object
• The bouncing off effect.