Transcript Adrian Buckley Evan Defibaugh John Duhamel Alex Kaiser Darnell Parker Craig Riggins T Q ~Q Project Objectives - Relay real time Nintendo 64 session data between multiple hosts for network play Provide.

Adrian Buckley
Evan Defibaugh
John Duhamel
Alex Kaiser
Darnell Parker
Craig Riggins
T
Q
~Q
Project Objectives
-
Relay real time Nintendo 64 session data
between multiple hosts for network play
Provide an interface for a Microsoft Kinect
frontend in place of tradition controller
Utilize non-obtrusive electronics
Minimize cost and maximize functionality
Power consumption is a secondary concern
System Architecture
Player 1
Controller Signal
Player 2
Controller Signal
Client 1
Server
Player 1
Controller Signal
Client 2
Player 2
Controller Signal
Nintendo 64
N64 RCA
Signal
Client 1
N64 RCA
Signal
Player 1
Television
N64 RCA
Signal
Server
N64 RCA
Signal
Client 2
N64 RCA
Signal
Player 2
Television
Client Architecture
Microsoft
Kinect
Kinect
USB Signal
Regular N64
Controllers
N64
Controller Signal
ARM Cortex
MCU
Digital
Controller Signal
Analog RCA
Signal
Television
Digital
RCA Signal
Ethernet IO
MCU takes input from 4 N64 controllers
 Microsoft Kinect can track 2 people

Server Architecture
Nintendo 64
Analog
RCA Signal
N64 Controller
Signal
ARM Cortex
MCU
Digital
RCA Signal
Digital
Controller Signal
Ethernet IO
Can communicate with 1-4 clients
 Moves controller signal from client to N64
and RCA signal from N64 to client

Unified Hardware
Nintendo 64
RCA
Signal
N64 Controllers /
Microsoft Kinect
Controller Signal
Controller
Signal
ARM Cortex
MCU
Controller
Signal
RCA Signal
Television
RCA
Signal
Ethernet IO

1 hardware architecture will be able to
operate as either the client or the server
Advanced System Architecture
Player 2
Controller Signal
Client 1 /
Server
Player 1
Controller Signal
Client 2
Player 2
Controller Signal
Nintendo 64
N64 RCA
Signal
Client 1 /
Server
N64 RCA
Signal
Player 1
Television
N64 RCA
Signal
Client 2
N64 RCA
Signal
Player 2
Television
Kinect Integrated Peripheral
Kinect
XBOX Kinect
Processor
Processor
Linux Microcontroller
Linux
Microcontroller
Kinect Sensor

I/O
 2 Cameras (RGB and IR Depth)
 IR Output
 4 Array Microphone
 USB Port
Possible Implementations
ROS: BeagleBone
2010: PC
VS C# 2010: Netduino
VS C#
Implementing Kinect
via ROS on Beaglebone
ROS (Robot Operating System) is an open
source project focusing on interfacing
microprocessors with robots (in our case the
Kinect and a peripheral Linux microcontroller)
Kinect Processor
Control Node
Implementing Kinect
via VS C# 2010 on Netduino
The Netduino operates on Microsoft Visual
Studios C# 2010 via Net. Micro workforce
Minor Hiccups in Progression
Using the Netduino
Using the Robot OS
• Netduino requires
 ROS tutorial has
Net. Micro
incorrect/missing
framework, where
info
Kinect requires Net.
 Open source, thus
Framework
compatibility issues
from node to node • Trouble converting
from standard to
 Unintuitive to
micro
navigate
There is no problems implementing Kinect via PC
Last Resort and Additional info
As a last resort the Kinect can be routed
through a Windows PC for processing then
sent to microcontroller.
More ROS info
More Net. Micro workforce
http://www.ros.org/wiki/kinect
(Install ROS)
http://www.ros.org/wiki/ROS/Tutorials
(ROS Tutorial)
http://beagleboard.org/bone
(Beaglebone Description)
http://www.ros.org/browse/list.php
(ROS software)
http://docs.com/BTN4
(Installing OpenNI)
http://www.netduino.com/netduino/schematic.pdf
(Netduino Schematics)
http://informatix.miloush.net/Microframework/Arti
cles/WpfEmulators.aspx
(.net micro install)
http://informatix.miloush.net/microframework/Arti
cles/Reflector.aspx
(.net micro reflector)
http://netduino.com/netduino/specs.htm
(Netduino Specs)
Gesture and Speech Algorithms
Joint Data is returned in X, Y, Z, and W
(confidence) data
 Evaluate position of joints relative to another
and trigger button pressed event
 For audio grammar files must be written
 Compare text output of audio and set trigger
for the controller or the sensor as being used
 Output control trigger and button triggers

Kinect Processor Interface
The Processor will output triggers in the form
of buttons and sensor/controller triggers for
both players one and two
 The triggers will be relayed to the Linux
microcontroller that will convert button trigger
to N64 controller output format and use the
control trigger to decide if forwarding the N64
controller data or sensor converted data

N64 Controller Tutorial
How the N64 Controller Sends Its Data


The N64 controller only has three pins: Ground, data, and
VCC. It takes a +3.6V supply.
The N64 uses a protocol in which bits are sent serially to and
from the controller as 4µs-wide pulses. ‘Low’ is 3 µs low and
1 µs high, and ‘High' is 1 µs low and 3 µs high.
N64 Data Polling
The N64 continually polls the controller using an 8-bit
command (0x01), and the controller reports the state of its
buttons and analog joystick with a 32-bit response: 16 bits for
the buttons, signed 8 bits for horizontal joystick position, and
signed 8 for vertical. Each bit packet ends with a stop bit (1).
Button Configuration
The buttons are mapped to their respective bits as follows:
Bit
0
1
2
3
4
5
6
7
Button
A
B
Z
Start
D-Up
D-Down
D-Left
D-Right
Bit
8
9
10
11
12
13
14
15
Null
Null
L
R
C-Up
C-Down
C-Left
C-Right
Button
Sample Joystick Positions and respective bits:
Left Extreme
1
0
0
0
0
0
0
1
Left Off Center
1
1
1
1
1
1
1
1
Right Off Center
0
0
0
0
0
0
0
1
Right Extreme
0
1
1
1
1
1
1
1
Up Off Center
0
0
0
0
0
0
0
1
Down Off Center
1
1
1
1
1
1
1
1
Joystick Off-Center Left & Down
The Translator Box

The Kinect will take player movement/sound data and
send over button and trigger data to the “translator box,”
which then converts that data into the 64-bit struct
format shown below.
Button Data
Joystick Data
Which Player?
More Buttons?
Time Delay
16 bits
16 bits
1 bit (0 or 1)
1 bit
Int (30-bit)


There will be a physical master switch on the translator
box, to switch between Kinect control and N64 controller
control. When this switch is set to “Kinect,” the
translator will await trigger and button data from the
Kinect to process.
If the switch is set to “N64,” the translator will need to
keep track of one or two controllers, simultaneously
converting their button presses into the shown format,
but with the “More Buttons” bit and time delay
consistently set to zero.
Server to Translator to N64


Another Translator Box is needed to translate the
Server’s data into N64 controller data for the N64.
This translator will read in the controller data, and send
the correct button sequence via serial ports to the N64,
correctly timed. If the N64 polls the “controller,” but the
translator has not received button data from the server,
the translator will automatically respond with a 32-bit ‘0’
analog stream and continue doing so until new button
data appears.
Audio/Visual Signal


The video output of the Nintendo 64 carries
visual information via an analog signal, plus
two synchronizing signals for aligning the
video signal on a television.
The audio output, also analog, is split into
two signals for stereo speakers in a
television.
Audio/Visual Waveforms
Video
Audio
Audio/Visual Sampling


The audio and visual output of the Nintendo
64 must be digitized and formatted for input
into the server for transport over the internet.
Once the audio/visual information is sent
over the internet it must be converted back
into an analog signal to be displayed on a
player's television.
A Series of Tubes – Hardware

Initial design and testing will be done on an
Arduino with an Ethernet shield.
 Ardiuno is easy to program to send and receive
data packets.

After getting a fully functional system with the
Premade hardware we will move on to a
custom PCB with a Linux OS.
Our Internet Objectives

We need the data received from the server
to be real time
 If there is missing data we just move to the next
data packet.
 We know that the receiver is ready to receive
packets.
TCP/UDP

TCP/IP uses an established connection
between sender and receiver.
 This requires an acknowledge signal from the
receiver which takes time.

UDP instead sends packets in a continuous
stream and doesn’t care if packets are
missing on the receiver side.
 Instead of waiting for acknowledgement UDP just
tags the packet with the correct receiver IP
address and sends it on its way.
UDP in depth

UDP sends data with a header, and within
the header a 2byte portion telling the
receiver how long the data section will be.
 This 2byte section allows for a maximum of 64Kb
section of data which is more than enough for
sending the n64 signal and frames of an RCA
signal.
 The other parts of the header are receiver IP and
port number.
Putting it all Together
Using UDP because we need a real time
connection and don’t mind dropped packets.
 Using a static IP for our server so that we
have something to aim at for our UDP
packets.
 Using Arduino and Ethernet shield to
prototype and onto custome PCB once all
kinks are worked out.

Project Schedule
Longest Design
Period
37 Days for Integration
11 Days for Testing
Technical Reference Manual
Division of Labor
Task
Adrian
Program Mgmt &
Documentation
P
Kinect Programming
S
N64 Translation
Evan
John
Alex
Darnell
S
P
S
Ethernet Interface
P
S
S
Audio/Video Signal
Conversion
S
P
S
N64 Gameplay Action
P
P
P
P = Primary
S
S
P
Controller Programming
Craig
S
P
P
P
S = Secondary
P
P
Advanced Parts List
Hardware
Television
XBOX Kinect
Nintendo 64
Nintendo 64 Controller
Nintendo 64 Controller
Nintendo 64 Controller Cable Extension
Nintendo 64 ROMS
Netduino
Beagle Bone
Modem
Parts for Printed Circuit Board
XBOX Kinect / Nintendo 64 Shield
RCA Cable
Case
PCB Development / Revisions
Cost Quantity Extended Cost Possible Substitutions
Provided
Provided
Provided
Provided
$19
$10
Provided
$60
$90
Provided
$50
$14
Provided
$10
$500
2
2
2
3
4
2
7
2
3
1
1
2
1
1
TBD
Provided
Provided
Provided
Provided
$76
$20
Provided
$120
$270
Provided
$50
$28
Provided
$10
$500
Beagle Board
Coax Cable
TOTAL ESTIMATED PROJECT COST = $1,074
*NOTE* – ALL EXPENSES PAID TRIP TO LAS VEGAS FOR “ELECTRONICS
SEMINAR” NOT INCLUDED
Goals
Primary:
• Control Nintendo 64 game exclusively via an XBOX Kinect
Secondary:
• Have one N64 controller communicate with one N64
console over the internet
• Control the N64 game using either the Kinect or the N64
controller remotely
Tertiary:
• Play a complete N64 game with 2 players at one location
using the Kinect, and the other 2 players at a different
location using N64 controllers
• Allow functionality with a variety of different controllers
Milestone 1 Objectives
Kinect Translation
• Kinect translator sends simple gesture data to
the Linux µController, in correct format
Nintendo 64
• Send properly timed Nintendo 64 analog signal
from the µController to the N64 console
Network Configuration
• Send data packets across the network to and
from each user using the µController
Audio/Video
• Convert analog input into a digital code
Milestone 2 Objectives
Kinect
 Determine ideal Kinect gestures for
satisfactory user interface
 Transmit verbal commands through Kinect
Network Configuration
 Send and receive N64 and audio/video
signals across server
Audio/Video
 Output signal from server to television
Risks and Contingency Plan
Risks
 Syncing the N64 timing sequence with the Netduino
microcontroller’s internal timer
 Busy internet path/lost data packets
 Delay between Kinect / N64 input and television
output
 Data corruption or potential injury resulting from
falling or player collisions while playing with a
motion sensitive controller
Solutions
 Use only N64 controllers (no Kinect) for game
control
 Leave out server / direct connection from
microprocessor to N64
 User plays video game at own risk!!
Questions???