Facilitating User Interaction with Complex Systems via Hand Gesture Recognition Joshua R. New, Erion Hasanbelliu, and Mario Aguilar MCIS Department Knowledge Systems Laboratory Jacksonville State University Knowledge Systems.
Download ReportTranscript Facilitating User Interaction with Complex Systems via Hand Gesture Recognition Joshua R. New, Erion Hasanbelliu, and Mario Aguilar MCIS Department Knowledge Systems Laboratory Jacksonville State University Knowledge Systems.
Facilitating User Interaction with Complex Systems via Hand Gesture Recognition Joshua R. New, Erion Hasanbelliu, and Mario Aguilar MCIS Department Knowledge Systems Laboratory Jacksonville State University Knowledge Systems Lab JN 11/6/2015 Outline • • • • • • Motivation System Architecture Implementation Overview Proposed Approach Demonstration Future Directions Knowledge Systems Lab JN 11/6/2015 Motivation • Gesturing is a natural form of communication • Interaction problems with the mouse – Have to locate cursor – Hard for some to control (Parkinsons or people on a train) – Limited forms of input from the mouse Knowledge Systems Lab JN 11/6/2015 Motivation (2) • Interaction Problems with the Virtual Reality Glove – Reliability – Always connected – Encumbrance Knowledge Systems Lab JN 11/6/2015 System Architecture User Rendering Hand Movement Update Object User Interface Display Gesture Recognition System Image Capture Image Input Standard Web Camera Knowledge Systems Lab JN 11/6/2015 Implementation Overview • System: • • 1.6 Ghz AMD Athlon OpenCV and IPL libraries (from Intel) • Input: • • 640x480 video image Hand calibration measure • Output: • • • • Rough estimate of centroid Refined estimate of centroid Number of fingers being held up Manipulation of 3D skull in QT interface in response to gesturing Knowledge Systems Lab JN 11/6/2015 Implementation Overview (2) • Hand Calibration Measure: • Max hand size in x and y orientations in # of pixels Knowledge Systems Lab JN 11/6/2015 Implementation Overview (3) Saturation Channel Extraction (HSL space): Original Image Hue Lightness Saturation Knowledge Systems Lab JN 11/6/2015 Proposed Approach Knowledge Systems Lab JN 11/6/2015 Proposed Approach (2) Knowledge Systems Lab JN 11/6/2015 Proposed Approach (3) Radius 0.19* ( HandSizeX HandSizeY) • The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses • A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance) • Total fingers is number of fingers minus 1 for the hand itself Knowledge Systems Lab JN 11/6/2015 Proposed Approach (4) System Runtime: • • Current time – 41 ms for one image from camera Processing Capability on 1.6 Ghz Athlon: • 24 fps Process Steps Time (ms) 1) Extract Sat Channel 9 2) Threshold 3 3) Connected Contour Fill 14 4) Centroid 2 5) Segment Hand From Arm 9 6) Refined Centroid 4 7) Count Number of Fingers 0 Total Time 41 Athlon XP 1900 (1.6 Ghz) Knowledge Systems Lab JN 11/6/2015 Demonstration System Configuration System GUI Layout Knowledge Systems Lab JN 11/6/2015 Demonstration (2) Gesture to Interaction Mapping Number of Fingers: 2 – Roll Left 3 – Roll Right 4 – Zoom In 5 – Zoom Out Knowledge Systems Lab JN 11/6/2015 Demonstration (3) Knowledge Systems Lab JN 11/6/2015 Demonstration (4) Knowledge Systems Lab JN 11/6/2015 Future Directions • • • • • Optimization Calibration Phase Defining Hand Orientation Learning System Interface Extensions For additional information, please visit http://ksl.jsu.edu. Knowledge Systems Lab JN 11/6/2015