Transcript O25_1430 Julie Hengst - ISCAR 2014 Presentations

Designing ‘Robo-Buddies': Technologies
to Mediate Communication in the Wild
Julie A. Hengst, Laura S. DeThorne,
Hillary Valentino &
Maeve McCartin
Oral Session 25
1430 – 1530 Platinum Lounge
University of Illinois
Champaign, IL, USA
“Robo-Buddies” Project
• Participatory Design Project Team
– Engineers, computer scientists, communication specialists,
participant-users, disability specialists
• Collaborating to develop communication technologies that
function as pseudo-intelligent mediators of interactions
– to improve communication between diverse communicators (
e.g., students on campus with and without disabilities)
– by blending strengths of human mediators with features of AAC
devices.
Augmentative & Alternative Communication (AAC)
• Forms of communication other than oral speech
• e.g., facial expressions, gestures, symbols,
pictures, or writing;
• people with communication disorders may rely
on AAC
• AAC Devices
• specialized assistive technologies
• used by people to supplement or replace
impaired speech abilities.
• e.g., communication boards, dedicated
computer systems
http://www.asha.org/public/speech/di
sorders/AAC.htm
www.hawking.org.uk
/images.html
http://uk.dynavoxtech.com/success/cerebralpalsy/details.aspx?id=92
Activity-based Approach to AAC
Prosthetic Approach
• Grounded in transmission models
of communication
– Design technology to replace or
augment impaired body part or
function
– For individual user
– To produce linguistic messages
• Proven, but limited, approach
for…
– speech production in pre-planned,
predictable, controlled activities
Activity Approach
• Grounded in CHAT theories
– Design technology to mediate
interactions and support activities
– For functional system
– To support interactional alignments
in routine activities
• Novel approach aimed at…
– supporting interaction in dynamic,
emergent, less-controlled activities
Collecting Interactional Data
Participants
• 13 primary participants (7 AT users with disabilities; 6 comparisons)
• 7 secondary participants; 51 incidental participants
Data Collection—42 video recorded sessions
• 13 semi-structured interviews (one with each primary participant)
• 24 On-campus observations (e.g., scavenger hunts; working with PA
in dorm; navigating campus)
• 5 lab trials (e.g., working with trial technologies)
Analyzing Interactional Patterns
• 17 on-campus observations
– 4 with “David”; 3 w/ “Artemesia” & “Iris”;
2 w/ “Chip”, “Jester” & “Jesse”; 1 w/
“Izzy”.
“Artemesia”
• 11 hrs 20 mins, video data.
• Analysis:
– Transcribing all sessions.
– Coding all sessions for interactional
patterns.
– Completing situated analysis of selected
interactions.
“Iris”
“David”
Coding Interactional Patterns
• Conversational Features:
– Conversational trouble source (TS) disruptions in conversational flow, may or
may not lead to a breakdown.
– Conversational repetition, or reformulation (R) marked repetition, reported
speech, and replays, within or across modalities .
• Interactional Discourse Resources:
– Humor, or playful episodes (PE) utterances sharing a common playful theme.
– Conversational narratives (CN) Verbal or nonverbal telling of event(s) displaced
from the moment of telling, plus evaluation.
– Procedural discourse (PD) telling how to do something (e.g., giving directions).
• Object Specific Codes:
– Contextual orchestration with objects (CO) objects within activity.
– Fidgets (F), fidgeting with objects, unrelated to communicative activity.
Coding Summary: Average #/10mins of observation by participant.
(Note: Total counts across participants in grey.)
Participant Time
et al.
(mins)
TS
R
PE
CN
PD
CO
F
Artemesia
115
2.5
30
4.6
2.3
1.6
5.3
0.6
Iris
David
Chip
Jester
Jessie
Izzy
112
152
93
34
124
50
1.9
3.7
2.7
3.2
2.7
1.0
42.8
31.6
69.8
49.1
36.6
25.4
6.9
1.1
6.3
2.4
0.5
3.0
5.4
0.5
3.5
3.2
3.7
3.0
2.0
2.2
1.1
4.7
1.6
1.4
5.1
9.5
10.7
13.5
3.9
3.8
0.8
0.1
2.4
0
0.4
0.2
Total
Counts
680
181
2701
235
200
126
475
45
Interactional
Profiles
• Listed in order from
most to least.
• Overall frequency of
codes varies (e.g.,
96.5-37.8 /10 min).
• Frequency of specific
codes varies (e.g., PD
varied from 3rd - 7th)
Average of All
(60.0*): R>CO>PE>CN>TS>PD>F
Chip (96.5):
Jester (76.1):
Iris (64.9):
Jessie (49.4):
David (48.7):
Artem. (46.9):
Izzy (37.8):
R, CO, PE, CN, TS, F, PD
R, CO, PD, CN, TS, PE, F
R, PE, CN, CO, PD, TS, F
R, CO, CN, TS, PD, PE, F
R, CO, TS, PD, PE, CN, F
R, CO, PE, TS, CN, PD, F
R, CO, CN, PE, PD, TS, F
*#codes/10 minutes of observation
Interactional
Profiles
Average of All
(60.0*): R>CO>PE>CN>TS>PD>F
• Data Excerpts:
• Artemesia & Jessie
• Service Encounters,
with PA’s included
• Examples of:
• Repetition
• Contextual use of
objects
• Trouble source
• Mediational roles of
the PAs
Chip (96.5):
Jester (76.1):
Iris (64.9):
Jessie (49.4):
David (48.7):
Artem. (46.9):
Izzy (37.8):
R, CO, PE, CN, TS, F, PD
R, CO, PD, CN, TS, PE, F
R, PE, CN, CO, PD, TS, F
R, CO, CN, TS, PD, PE, F
R, CO, TS, PD, PE, CN, F
R, CO, PE, TS, CN, PD, F
R, CO, CN, PE, PD, TS, F
“Artemesia”
• 38-year-old history major;
– Employs multiple PA’s to assist
24hrs/day with physical needs
due to CP
• 2 on-campus observations
– Dining hall with a PA;
– Outside dining all, with a PA
Profile: R, CO, PE, TS, CN, PD, F
Dining Hall: “PA
interpreting story”
Dining Hall:
“selecting
meal items
with PA”
Service encounter in
dining hall with familiar
partner.
Repetition for confirming.
Artemesia directing PA to
selecting items; PA
questions some choices.
“Jessie”
• 30-yr-old, doctoral student in
Community Health &
Disability Studies;
At SHS
Building:
“story of first
meeting”
– Employs 6 PA’s to assist with
physical needs due to CP
• 2 on-campus observations
– Student Union bowling alley;
– w/ his PA, Tech Store in Union
Profile: R, CO, CN, TS, PD, PE, F
At Tech
Store:
“clickers”
Service encounter,
shopping in store with
unfamiliar clerk.
Repetition helps to
resolve trouble
source.
PA is co-shopper.
CHAT Approaches to AAC
• Shift in unit of analysis
– from “replacing” individual abilities to mediating activities of
functional systems
• Interactional data and profiles
– empirically grounded discussions
– distributed, flexible, and persistent nature of functional systems
• Imagining mediational roles for technologies
– connecting people to histories and broader contexts
– reformulating words/actions in the moment
– layering ongoing activities with interpretive frames
Acknowledgements
Grant Support:
M. Hasegawa-Johnson (PI), J.A. Hengst, L.S. DeThorne, T. Huang, P. Malik, and T.
Gunderson. Pseud-intelligent mediators ("Robo-Buddies") to improve communication
between students with and students without disabilities. UIUC IN3 Grant. Funded August
2012 - August 2014; Amount: $199,898.
Research Team and Lab Assistants:
Human Computer Interface
Karrie Karahalios, Jennifer Kim, Ha
Kyung Kong
Speech and Hearing Science
Sara Small, Katie Lester, Maeve
McCartin, Hillary Valentino, Monique
Kammo, Shanthi Sivasankaran, Carissa Disability Resource Educational Services
Ernat, Caroline Eichelberger, Gena
Electrical & Computer Engineering
Carpenter, Mariana Aparicio, Suma
Xuesong Yang, Yang Zhang, Dennis Lin,
Devanga, Christina Bronson-Lowe,
Pooya Khorrami
Participants: A total of 71 individuals as primary, secondary, or incidental participants.
References