Transcript The Portable Sign Language Translator (PSLT): from wishful

The Portable Sign Language Translator (PSLT):
from wishful thinking
to commercial product
Ernesto Compatangelo, Ryan Russell
PSLT project (http://www.pslt.org)
Technabling Ltd
PSLT development is funded by the Small Business Research Initiative (SBRI)
sponsored by the Department for Business Innovation and Skills (BIS) and the Technology Strategy Board (TSB)
and managed by JISC TechDis, a leading UK advisory service on technology and inclusion.
The PSLT in a nuthell
●
●
NOW: translates sign languages (BSL, CSL) into
text;
2014: text into sign languages (using an Avatar)
NOW: uses Customisable SL (CSL) to control
devices
●
BCS, 16/10//2013
2 of 20
Talk structure
●
Motivations and background
●
Relevant features
●
A short demo of the PSLT in action
●
Technical challenges
●
Social, cultural, and mundane issues
●
Wider applicability of video-based analysis
BCS, 16/10//2013
3 of 20
Motivating the PSLT (1)
●
●
●
There are around 100,000 Sign Language (SL)
users
in the UK only (mostly BSL); millions worldwide
Profoundly deaf SL users only communicate w/
sign
Most Voice & Text (V&T) users do not sign
No signer-to-speaker communication gaps in
order
BCS, 16/10//2013
to remove educational, workplace, lifestyle
●
4 of 20
Motivating the PSLT (2)
●
●
●
Sign interpreters are few and expensive – only use
when strictly needed
A tool to self-learn SL: low cost, anywhere, any
time
Enabling independent life means allowing gesture
and vocabulary customisation, not just translation
BCS, 16/10//2013
5 of 20
Motivating the PSLT (3)
The Small Business Research Initiative (SBRI)
●
UK Gov needs new products for wider society
●
1 - Contract awarded to small businesses for R&D
●
2 - Follow-up for market-ready commercial product
●
Cost-effective, delivering value for money product
●
Developed technologies owned by businesses
●
NOT for the academic sector
BCS, 16/10//2013
6 of 20
Requirements & desiderata
●
Sign to text (the real challenge) - no text to sign
●
Works on “mobile” technology
●
Open source components only – no strings
attached
●
Output: text/sound + execution of commands
●
Discreet, unobtrusive, flexible, to run on std HW
BCS, 16/10//2013
7 of 20
NO-GO areas (1)
What about these as a translation device?
Signing gloves for computerised sign-to-text translation
BCS, 16/10//2013
8 of 20
NO-GO areas (2)
What about this as a translation device?
Prototype device for gesture control
BCS, 16/10//2013
9 of 20
NO-GO areas (3)
What about this as (part of) a translation device?
Kinect, a motion sensing input device by Microsoft
for the Xbox 360 video game console and Windows
PCs
BCS, 16/10//2013
10 of 20
Core PSLT feature
●
●
●
●
●
No weird/costly devices – just off-the-shelf HW
Translates BSL + CSL (combined) + Makaton into
text
Has provisions for other SLs & regional variations
Fully customisable: users can create their own
signs and give them specialist meanings not in
other SLs
Works on many devices, from smartphones to PCs
BCS, 16/10//2013
●
11 of 20
Signs translated into ambient device commands
Time for a demo... the PSLT in action
BCS, 16/10//2013
12 of 20
Problems tackled:
what others avoided to consider
●
Dynamic background: light, movement
●
Ind. form factors: skin, body shape, signing style
●
The inherent fuzziness of the signing process
●
●
Recognise & disambiguate ALL hand gestures
(~1200)
Individual signs, profiles, & regional variations
Create your gestures with your specialist meaning
13 of 20
●
BCS, 16/10//2013
The challenges of image recognition
Identify & track moving shapes in the “foreground”
against a noisy background with sudden light
changes
(the signer does also move while signing!)
BCS, 16/10//2013
14 of 20
The technical challenges
●
●
●
●
Recognise (i) individual signs and (ii) hand
gestures
Disambiguate meanings if gestures are overloaded
Articulate adverbs, verbs, adjectives, subjects,
objects
in a grammatically correct and meaningful
sentence
Bridge SL syntax with written text syntax
BCS, 16/10//2013
●
Run under Win, Linux, Android with basic HW
15 of 20
Typical objections (not from real
users)
●
●
●
●
It can't be done - Academic linguists
It won't work – there is “much more” to SLs than
just hand gestures (e.g., facial expression)
- Academic linguists + worried interpreters
You are offending the sign language – as above
Money spent on technology should be better spent
on extra interpreters – interpreters, social workers
– some members of “the voice of deaf people” +
interpreters
Can
you
spot
vested
interests
/
marginalised
BCS, 16/10//2013
complainers?
16 of 20
The position of “real” stakeholders
●
“PSLT has the potential to revolutionise the way
deaf and hearing impaired individuals can
communicate in every day situations that hearing
people take for gramnted” - Jon Smith, Vice Principal,
Communication Specialist College, Doncaster
●
“Not having to rely on an interpreter all the time
would be good” - Young learner, Communication Specialist College,
Doncaster
●
“I can see the benefits and impact rhis will have on
my ability to communicate in a range of
community settings” - Brian Kokoruwe, inclusion consultant and BSL
user
BCS, 16/10//2013
17 of 20
The near future
●
Facial expressions (technology there, needs time)
●
Performance (speed), particularly on Android
●
Automated profile upload/download
●
●
BSL word usage frequencies for overloaded
gestures
(joint project with UCL, end of 2013)
Text-to-sign Avatar (proof of concept successful)
BCS, 16/10//2013
18 of 20
Thank you for your time
Any questions?
Please ask
BCS, 16/10//2013
19 of 20
Other relevant PSLT features
●
●
●
●
Gestures can have multiple meanings: signers
choose
Text sent to PSLT-hosting display or other devices
Avatar for text-to-sign available on smartphones
too
Signs translated into ambient device commands
and
local/remote appliances (via Bluetooth, WiFi, GSM)
BCS, 16/10//2013
20 of 20