Transcript Pen Research Jay Pittman Development Lead Tablet PC Handwriting Recognition Microsoft Corporation TDNN Time-Delayed Neural Network Ink is cut into segments via simple algorithm For Latin script,

Pen Research
Jay Pittman
Development Lead
Tablet PC Handwriting Recognition
Microsoft Corporation
TDNN
Time-Delayed Neural Network
Ink is cut into segments via simple algorithm
For Latin script, we just cut at bottoms
For Arabic or Hindi, we may need a different algorithm
Features are computed per segment
Chebychev coefficients
Variety of size metrics: width, height, offsets to neighbors, etc.
Each supported letter or character has an output node
Many have 2 outputs, for beginning of character and
continuation of character
Very large training sets are collected from tens of
thousands of native writers
Beam Search
Very similar to speech recognition systems
System lexicon
Simple list of words (like a spellchecker)
Stored as a trie
Plus regular expressions for numbers, dates, times, currency,
phone numbers, etc.
Low-scoring sequences are “culled” from the trie as we
go
Converts character recognizer into word recognizer
We can recognize a word even if you mangle one of the letters
Supports sloppier (and therefore faster) handwriting
TDNN + Beam Search
Ink Segments
Top 10 List
TDNN
Space
TDNN
Lexicon
b
d
o
g
b
t
t
a
g
t
d*
d
923151 9 4720
1381 8 2 14 3
g*
g
711252 8 7913
1717 5 7 4390
o*
o
5318792857 6
7 1657914415
e
…
…
…
a
o
…
c
l
Beam Search
…
a
…
…
…
Output Matrix
a* 88 8 68226357
23 4 44615757
a
dog
68
clog
57
dug
51
doom
42
divvy
37
ooze
35
cloy
34
doxy
29
client
22
dozy
13
g 57
a 88
o 65
b 14
t 12
l 76
c 86
a 73
d 92
b6
t5
o 77
g 68
t8
Personalization
New In Vista
Shape adaptation
Collect samples from you
Simple idea: continue same training we do at
Microsoft, but only on your samples
Implicit
Explicit
Capital I?
Text adaptation
Collect your personal words from Word and
outgoing emails
dev RTM KKOMO dogfooding
Qi trie featurize Herry
Open System
Recognizer API is published
Any recognizer may support this API
A non-Microsoft recognizer can be installed, and it will be
invoked by the inking platform
Non-Microsoft recognizers available now in Japan,
China, and Russia
Compete with my group
Or cover languages we don’t cover
Or cover non-text (music, math, chemistry)
Sorry: recognition result API is very text oriented
Strings in Unicode
Perhaps your text might be XML
http://www.microsoft.com/downloads/details.aspx?FamilyID=69640b5c-0ee9-421e-8d5c-d40debee36c2&displaylang=en
Start / All Programs / Microsoft Tablet PC Platform SDK / Microsoft Tablet PC Platform SDK Documentation
Microsoft Tablet PC / Tablet PC Platform / Programming the Tablet PC / Creating a Recognizer
Latin Orthography
XP has U.S. English, U.K. English, German, French, Spanish, and
Italian
Vista adds Dutch and Brazilian Portuguese
Currently working on Swedish, Danish, Norwegian (Bokmal),
Finnish, Polish, Czech, Portugal Portuguese, Catalan, Romanian,
Croatian, and Serbian
No ship estimates are available
Serbian is written in both Latin and Cyrillic
Making plans for the next batch: Probably Bahasa Indonese (Indonesian),
Hungarian, Turkish, Slovak, Slovene, Lithuanian, Estonian, Latvian,
Vietnamese, Tagalog (Filipino), others TBD
World’s largest orthography
Largest count of languages
Largest combined count of literates
East Asian Orthographies
Ideographic Orthographies
Completely different code base
Focus on large count of characters
XP has Japanese, Chinese (Simplified),
Chinese (Traditional), and Korean
Vista adds personalization, and improves
cursive recognition
Cyrillic and Greek Orthographies
Same TDNN + beam search technology works
equally well in Greek and Cyrillic
But we must collect new training sets
Cyrillic has more upper/lowercase confusion
Russian Collection completed
Currently working on recognizer
No estimate on shipping date
Some work also underway in Serbian
Cyrillic: Russian, Ukrainian, Bulgarian, Serbian, Byelorussian,
Macedonian, Kazakh
Other former-Soviet Turkic republics are in a state of transition back to
Latin scripts
Serbian is written in both Latin and Cyrillic
Greek
Τη
γλώσσα
μου
έδωσαν
Ελληνική
I was given a language that is Hellenic
Ink by Dr. John Drakopoulos
Russian
Кремль,
Кремль. Ото всех я
слышал
Kremlin, Kremlin. I’ve heard from everyone about [it]
Ink by Vladimir Smirnov
про
Bi-Directional Orthographies
Arabic and Hebrew
Text written right-to-left, but numbers written left-to-right
Arabic has its own digits
Hebrew uses “western” digits
No uppercase / lowercase distinction
Arabic is cursive-only
Up to 4 forms per letter (initial, medial, final, isolated)
Hebrew is print-only
5 letters have a separate final form
Abjabs (consonant alphabets)
We have collections underway
Some initial experimentation in both languages
No estimate on shipping date
Arabic script variations used for Farsi (Persian), Urdu, Kurdish, Azeri (in Iran),
Pashto, Darwi, Baluchi, Sindhi (in Pakistan), Kashmiri (in Pakistan)
Bi-Directional Example
Arabic
ً‫عاما‬
Year[s]
١٢٣
123
‫عمره‬
[his] age
Ink by Ahmed Kamal
Bi-Directional Example
Hebrew
.‫מבבל‬
322
‫פלימפטון‬
Plimpton 322 [is] from Babylonia.
Ink by Ethan Zoller
Brahmic Orthographies
Left to right, no uppercase / lowercase
Abugidas (syllabic alphabets)
Default vowel (short “Ə” or “uh” sound) is not written
All other vowels require a vowel sign added to the consonant
This includes a “no-vowel sign”
Called halant (“choked”) in Hindi, virama in other Indic languages
Vowel sound follows consonant sound
There are also independent vowels
Hindi Devanagari collection in progress
Some initial experimentation
No estimate on shipping date
Brahmic scripts: Devanagari (Hindi, Marathi), Thai, Bengali, Gujarati,
Gurmukhi (Punjabi), Tamil, Telugu, Kannada, Malayalam, Oriya, Sinhala,
Khmer, Lao, Tibetan, Myanmar, Sindhi (in India), Kashmiri (in India)
Vowel Signs
“KA”
“KAA”
“KAY”
“KAI”
“KO”
“KOW”
“KE”
“KU”
“KEE”
“KUU”
“K”
“KR”
Consonant Clusters
A transliteration of the English word “string”
Unicode buffer:
S
halant
T
halant
R I N G
Brahmic Comparison
K Kh G Gh C Ch
Devanagari:
Bengali:
Gurmukhi:
Gujarati:
Tamil:
Telugu:
Kannada:
कखगघचछ
ক খ গঘ চ ছ
ਕ ਖਗਘਚਛ
ક ખગઘચછ
க
ச
క ఖ గఘచఛ
ಕ ಖ ಗಘಚಛ
Demo Title
Name
Title
Group
Early Feedback
As for Vista, I am in love with the B2 Tablet
implementation! I can see where the effort is
going! The TIP is awesome and the handwriting
recognition is vastly improved. There is hardly a
time where I have to correct my input - no
matter how sloppy I am.
Unsolicited feedback from a Vista Beta II user
whose identity I do not know
© 2006 Microsoft Corporation. All rights reserved.
Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation.
Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft,
and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Backup Slides
Devanaguri Example
headline
headline
headline
period
[abbreviation]
प
“PA”
[vowel sign]
[R sign]
“AA”
ल
[vowel sign]
“LA” “I”
period
[abbreviation]
Pr. Ltd. [Private Limited]
ि
म
“MA”
ि
[vowel sign]
“I”