Transcript Pattern Analysis & Machine Intelligence

From Anthrax to
ZIP CodesThe Handwriting is on the Wall
Venu Govindaraju
Dept. of Computer Science &
Engineering
University at Buffalo
[email protected]
Outline






Success in Postal Application
Role of Handwriting Recognition
Recognition Models
Interactive Cognitive Models
New Research Areas
Other Applications
USPS HWAI Background






Postal Sponsorship Started – 1984
370 Academic Articles Published
Millions of Letters Examined
Many Experimental Systems Built and Tested
Migrated from Hardware to Software System
Only Postal Research Continuously Funded
Pattern Recognition Tasks
Items to be Recognized, Read, and
Evaluated (Machine printed and Script)

Delivery address, sender´s address, endorsements

Linear Codes, Mail Class

Indicia (2D-Codes, Meter Marks)
Meter
Mark
Sender’s Address
Endorsem
ent
In Case of Undeliverable as Addressed Return to Sender
Linear
Code
Delivery Address
Digital
Post Mark
Deployed..

USA





UK




250 P&DC sites
27 Remote Encoding Centers
25 Billion Images Processed Annually
89% Automated Bar-coding
67 Processing Centers
27 Million Pieces Per Day,
9.7 Million Pieces Per Hour Peak
Australia
RCR Overview
Advanced
Facer
Canceler
Multi-Line
OCR
Image
RCR
Remote
Encodin
g
Bar Code Sorter
At the Right Price
Processing Type Cost/1000 Pieces
Manual
$47.78
Mechanized
$27.46
Automated
$5.30
80% encode rate and
counting!
Encode Rate
Handwriting Encode Rate
80%
70%
60%
50%
40%
30%
20%
10%
0%
J
6
-9
n
a
7
-9
n
Ja
8
-9
n
Ja
9
-9
n
Ja
Date
J
0
-0
n
a
J
1
-0
n
a
Impact

Applications of CEDAR research helping to
automate tasks at IRS and USPS



1st year that USPS used CEDAR-developed
software to read handwritten addresses on
envelopes, saved $100 million
1997-1999 USPS deployment of CEDAR-developed
RCRs, USPS saved 12 million work hours and over
$340 million
500 scientific publications and 10 patents
Outline






Success in Postal Application
Role of Handwriting Recognition
Recognition Models
Interactive Cognitive Models
New Research Areas
Other Applications
Role Handwriting Recognition
in Address Interpretation
Context Provided by Postal Directories
• <ZIP Code, Primary Number>
– Create street name lexicon
<06478, 110>
• DPF yields 8 street names
• ZIP+4 yields 31 street names
(on average about 5 times more)
HAWLEY
NEWGATE
BEE MOUNTAIN
DORMAN
BOWERS HILL
FREEMAN
PUNKUP
PARK
RD
RD
RD
RD
RD
RD
RD
RD
1034
1533
1615
1642
1757
1781
1784
6124
CEDAR
Context



One record per delivery point in USA
Provided weekly by USPS, San Mateo
Raw DPF




138 million records
15 GB (114 bytes per record);
41,889 ZIP Code files
Fields of interest to HWAI

ZIP Code, street name, primary number,
secondary number, add-on
CEDAR
Power of Context

ZIP Code
 30% of ZIP Codes contain a single street name
 5% of ZIP Codes contain a single primary number
 2% of ZIP Codes contain a single add-on

<ZIP Code, primary number>
 Maximum number of records returned is 3,071

<ZIP Code, add-on>
 Maximum number of records returned is 3,070
Outline






Success in Postal Application
Role of Handwriting Recognition
Recognition Models
Interactive Cognitive Models
New Research Areas
Other Applications
Handwriting Recognition
Context
Ranked
Lexicon
Multiple Choice Question
Context
Ranked
Lexicon
Lexicon Driven Model
1
2
3
4
5
6
7
8
w[5.0]
1
2
o[6.6]
w[7.2]
w[8.6]
9
o[7.7]r[5.8]
o[6.1]
w[5.0]
w[7.6]
Distance between lexicon entry
‘word’ first character ‘w’ and the
image between:
- segments 1 and 4 is 5.0
- segments 1 and 3 is 7.2
- segments 1 and 2 is 7.6
o[6.0]
3
o[8.3]
4
r[7.6]
d[4.9]
r[6.4]
r[7.5]
o[7.6]r[6.3]
5
6
7
8
r[3.8]
o[7.2]
o[10.6]
9
d[4.4]
d[6.5]
o[7.8]r[8.6]
Find the best way of accounting for characters
‘w’, ‘o’, ‘r’, ‘d’ buy consuming all segments 1
to 8 in the process
Lexicon Free Model
1
4
2
3
6
7
5
8
-Image from 1 to 3 is a in with
0.5 confidence
-Image from segment 1 to 4 is
a ‘w’ with 0.7 confidence
-Image from segment 1 to 5 is
a ‘w’ with 0.6 confidence and
an ‘m’ with 0.3 confidence
w[.6], m[.3]
w[.7]
1
i[.8], l[.8]
2
i[.7]
d[.8]
o[.5]
u[.5], v[.2]
3
4
u[.3]
m[.2]
5
6
r[.4]
7
m[.1]
Find the best path in
graph from segment 1 to 8
word
8
Holistic Features
Slant Norm
Turn Points
Position Grid
and gaps
Reference
Lines
Ascender
Descender
Lexicon Reduction and
Verification
Outline






Success in Postal Application
Role of Handwriting Recognition
Recognition Models
Interactive Cognitive Models
New Research Areas
Other Applications
Grapheme Models
Structural Features
BAG
End
Loops
Junction
End
Turns
Loop
Feature Extraction and
Ordering
Critical node: removal disconnects a connected component.
Loops
End
Turns
End
Junction
Turns
Loop
2-degree critical nodes keep feature ordering from left to right.
Left
Component
Right
Component
Continuous Attributes
graphe
me
pos
orientation
Down
cusp
3.0
-90o
Up loop
Down arc
angle
Stochastic Model
Observations
Results
Lex size Top
10
1
2
100
1
2
1000
1
2
20000
1
2
WMR %
96.86
98.80
91.36
95.30
79.58
88.29
62.43
71.07
SM CA%
96.56
98.77
89.12
94.06
75.38
86.29
58.14
66.49
Interactive Models
[McClelland and Rumelhart, Psychological Review, 1981]
ABLE
A
TRAP
TRIP
Words
N
T
Letters
Features
Interactive Recognition
Lexicon 1
Lexicon 2
Lexicon 3
West Central Street
West Main Street
Sunset Avenue
West Central Street
East Central Street
Sunset Avenue
West Central Street
West Central Avenue
Sunset Avenue
Interactive Model
features
T-crossings, loops, ascenders, descenders, length
image
Adaptive Character Recognition
[Park and Govindaraju, IEEE CVPR 2000]
•Adaptive selection of features
•Adaptive number of features
•Adaptive resolutions
•Adaptive sequencing of features
•Adaptive termination conditions
Features
4 gradient features
5 moment features
Vector code book
Feature Space





|V| x |Nc| x |Ixy|
29 x 10 x 85 (quad tree, 4 levels)
Recognition rate and feature |V|
GSC: |V| : 2512
Tradeoffs: space vs accuracy

Hierarchical space with additional
resolution and features as needed
Active Recognition Using Quad Trees
Experimental Results
Results
Classifier
Active Model
Neural Net KNN
Top 1%
95.7 %
96.4%
95.7%
Templates
612
976
3,777
Msec/char
1.45
11.5
384
Training hrs
1
24
1
25656 training and 12242 test (Postal +NIST)
Outline






Success in Postal Application
Role of Handwriting Recognition
Recognition Models
Interactive Cognitive Models
New Research Areas
Other Applications
Fast Recognition
-Reuse matched
characters
-Reuse matched
sub-strings
-Parallel processing
Combination and Dynamic Selection
[Govindaraju and Ianakiev, MCS 2000]
image
WR 1
Lexicon
Top 50
<55
WR 2
Top 5
•Optimization problem
•Combinatorial explosion in
•arrangement of recognizers
•lexicon reduction levels
WR 3
+
1
Lexicon Density
[Govindaraju, Slavik, and Xue, IEEE PAMI 2002]
Lexicon 1
Lexicon 2
Me
He
So
To
In
Me
Memo
Memory
Memoirs
Mellon
Classifier Performance Prediction
[Xue and Govindaraju, IEEE PAMI 2002]
q: probability that recognizer make a unit distance errors
D: average distance between any two words in the lexicons
n: lexicon size; p: performance; a, k,: model parameters
ln (-ln p) = (ln q) D + a ln ln n + ln k
Outline






Success in Postal Application
Role of Handwriting Recognition
Recognition Models
Interactive Cognitive Models
New Research Areas
Other Applications
Bank Check
Recognition
PCR Trend Analysis
NYS EMS PCR Form
NYS PCR Example
Thousands are filed a day.
Passed from EMS to Hospital.
PCR Purpose:
– Medical care/diagnosis
– Legal Documentation
– Quality Assurance
EMS Abbreviations
COPD Chronic Obstructive Pulmonary
Disease
CHF Congestive Heart Failure
D/S
Dextrose in Saline
PID
Pelvic Inflammatory Disease
GSW Gunshot Wound
NKA No known allergies
KVO Keep vein open
NaCL Sodium Chloride
Medical Text Recognition and
Data Mining
Reading Census Forms
Lexicon Anomalies
Space: “sales man” and “salesman”
Morphology: “acct manager” and
“account management”
Abbreviation
Plural: “school” and “schools”
Typographical: “managar” and
“manager”
Binarization
Historic Manuscripts
Summary






Handwriting recognition technology
Pattern recognition task
Lexicon holds domain specific
knowledge
Adaptive methods
Classifier combination methods
Many applications