Transcript Indexing of Graphic Document Images: a Perceptive Approach

FRE 2645
Indexing of Graphic Document Images :
a Perceptive Approach
Mathieu Delalandre¹,²
Thursday 16th June 2005
¹ PSI Laboratory, Rouen University, France
² SCSIT, Nottingham University, UK
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Who I am ?
 Mathieu Delalandre






Thesis:
Fourth year of PhD (defence in September)
Lab:
PSI Laboratory, Rouen city, France
Super:
E. Trupin, J.M. Ogier, J. Labiche
Team:
S. Adam, H. Locteau, P. Héroux, E. Barbu, Y. Lecourtier
Field:
Document Image Analysis (Graphics Recognition)
Postdoc: IPI, SCSIT, from April to September (4-5 months) with
Tony Pridmore
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Indexing of Graphic Document Images :
a Perceptive Approach




Introduction
Systems Overview
The Knowledge Level
Conclusion
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Introduction
Indexing & Retrieval (I & R)
-Indexing & Retrieval (I & R)
-Categorization of Images
-I & R of Document Images
-My Topic
 Indexing & Retrieval [Greengrass’00]
 Indexing: Identification and recording of attributes of data that will aid
retrieval.
 Retrieval: Ability of a database management system to get back data
that were stored there previously.
 Applications




videos (MPEG, AVI, …)
Web pages (XML, XHTML, …)
structured documents (PDF, PS, Word, …)
images (JPG, GIF, …)
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Introduction
-Indexing & Retrieval (I & R)
-Categorization of Images
-I & R of Document Images
-My Topic
Categorization of Images
photographies
document images
trademark
logo
heading
journal
manual
foreground/background images
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Introduction
I & R of Document Images (1/3)



-Indexing & Retrieval (I & R)
-Categorization of Images
-I & R of Document Images
-My Topic
Today, document images are not indexed by search engines due of complexity of
Document Image Analysis (DIA) task [Doerman’98][Walker’00][Baird’03]
Is indexing of document images really needed ?  two questions
Question : How many document images and where [Spring’95] [Cleveland’98]
[Steve’99] [Ouf’01] [Baird’03] [Hu’04] ?
Web (8.1015ko)
99.3%
0.3%
Web Pages
30%
Images
60%
Document Images
Logos, Headings, …
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Deep Web
large (or main) part
70%
Markup Languages
HTML, XHTML, ..
40%
Digital Libraries
main part
Others
Softwares, Data Bases, …
minor part
Document Images Structured Documents
Photographies
Introduction
I & R of Document Images (2/3)
-Indexing & Retrieval (I & R)
-Categorization of Images
-I & R of Document Images
-My Topic
Question : New or just old document images ?
Paper (and image) has too
many desirable properties,
document images and
structured documents
will increasingly co-exist in
the future [Breul’04]
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Introduction
I & R of Document Images (3/3)
-Indexing & Retrieval (I & R)
-Categorization of Images
-I & R of Document Images
-My Topic
 To Conclude :
 (1) DIA is needed (and will be needed) in the future of I &
R of documents [Baird’03] [Breul’04]
 (2) DIA must come back today under the way of I & R
[Baird’03]
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Introduction
-Indexing & Retrieval (I & R)
-Categorization of Images
-I & R of Document Images
-My Topic
My Topic
 Indexing of graphic document images
 Indexing & Retrieval  Indexing
 Identification and recording of attributes of data that will aid
retrieval
 First step before retrieval
 document images  graphic document images
line drawing
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
symbol
logo
asian script
historical
heading
Indexing of Graphic Document Images :
a Perceptive Approach




Introduction
Systems Overview
The Knowledge Level
Conclusion
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Systems Overview
Introduction
-Introduction
-Graphics Recognition Systems
-Graphics Indexing Systems
-Open Problems
 Overview of systems to index graphic document images
 we talk about Graphics Indexing Systems
 Graphics Indexing Systems are specialized from DIA
systems applied to recognition and understanding of graphic
document images [Tombre’03]
 we talk about Graphics Recognition Systems
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Systems Overview
Graphics Recognition Systems (1/3)
-Introduction
-Graphics Recognition Systems
-Graphics Indexing Systems
-Open Problems
 Graphics Recognition Systems :
 graphic document images  structured documents
symbol
linear
text
 Applications deal with graphics parts (symbol and linear)
 text/graphics segmentation [Tombre’02], vectorisation [Mejbri’02],
symbol recognition [Llados’02], document interpretation (or
understanding) [Ablameko’00], …
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Systems Overview
-Introduction
-Graphics Recognition Systems
-Graphics Indexing Systems
-Open Problems
Graphics Recognition Systems (2/3)
 Graphics are structured and connected
symbol and its
structure
line
connect
point
connected symbol
in drawing
 Graphics Recognition Systems are based on structural methods
 “relational organization of low-level features (graphic primitives)
into higher-level structures (graph)” [Tombre’96] [Shi’89]
low level features
graphic primitives
higher-level structure
graph
line
connect point
T link
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
symbol recognition
line
connect edge
T edge
Systems Overview
Graphics Recognition Systems (3/3)

Architecture of Graphics Recognition Systems :
Graphic Primitive
Extraction
document images

Recognition
graph of graphic
primitives
Graphic
Models
<network>
<part id=”1”>
<symbols>
<labels>
</labels>
</symbols>
</part>
</network>
structured
document
Graphic Primitive Extraction, some methods [Wenyin’98] [Delalandre’04] :


-Introduction
-Graphics Recognition Systems
-Graphics Indexing Systems
-Open Problems
skeletonization [Hilaire’04], contouring [Ramel’00], tracking [Song’00], labelling [Badawy’02],
transform [Couasnon’01], meshes [Vaxiviere’95], region segmentation [Cao’00], run-length
[Burge’98], …
Recognition

Graph Matching [Bunke’00], Graph Transform [Blostein’05], Primitive Matching [Foggia’99], …
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Systems Overview
-Introduction
-Graphics Recognition Systems
-Graphics Indexing Systems
-Open Problems
Graphics Indexing Systems (1/3)

Graphics Indexing Systems [Doerman’98] [Tombre’03], 3 classes :
Title block recognition
[Arias’98], [Najman’01],
[Lamiroy’02], …
Graphics indexing
Statistical framework
[Samet’96], [Worring’99],
[Tabbone’03], [Terrades’03], …
[Kasturi’88], [Lorenz’95],
[Huang’97], [Hu’97],
[Barbu’04], [Valasoulis’04], …
Partial
matching
Connected
so no
matched
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Systems Overview
Graphics Indexing Systems (2/3)

-Introduction
-Graphics Recognition Systems
-Graphics Indexing Systems
-Open Problems
Architecture of Graphics Indexing Systems :
document links
Graphic
Primitive
Extraction
Index
Indexing
Graph of graphic primitives
attributes+
document links
indexing attributes
specific set of graphic
primitives
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Systems Overview
Graphics Indexing Systems (3/3)
-Introduction
-Graphics Recognition Systems
-Graphics Indexing Systems
-Open Problems
Works
Graphic Primitives
Extraction
Graph of
Graphic Primitives
Indexing
[Kasturi’88]
thinning and
chaining
line graph of skeleton
cycle search, width and length
matching of lines
[Lorenz’95]
run length encoding
and polygonisation
straight line graph of
contours and skeleton
Fourier approximation
of line graph
[Huang’97]
contouring and
polygonisation
2-D strings of contours
string matching
[Barbu’04]
thinning and neighbour
analysis of skeleton’s pixels
region adjacency graph
graph mining
[Hu’04]
thinning, chaining,
and polygonisation
set of straight line
of skeleton
string matching
[Dosh’04]
thinning,
chaining, and polygonisation
set of straight line
of skeleton
vectorial signature
thinning
skeleton graph
structural
contouring
region graph
statistical
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Systems Overview
-Introduction
-Graphics Recognition Systems
-Graphics Indexing Systems
-Open Problems
Open Problems (1/2)

All these systems use a Lexical/Syntactic (or Bottom/Up) approach [Tombre’96]
 Lexical (Bottom) : Extraction from images of graphical primitives in an fixed way
 Syntactic (Up) : Analysis of graphical primitives without returns on image

So, all these systems use a Document Understanding Approach, but I & R is not an
Understanding problem
Criterion
Understanding
I&R
Image Size
Data Base Size
Process Execution
large
small
one shot
small and medium
large
every-time
complexity
Graphic Primitives
Noise Level
accurate
high and medium
approximated
low and medium
robustness
Prior Knowledge
Document Class
yes
few and known
no
several and unknown
content adaptation
content adaptation is the most important feature of I & R systems
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Systems Overview
Open Problems (2/2)

-Introduction
-Graphics Recognition Systems
-Graphics Indexing Systems
-Open Problems
Examples of Content Adaptation
 A broad class of document
region based
[Roque’03]
both based
[Ramel’00]
line based
[Hilaire’04]
 Context
text/graphics
segmentation

noise
adaptation
To conclude
 A I & R must deal with the content adaptation
 Content adaptation can’t be solved without a knowledge based approach
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Indexing of Graphic Document Images :
a Perceptive Approach




Introduction
Systems Overview
The Knowledge Level
Conclusion
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
The Knowledge Level
Introduction

-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
Some (general) definitions [Tuthill’90] [Holsapple’04]
 Knowledge : human mental grasp of reality
 Representation : placement (and meaning) of knowledge into (from) computer
memory
 Formalism : a set of symbols corresponding to knowledge inside computers
Knowledge
meaning
placement
Formalism(s)

Different types of knowledge
 on strategies []
 on case based reasoning []
 on ontologies []
 Nottingham
….
SCSIT Talk,
University,
Thursday 16th June 2005
Human
Human/Computer
Computer
The Knowledge Level
Graphical Knowledge (1/2)
-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
 Graphical Knowledge [Delalandre’05] : It is a type of knowledge
corresponding to human mental grasp of graphics
Levels of Graphical Knowledge
it is a gate !
interpretation
symbol
high-level
objects
graph-based
formalisms
graphic
primitives
pixel-based
formalisms
image
perception
abstraction levels
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
vector-based
formalisms
formalism levels
The Knowledge Level
Graphical Knowledge (2/2)
-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
 Two formalism levels [Tombre’96]
 Graphic Primitives [Murray’96]
 Pixel-based formalism : pixel, raster,
run, connected component, …
 Vector-based formalism : vector, arc,
curve, ellipsis, square, …
primitives
line images
 Graph-based formalisms [Sowa 99]:
Relational Attributed Graphs (RAG),
Frames, Object-Oriented Languages, …
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Relational Attributed Graphs
[Seong’93]
The Knowledge Level
Graphics Model (1/2)


-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
Model [Seguela’01] : a knowledge representation using given formalisms
and for given system’s purposes
Graphics Model [Delalandre’05] : model used to represent the graphical
knowledge
a (simple) shape
graphic primitives
extremity
junction
line
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
line based model
junction edge
line
junction based model
extremity
junction
line edge
The Knowledge Level
Graphics Model (2/2)

-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
One system = one model  a considerable number of models
 [Joseph’92] [Pasternak’93] [Han’94] [Burgue’95] [Yu’97] [Lee’98] [Ramel’00]
[Couasnon’01] [Badawy’02] [Yan’04] …

Models depend of extracted graphic primitives, we can defined a graphics
model taxonomy into 3 classes [Delalandre’05]
region-based models
contour based models
skeleton based models
component
loop
neighbour
include
quadrilateral
Line link
Junction link
extremity
junction
line edge
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
The Knowledge Level
a Perceptive Approach (1/6)
Perception Level
of Representations
-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
two links between
levels
Global
Region Level
Contour Level
Skeleton Level
Local
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
specialisation
aggregation
The Knowledge Level
a Perceptive Approach (2/6)
Perception Level
of Representations
Global
Region Level
classic models
Contour Level
hybrid models
perceptive approach
(jump or browse)
Skeleton Level
Local
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
The Knowledge Level
a Perceptive Approach (3/6)

-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
First step, the region level : connected component analysis [Alnuweiri’92]
foreground
background
main background
loops
foreground’s
components
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
background’s
components
The Knowledge Level
a Perceptive Approach (4/6)
-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
 Six Features
 (F) Foreground
 (N) Neighboring
 (B) Background
 (S) Size
 (R) Resolution (ie. distance)
 (I) Inclusion
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
The Knowledge Level
a Perceptive Approach (5/6)
-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
 Use-Case Queries
started image
FR1
FR2
BR2
BR2S2
BR2S2N2
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
The Knowledge Level
a Perceptive Approach (6/6)

True-Life Query
FS1 BR2 N>2
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
-Introduction
-Graphical Knowledge
-Graphics Model
-a Perceptive Approach
Indexing of Graphic Document Images :
a Perceptive Approach




Introduction
Systems Overview
The Knowledge Level
Conclusion
SCSIT Talk, Nottingham University,
Thursday 16th June 2005
Conclusion
 Conclusion
 It is just a bibliography study and ideas
 Start on this ideas ?
 Perspectives
 Contour and skeleton levels ?
 System to control the representation building ?
SCSIT Talk, Nottingham University,
Thursday 16th June 2005