Transcript CAD-CBIR-inglês.ppt

Content-based image retrieval and
Computer-aided diagnosis systems
Paulo Mazzoncini de Azevedo Marques - PhD
([email protected])
Science of Images and Medical Physics Center
School of Medicine of Ribeirão Preto
University of São Paulo
DIAGNOSIS
Signal Detection Theory – Decision Matrix
Confirmed Confirmed
Abnormal Normal
Diagnosed
as
Abnormal
Diagnosed
as
Normal
True
Positive
(TP)
False
Negative
(FN)
False
Positive
(FP)
True
Negative
(TN)
The Essential Physics Of Medical Imaging. Bushberg JT,
Seibert JA, Leidholdt Jr. EM, Boone JM. Lippincott
Williams  Wilkins, Philadelphia, USA, 2002.
DIAGNOSIS
PERFORMANCE MEASUREMENTS
Confirmed Confirmed
Abnormal Normal
Diagnosed
as
Abnormal
Diagnosed
as
Normal
True
Positive
(TP)
False
Negative
(FN)
False
Positive
(FP)
True
Negative
(TN)
True Positive Fraction (TPF)
TPF = TP/(TP+FN)
Sensitivity = TP/(TP+FN) = TPF
Specificity = TN/(TN+FP) = (1-FPF)
False Positive Fraction (FPF)
FPF = FP/(FP+TN)
Accuracy = (TP+TN)/(TP+TN+FP+FN)
DIAGNOSIS
PERFORMANCE MEASUREMENTS
ROC curves (receiver operating characteristic)
Az
The Essential Physics of Medical Imaging. Bushberg JT,
Seibert JA, Leidholdt Jr. EM, Boone JM. Lippincott
Williams  Wilkins, Philadelphia, USA, 2002.
Computer-aided Diagnosis
Definition:
(CAD)
A diagnosis made by a radiologist using the output of a
computerized scheme for automated image analysis as a
diagnostic aid (second opinion).
K. Doi - Computerized Medical Imaging and Graphics 31 (2007) 198–211
With CAD, the performance by
computers does not have to be
comparable to or better than
that by physicians, but needs
to be complementary to that
by physicians (synergy).
Nishikawa RM - Applied Radiology, Suplement November 2001:14-16
CAD
TYPES OF AID

Computer-aided Detection (CADe)
– usually confined to marking suspicious structures and
sections
– Initially approved by FDA-USA in 1998 for mammography
CAD
TYPES OF AID

Computer-aided Diagnosis (CADx)
– usually focused on to classify detected structures or
regions (more academic).
CAD
KNOWLEDGE INVOLVED

Computer Vision
(quantitative features extraction)
– Preprocessing (noise reduction and
enhancement)
– Segmentation (regions, edges, structures)
– Structure/ROI Analyze (form, size and
location, texture, topology)

Artificial Intelligence
– Features selection
– Classification
(classification)
CAD- EXAMPLE
CAD in Orthopedic Radiology:
Quantitative Evaluation of Vertebral Morphometry
Eduardo A. Ribeiro, Marcello H. Nogueira-Barbosa,
Rangaraj M. Rangayyan, Paulo M. Azevedo-Marques
School of Medicine of Ribeirão Preto, University of São Paulo,
Ribeirão Preto, São Paulo, Brazil
Department of Electrical & Computer Engineering, University of Calgary,
Calgary, Alberta, Canada
Vertebral fractures are important indicators of
osteoporosis.
Insufficiency fractures of the vertebrae are usually seen
as a partial collapse of the vertebral body.
Both semi-quantitative and quantitative analysis of spinal
and vertebral deformities could assist in the diagnostic
decision-making process and in guiding therapeutic
procedures.
Grading of
Vertebral Fractures
(Genant)
Genant HK et al. Journal of Bone and Mineral Research, 8:1137–1148, 1993.
Manual quantitative morphometric
analysis is labor-intensive and subject
to inter-observer and intra-observer
variability
CAD - Pipeline
Image
Acquisition
(film digitization)
Vertebral Plateau
Segmentation
(Gabor Filters and ANN)
Vertebral Morphometry
(vertebral height measurement)
Genant Grading
Analysis of Vertebral
Height
(rule-based classification)12
Marking Reference Points
Five points, P1–P5, were manually marked
near the middle of the intervertebral spaces
spanning the range of L1–L4 by using a
pointer.
The distances between the points were
calculated automatically:
D(1,2), D(2,3), D(3,4), and D(4,5).
Using 75% of each distance measure, the
corresponding line joining the manually
marked points was shifted in either
direction along its perpendicular to create a
quadrilateral for each vertebra.
Segmentation
F
S
Segmentation is based on the detection and
characterization of oriented edges using Gabor filters
and classification using a neural network.
F. J. Ayres and R. M. Rangayyan. Journal of Electronic Imaging, 16(2):023007:1–12, 2007.
Each image was filtered with a bank of 180 Gabor
filters (sinusoidally modulated Gaussian functions) in
steps of 1 degree
Width = 4 pixels and elongation factor = 8.
For each pixel, the magnitude response and angle of
the Gabor filter providing the highest output were used
to compose a Gabor magnitude image and an
orientation field.
Result of Gabor Filters
original image
Gabor magnitude
response
coherence image
Manual Delineation of
Vertebral Plateaus
5-pixel thick lines drawn for L1-L4
Detection of Vertebral Plateaus
with a Neural Network
Pixels in regions corresponding to L1-L4 were
obtained from the original image, the Gabor
magnitude response, and the coherence image
for analysis using a logistic sigmoid neural
network.
A leave-one-out training and testing procedure
was used.
The output of the neural network for each pixel
was used to label the pixel as belonging to a
vertebral plateau or not.
Detection of Vertebral Plateaus
with a Neural Network
Original image
Output of neural network
Manual annotation
Vertebral Morphometry
convex
skeleton
hull
skeleton
apply
19
skeleton
to plateaus
remove
spurs
Measurement of
Vertebral Height
Measures of height
obtained for a normal
vertebral body
Measures of height obtained
for an abnormal vertebral
body
Initial Results of CAD
Results of computer-aided grading of vertebral fracture using
the method proposed by Genant.
Values along the main diagonal correspond to correct
classification by the CAD (86%).
Content-Based Image Retrieval
Definition:
CBIR
Content-based image retrieval (CBIR), also known as
query by image content (QBIC) is the application of
computer vision techniques to the image retrieval
problem, that is, the problem of searching for similar
images in large databases.
Content-based means that the search will analyze the
actual contents of the image rather than the metadata
such as keywords, tags, and/or descriptions associated
with the image.
Müller H. et al. International Journal of Medical Informatics (2004) 73, 1—23
CBIR Framework
Extracted Features
ID of retrieved
features
Color
Query
texture
Features of the
query object
by Similarity
Module
Similar Features +
distances + Images
ID
Features
shape
Extraction
...
Indexing
feedback
Query Image
structure
Similar
Images
Computer Vision
Feature
Extraction
Original
Image

Feature
Vector
X1
X2
.
.
.
XN
Image Processing Techniques
– Feature Extraction

Feature Vector (based on shape, texture,
color or others techniques)
Similarity Searches
Data Domains
– MAM-Metric Access Methods
Multi-dimensional Domains
 Adimensional Domains

– Fingerprints, words and so on.

Example
– mvp-tree, vp-tree, M-tree, Slim-Tree
SLIM-TREE armazenando
SLIM
SLIM-TREE
armazenando 17
17 objetos
objetos
(query by example)
Similarity Searches
Metric Space


Metric Space is a pair: M=(D,d) where:
– D is the characteristic domain of objects
– d is a metric distance function.
Properties of the distance function d():
– symmetry:

d(x,y) = d(y,x)
– non-negativity:

0 < d(x,y) < , x  y e d(x,x) = 0
– triangle inequality:
d(x,y)  d(x,z) + d(z,y)
Where x, y e z are objects of D

Minkowski Function
Similarity Searches
Query Definitions

Range Query:
“ Find all the images that are
within 10 units of distance
from image1.”

Nearest Neighbor Query (k-NN):
"Find the 5 nearest images to image1”
CBIR
PERFORMANCE MEASUREMENTS
Precision X Recall curves
a) 15% of database
b) 10% of database
1
c) 5% of database
1
MRHead500
0.98
1
MRHead500
0.98
0.96
0.96
0.94
0.94
0.94
0.92
0.92
0.9
0.9
0.92
0.88
0.88
0.9
0.88
Recall
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Recall
0.84
1
0.82
0
e) 1% of database
d) 2% of database
1
0.98
0.86
Recall
0.86
0.84
MRHead500
0.98
0.96
0.1
0.2
0.3
0.4
1
0.6
0.7
0.8
0.9
1
1
MRHead500
MRHead500
MRHead500
0.95
0.95
0.92
0.9
0.9
0.9
0.88
0.85
0.85
0.96
0.5
f) 0.5% of database
0.94
0.86
0.8
0.1
0.8
0.8
0.84
0.82
Recall
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Recall
1
0.75
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Recall
1
0.75
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
CBIR- EXAMPLE
Content-based retrieval of color images of
dermatological ulcers.
Silvio Moreto Pereira, Marco Andrey C. Frade,
Rangaraj M. Rangayyan, Paulo M. Azevedo-Marques
School of Medicine of Ribeirão Preto, University of São Paulo,
Ribeirão Preto, São Paulo, Brazil
Department of Electrical & Computer Engineering, University of Calgary,
Calgary, Alberta, Canada
Dermatological Ulcers
Ulcers may appear on the legs due to chronic diseases
such as diabetes and venous insufficiency.
Visual assessment of pathological regions and
evaluation of macroscopic features are used for the
diagnosis of skin lesions in clinical practice.
The appearance of a lesion provides important clues
regarding the diagnosis, severity, and prognosis.
The red-yellow-black-white (RYKW) model of tissue
composition is useful as a descriptive tool.
Ulcer Tissue Types
Granulation (red)
Scar or necrosis (black)
Fibrin (yellow)
Mixed
Imaging of Ulcers
32
Representation of
Color Images
Each color image was represented using the
standard representations as
• [red, green, blue] or RGB,
• [hue, saturation, intensity] or HSI, and
• L*u*v* .
Segmentation of
Ulcer Images
Original image
Hue-saturation
histogram
Red regions
Yellow regions
S>0.4 and
S>0.2 and
H 300º to 0 to 30º
H 30º to 90º
Black regions
S<0.2 and
I<0.25*max
Ulcer regions
Features Extraction
Multispectral cooccurrence matrices (CCMs) obtained from the
RGB, HS, u*v*, and a*b* components.
a total of 111 statistical features were extracted from the R, G, B, H, S,
u*, v*, a*, and b* components to characterize each color image
KNN Based Retrieval using
Cosine Distance
CAD-CBIR/PACS INTEGRATION
Visualization
Imaging
Workstation
Modality
Speech Recognition
PACS
RIS
DB
DICOM
HL-7
High-Speed
DICOM
Network
HIS/MIS
Firewall
RAID
Web-based
RIS/PACS/EMR
Archive
PACS – Picture Archiving and Communication System
PACS AND IMAGING
INFORMATICS: Basic Principles and
Applications - H.K. Huang, New
Jersey - USA, 2004
Example of CAD/PACS integration framework:
– Communication services (DICOM functionalities)
– Image-processing pipeline (CAD-CBIR server)
Azevedo Marques PM et. al. International Journal of Computer Assisted Radiology and Surgery. 2009,
v. 4. p. S-180-S-181.
Example of CAD-PACS integration
cores.put("normal", Color.WHITE);
cores.put("ground-glass", Color.BLUE);
cores.put("reticular-linear", Color.GREEN);
cores.put("micronodules", Color.RED);
cores.put("honeycombing", Color.YELLOW);
cores.put("emphysematous", Color.MAGENTA);
cores.put("consolidation", Color.CYAN);
Azevedo Marques PM, et. al. International Journal of Computer Assisted Radiology and
Surgery. 2009, v. 4. p. S-180-S-181.
Example of CAD/CBIR-PACS integration
CAD scheme using CBIR approach.
Bin Zheng. Computer-Aided Diagnosis in
Mammography Using Content based Image
Retrieval Approaches: Current Status and
Future Perspectives.
Algorithms. 2009 June 1; 2(2): 828–849.
Example of applying a CAD scheme using CBIR approach to detect and classify a suspicious breast
mass region. A suspicious mass is automatically detected by CAD scheme and queried by the observer
(pointed by the arrow). In CAD workstation, the mass region segmentation (boundary contour), 12
CBIR-selected similar ROIs, and both detection and classification scores are displayed. Among the 12
similar ROIs, 8 depict malignant masses (marked by Red frame), 2 depict benign masses (marked by
Green frame), and 2 depict CAD-cued false-positive regions (marked by Blue frame).
CONCLUSION
Computer-aided diagnosis has become a part of clinical work in the detection
of breast cancer by use of mammograms, but is still in the infancy of its
full potential for applications to many different types of lesions obtained
with various modalities.
Content-based image retrieval is an alternative and complementary approach
for image retrieval based on key-words and metadata. Initial results are
very promising about using CBIR as a diagnostic support tool
In the future, it is likely that CAD and CBIR schemes will be incorporated into
PACS
CAD and CBIR will be employed as useful tools for diagnostic examinations in
daily clinical work.
THANK YOU!
[email protected]