Transcript Xilinx XC4000 FPGA devices - Mahanakorn University of

Introduction and Overview
Image and Video Processing
By Theerayod Wiangtong, PhD., DIC
Mahanakorn University of Technology
1
Human Visual Perception
2
Eye Anatomy
3
Eye Vs. Camera
4
The Human Visual System (HVS)
left eye
right eye
retina
retina
pupil
cornea
lens
LGN
retina
primary visual
cortex
fovea
optic nerve
visual axis
higher level vision
and cognition
5
Human Perception of Color
• Retina contains two classes
of receptors:
– Cones (6-7 millions):
• Located in the fovea
•
•
•
•
Each one connected to its own nerve end
Day vision, can perceive color tone
Red, Green, and Blue cones.
Different cones have different freq. responses
– Rods (75-100 millions):
• Several rods are connected to a single nerve
• Night vision, perceive brightness only
6
Range of Perception
Electromagnetic Spectrum
cosmic
rays
-4
10
gamma
rays
-2
10
radio
frequency
microwave
(SAR)
visible
X-Rays
1
UV
10
2
IR
10
4
10
6
8
10
10
10
12
10
10
wavelength (Angstroms)
1 Å = 10
-1 0
m
• Light consists of an electromagnetic wave with wavelength in
the range 380-780 nm.
7
Light Sources
Illuminating Source
•
•
•
Emits light (e.g. sun, bulb, TV)
Perceived color depends on spectral
contents of the emitted light
Follows the additive rule: (the perceived
color of several mixed illuminating light
sources depends on the sum of the
spectra of the light sources)
Reflecting Source
•
•
•
Reflects incoming light (e.g. all nonilluminating objects)
Perceived color depends on spectral
contents of the reflected light
Follows the subtractive rule: (the
perceived color of several mixed
reflective light sources depends on the
remaining (unabsorbed) wavelengths)
8
Trichromatic color mixing theory
• Trichromatic color mixing theory
– Any color can be obtained by mixing three primary colors with a right
proportion
• Primary colors for illuminating sources:
– Red, Green, Blue (RGB)
– Color monitor works by exciting red, green, blue phosphors using
separate electronic guns
• Primary colors for reflecting sources (also known as secondary
colors):
– Cyan, Magenta, Yellow (CMY)
– Color printer works by using cyan, magenta, yellow.
9
Primary Vs Secondary Colors
Primary Colors
Secondary Colors
10
HSV Visual Illusion
11
HSV Visual Illusion
Which lines are straight?
12
Motion Illusion
13
Motion Perception is Tricky
14
Where is Human Head
15
Test
16
17
Image Processing
)
18
Digital Image Formation
19
Matrix Representation
183
183

179
177
178

179
179

180
160
153
168
177
178
180
179
179
94
116
171
179
179
180
180
181
153
176
182
177
176
179
182
179
194
187
179
179
182
183
183
181
163
166
170
165
164
169
170
170
132
130
131
131
130
132
129
130
165
169
167
167
171

169
173

169
Divide into
8x8 blocks
H=256
W=256
20
Image Resolution
21
Image Resolution
22
Bitplanes
Original 8bits/pixel
one 8-bit byte
Bitplane 7
Bitplane 6
Bitplane 5
Bitplane 4
Bitplane 7
Bitplane 0
23
Bitplanes
Original 8bits/pixel
one 8-bit byte
Bitplane 3
Bitplane 2
Bitplane 1
Bitplane 0
Bitplane 7
Bitplane 0
24
Dimensionality of Digital Images
• Images and videos are multi-dimensional (≥ 2
dimensions) signals.
Dimension 3
Dimension 2
2-D image
Dimension 2
3-D Image
Sequence
or video
Dimension 1
Dimension 1
25
Color
26
Color Space Model
YDbDr
CMYK
RGB
L*a*b*
YCrCb
YUV
YIQ
27
Color: RGB Cube
28
Conversion between RGB and
YIQ in NTSC
• RGB -> YIQ
Y = 0.299 R + 0.587 G + 0.114 B
I = 0.596 R -0.275 G -0.321 B
Q = 0.212 R -0.523 G + 0.311 B
• YIQ -> RGB
R =1.0 Y + 0.956 I + 0.620 Q,
G = 1.0 Y - 0.272 I -0.647 Q,
B =1.0 Y -1.108 I + 1.700 Q.
29
YIQ Component
Color Image
I image (orange-cyan)
Y image
Q image (green-purple)
30
YUV in PAL
31
Y’DbDr in SECAM
32
Y’CbCr in Digital Image/Video
Analog video
Digital video
33
Different Color TV Systems
Parameters
NTSC
PAL
SECAM
Field Rate (Hz)
59.95 (60)
50
50
Line Number/Frame
525
625
625
Line Rate (Line/s)
15,750
15,625
15,625
Color Coordinate
YIQ
YUV
YDbDr
Luminance Bandwidth (MHz)
4.2
5.0/5.5
6.0
Chrominance Bandwidth (MHz)
1.5(I)/0.5(Q)
1.3(U,V)
1.0 (U,V)
Color Subcarrier (MHz)
3.58
4.43
4.25(Db),4.41(Dr)
Color Modulation
QAM
QAM
FM
Audio Subcarrier
4.5
5.5/6.0
6.5
Total Bandwidth (MHz)
6.0
7.0/8.0
8.0
34
Applications
Segmentation
Manipulation
Tracking
Digital Image
Processing
Coding
Detection
Recognition
35
Image Processing: Image Enhancement
Enhance
36
Image Processing: Image De-noising
Denoise
37
Image Processing: Image Deblurring
Deblur
38
Image Processing: Image Inpainting
39
Image Analysis: Edge Detection
40
Image Analysis: Face Detection
41
Image Analysis: Image Segmentation
42
Image Analysis: Image Matching
43
Image Coding: Image Compression
original image
262144 Bytes
image
encoder
compressed bitstream
00111000001001101…
(2428 Bytes)
image
decoder
compression ratio (CR) = 108:1
44
Video Processing
45
Digital Video Processing
• Digital
– Use a digital data format
• Video
– a sequence of images along the temporal axis
– We will talk this more next
• Processing
– A running software program or computing operation
– Driven by the real-world applications (e.g., compression,
filtering, retrieval)
46
Overview of Video Processing
Video
Manipulation
Video
Acquisition
Computer
Graphics
Video
Compression
Video
Display
Video
Database
Video
Transmission
Video
Analysis
Computer
Vision
47
Video Acquisition
Video camera
VHS digitization
computer-generated
48
Acquisition-related Problems
• Video camera
– What if camera is not kept still? (e.g., jittering
in carphone sequence)
– Why is it difficult to improve the spatial
resolution of video cameras?
• VHS digitization
– What if VHS contains some scratches?
– How to handle interlaced video?
• Computer-generated
– How is this type of video different? Shouldn’t
we have a separate coding algorithm for this
type of video?
49
Interlaced vs Progressive scan
Interlaced
Progressive
50
Digital Video
Sampling
Quantization
51
Sampling and Quantization
52
Amplitude
Sampling
5
Quantization
4
Minimum of 3 bits!
3
2
1
0
1
2
3
4
5
6
7
Time
53
BT.601* Video Format
858 pels
864 pels
Active
Area
122
pel
16
pel
525/60: 60 field/s
576 lines
720 pels
625 lines
480 lines
525 lines
720 pels
Active
Area
132
pel
12
pel
625/50: 50 field/s
* BT.601 is formerly known as CCIR601
54
Chrominance Subsampling Formats
Y sample
Cb sample
Cr sample
4:4:4
4:2:2
4:2:0
55
Digital Video Formats
Video Format
Y Size
Color
Sampling
Frame Rate
(Hz)
Raw Data Rate
(Mbps)
HDTV Over air. cable, satellite, MPEG2 video, 20-45 Mbps
SMPTE296M
1280x720
4:2:0
SMPTE295M
1920x1080
4:2:0
24P/30P/60P
24P/30P/60I
265/332/664
597/746/746
Video production, MPEG2, 15-50 Mbps
BT.601
720x480/576
BT.601
720x480/576
60I/50I
60I/50I
249
166
High quality video distribution (DVD, SDTV), MPEG2, 4-10 Mbps
BT.601
720x480/576
4:2:0
60I/50I
124
Intermediate quality video distribution (VCD, WWW), MPEG1, 1.5 Mbps
SIF
352x240/288
4:2:0
30P/25P
30
Video conferencing over ISDN/Internet, H.261/H.263, 128-384 Kbps
CIF
352x288
4:2:0
30P
37
Video telephony over wired/wireless modem, H.263, 20-64 Kbps
QCIF
176x144
4:2:0
30P
9.1
4:4:4
4:2:2
56
Overview of Video Processing
Video
Manipulation
Video
Acquisition
Computer
Graphics
Video
Compression
Video
Display
Video
Database
Video
Transmission
Video
Analysis
Computer
Vision
57
Video Manipulation
• Why?
– Fight against a non-ideal video acquisition (e.g.,
analog heritage, film scratches, limited resolution)
or transmission environment
– Create new and artificial video content (e.g.,
spatio-temporal interpolation,
background/foreground modification)
58
Video De-jittering
PDE-based approach by Jackie Shen
http://epubs.siam.org/sam-bin/dbq/article/41869
59
Video Inpainting
Cool application: remove the annoying texts added by
various video conversion software
60
Error Concealment
some blocks are corrupted
due to channel errors
corrupted blocks are recovered
From surrounding neighbors
in space and time
61
Super Resolution
LR sequence
…
…
…
…
HR sequence
62
Post-processing
Deblocking: suppress block artifacts in video
decoded video frame
at very low bit rate
processed video frame
after deblocking
63
Overview of Video Processing
Video
Manipulation
Video
Acquisition
Computer
Graphics
Video
Compression
Video
Display
Video
Database
Video
Transmission
Video
Analysis
Computer
Vision
64
Video Coding Overview
• The grand challenge
– We still face insufficient storage space for video data even
with Gigabyte hard disks
– Video transmission through limited bandwidth channels
• Existing approaches
– Three-dimensional waveform coding
– Motion-compensated hybrid coding
– Model-based coding
65
Video Coding Standards
66
Overview of Video Processing
Video
Manipulation
Video
Acquisition
Computer
Graphics
Video
Compression
Video
Display
Video
Database
Video
Transmission
Video
Analysis
Computer
Vision
67
Video Transmission (I)
• Downloading
– Pro: you can have your own copy and can watch it
offline
– Con: you have to wait!!!
• Streaming
– Pro: no need to store (we seldom watch a movie
again and again)
– Con: you have to have a good network connection
and pray for less traffic
68
Video Transmission (II)
6Mbps
3Mbps
6Mbps 3Mbps
1.5Mbps
1.5Mbps
Multi-cast mode
Scalable mode
69
Scalable vs. Multicast
• What is scalable coding?
foreman.yuv
foreman128k.cod
foreman256k.cod
foreman512k.cod
foreman1024k.cod
foreman.yuv
foreman.cod
128 256
Multicast
512
1024
Scalable coding
70
Spatial scalability
1 0 1 1 1 …0 1 0 1 0 0 0 …1 1 0 1 0 0
71
Temporal scalability
1 0 1 1 1 …0 1 0 1 0 0 0 …1 1 0 1 0 0
Frame 0,4,8,12,…
7.5Hz
Frame 0,2,4,6,8,…
15Hz
Frame 0,1,2,3,4,5,…
30Hz
72
SNR (Rate) scalability
1 0 1 1 1 …0 1 0 1 0 0 0 …1 1 0 1 0 0
PSNRavg=30dB
PSNRavg
PSNRavg=35dB
1

N
PSNRavg=40dB
N
 PSNR
i 1
i
PSNRi: PSNR of frame i
73
Overview of Video Processing
Video
Manipulation
Video
Acquisition
Computer
Graphics
Video
Compression
Video
Display
Video
Database
Video
Transmission
Video
Analysis
Computer
Vision
74
Motion Tracking
75
Video-based Lifeguard
Application in swimming pool monitoring to prevent drowning
76
Overview of Video Processing
Video
Manipulation
Video
Acquisition
Computer
Graphics
Video
Compression
Video
Display
Video
Database
Video
Transmission
Video
Analysis
Computer
Vision
77
Video Database Management
• Database management
– Indexing, parsing, browsing, querying
– Retrieval
• What is special about video?
– Formidable amount of data
– Difficulty with query (content-based)
• MPEG-7 is designed for database management
78
Content-Based Video Retrieval
(CBVR)
• How to provide a compact and complete video
sequence representation?
– Spatial analysis (histogram, color, texture)
– Temporal analysis (cut, dissolve, wipe)
• How to provide easy-to-use and efficient query
interface to user
– Video browsing (slide vs. 3D)
– Video querying (example-based, text-based)
79
Video Summarization
Extract “important” motion pictures such as home-runs
80