Transcript 91.204.201_02_OpenCVCore

91.204.201
Computing IV
Chapter Two: Core Module. The Core
Functionality
Xinwen Fu
References
Application Development in Visual Studio
 Reading assignment: Chapter 2


An online OpenCV Quick Guide with nice
examples
By Dr. Xinwen Fu
CS@UML
2
A few things
Blackboard submission
 Report format
 Sreenshots

By Dr. Xinwen Fu
CS@UML
3
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
4
2.1 Mat - The Basic Image Container


We have multiple ways to acquire digital images from the
real world: digital cameras, scanners, computed tomography
or magnetic resonance imaging to just name a few. In every
case what we (humans) see are images.
When transforming this to our digital devices what we record
are numerical values for each of the points of the image.
By Dr. Xinwen Fu
CS@UML
5
Storing Images


A black-white image is nothing more than a
matrix containing all the intensity values of the
pixel points.
How we get and store the pixels values may vary
according to what fits best our need


In the end all images inside a computer world may be
reduced to numerical matrices and some other
information describing the matric itself.
OpenCV is a computer vision library whose main
focus is to process and manipulate these
information to find out further ones.

The first thing to learn and get accommodated with is
how OpenCV stores and handles images.
By Dr. Xinwen Fu
CS@UML
6
Mat


Basically a class with two data parts:

the matrix header (containing information such
as the size of the matrix, the method used for
storing, at which address is the matrix stored
and so on)

a pointer to the matrix containing the pixel
values (may take any dimensionality
depending on the method chosen for storing)
OpenCV is an image processing library,
doing image processing with its functions
By Dr. Xinwen Fu
CS@UML
7
Copy Mat

OpenCV uses a reference counting system.
The idea is that each Mat object has its own
header.

However the matrix may be shared between
two instance of them by having their matrix
pointer point to the same address.

Copy operators will only copy the headers,
and as also copy the pointer to the large
matrix too, however not the matrix itself.
By Dr. Xinwen Fu
CS@UML
8
1.
Mat A, C; // creates just the header parts
2.
// here we'll know the method used (allocate matrix)
A=imread(file, CV_LOAD_IMAGE_COLOR);
3.
4.
5.



Mat B(A); // use the copy constructor
C=A; // assignment operator
All the above objects, in the end point to the same
single data matrix.
Their headers are different, however making any
modification using either one of them will affect all the
other ones too.
In practice the different objects just provide different
access method to the same underlying data.
Nevertheless, their header parts are different.
By Dr. Xinwen Fu
CS@UML
9
Refer only to a subsection of the full data
The real interesting part comes that you
can create headers that refer only to a
subsection of the full data.
 For example, to create a region of interest
(ROI) in an image you just create a new
header with the new boundaries:

Coordinates of the top-left corner
By Dr. Xinwen Fu
CS@UML
Rectangle width and height
10
Cleaning Mat

You may ask if the matrix itself may belong to
multiple Mat objects who will take responsibility
for its cleaning when it’s no longer needed.


The short answer is: the last object that used it.
For this a reference counting mechanism is used

Whenever somebody copies a header of a Mat object a
counter is increased for the matrix.

Whenever a header is cleaned this counter is decreased.

When the counter reaches zero the matrix too is freed.
By Dr. Xinwen Fu
CS@UML
11
Copy the matrix itself

Because, sometimes you will still want to
copy the matrix itself too, there exists the
clone() or the copyTo() function.

Now modifying F or G will not affect the
matrix pointed by the Mat header
By Dr. Xinwen Fu
CS@UML
12
Tips of using Mat

Output image allocation for OpenCV functions
is automatic (unless specified otherwise).

No need to think about memory freeing with
OpenCVs C++ interface.

The assignment operator and the copy
constructor (ctor)copies only the header.

Use the clone() or the copyTo() function to
copy the underlying matrix of an image.
By Dr. Xinwen Fu
CS@UML
13
Storing methods for pixel values.

You can select color space and data type used



The color space refers to how we combine color
components in order to code a given color.
The simplest one is the gray scale.
For colorful ways we have a lot more of methods
to choose from. However, every one of them
breaks it down to three or four basic components
and the combination of this will give all others.


The most popular one is RGB, mainly because this is
also how our eye builds up colors in our eyes. Its base
colors are red, green and blue.
To code the transparency of a color sometimes a fourth
element: alpha (A) is added.
By Dr. Xinwen Fu
CS@UML
14
Color Systems


RGB is the most common as our eyes use
something similar, used by our display systems.
The HSV and HLS decompose colors into their
hue, saturation and value/luminance
components, which is a more natural way for us
to describe colors.



You may dismiss last component, making your algorithm
less sensible to light conditions of the input image.
YCrCb is used by the popular JPEG image format.
CIE L*a*b* is a perceptually uniform color space,
which comes handy if you need to measure the
distance of a given color to another color.
By Dr. Xinwen Fu
CS@UML
15
Data Types for Color


Each of building components has their own valid
domains. This leads to the data type used.
The smallest data type possible is char, which
means one byte or 8 bits


In case of three components this gives 16 million
possible colors to represent (like in case of RGB)


This may be unsigned (so can store values from 0 to
255) or signed (values from -127 to +127).
Even finer control by using float (4 byte = 32 bit) or
double (8 byte = 64 bit) data types for each component.
However, increasing the size of a component also
increases the size of the whole picture in the
memory.
By Dr. Xinwen Fu
CS@UML
16
Creating explicitly a Mat object

Although Mat is a great class as image
container it is also a general matrix class.

Therefore, it is possible to create and
manipulate multidimensional matrices.

You can create a Mat object in multiple
ways

For two dimensional and multichannel
images we first define their size: row and
column count wise.
By Dr. Xinwen Fu
CS@UML
17

We need to specify the data type to use for storing the elements
and the number of channels per matrix point. To do this we have
multiple definitions made according to the following convention:




CV_[The number of bits per item][Signed or Unsigned][Type Prefix]C[The
channel number]
CV_8UC3 means we use unsigned char types that are 8 bit long and each
pixel has three items of this to form the three channels. This are predefined
for up to four channel numbers.
The Scalar is four element short vector.
Only 2 dimension matrix can use cout
By Dr. Xinwen Fu
CS@UML
18
By Dr. Xinwen Fu
CS@UML
19
Data Type

A primitive OpenCV data type is one of unsigned char, bool,
signed char, unsigned short, signed short, int, float, double,
or a tuple of values of one of these types, where all the
values in the tuple have the same type.

Any primitive type from the list can be defined by an
identifier in the form


CV_<bit-depth>{U|S|F}C(<number_of_channels>), for example:
uchar ~ CV_8UC1, 3-element floating-point tuple ~ CV_32FC3
A universal OpenCV structure that is able to store a single
instance of such a primitive data type is Vec. Multiple
instances of such a type can be stored in a std::vector,
Mat, Mat_, SparseMat, SparseMat_, or any other container
that is able to store Vec instances.
By Dr. Xinwen Fu
CS@UML
20
By Dr. Xinwen Fu
CS@UML
21
Print for other common items
OpenCV offers support for print of other
common OpenCV data structures too via
the << operator like
 2D Point


3D Point

std::vector via cv::Mat
By Dr. Xinwen Fu
CS@UML
22
Print (Cont’d)

std::vector of points
By Dr. Xinwen Fu
CS@UML
23
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
24
Goal
How to go through each and every pixel of
an image?
 How is OpenCV matrix values stored?
 How to measure the performance of our
algorithm?
 What are lookup tables and why use
them?

By Dr. Xinwen Fu
CS@UML
25
Color space reduction
Divide the color space current value with a
new input value to end up with fewer
colors
 For instance every value between zero
and nine takes the new value zero, every
value between ten and nineteen the value
ten and so on.

By Dr. Xinwen Fu
CS@UML
26
How the image matrix is stored in the
memory? - gray scale image

The size of the matrix depends of the color
system used. More accurately, it depends
from the number of channels used.
By Dr. Xinwen Fu
CS@UML
27
How the image matrix is stored in the
memory? - RGB color system
For multichannel images the columns
contain as many sub columns as the
number of channels
 Note that the order of the channels is
inverse: BGR instead of RGB

By Dr. Xinwen Fu
CS@UML
28
Color reduction formula

When you divide an uchar (unsigned char - aka
values between zero and 255) value with an int
value the result will be also char. These values
may only be char values.


Therefore, any fraction will be rounded down.
Taking advantage of this fact the upper operation
in the uchar domain may be expressed as:
By Dr. Xinwen Fu
CS@UML
29
Measure time code runs


Another issue is how do we measure time?
OpenCV offers two simple functions to achieve this
getTickCount() and getTickFrequency().


The first returns the number of ticks of your systems CPU
from a certain event (like since you booted your system).
The second returns how many times your CPU emits a tick
during a second. So to measure in seconds the number of
time elapsed between two operations is easy as:
By Dr. Xinwen Fu
CS@UML
30
Lookup table for color reduction

how_to_scan_images imageName.jpg
intValueToReduce [G]

The final argument is optional. If given the image will be
loaded in gray scale format, otherwise the RGB color way is
used. The first thing is to calculate the lookup table.
By Dr. Xinwen Fu
CS@UML
31
The Efficient Way
By Dr. Xinwen Fu
CS@UML
32
Iterator
By Dr. Xinwen Fu
CS@UML
33
On-the-fly address calc


In case of color images we have three uchar items per
column.
This may be considered a short vector of uchar items, that
has been baptized in OpenCV with the Vec3b name.
By Dr. Xinwen Fu
CS@UML
34
On-the-fly address calc
By Dr. Xinwen Fu
CS@UML
35
The Core Function
By Dr. Xinwen Fu
CS@UML
36
Performance Difference


For the best result compile the program and run it on your
own speed. For showing off better the differences I’ve used
a quite large (2560 X 1600) image.
The performance presented here are for color images. For a
more accurate value I’ve averaged the value I got from the
call of the function for hundred times.
By Dr. Xinwen Fu
CS@UML
37
Mat_



If you need multiple lookups using this method for an
image it may be troublesome and time consuming to enter
the type and the at keyword for each of the accesses. To
solve this problem OpenCV has a Mat_ data type.
It’s the same as Mat with the extra need that at definition
you need to specify the data type through what to look at
the data matrix, however in return you can use the
operator() for fast access of items.
To make things even better this is easily convertible from
and to the usual Mat data type. A sample usage of this you
can see in case of the color images of the upper function.
Nevertheless, it’s important to note that the same
operation (with the same runtime speed) could have been
done with the at() function. It’s just a less to write for the
lazy programmer trick.
By Dr. Xinwen Fu
CS@UML
38
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
39
Mask operation

Recalculate each pixels value in an image according to
a mask matrix (also known as kernel).


This mask holds values that will adjust how much influence
neighboring pixels (and the current pixel) have on the new
pixel value.
From a mathematical point of view we make a weighted
average, with our specified values.
By Dr. Xinwen Fu
CS@UML
40
Basic Method
1.
2.
3.
void Sharpen(const Mat& myImage,Mat& Result)
{
CV_Assert(myImage.depth() == CV_8U); // accept only uchar images
4.
5.
6.
7.
8.
9.
10.
11.
const int nChannels = myImage.channels();
Result.create(myImage.size(),myImage.type());
for(int j = 1 ; j < myImage.rows-1; ++j)
{
const uchar* previous = myImage.ptr<uchar>(j - 1);
const uchar* current = myImage.ptr<uchar>(j );
const uchar* next
= myImage.ptr<uchar>(j + 1);
uchar* output = Result.ptr<uchar>(j);
By Dr. Xinwen Fu
CS@UML
41
12.
for(int i= nChannels;i < nChannels*(myImage.cols-1); ++i)
13.
{
*output++ = saturate_cast<uchar>(5*current[i]
14.
-current[i-nChannels] - current[i+nChannels] - previous[i] - next[i]);
15.
}
16.
17.
}
18.
Result.row(0).setTo(Scalar(0));
19.
Result.row(Result.rows-1).setTo(Scalar(0));
20.
Result.col(0).setTo(Scalar(0));
21.
Result.col(Result.cols-1).setTo(Scalar(0));
22.
}
By Dr. Xinwen Fu
CS@UML
42
filter2D function

Applying such filters is so common in image
processing that in OpenCV there exist a function
that will take care of applying the mask (also
called a kernel in some places).
1.
define a Mat object that holds the mask:
2.
Then call the filter2D function specifying the input, the
output image and the kernell to use:
By Dr. Xinwen Fu
CS@UML
43
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
44
Theory
From our previous tutorial, we know
already a bit of Pixel operators.
 An interesting dyadic (two-input) operator
is the linear blend operator:


By varying from 0 to 1 this operator can
be used to perform a temporal crossdissolve between two images or videos, as
seen in slide shows and film productions
By Dr. Xinwen Fu
CS@UML
45
Example
1.
2.
3.
4.
5.
6.
7.
8.
#include <opencv/cv.h>
#include <opencv/highgui.h>
#include <iostream>
using namespace cv;
int main( int argc, char** argv )
{
double alpha = 0.5; double beta; double input;
Mat src1, src2, dst;
9.
10.
11.
12.
13.
14.
/// Ask the user enter alpha
std::cout<<" Simple Linear Blender "<<std::endl;
std::cout<<"-----------------------"<<std::endl;
std::cout<<"* Enter alpha [0-1]: ";
std::cin>>input;
By Dr. Xinwen Fu
CS@UML
46
/// We use the alpha provided by the user iff it is between 0 and 1
if( alpha >= 0 && alpha <= 1 ) { alpha = input; }
15.
Example
16.
/// Read image ( same size, same type )
src1 = imread("../../images/LinuxLogo.jpg");
src2 = imread("../../images/WindowsLogo.jpg");
if( !src1.data ) { printf("Error loading src1 \n"); return -1; }
if( !src2.data ) { printf("Error loading src2 \n"); return -1; }
17.
18.
19.
20.
21.
25.
/// Create Windows
namedWindow("Linear Blend", 1);
beta = ( 1.0 - alpha );
addWeighted( src1, alpha, src2, beta, 0.0, dst);
26.
imshow( "Linear Blend", dst );
27.
waitKey(0);
return 0;
22.
23.
24.
28.
29.
}
By Dr. Xinwen Fu
CS@UML
47
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
48
Image Processing

A general image processing operator is a
function that takes one or more input
images and produces an output image.

Image transforms can be seen as:


Point operators (pixel transforms)
Neighborhood (area-based) operators
By Dr. Xinwen Fu
CS@UML
49
Pixel Transforms

In this kind of image processing transform,
each output pixel’s value depends on only the
corresponding input pixel value (plus,
potentially, some globally collected
information or parameters).

Examples of such operators include
brightness and contrast adjustments as well
as color correction and transformations.
By Dr. Xinwen Fu
CS@UML
50
Brightness and contrast adjustments

Two commonly used point processes are
multiplication and addition with a constant:


The parameters α > 0 and β are often called the
gain and bias parameters


g(x) = α f(x) + β
Sometimes these parameters are said to control contrast
and brightness respectively.
You can think of f(x) as the source image pixels
and g(x) as the output image pixels. Then, more
conveniently we can write the expression as:

g(i; j) = α f(i; j) + β
where i and j indicates that the pixel is located in the ith row and j-th column.
By Dr. Xinwen Fu
CS@UML
51
Example
3.
#include <opencv/cv.h>
#include <opencv/highgui.h>
#include <iostream>
4.
using namespace cv;
5.
double alpha; /**< Simple contrast control */
int beta; /**< Simple brightness control */
1.
2.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
int main( int argc, char** argv )
{
/// Read image given by user
Mat image = imread( argv[1] );
Mat new_image = Mat::zeros( image.size(), image.type() );
/// Initialize values
std::cout<<" Basic Linear Transforms "<<std::endl;
std::cout<<"-------------------------"<<std::endl;
std::cout<<"* Enter the alpha value [1.0-3.0]: ";std::cin>>alpha;
std::cout<<"* Enter the beta value [0-100]: "; std::cin>>beta;
CS@UML
By Dr. Xinwen Fu
52
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
/// Do the operation new_image(i,j) = alpha*image(i,j) + beta
for( int y = 0; y < image.rows; y++ )
{
for( int x = 0; x < image.cols; x++ )
{
for( int c = 0; c < 3; c++ )
{
new_image.at<Vec3b>(y,x)[c] =
saturate_cast<uchar>( alpha*( image.at<Vec3b>(y,x)[c] ) + beta );
}
}
}
/// Create Windows
namedWindow("Original Image", 1);
namedWindow("New Image", 1);
/// Show stuff
imshow("Original Image", image);
imshow("New Image", new_image);
/// Wait until user press some key
36. waitKey();
37. return 0;
CS@UML
38. }
35.
By Dr. Xinwen Fu
53
Who is Lena?

Lena
Söderberg, a
Swedish model

cropped from
the centerfold
of November
1972 issue of
Playboy
magazine
By Dr. Xinwen Fu
CS@UML
54
Core function

Instead of using the for loops to access each
pixel, we could have simply used this command:
image.convertTo(new_image, -1, alpha, beta);
where convertTo would effectively perform
new_image = a*image + beta.

However, we wanted to show you how to access
each pixel. In any case, both methods give the
same result.
By Dr. Xinwen Fu
CS@UML
55
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
56
Example in Visual Studio 2010

Use Point to define 2D points in an image.

Use Scalar and why it is useful

Draw a line by using the OpenCV function line

Draw an ellipse by using the OpenCV function ellipse

Draw a rectangle by using the OpenCV function rectangle

Draw a circle by using the OpenCV function circle

Draw a filled polygon by using the OpenCV function fillPoly
By Dr. Xinwen Fu
CS@UML
57
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
58
Example in Visual Studio 2010

Use the Random Number generator class
(RNG) and how to get a random number
from a uniform distribution.



RNG rng( 0xFFFFFFFF );
rng.uniform(a,b); // This generates a
randomly uniformed distribution between the
values a and b (inclusive in a, exclusive in b).
Display text on an OpenCV window by
using the function putText
By Dr. Xinwen Fu
CS@UML
59
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
60
Skipped

What is a Fourier transform and why use
it?

How to do it in OpenCV?

Usage of functions such as:
copyMakeBorder(), merge(), dft(),
getOptimalDFTSize(), log() and
normalize() .
By Dr. Xinwen Fu
CS@UML
61
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
62
Skipped

How to print and read text entries to a file
and OpenCV using YAML or XML files?

How to do the same for OpenCV data
structures?

How to do this for your data structures?

Usage of OpenCV data structures such as
FileStorage, FileNode or FileNodeIterator.
By Dr. Xinwen Fu
CS@UML
63
Outline










2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time
measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu
CS@UML
64
Skipped

What changed with the version 2 of
OpenCV in the way you use the library
compared to its first version

How to add some Gaussian noise to an
image

What are lookup tables and why use
them?
By Dr. Xinwen Fu
CS@UML
65
References

OpenCV documentation
By Dr. Xinwen Fu
CS@UML
66