Transcript Mat

Introduction to C & C++
Lecture 10 – Image Processing with
OpenCV
JJCAO
OpenCV (Open Source Computer Vision) is a library of programming functions for
real time computer vision.
Supported Development Environment
• Linux, Windows, Android
• GCC, Eclipse, Visual Studio,
Install OpenCV
1. Basic Info: http://opencv.willowgarage.com/wiki/InstallGuide
2. Necessary steps for Qt Creator (Similar with
VS): http://www.laganiere.name/opencvCookbook/chap1.shtml
3. Installation in Windows (too much of details):
http://opencv.itseez.com/doc/tutorials/introduction/windows_install/windows_install.htm
l#windows-installation
How to build applications with OpenCV
inside the Microsoft Visual Studio
http://opencv.itseez.com/doc/tutorials/introduction/windows_vi
sual_studio_Opencv/windows_visual_studio_Opencv.html#wind
ows-visual-studio-how-to
P11 of
OpenCV.2.Computer.Vision.Application.Programming.Cookbook_
2011
Load & Display an Image
1. Load an image (using imread)
2. Create a named OpenCV window
(using namedWindow)
3. Display an image in an OpenCV window
(using imshow)
1. Load an image (using imread)
cv::Mat image;
image = cv::imread(argv[1], CV_LOAD_IMAGE_COLOR); // Read the file
if( img.empty() )
// Check for invalid input
{
std::cout << "Could not open or find the image" << std::endl ;
return -1;
}
Image formats supported: Bmp, jpg, png, tif, ppm, …
CV_LOAD_IMAGE_UNCHANGED (<0)
loads the image as is (including the alpha channel if present)
CV_LOAD_IMAGE_GRAYSCALE (0)
loads the image as an intensity one
CV_LOAD_IMAGE_COLOR (>0)
loads the image in the RGB format
2. Create a named OpenCV window
(using namedWindow)
cv::namedWindow( "Display window",
CV_WINDOW_NORMAL|CV_WINDOW_FREERATIO );// Create a window for
display.
CV_WINDOW_AUTOSIZE
is the only supported one if you do not use the Qt backend. In this case the window
size will take up the size of the image it shows. No resize permitted!
CV_WINDOW_NORMAL
on Qt you may use this to allow window resize. The image will resize itself according
to the current window size. By using the | operator you also need to specify if you
would like the image to keep its aspect ratio (CV_WINDOW_KEEPRATIO) or not
(CV_WINDOW_FREERATIO).
3. Display an image in an OpenCV window
(using imshow)
cv::imshow( "Display window", image ); // Show our image inside it.
cv::waitKey(0); // Wait for a keystroke in the window
• Because we want our window to be displayed until the user presses a key
(otherwise the program would end far too quickly), we use the waitKey function
whose only parameter is just how long should it wait for a user input (measured
in milliseconds).
• Zero means to wait forever.
Necessary Head Files
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
using namespace cv;
using namespace std;
You’ll almost always end up using the:
• core section, as here are defined the basic building
blocks of the library
• highgui module, as this contains the functions for input,
output & GUI operations
Necessary libraries
#ifdef _DEBUG
#pragma comment(lib, "opencv_core230d.lib“ )
#pragma comment(lib, "opencv_highgui230d.lib“ )
#else
#pragma comment(lib, "opencv_core230.lib“ )
#pragma comment(lib, "opencv_highgui230.lib“ )
#endif
Congratulation!
From C to C++
• OpenCV has been around ever since 2001. In those
days the library was built around a C interface.
– user is responsible for taking care of memory allocation
and de-allocation
– Lots of old tutorials written in C
• Once your code start to grow larger & larger, more &
more a struggle to handle this rather than focusing on
solving your goal
• Finally C++
– automatic memory management (more or less)
– less to write, to achieve more
class CV_EXPORTS Mat {
public:
// ... a lot of methods ... ...
/*! includes several bit-fields:
- the magic signature
- continuity flag
- depth
- number of channels */
int flags;
int dims; //! the array dimensionality, >= 2
int rows, cols; //! the number of rows and columns or (-1, -1) when the
array has more than 2 dimensions
Head
int* refcount; //! pointer to the reference counter;
user-allocated data, the pointer is NULL
// other members ...
uchar* data; //! pointer to the data
};
when array points to
Create a cv::Mat
Mat A, C; // creates just the header parts
A = imread(argv[1], CV_LOAD_IMAGE_COLOR); // the
method will allocate matrix (deep copy)
Mat B(A); // Use the copy constructor, without copying the data
(shadow copy)
C = A; // Assignment operator, shadow copy
Mat roi(A, Rect(10,10,100,100)); // select a ROI
roi = Scalar(0,255,0); // fill the ROI with (0,255,0) (which is green
in RGB space); the original A will be modified; see next page.
cv::Scalar
template <typename _Tp> class Scalar_ : public
Vec<_Tp, 4> { ... };
typedef Scalar_<double> Scalar;
Being derived from Vec<_Tp, 4> , Scalar_ and Scalar can
be used just as typical 4-element vectors.
Deep Copy
Mat F = A.clone();
Mat G;
A.copyTo(G);
Now modifying F or G will not affect the matrix
pointed by the Mat header.
What you need to remember
1. Output image allocation for OpenCV functions is
automatic (unless specified otherwise).
– Example (next page)
2. No need to think about memory freeing with
OpenCVs C++ interface.
3. The assignment operator and the copy constructor
(ctor)copies only the header.
4. Use the clone() or the copyTo() function to copy the
underlying matrix of an image.
Output image allocation for OpenCV
functions is automatic
• instead of writing:
Mat color;
...
Mat gray(color.rows, color.cols, color.depth());
cvtColor(color, gray, CV_BGR2GRAY);
• you can simply write:
Mat color;
...
Mat gray;
cvtColor(color, gray, CV_BGR2GRAY);
How to scan images, lookup table & time
measurement
•
•
•
•
1.
2.
3.
4.
How to go through each and every pixel of an image?
How is OpenCV matrix values stored?
How to measure the performance of our algorithm?
What are lookup tables and why use them?
Basic Mat info
Storing methods
Data type conversion
Accessing Pixel Values
a simple color reduction method
how_to_scan_images imageName.jpg divideWith [G]
uchar
if( argc == 4 && !strcmp(argv[3],"G") )
I = imread(argv[1], CV_LOAD_IMAGE_GRAYSCALE);
else
I = imread(argv[1], CV_LOAD_IMAGE_COLOR);
CV_8U
CV_8U3
Storing methods
• How to store pixel values?
– Color space
• Gray-level (Black-and-White)
– unsigned 8-bit values
– 0: black, 255: white
• RGB
– A triplet of unsigned 8-bit values
– [0, 0, 0]: black, [255, 0, 0]: red, [255, 255, 255]: white
• HSV, HLS, …, CIE
– Data type
bool Mat::isContinuous()
• Matrices created with Mat::create() are always
row1
row2
…
row n
continuous:
• may no longer continuous
– extract a part of the matrix
using Mat::col(), Mat::diag() , and so on,
– or constructed a matrix header for externally
allocated data.
Storing methods
• How to store pixel values?
– Color space
• Gray-level (Black-and-White)
– unsigned 8-bit values
– 0: black, 255: white
• RGB
– A triplet of unsigned 8-bit values
– [0, 0, 0]: black, [255, 0, 0]: red, [255, 255, 255]: white
• HSV, HLS, …, CIE
– Data type
CV_8U - 8-bit unsigned integers ( 0..255 )
CV_8S - 8-bit signed integers ( -128..127 )
CV_16U - 16-bit unsigned integers ( 0..65535 )
CV_16S - 16-bit signed integers ( -32768..32767 )
CV_32S - 32-bit signed integers ( -2147483648..2147483647 )
CV_32F - 32-bit floating-point numbers ( -FLT_MAX..FLT_MAX, INF, NAN )
CV_64F - 64-bit floating-point numbers ( -DBL_MAX..DBL_MAX, INF, NAN )
Inquire Mat Info
1. int Mat::depth() const
2. int Mat::channels() const
3. int Mat::type() // mixture of depth & channels
–
–
–
–
–
–
#define CV_8UC1 CV_MAKETYPE(CV_8U,1)
#define CV_8UC2 CV_MAKETYPE(CV_8U,2)
#define CV_8UC3 CV_MAKETYPE(CV_8U,3)
#define CV_8UC4 CV_MAKETYPE(CV_8U,4)
#define CV_8UC(n) CV_MAKETYPE(CV_8U,(n))
…
4. size_t Mat::elemSize() // matrix element (pixel) size in bytes
–
if the matrix type is CV_16SC3 , the method
returns3*sizeof(short) or 6.
5. size_t Mat::elemSize1() // element size of a channel in bytes
– if the matrix type is CV_16SC3 , the method
returns sizeof(short) or 2.
6. M.step[]
7. size_t Mat::step1() // a matrix step divided by Mat::elemSize1()
Accessing Pixel Values
1. The efficient way: c style access
p = I.ptr<uchar>(i);
p[j]
2. The iterator (safe) method
3. On-the-fly address calculation with reference returning
I.at<uchar>(i,j)
• Performance Difference
quite large (2560 X 1600) image
Performance Difference
Debug
Release
More efficient Algorithm
1. divide and multiplication operations are
bloody expensive for a system.
2. cheaper operations such as a few
subtractions, addition or in best case a
simple assignment
3. limited number of input values, 256 to be
exact in this problem
Lookup table
int divideWith; // convert our input string to number - C++ style
stringstream s;
s << argv[2];
s >> divideWith;
if (!s) {
cout << "Invalid number entered for dividing. " << endl;
return -1;
}
uchar table[256];
for (int i = 0; i < 256; ++i)
table[i] = divideWith* (i/divideWith);
Mat& ScanImageAndReduceC(Mat& I, const uchar*
const table)
{
int i,j;
uchar* p;
for( i = 0; i < nRows; ++i)
{
p = I.ptr<uchar>(i);
for ( j = 0; j < nCols; ++j)
{
p[j] = table[p[j]];
}
}
return I;
// accept only char type matrices
CV_Assert(I.depth() != sizeof(uchar));
int channels = I.channels();
int nRows = I.rows;
int nCols = I.cols * channels;
if (I.isContinuous())
{
nCols *= nRows;
nRows = 1;
}
}
Basic Mat Info
1.
2.
3.
4.
bool Mat::empty()
size_t Mat::total()
int Mat::rows, Mat::cols
Size Mat::size()
Mat& ScanImageAndReduceIterator(Mat& I, const
uchar* const table)
{
// accept only char type matrices
CV_Assert(I.depth() != sizeof(uchar));
const int channels = I.channels();
switch(channels)
{
case 1: {
MatIterator_<uchar> it, end;
for( it = I.begin<uchar>(), end = I.end<uchar>(); it != end; ++it)
*it = table[*it];
break;
}
case 3: {
MatIterator_<Vec3b> it, end;
for( it = I.begin<Vec3b>(), end = I.end<Vec3b>(); it != end; ++it) {
(*it)[0] = table[(*it)[0]];
(*it)[1] = table[(*it)[1]];
(*it)[2] = table[(*it)[2]];
}
}
}
return I;
}
Mat& ScanImageAndReduceRandomAccess(Mat& I,
const uchar* const table)
{
case 3: {
Mat_<Vec3b> _I = I;
// accept only char type matrices
CV_Assert(I.depth() != sizeof(uchar));
const int channels = I.channels();
switch(channels)
{
for( int i = 0; i < I.rows; ++i)
for( int j = 0; j < I.cols; ++j ) {
_I(i,j)[0] = table[_I(i,j)[0]];
_I(i,j)[1] = table[_I(i,j)[1]];
_I(i,j)[2] = table[_I(i,j)[2]];
}
I = _I;
break;
case 1: {
for( int i = 0; i < I.rows; ++i)
for( int j = 0; j < I.cols; ++j ){
I.at<uchar>(i,j) =
table[I.at<uchar>(i,j)];
}
break;
}
}
}
return I;
}
The Core Function: cv::LUT()
LUT: replace all of given image values to some other
values
Mat lookUpTable(1, 256, CV_8U);
uchar* p = lookUpTable.data;
for( int i = 0; i < 256; ++i)
p[i] = table[i];
for (int i = 0; i < times; ++i)
cv::LUT(I, lookUpTable, J);
Conclusion
1. If possible, use the already made functions of OpenCV
(instead reinventing these).
2. The fastest method turns out to be the LUT function.
This is because the OpenCV library is multi-thread
enabled via Intel Threaded Building Blocks.
3. However, if you need to write a simple image scan
prefer the pointer method. The iterator is a safer bet,
however quite slower.
4. Using the on-the-fly reference access method for full
image scan is the most costly in debug mode. In the
release mode it may beat the iterator approach or not,
however it surely sacrifices for this the safety trait of
iterators.
Type Conversion
#include <opencv2/imgproc/imgproc.hpp>
cvtColor(I, J, CV_RGB2GRAY);
for( int i = 0; i < 0.5*I.rows; ++i)
for( int j = 0; j < 0.5*I.cols; ++j ) {
//J.at<uchar>(i,j) = 0;
J.at<float>(i,j) = 0;
}
Mat::convertTo()
void Mat::convertTo(OutputArray m, int rtype,
double alpha=1, double beta=0 )
• Converts an array to another datatype with optional scaling.
• Parameters:
– m – Destination matrix. If it does not have a proper size or type before the
operation, it is reallocated.
– rtype – Desired destination matrix type or, rather, the depth since the number
of channels are the same as the source has. If rtype is negative, the destination
matrix will have the same type as the source.
– alpha – Optional scale factor.
– beta – Optional delta added to the scaled values.
• The method converts source pixel values to the target
datatype. saturate_cast<> is applied at the end to avoid possible
overflows:
One More Example: gradient
infile='data\HappyFish.jpg';
im = imread(infile);
info = imfinfo(infile);
if
~strcmp(info.ColorType,'grayscale')
im = double(rgb2gray(im)) ;
end
[gx,gy] = gradient(im);
figure;colormap(gray);
imagesc(im);hold on;
[x,y] = meshgrid(1:n,1:m);
quiver(x, y, gx,gy);
See the cvMatlab and cvMatlabTest
example
void jj::gradient(cv::InputArray _src, cv::OutputArray _dst, int xorder)
{
cv::Mat src=_src.getMat();
_dst.create( src.size(), CV_MAKETYPE(src.depth(), src.channels()) );
cv::Mat dst = _dst.getMat();
…
}
See the cvMatlab and cvMatlabTest
example
• Difference in x direction
// single channel
Mat src, dest; // same size, diff depth
for( int i = 0; i < src.rows; ++i)
for( int j = 1; j < src.cols-1; ++j ) {
dest.at<uchar>(i,j) = (src.at<uchar>(i,j+1) - src.at<uchar>(i,j-1) )
/2.0;
}
// 3 channels
dst.at<cv::Vec3b>(i,j)[0] = (src.at<cv::Vec3b>(i,j+1)[0] - src.at<cv::Vec3b>(i,j-1)[0] ) /2.0;
dst.col(j)=(src.col(j+1)-src.col(j-1))/2.0;
Create a Mat object 1
• Mat()
Mat M(2,2, CV_8UC3, Scalar(0,0,255));
cout << "M = " << endl << " " << M << endl << endl;
• Create a matrix with more than two dimensions
int sz[3] = {2,2,2};
Mat L(3,sz, CV_8UC(1), Scalar::all(0)); // Specify its dimension, then pass
a pointer containing the size for each dimension and the rest remains
the same.
• Create a header for an already existing IplImage pointer
IplImage* img = cvLoadImage("greatwave.png", 1);
Mat mtx(img); // convert IplImage* -> Mat
Create a Mat object 2
• Create() function
M.create(4,4, CV_8UC(2));
cout << "M = "<< endl << " " << M << endl << endl;
// You cannot initialize the matrix values with this construction. It will
only reallocate its matrix data memory if the new size will not fit into
the old one.
• MATLAB style initializer: zeros(), ones(), :eyes()
Mat E = Mat::eye(4, 4, CV_64F);
Mat O = Mat::ones(2, 2, CV_32F);
Mat Z = Mat::zeros(3,3, CV_8UC1);
Create a Mat object 3
• For small matrices
Mat C = (Mat_<double>(3,3) << 0, -1, 0, -1, 5, -1, 0, -1, 0);
• Create a new header for an existing Mat object
and clone() or copyTo() it
Mat RowClone = C.row(1).clone();
Create a Mat object 4
• Create a random matrix with randu()
Mat R = Mat(3, 2, CV_8UC3);
randu(R, Scalar::all(0), Scalar::all(255));
cout << "R (default) = " << endl << R << endl << endl;
cout << "R (python) = " << endl << format(R,"python") << endl << endl;
cout << "R (csv) = " << endl << format(R,"csv" ) << endl << endl;
…
Resources
• http://opencv.willowgarage.com/wiki/
• C++ Cheatsheet.pdf
• Online reference manual: 2.3 documentation is here
2008
2011
References
• OpenCV 2 Computer Vision Application
Programming Cookbook, 2011.