Transcript 9 Robot Vision

9 Robot Vision
Intro to Robots
Processing Pictures:
• Without the fluke board we can’t process pictures taken
by the robot.
• However we can process regular .jpg files.
• Caution: Processing can take a long time so keep your
pictures to 600 x 600 pixels. If you are just experimenting
try to keep your pictures even smaller – 200x200 pixels.
Intro to Robots
Picture Functions:
from myro import *
picture = makePicture(“somefile.jpg”) # read from a .jpg file
show(picture)
newPic = pictureCopy(picture)
picWidth = getWidth(picture); picHeight = getHeight(picture)
<list of pixels> = getPixels(picture)
<some Pixel> = getPixel(picture,x,y)
setRed(pix, <n>)
setGreen(pix, <n>)
setBlue(pix, <n>)
# 0 <= n <= 255, ‘red’ == 255
# 0 <= n <= 255, ‘red’ == 0
# 0 <= n <= 255, ‘red’ == 0
redValue = getRed(pix); greenValue = getGreen(pix); blueValue = getBlue(pix)
x = getX(pix); y = getY(pix)
savePicture(picture,”anotherfile.jpg”) # save to a .jpg file
Intro to Robots
Robot Vision:
• Without the Fluke board, Scribbler can not take pictures.
• However we can explore how we would process
photographs taken by Scribbler and to use results of that
analysis.
Intro to Robots
Making your own Picture:
newPic = makePicture(100,100)
show(newPic)
black by default
onePixel = getPixel(newPic,10,5)
print getRed(onePixel) #initial red value
setRed(onePixel,255)
print getRed(onePixel) #new red value
show(newPic)
Intro to Robots
.
Making your own Picture:
for yValue in range(0, getHeight(newPic)):
aPixel = getPixel(newPic, 10, yValue)
setRed(aPixel,255)
setGreen(aPixel,0)
setBlue(aPixel,0)
show(newPic)
# or make it into a function
def drawVerticalRedLine( picture, xPos ):
for yPos in range(0, getHeight(picture) ):
aPixel = getPixel( picture, xPos, yPos)
setRed(aPixel,255)
setGreen(aPixel,0)
setBlue(aPixel,0)
drawVerticalRedLine(newPic,10)
show(newPic)
Intro to Robots
height of picture
Real Photograph
• Photo taken by Scribbler
applePic = takePicture()
OR
applePic = loadPicture(‘apple.jpg’)
drawVerticalRedLine(applePic,10)
show(applePic)
Intro to Robots
Scan Real Photograph
• If we draw the vertical red line continuously across the
photo we will cover up the photo.
run
• We would rather have the following behaviour.
• What we are doing is:
– drawing a red line,
– waiting a fraction of a second,
– putting back the original pixels and
– drawing the line again further on.
Intro to Robots
Robot/Program Vision:
• The robot and our program don’t see purple, they each
see a combination of red, green and blue.
r,g,b = getColors(pixel)
• Colors with low values of red, green and blue are
generally dark and with high values, generally light.
run
width=height=500
myCanvas = GraphWin("Circles", width, height)
myCanvas.setBackground("white")
for i in range(1,25):
c = Circle(Point(randrange(0,width),
randrange(0,height)),20)
red = 10*i
c.setFill(color_rgb(red, red/2, red/4))
c.draw(myCanvas)
wait(1)
wait(5)
myCanvas.close()
Intro to Robots
Filtering Photographs:
• Have you ever heard the expression, “The world is not
black and white, but various shades of gray”.
• Some times seeing things as black and white helps
filter out other irrelevant or confusing messages.
• Go back to our original picture
and ask, “What parts of the
photo are brightest?”
Intro to Robots
Brightness Filter:
• We know that higher red, green and blue values mean
brighter object.
• Brightness: average color value > 175
myPicture = takePicture()
show(myPicture)
for pixel in getPixels(myPicture):
rValue = getRed(pixel)
gValue = getGreen(pixel)
bValue = getBlue(pixel)
avgValue = ( rValue + gValue + bValue) / 3.0
if avgValue > 175 :
#Turn the pixel white
setRed(pixel,255);setGreen(pixel,255)
setBlue(pixel,255)
else:
#Otherwise, turn it black.
setRed(pixel,0);setGreen(pixel,0)
setBlue(pixel,0)
show(myPicture)
Intro to Robots
before
after
Redness Filter:
• Suppose you want to find the reddest thing in a photo
(and if the photo was taken by the robot, have the robot
turn towards that thing).
• In the previous example most of the filtered apple was
black, why?
• Redness:
– high red value (> 175),
– low green and blue values (< 175).
Intro to Robots
Redness Filter:
• Program for highlighting red
myPicture = takePicture()
show(myPicture)
def filterRed(picture):
for pixel in getPixels(picture):
rValue = getRed(pixel)
if rValue > 175:
#Turn the pixel white
setRed(pixel,255)
setGreen(pixel,255)
setBlue(pixel,255)
else:
#Otherwise, turn it black.
setRed(pixel,0)
setGreen(pixel,0)
setBlue(pixel,0)
filterRed(myPicture)
show(myPicture)
an awful lot of red
Intro to Robots
Greenness Filter:
• Same program for green
myPicture = takePicture()
show(myPicture)
for pixel in getPixels(myPicture):
gValue = getGreen(pixel)
if gValue > 175:
#Turn the pixel white
setRed(pixel,255)
setGreen(pixel,255)
setBlue(pixel,255)
else:
#Otherwise, turn it black.
setRed(pixel,0)
setGreen(pixel,0)
setBlue(pixel,0)
show(myPicture)
funny how a green filter makes a red apple stand out
Intro to Robots
Primarily Red Filter:
myPicture = takePicture()
show(myPicture)
Def findRedAreas(picture):
for pixel in getPixels(myPicture):
rValue = getRed(pixel)
gValue = getGreen(pixel)
bValue = getBlue(pixel)
if rValue > 175 and
gValue < 175 and bValue < 175:
#Turn the pixel white
setRed(pixel,255)
setGreen(pixel,255)
setBlue(pixel,255)
else:
#Otherwise, turn it black.
setRed(pixel,0)
setGreen(pixel,0)
setBlue(pixel,0)
show(myPicture)
Intro to Robots
more like it; just two red
“hot spots”
Find the Center of Redness:
• Simple Technique: Find the average x-coordinate of all
white pixels in the filtered picture.
• This will point to the center of redness, which may
unfortunately be somewhere in between two major red
(now white) areas
average x-coordinate
Intro to Robots
Center of Redness of our Photography
def AverageXofWhitePixels(picture):
sumX = 0.0 # We use floating point values.
counter = 0.0 # We use floating point values.
for xPos in range(0, getWidth(picture) ):
for yPos in range(0, getHeight(picture) ):
pixel = getPixel(picture,xPos,yPos)
value = getGreen(pixel)
if value > 0 :
# add x-coord of every white pixel
sumX = sumX + xPos
counter = counter + 1
averageX = sumX / counter
return int(averageX) #Return an Integer
myPicture = takePicture()
picture = copyPicture(myPicture)
picture = findRedAreas(picture)
XposAvg = AverageXofWhitePixels(picture)
drawVerticalRedLine(myPicture,XposAvg)
show(myPicture)
Intro to Robots
Vertical line to one side because
redness of the apple shadow is
pulling the red center to the right.
Processing Red Areas Individually:
• Call each red area “a blob”.
• Our problem is:
– find a blob
– count its pixels
– calculate its x-coordinate center
• Easiest way to traverse the picture is
for pix in getPixels(picCopy):
...
two blobs
one blob
Intro to Robots
Red Blobs one-at-a-time:
• Problem: Processing the pixels left-to-right, top-tobottom we won’t know when we process the line
indicated by the red arrow if the blobs are two or one.
• Solution: Keep a list of partially-built blobs and as we
find that two blobs are really one, combine them into
one.
no
processing
continue
longer processing
on
aanew
same
blob
blob
processing
newblob;
blob
or is it the same blob?
two blobs
one blob
Intro to Robots
BlobList Data Structure:
• A BlobList is a data structure that contains a list of blobs
and has many methods for accessing that list.
from blob import *
class BlobList:
def __init__(self):
self.contents = [ ]
def add(self,b):
self.contents.append(b)
def findBlob(self,c):
for b in self.contents:
if b.inBlob(c):
return b
return None
def getIndex(self,b):
for i in range(len(self.contents)):
if b == self.contents[i]:
return i
return -1
Intro to Robots
def dropBlob(self, b):
i = self.getIndex(b)
del self.contents[i]
def writeOut(self):
for b in self.contents:
b.writeOut()
print ""
def getBiggestBlob(self):
maxNdx = -1
maxSz = 0
for i in range(len(self.contents)):
b = self.contents[i]
size = b.getSize()
if size > maxSz:
maxSz = size
maxNdx = i
return self.contents[maxNdx]
Blob Data Structure:
• A Blob is a contiguous red section of a filtered photo. All
pixels are connected either vertically or horizontally.
• A blob is built by finding a red point, not already part of
another blob and adding it and all the connected red
points on the same vertical line to the same blob.
x
As the point, x, is added
to a new blob, we continue
to add all the red pixels
on the same vertical line as
the point, x, and connected
to x
.
Intro to Robots
Complicated Blobs
x
.
these points are part of the
initial blob containing x
A
these points are not part of
the initial blob containing x
because although on the same
vertical line, the two lines are
not connected by red pixels
once the line labeled A is part of the
same blob as x, the extension of the
vertical line containing x will also
become part of the same blob
eventually the entire donut will be
a single blob
Intro to Robots
total blob
in picture
How to Build a Blob:
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1st
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
2.nd .
. .
. .
. .
. .
. .
. .
. .
. .
. .
. .
. .
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Which is the last pixel to be added to the program blob?
Intro to Robots
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
pixels added
to program
blob
Blob Strategy
• Each time we add a pixel to a blob we give it a color that
is not red and not black and not used in any other blob.
combined blob
previous blob new blob
select a red pixel
create a new empty blob with new color
while pixel is red:
add pixel to current blob (change color)
if pixel to the left is already in another blob:
combine the two blobs
if pixel to the right is already in another blob:
combine the two blobs
get the pixel immediately above the previous pixel
select the pixel immediately below the original red pixel
while pixel is red:
add pixel to current blob (change color)
if pixel to the left is already in another blob:
combine the two blobs
if pixel to the right is already in another blob:
combine the two blobs
get the pixel immediately below the previous pixel
Intro to Robots
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Blob Code:
.....
.....
.....
.....
from myro import *
class Blob:
xCoordinates
12 13 14
vertcialCounts = {12:2, 13:3, 14:1}
def __init__(self, c=None,x=None):
self.colors = [ ]; self.blobPixelSum = 0
### blob consists of a list of colors in use
self.verticalCounts = { }; self.midPoint = -1 ### and a dictionary of vertical counts
if c != None:
### key = xCoord; value = number yPixels
self.colors.append(c)
if x != None:
self.verticalCounts[x] = 0
def incPixelCount(self,xCoordinate,cnt):
self.blobPixelSum = self.blobPixelSum + cnt
if self.verticalCounts.has_key(xCoordinate): ### xCoord already belongs to blob
self.verticalCounts[xCoordinate] =
### handles the donut case
self.verticalCounts[xCoordinate] + cnt
else:
self.verticalCounts[xCoordinate] = cnt
def inBlob(self,c):
return c in self.colors ### tests if a color is already in the blob; if the same color
### is found in two blobs, they are the same blob
Intro to Robots
Blob Code:
def mergeDictionaries(self,b):
newD = { }
d1 = self.verticalCounts ### {x1:c1, x2:c2, …, xn:cn}
d2 = b.verticalCounts
for d in d1.keys():
newD[d] = d1[d]
if d2.has_key(d):
newD[d] = newD[d] + d2[d] ### add vertical counts if xCoord in both blobs
del d2[d]
for d in d2.keys(): ### get the remaining xCoords not gotten in first loop
newD[d] = d2[d]
return newD
def combine(self,b):
newB = Blob()
newB.colors = self.colors + b.colors ### combine disjoint color lists
newB.blobPixelSum = self.blobPixelSum +
b.blobPixelSum
newB.verticalCounts = self.mergeDictionaries(b) ### combine vertical counts
return newB
Intro to Robots
Blob Code:
def calculateMidPoint(self):
sum = 0
for x in self.verticalCounts.keys(): ### same calculation as the single blob problem
sum = sum + x*self.verticalCounts[x]
self.midPoint = sum/self.blobPixelSum
def getMidPoint(self):
if self.midPoint == -1:
self.calculateMidPoint()
return self.midPoint
def getSize(self):
return self.blobPixelSum
def writeOut(self):
print self.colors
print "sum of pixels: ", self.blobPixelSum
self.calculateMidPoint()
print 'midPoint ' , self.midPoint
print 'color list', self.colors
Intro to Robots
Generate New Colors:
• Our filtered picture is all black (0,0,0) or red (255,0,0).
• We have 224 – 2 other colors to choose from.
• More than enough for the pictures we will process.
from myro import *
class AvailableColors:
def __init__(self):
self.g = 100; self.b = 0
generates blue-green colors
on the fly in the ranges:
100 <= green <= 255
0 <= blue <= 255
approximately 37500 different
colors
def getNext(self):
g = self.g; b = self.b
self.b = self.b + 1
if self.b > 255:
self.g = self.g+1
self.b = 0
return Color(0,g,b)
Intro to Robots
Alternate AvailableColors Implementation:
from myro import *
class AlternativeAvailableColors:
def __init__(self):
self.contents = [ ]
self.index = 0
for i in range (100,256):
for j in range(0,256):
self.contents.append(Color(0,i,j))
def getNext(self):
c = self.contents[self.index]
index = index + 1
return c
Intro to Robots
creates a complete lost of colors
up front and then returns these
colors one at a time as requested
Data/Code Comparisons:
• Both AvailableColors and AlternativeAvailableColors
have exactly the same interface:
ac.getNext()
• But their implementations are very different.
• AvailableColors has almost no data (variables g and b)
and a complicated algorithm for getNext().
• AlternativeAvailableColors has a complicated data
structure (self.contents, a list of all available colors) but a
very simple implementation of getNext().
• Lesson to Learn: There is a trade-off between data
structure and algorithm; if one is simple the other is
usually complex.
Intro to Robots
Main Program:
leftPixel
= ==
rightPixel
rightPixel
getPixel(pictureCopy,x-1,yNdx)
getPixel(pictureCopy,x-1,yNdx)
getPixel(pictureCopy,x+1,yNdx)
getPixel(pictureCopy,x+1,yNdx)
ac=
AvailableColors()
clr
yNdx
=yCount
y+1= ac.getNext()
= yCount
+ and
yNdx
ififwhile
getRed(leftPixel)
====
0==
0
if
getRed(rightPixel)
0and
and- y -1
bl
=getRed(rightPixel)
BlobList()
x =yNdx
getX(pix)
>=
0
and
picHeight
=
getHeight(pictureCopy)
yNdx
yNdx
==yNdx
yNdx
+
1
while
<- 1picHeight
and> >0:>0:0:
b.incPixelCount(x,yCount)
getGreen(leftPixel)
getGreen(leftPixel)
getGreen(rightPixel)
getGreen(rightPixel)
> 0:
y<=picHeight:
getY(pix)
getRed(pixel)
>
maxRed:
if pixel
yNdx
pixel
=
=
getPixel(pictureCopy,x,yNdx)
getPixel(pictureCopy,x,yNdx)
getRed(pixel)
>
maxRed:
bl.add(b)
leftColor
==getColor(leftPixel)
rightColor
=getColor(leftPixel)
rightColor
=getColor(rightPixel)
getColor(rightPixel)
for leftColor
pix
bin
= getPixels(pictureCopy):
Blob(clr,x)
setColor(pixel,clr)
setColor(pixel,clr)
pixel
=
getPixel(pictureCopy,x,yNdx)
if
if
not
not
b.inBlob(leftColor):
b.inBlob(leftColor):
ifpixel
notb.inBlob(rightColor):
b.inBlob(rightColor):
redValue
getRed(pix)
==pix
oldB
=
bl.findBlob(rightColor)
oldB >=bl.findBlob(leftColor)
bl.findBlob(rightColor)
if redValue
maxRed:
bb==b.combine(oldB)
b.combine(oldB)
bl.dropBlob(oldB)
bl.dropBlob(oldB)
for each pixel
if pixel is red:
create a new empty blob and new color
while pixel is red:
add pixel to current blob (change color)
if pixel to the left is already in another blob:
combine the two blobs
if pixel to the right is already in another blob:
combine the two blobs
get the pixel immediately above the previous pixel
select the pixel immediately below the original red pixel
while pixel is red:
add pixel to current blob (change color)
if pixel to the left is already in another blob:
combine the two blobs
if pixel to the right is already in another blob:
combine the two blobs
get the pixel immediately below the previous pixel
add blob to blob list
Intro to Robots