Transcript ACM SIGCSE 2003: Multimedia Construction Projects Mark Guzdial College of Computing Georgia Institute of Technology [email protected] http://www.cc.gatech.edu/~mark.guzdial.

ACM SIGCSE 2003:
Multimedia
Construction Projects
Mark Guzdial
College of Computing
Georgia Institute of Technology
[email protected]
http://www.cc.gatech.edu/~mark.guzdial
Plan

7-7:15: Introduction
What’s on your CD
 Why media computation







7:15-7:45: Picture manipulations in Python
7:45-8:15: Sound manipulations in Python
8:15-8:30: Break
8:30-8:45: How to do this in Java and Squeak
8:45-9:45: You Play
9:45-10: Wrap-up
What’s on your CD

Materials from our Introduction to Media
Computation course





Jython Environment for Students (JES)
Book
Course slides
MediaTools: Squeak-based media exploration tools
Material for this workshop



Java and Squeak API slides
Java source code
Squeak computer music essays
Computer science is more
important than Calculus

In 1961, Alan Perlis argued
that computer science is more
important in a liberal
education than calculus



Explicitly, he argued that
all students should learn to
program.
Calculus is about rates, and
that’s important to many.
Computer science is about
process, which is important to
everyone
How close are we to being
able to teach everyone CS?

Not very


At many departments, CS retention rates are lower than the
rest of campus



CS1 is one of the most despised courses for non-majors
At Georgia Tech: 65% for 1995 cohort, vs. 73% for
Engineeering
Drop-out rates near 50% at many institutions
Female enrollment in CS has been dropping nationally
Why?

Several recent studies and books
claim that CS instruction tends to
dissuade anyone but white males

“Tedious,”
“taught without application
relevance,”
“boring,”
“lacking creativity,”
“asocial”
The potential of computing


How can we realize the potential impact of
computing if future professionals in other
disciplines despise computer science?
To realize that potential, we need practitioners in
other disciplines to understand computing.

At least some of those practitioners need to
understand it as a creative, exciting venture—as we
do!
The best uses for computing
technologies will come from others



Thomas Edison vs. D.W. Griffith
If we want computing technologies to become
useful, they have to get out of our hands.
It can’t be just through applications


That presumes that the current technologists
know how everyone else wants to do things and
should do things
Suggestion: D.W. Griffith knew things that Edison
didn’t.
An attempt at relevant, creative computing:
Introduction to Media Computation

A course for non-CS and non-Engineering majors


120 students this semester,
planning 400-600 in the Fall


International Affairs, Literature, Public Policy,
Architecture, Management, Biology, etc.
2/3 female in this semester’s CS1315
Focus: Learning programming within the context of
media manipulation and creation
Motivating the Computing



As professionals, these students will often the use
the computer as a communications medium.
All media are going digital,
and digital media are manipulated with software.
Knowing how to program, then, is a
communications skill.
Python as the programming
language


Huge and contentious issue
Use in commercial contexts legitimatizes the choice



Industrial Light & Magic, Google, Nextel, etc.
Minimal syntax
Looks like other programming languages
Potential for knowledge transfer for students
 “Executable pseudocode” for this workshop


Actually using Jython (http://www.jython.org) for Java
class libraries
JES - Jython Environment for
Students
Simple Python in JES
>>> print 34 + 56
90
>>> print 34.1/46.5
0.7333333333333334
>>> print 22 * 33
726
>>> print 14 - 15
-1
>>> print "Hello"
Hello
>>> print "Hello" + "Mark"
HelloMark
Command Area Editing


Up/down arrows walk through command history
You can edit the line at the bottom

Just put the cursor at the end of the line before
hitting Return/Enter.
Basic Picture Functions



makePicture(filename) creates and returns a
picture object, from the JPEG file at the filename
show(picture) displays a picture in a window
We’ll learn functions for manipulating pictures
later, like getColor, setColor, and repaint
Basic Sound Functions




makeSound(filename) creates and returns a sound
object, from the WAV file at the filename
play(sound) makes the sound play (but doesn’t
wait until it’s done)
blockingPlay(sound) waits for the sound to finish
We’ll learn more later like getSample and
setSample
Demonstrating simple JES
>>> print pickAFile()
/Users/guzdial/mediasources/barbara.jpg
>>> print makePicture(pickAFile())
Picture, filename /Users/guzdial/mediasources/barbara.jpg height 294 width 222
>>> print pickAFile()
/Users/guzdial/mediasources/hello.wav
>>> print makeSound(pickAFile())
Sound of length 54757
>>> print play(makeSound(pickAFile()))
None
>>> myfilename = pickAFile()
>>> print myfilename
/Users/guzdial/mediasources/barbara.jpg
>>> mypicture = makePicture(myfilename)
>>> print mypicture
Picture, filename /Users/guzdial/mediasources/barbara.jpg height 294 width 222
>>> show(mypicture)
Writing a recipe:
Making our own functions





To make a function, use the
command def
Then, the name of the function,
and the names of the input values
between parentheses (“(input1)”)
End the line with a colon (“:”)
The body of the recipe is indented
(Hint: Use two spaces)
Your function does NOT
exist for JES until you
load it
Functions for picking and
playing/showing files
def pickAndShow():
myfile = pickAFile()
mypict = makePicture(myfile)
show(mypict)
def pickAndPlay():
myfile = pickAFile()
mysound = makeSound(myfile)
play(mysound)
Functions that take input
def playNamed(myfile):
mysound = makeSound(myfile)
play(mysound)
def showNamed(myfile):
mypict = makePicture(myfile)
show(mypict)
CS1315:
Introduction to
Media Computation
Picture encoding and manipulation
Pictures and Pixels

A picture is a matrix of pixels


A picture has two dimensions: Width and Height
Pixels are picture elements
pixel object knows its color using
RGB encoding
 It also knows where it is in its picture
 Each
RGB

In RGB, each color has three
component colors:





Amount of redness
Amount of greenness
Amount of blueness
Each does appear as a separate dot
on most devices, but our eye
blends them.
In most computer-based models
of RGB, a single byte (8 bits) is
used for each

So a complete RGB color is 24
bits, 8 bits of each
Encoding RGB


Each component color (red,
green, and blue) is encoded as
a single byte
Colors go from (0,0,0) to
(255,255,255)

If all three components are
the same, the color is in
greyscale

(50,50,50) at (2,2)
(0,0,0) (at position (1,2) in
example) is black
 (255,255,255) is white

Manipulating pixels
getPixel(picture,x,y) gets a single pixel.
getPixels(picture) gets all of them in an array.
(Square brackets is a standard array reference
notation—which we’ll generally not use.)
>>> pixel=getPixel(picture,1,1)
>>> print pixel
Pixel, color=color r=168 g=131 b=105
>>> pixels=getPixels(picture)
>>> print pixels[0]
Pixel, color=color r=168 g=131 b=105
What can we do with a
pixel?
• getRed, getGreen, and getBlue are
functions that take a pixel as input and
return a value between 0 and 255
• setRed, setGreen, and setBlue are
functions that take a pixel as input and
a value between 0 and 255
We can also get, set, and
make Colors





getColor takes a pixel as input and returns a Color object with the
color at that pixel
setColor takes a pixel as input and a Color, then sets the pixel to that
color
makeColor takes red, green, and blue values (in that order) between
0 and 255, and returns a Color object
pickAColor lets you use a color chooser and returns the chosen color
We also have functions that can makeLighter and makeDarker an
input color
We can change pixels directly…
>>> file="/Users/guzdial/mediasources/barbara.jpg"
>>> pict=makePicture(file)
>>> show(pict)
>>> setColor(getPixel(pict,10,100),yellow)
>>> setColor(getPixel(pict,11,100),yellow)
>>> setColor(getPixel(pict,12,100),yellow)
>>> setColor(getPixel(pict,13,100),yellow)
>>> repaint(pict)
But that’s really dull and boring…
That’s the subject of the next lecture
If you make something you like…


writePictureTo(picture,”filename”)
Writes the picture out as a JPEG


Be sure to end your filename as “.jpg”!
If you don’t specify a full path,
will be saved in the same directory as JES.
How do you find out what RGB
values you have? And where?

Use the MediaTools!
Use a loop!
Our first picture recipe
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
Used like this:
>>> file="/Users/guzdial/mediasources/barbara.jpg"
>>> picture=makePicture(file)
>>> show(picture)
>>> decreaseRed(picture)
>>> repaint(picture)
getPixels returns a sequence of
pixels



Each pixel knows its color and its original picture
Change the pixel, you change the picture
So the loop below assigns the index variable p to
each pixel in the picture picture, one at a time.
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
Let’s walk that through slowly…
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
Here we get a picture object
in as input and call it
picture
picture
Now, get the pixels
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
Pixel,
color
r=168
g=131
b=105
p
Pixel,
color
r=160
g=131
b=105
Pixel,
color
r=168
g=132
b=106
We get all the pixels from
the picture, then make p be
the name of each one one at
a time
picture
getPixels()
…
Get the red value from pixel
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
We get the red value of
pixel p and name it value
picture
Pixel,
color
r=168
g=131
b=105
p
Pixel,
color
r=160
g=131
b=105
Pixel,
color
r=168
g=132
b=106
value = 168
…
Now change the pixel
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
Pixel,
color
r=84
g=131
b=105
p
Pixel,
color
r=160
g=131
b=105
Pixel,
color
r=168
g=132
b=106
value = 168
Set the red value of pixel p
to 0.5 (50%) of value
picture
…
Then move on to the next pixel
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
Move on to the next pixel
and name it p
picture
Pixel,
color
r=84
g=131
b=105
Pixel,
color
r=160
g=131
b=105
p
Pixel,
color
r=168
g=132
b=106
value = 168
…
Get its red value
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
Change value to the new
red value at the new p
picture
Pixel,
color
r=84
g=131
b=105
Pixel,
color
r=168
r=160
g=131
b=105
p
Pixel,
color
r=168
g=132
b=106
value = 160
…
And change this red value
def decreaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*0.5)
Pixel,
color
r=84
g=131
b=105
Pixel,
color
r=80
g=131
b=105
p
Pixel,
color
r=168
g=132
b=106
value = 160
Change the red value at
pixel p to 50% of value
picture
…
And eventually, we do all pixels

We go from this…
to this!
Increasing Red
def increaseRed(picture):
for p in getPixels(picture):
value=getRed(p)
setRed(p,value*1.2)
What happened here?!?
Remember that the limit
for redness is 255.
If you go beyond 255, all
kinds of weird things can
happen: Wrap around
Clearing Blue
def clearBlue(picture):
for p in getPixels(picture):
setBlue(p,0)
Combining into a sunset function


How do we turn this beach
scene into a sunset?
What happens at sunset?

At first, I tried increasing
the red, but that made
things like red specks in
the sand REALLY
prominent.


That can’t be how it really
works
New Theory: As the sun
sets, less blue and green is
visible, which makes things
look more red.
A Sunset-generation Function
def makeSunset(picture):
for p in getPixels(picture):
value=getBlue(p)
setBlue(p,value*0.7)
value=getGreen(p)
setGreen(p,value*0.7)
Creating a negative

Let’s think it through


R,G,B go from 0 to 255
Let’s say Red is 10. That’s very light red.



What’s the opposite? LOTS of Red!
The negative of that would be 245: 255-10
So, for each pixel, if we negate each color
component in creating a new color, we negate the
whole picture.
Recipe for creating a negative
def negative(picture):
for px in getPixels(picture):
red=getRed(px)
green=getGreen(px)
blue=getBlue(px)
negColor=makeColor( 255-red, 255-green, 255-blue)
setColor(px,negColor)
Converting to greyscale

We know that if red=green=blue, we get grey



But what value do we set all three to?
What we need is a value representing the darkness of the
color, the luminance
There are lots of ways of getting it, but one way that
works reasonably well is dirt simple—simply take the
average:
Converting to greyscale
def greyScale(picture):
for p in getPixels(picture):
intensity = (getRed(p)+getGreen(p)+getBlue(p))/3
setColor(p,makeColor(intensity,intensity,intensity))
But that’s not really the best
greyscale

In reality, we don’t perceive red, green, and blue
as equal in their amount of luminance: How
bright (or non-bright) something is.



We tend to see blue as “darker” and red as
“brighter”
Even if, physically, the same amount of light is
coming off of each
Photoshop’s greyscale is very nice: Very similar
to the way that our eye sees it

B&W TV’s are also pretty good
Building a better greyscale

We’ll weight red, green, and blue based on how
light we perceive them to be, based on laboratory
experiments.
def greyScaleNew(picture):
for px in getPixels(picture):
newRed = getRed(px) * 0.299
newGreen = getGreen(px) * 0.587
newBlue = getBlue(px) * 0.114
luminance = newRed+newGreen+newBlue
setColor(px,makeColor(luminance,luminance,luminance))
Comparing the two greyscales:
Average on left, weighted on right
Let’s try making Barbara a redhead!

We could just try increasing the redness, but as
we’ve seen, that has problems.



Overriding some red spots
And that’s more than just her hair
If only we could increase the redness only of the
brown areas of Barb’s head…
Making Barb a redhead
def turnRed():
brown = makeColor(57,16,8)
file = r"C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\barbara.jpg"
picture=makePicture(file)
for px in getPixels(picture):
color = getColor(px)
if distance(color,brown)<50.0:
redness=getRed(px)*1.5
setRed(px,redness)
show(picture)
return(picture)
Original:
Tuning our color replacement

If you want to get more of Barb’s hair, just
increasing the threshold doesn’t work


Wood behind becomes within the threshold value
How could we do it better?


Lower our threshold, but then miss some of the hair
Work only within a range…
Introducing the function range

Range returns a sequence between its first two
inputs, possibly using a third input as the
increment
>>> print range(1,4)
[1, 2, 3]
>>> print range(-1,3)
[-1, 0, 1, 2]
>>> print range(1,10,2)
[1, 3, 5, 7, 9]
Replacing colors
in a range
Get the range
using
MediaTools
def turnRedInRange():
brown = makeColor(57,16,8)
file=r"C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\barbara.jpg"
picture=makePicture(file)
for x in range(70,168):
for y in range(56,190):
px=getPixel(picture,x,y)
color = getColor(px)
if distance(color,brown)<50.0:
redness=getRed(px)*1.5
setRed(px,redness)
show(picture)
return(picture)
Mirroring



Imagine a mirror horizontally across the picture,
or vertically
What would we see?
How do generate that digitally?

We simply copy the colors of pixels from one place
to another
Mirroring a picture


Slicing a picture down the middle and sticking a mirror on the slice
Do it by using a loop to measure a difference


The index variable is actually measuring distance from the
mirrorpoint
Then reference to either side of the mirror point using the difference
Recipe for mirroring
def mirrorVertical(source):
mirrorpoint = int(getWidth(source)/2)
for y in range(1,getHeight(source)):
for x in range(1,mirrorpoint):
p = getPixel(source, x+mirrorpoint,y)
p2 = getPixel(source, mirrorpoint-x,y)
c = getColor(p2)
setColor(p,c)
Can we do it with a horizontal
mirror?
def mirrorHorizontal(source):
mirrorpoint = int(getHeight(source)/2)
for y in range(1,mirrorpoint):
for x in range(1,getWidth(source)):
p = getPixel(source,x,y+mirrorpoint)
p2 = getPixel(source,x,mirrorpoint-y)
setColor(p,getColor(p2))
Of course!
What if we wanted to copy bottom
to top?

Very simple: Swap the p and p2 in the bottom line

Copy from p to p2, instead of from p2 to p
def mirrorHorizontal(source):
mirrorpoint = int(getHeight(source)/2)
for y in range(1,mirrorpoint):
for x in range(1,getWidth(source)):
p = getPixel(source,x,y+mirrorpoint)
p2 = getPixel(source,x,mirrorpoint-y)
setColor(p2,getColor(p))
Messing with Santa some more
Copying pixels

In general, what we want to do is to keep track of
a sourceX and sourceY, and a targetX and targetY.

We increment (add to them) in pairs
sourceX and targetX get incremented together
 sourceY and targetY get incremented together


The tricky parts are:
Setting values inside the body of loops
 Incrementing at the bottom of loops

Copying Barb to a canvas
def copyBarb():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
canvasf = getMediaPath("7inX95in.jpg")
canvas = makePicture(canvasf)
# Now, do the actual copying
targetX = 1
for sourceX in range(1,getWidth(barb)):
targetY = 1
for sourceY in range(1,getHeight(barb)):
color = getColor(getPixel(barb,sourceX,sourceY))
setColor(getPixel(canvas,targetX,targetY), color)
targetY = targetY + 1
targetX = targetX + 1
show(barb)
show(canvas)
return canvas
Transformation =
Small changes in copying

Making relatively small changes in this basic
copying program can make a variety of
transformations.




Change the targetX and targetY, and you copy
wherever you want
Cropping: Change the sourceX and sourceY range,
and you copy only part of the program.
Rotating: Swap targetX and targetY, and you end up
copying sideways
Scaling: Change the increment on sourceX and
sourceY, and you either grow or shrink the image.
Copying into the middle of the
canvas
def copyBarbMidway():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
canvasf = getMediaPath("7inX95in.jpg")
canvas = makePicture(canvasf)
# Now, do the actual copying
targetX = 100
for sourceX in range(1,getWidth(barb)):
targetY = 100
for sourceY in range(1,getHeight(barb)):
color = getColor(getPixel(barb,sourceX,sourceY))
setColor(getPixel(canvas,targetX,targetY), color)
targetY = targetY + 1
targetX = targetX + 1
show(barb)
show(canvas)
return canvas
Rotating the copy
def copyBarbSideways():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
canvasf = getMediaPath("7inX95in.jpg")
canvas = makePicture(canvasf)
# Now, do the actual copying
targetX = 1
for sourceX in range(1,getWidth(barb)):
targetY = 1
for sourceY in range(1,getHeight(barb)):
color = getColor(getPixel(barb,sourceX,sourceY))
setColor(getPixel(canvas,targetY,targetX), color)
targetY = targetY + 1
targetX = targetX + 1
show(barb)
show(canvas)
return canvas
Cropping: Just the face
def copyBarbsFace():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
canvasf = getMediaPath("7inX95in.jpg")
canvas = makePicture(canvasf)
# Now, do the actual copying
targetX = 100
for sourceX in range(45,200):
targetY = 100
for sourceY in range(25,200):
color = getColor(getPixel(barb,sourceX,sourceY))
setColor(getPixel(canvas,targetX,targetY), color)
targetY = targetY + 1
targetX = targetX + 1
show(barb)
show(canvas)
return canvas
Scaling the picture down
def copyBarbsFaceSmaller():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
canvasf = getMediaPath("7inX95in.jpg")
canvas = makePicture(canvasf)
# Now, do the actual copying
sourceX = 45
for targetX in range(100,100+((200-45)/2)):
sourceY = 25
for targetY in range(100,100+((200-25)/2)):
color = getColor(getPixel(barb,sourceX,sourceY))
setColor(getPixel(canvas,targetX,targetY), color)
sourceY = sourceY + 2
sourceX = sourceX + 2
show(barb)
show(canvas)
return canvas
Scaling the picture up
def copyBarbsFaceLarger():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
canvasf = getMediaPath("7inX95in.jpg")
canvas = makePicture(canvasf)
# Now, do the actual copying
sourceX = 45
for targetX in range(100,100+((200-45)*2)):
sourceY = 25
for targetY in range(100,100+((200-25)*2)):
color = getColor(getPixel(barb,int(sourceX),int(sourceY)))
setColor(getPixel(canvas,targetX,targetY), color)
sourceY = sourceY + 0.5
sourceX = sourceX + 0.5
show(barb)
show(canvas)
return canvas
Background subtraction



Let’s say that you have a picture of someone, and
a picture of the same place (same background)
without the someone there, could you subtract out
the background and leave the picture of the
person?
Maybe even change the background?
Let’s take that as our problem!
Person (Katie) and Background
Background Subtraction Code
def swapbg(person, bg, newbg):
for x in range(1,getWidth(person)):
for y in range(1,getHeight(person)):
personPixel = getPixel(person,x,y)
bgpx = getPixel(bg,x,y)
personColor= getColor(personPixel)
bgColor = getColor(bgpx)
if distance(personColor,bgColor) < 10:
bgcolor = getColor(getPixel(newbg,x,y))
setColor(personPixel, bgcolor)
Putting Katie in a Jungle
But why isn’t it alot better?

We’ve got places where
we got pixels swapped
that we didn’t want to
swap


See Katie’s shirt stripes
We’ve got places where
we want pixels swapped,
but didn’t get them
swapped

See where Katie made a
shadow
Another way: Chromakey

Have a background of a known
color
Some color that won’t be
on the person you want to
mask out
 Pure green or pure blue is
most often used
 I used my son’s blue
bedsheet


This is how the weather people
seem to be in front of a map—
they’re actually in front of a
blue sheet.
Chromakey recipe
def chromakey(source,bg):
# source should have something in front of blue, bg is the new background
for x in range(1,getWidth(source)):
for y in range(1,getHeight(source)):
p = getPixel(source,x,y)
# My definition of blue: If the redness + greenness < blueness
if (getRed(p) + getGreen(p) < getBlue(p)):
#Then, grab the color at the same spot from the new background
setColor(p,getColor(getPixel(bg,x,y)))
Can also do this with getPixels()
def chromakey2(source,bg):
# source should have something in front of blue, bg is the new background
for p in getPixels(source):
# My definition of blue: If the redness + greenness < blueness
if (getRed(p) + getGreen(p) < getBlue(p)):
#Then, grab the color at the same spot from the new background
setColor(p,getColor(getPixel(bg,getX(p),getY(p))))
Example results
Just trying the obvious thing for
Red
def chromakey2(source,bg):
# source should have something in front of red, bg is the new
background
for p in getPixels(source):
if getRed(p) > (getGreen(p) + getBlue(p)):
#Then, grab the color at the same spot from the new background
setColor(p,getColor(getPixel(bg,getX(p),getY(p))))
Doesn’t always work as you expect
Let’s try that with green
def chromakeyGreen(source,bg):
# source should have something in front of green, bg is the new background
for x in range(1,getWidth(source)):
for y in range(1,getHeight(source)):
p = getPixel(source,x,y)
# My definition of green: If the greenness > redness + blueness
if getGreen(p) > getBlue(p) + getRed(p):
#Then, grab the color at the same spot from the new background
setColor(p,getColor(getPixel(bg,x,y)))
The same definition of green
doesn’t work
Tweaking Chromakey
def chromakeyGreen(source,bg):
# source should have something in front of green, bg is the new background
for x in range(1,getWidth(source)):
for y in range(1,getHeight(source)):
p = getPixel(source,x,y)
# My definition of green: If the greenness > redness and blueness
if getGreen(p) > getBlue(p) and getGreen(p) > getRed(p):
#Then, grab the color at the same spot from the new background
setColor(p,getColor(getPixel(bg,x,y)))
That looks better
New functions for drawing on
pictures




addText(pict,x,y,string) puts the string starting at
position (x,y) in the picture
addLine(picture,x1,y1,x2,y2) draws a line from position
(x1,y1) to (x2,y2)
addRect(pict,x1,y1,w,h) draws a black rectangle
(unfilled) with the upper left hand corner of (x1,y1) and a
width of w and height of h
addRectFilled(pict,x1,y1,w,h,color) draws a rectangle
filled with the color of your choice with the upper left
hand corner of (x1,y1) and a width of w and height of h
Making it easier to deal with
the media folder
>>> setMediaFolder()
New media folder: /Users/guzdial/mediasources/
>>> print getMediaPath("barbara.jpg")
/Users/guzdial/mediasources/barbara.jpg
>>> print getMediaPath("sec1silence.wav")
/Users/guzdial/mediasources/sec1silence.wav
Example picture
def littlepicture():
canvas=makePicture(getMediaPath("640x480.jpg"))
addText(canvas,10,50,"This is not a picture")
addLine(canvas,10,20,300,50)
addRectFilled(canvas,0,200,300,500,yellow)
addRect(canvas,10,210,290,490)
return canvas
CS1315:
Introduction to
Media Computation
Sound Encoding and Manipulation
How sound works:
Acoustics, the physics of sound

Sounds are waves of air
pressure
Sound comes in cycles
 The frequency of a wave is
the number of cycles per
second (cps), or Hertz



(Complex sounds have more
than one frequency in them.)
The amplitude is the
maximum height of the
wave
Volume and pitch:
Psychoacoustics, the psychology of
sound

Our perception of volume is related (logarithmically) to
changes in amplitude


If the amplitude doubles, it’s about a 3 decibel (dB)
change
Our perception of pitch is related (logarithmically) to
changes in frequency
Higher frequencies are perceived as higher pitches
 We can hear between 5 Hz and 20,000 Hz (20 kHz)
 A above middle C is 440 Hz

“Logarithmically?”

It’s strange, but our hearing works on ratios not
differences, e.g., for pitch.



We hear the difference between 200 Hz and 400 Hz,
as the same as 500 Hz and 1000 Hz
Similarly, 200 Hz to 600 Hz, and 1000 Hz to 3000 Hz
Intensity (volume) is measured as watts per meter
squared

A change from 0.1W/m2 to 0.01 W/m2, sounds the
same to us as 0.001W/m2 to 0.0001W/m2
Decibel is a logarithmic
measure

A decibel is a ratio between two intensities: 10 *
log10(I1/I2)




As an absolute measure, it’s in comparison to
threshold of audibility
0 dB can’t be heard.
Normal speech is 60 dB.
A shout is about 80 dB
Digitizing Sound: How do we
get that into numbers?


Remember in calculus,
estimating the curve by
creating rectangles?
We can do the same to
estimate the sound curve
Analog-to-digital conversion
(ADC) will give us the
amplitude at an instant as a
number: a sample
 How many samples do we
need?

Nyquist Theorem


We need twice as many samples as the maximum
frequency in order to represent (and recreate, later) the
original sound.
The number of samples recorded per second is the
sampling rate

If we capture 8000 samples per second, the highest
frequency we can capture is 4000 Hz


That’s how phones work
If we capture more than 44,000 samples per second, we
capture everything that we can hear (max 22,000 Hz)

CD quality is 44,100 samples per second
Digitizing sound in the computer


Each sample is stored as a number (two bytes)
What’s the range of available combinations?
16 bits, 216 = 65,536
 But we want both positive and negative values


To indicate compressions and rarefactions.
What if we use one bit to indicate positive (0) or negative
(1)?
 That leaves us with 15 bits
 15 bits, 215 = 32,768
 One of those combinations will stand for zero



We’ll use a “positive” one, so that’s one less pattern for positives
Each sample can be between -32,768 and 32,767
Working with sounds

We’ll use pickAFile and makeSound as we have before.





But now we want .wav files
We’ll use getSamples to get all the sample objects out of
a sound
We can also get the value at any index with
getSampleValueAt
Sounds also know their length (getLength) and their
sampling rate (getSamplingRate)
Can save sounds with writeSoundTo(sound,”file.wav”)
Demonstrating Working with
Sound in JES
>>> filename=pickAFile()
>>> print filename
/Users/guzdial/mediasources/preamble.wav
>>> sound=makeSound(filename)
>>> print sound
Sound of length 421109
>>> samples=getSamples(sound)
>>> print samples
Samples, length 421109
>>> print getSampleValueAt(sound,1)
36
>>> print getSampleValueAt(sound,2)
29
Demonstrating working with
samples
>>> print getLength(sound)
220568
>>> print getSamplingRate(sound)
22050.0
>>> print getSampleValueAt(sound,220568)
68
>>> print getSampleValueAt(sound,220570)
I wasn't able to do what you wanted.
The error java.lang.ArrayIndexOutOfBoundsException has occured
Please check line 0 of
>>> print getSampleValueAt(sound,1)
36
>>> setSampleValueAt(sound,1,12)
>>> print getSampleValueAt(sound,1)
12
Working with Samples



We can get sample objects out of a sound with
getSamples(sound) or
getSampleObjectAt(sound,index)
A sample object remembers its sound, so if you
change the sample object, the sound gets changed.
Sample objects understand getSample(sample)
and setSample(sample,value)
Example: Manipulating Samples
>>> soundfile=pickAFile()
>>> sound=makeSound(soundfile)
>>> sample=getSampleObjectAt(sound,1)
>>> print sample
Sample at 1 value at 59
>>> print sound
Sound of length 387573
>>> print getSound(sample)
Sound of length 387573
>>> print getSample(sample)
59
>>> setSample(sample,29)
>>> print getSample(sample)
29
Recipe to Increase the Volume
def increaseVolume(sound):
for sample in getSamples(sound):
value = getSample(sample)
setSample(sample,value * 2)
Using it:
>>> f="/Users/guzdial/mediasources/gettysburg10.wav"
>>> s=makeSound(f)
>>> increaseVolume(s)
>>> play(s)
>>> writeSoundTo(s,"/Users/guzdial/mediasources/louder-g10.wav")
Decreasing the volume
def decreaseVolume(sound):
for sample in getSamples(sound):
value = getSample(sample)
setSample(sample,value * 0.5)
This works just like
increaseVolume, but
we’re lowering each
sample by 50% instead of
doubling it.
Maximizing volume


How do we get maximal volume?
It’s a three-step process:


First, figure out the loudest sound (largest sample).
Next, figure out a multiplier needed to make that
sound fill the available space.
We want to solve for x where x * loudest = 32767
 So, x = 32767/loudest


Finally, multiply the multiplier times every sample
Maxing (normalizing) the sound
def normalize(sound):
largest = 0
for s in getSamples(sound):
largest = max(largest,getSample(s) )
multiplier = 32767.0 / largest
print "Largest sample value in original sound was", largest
print "Multiplier is", multiplier
for s in getSamples(sound):
louder = multiplier * getSample(s)
setSample(s,louder)
Could also do this with IF
def normalize(sound):
largest = 0
for s in getSamples(sound):
if getSample(s) > largest:
largest = getSample(s)
multiplier = 32767.0 / largest
print "Largest sample value in original sound was", largest
print "Multiplier is", multiplier
for s in getSamples(sound):
louder = multiplier * getSample(s)
setSample(s,louder)
Increasing volume by sample
index
def increaseVolumeByRange(sound):
for sampleIndex in range(1,getLength(sound)+1):
value = getSampleValueAt(sound,sampleIndex)
setSampleValueAt(sound,sampleIndex,value * 2)
This really is the same as:
def increaseVolume(sound):
for sample in getSamples(sound):
value = getSample(sample)
setSample(sample,value * 2)
Recipe to play a sound
backwards
def backwards(filename):
source = makeSound(filename)
target = makeSound(filename)
sourceIndex = getLength(source)
for targetIndex in range(1,getLength(target)+1):
sourceValue = getSampleValueAt(source,sourceIndex)
setSampleValueAt(target,targetIndex,sourceValue)
sourceIndex = sourceIndex - 1
return target
Note use of return for
returning the processed sound
Recipe for halving the
frequency of a sound
def half(filename):
source = makeSound(filename)
target = makeSound(filename)
This is how a
sampling synthesizer
works!
sourceIndex = 1
for targetIndex in range(1, getLength( target)+1):
setSampleValueAt( target, targetIndex,
getSampleValueAt( source, int(sourceIndex)))
sourceIndex = sourceIndex + 0.5
Here are the
play(target)
return target
pieces that
do it
Compare these two
def half(filename):
source = makeSound(filename)
target = makeSound(filename)
sourceIndex = 1
for targetIndex in range(1, getLength( target)+1):
setSampleValueAt( target, targetIndex,
getSampleValueAt( source,
int(sourceIndex)))
sourceIndex = sourceIndex + 0.5
play(target)
return target
def copyBarbsFaceLarger():
# Set up the source and target pictures
barbf=getMediaPath("barbara.jpg")
barb = makePicture(barbf)
canvasf = getMediaPath("7inX95in.jpg")
canvas = makePicture(canvasf)
# Now, do the actual copying
sourceX = 45
for targetX in range(100,100+((200-45)*2)):
sourceY = 25
for targetY in range(100,100+((200-25)*2)):
color = getColor(
getPixel(barb,int(sourceX),int(sourceY)))
setColor(getPixel(canvas,targetX,targetY), color)
sourceY = sourceY + 0.5
sourceX = sourceX + 0.5
show(barb)
show(canvas)
return canvas
Both of them are sampling

Both of them have three parts:


A start where objects are set up
A loop where samples or pixels are copied from one
place to another
To decrease the frequency or the size, we take each
sample/pixel twice
 In both cases, we do that by incrementing the index by
0.5 and taking the integer of the index


Finishing up and returning the result
Recipe to double the frequency
of a sound
Here’s the critical
piece: We skip
every other
sample in the
source!
def double(filename):
source = makeSound(filename)
target = makeSound(filename)
targetIndex = 1
for sourceIndex in range(1, getLength(source)+1, 2):
setSampleValueAt( target, targetIndex,
getSampleValueAt( source, sourceIndex))
targetIndex = targetIndex + 1
#Clear out the rest of the target sound -- it's only half full!
for secondHalf in range( getLength( target)/2, getLength( target)):
setSampleValueAt(target,targetIndex,0)
targetIndex = targetIndex + 1
play(target)
return target
What happens if we don’t “clear
out” the end?

Try it!
def double(filename):
source = makeSound(filename)
target = makeSound(filename)
targetIndex = 1
for sourceIndex in range(1, getLength(source)+1, 2):
setSampleValueAt( target, targetIndex, getSampleValueAt( source, sourceIndex))
targetIndex = targetIndex + 1
play(target)
return target
Splicing Sounds (page 81-85)


Splicing gets its name from literally cutting and
pasting pieces of magnetic tape together
Doing it digitally is easy, but not short



We find where the end points of words are
We copy the samples into the right places to make
the words come out as we want them
(We can also change the volume of the words as we
move them, to increase or decrease emphasis and
make it sound more natural.)
Finding the word end-points

Using MediaTools and
play before/after cursor,
can figure out the index
numbers where each word
ends
Now, it’s all about copying

We have to keep track of the source and target
indices.
targetIndex = Where-the-incoming-sound-should-start
for sourceIndex in range(startingPoint,endingPoint)
setSampleValueAt( target, targetIndex,
getSampleValueAt( source, sourceIndex))
targetIndex = targetIndex + 1
The Whole Splice
def splicePreamble():
file = "/Users/guzdial/mediasources/preamble10.wav"
source = makeSound(file)
target = makeSound(file) # This will be the newly spliced sound
targetIndex=17408
# targetIndex starts at just after "We the" in the new sound
for sourceIndex in range( 33414, 40052): # Where the word "United" is in the sound
setSampleValueAt(target, targetIndex, getSampleValueAt( source, sourceIndex))
targetIndex = targetIndex + 1
for sourceIndex in range(17408, 26726): # Where the word "People" is in the sound
setSampleValueAt(target, targetIndex, getSampleValueAt( source, sourceIndex))
targetIndex = targetIndex + 1
for index in range(1,1000):
#Stick some quiet space after that
setSampleValueAt(target, targetIndex,0)
targetIndex = targetIndex + 1
play(target)
#Let's hear and return the result
return target
What’s going on here?


First, set up a source and target.
Next, we copy “United” (samples 33414 to
40052) after “We the” (sample 17408)
That means that we end up at
17408+(40052-33414) =
17408+6638=24046
 Where does “People” start?


Next, we copy “People” (17408 to 26726)
immediately afterward.
Do we have to copy “of” to?
 Or is there a pause in there that we can
make use of?


Finally, we insert a little (1/441-th of a second)
of space – 0’s
What if we didn’t do that second
copy? Or the pause?
def spliceSimpler():
file = "C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\preamble10.wav"
source = makeSound(file)
target = makeSound(file) # This will be the newly spliced sound
targetIndex=17408
# targetIndex starts at just after "We the" in the new
sound
for sourceIndex in range( 33414, 40052): # Where the word "United" is in the
sound
setSampleValueAt( target, targetIndex, getSampleValueAt( source,
sourceIndex))
targetIndex = targetIndex + 1
play(target)
#Let's hear and return the result
return target
Can we generalize shifting a sound
into other frequencies?
This way does NOT work:
def shift(filename,factor):
source = makeSound(filename)
target = makeSound(filename)
sourceIndex = 1
for targetIndex in range(1, getLength( target)+1):
setSampleValueAt( target, targetIndex, getSampleValueAt( source, int(sourceIndex)))
sourceIndex = sourceIndex + factor
play(target)
return target
Watching it not work
It’ll work for shifting down, but not shifting up. Why?
>>> hello=pickAFile()
>>> print hello
/Users/guzdial/mediasources/hello.wav
>>> lowerhello=shift(hello,0.75)
>>> higherhello=shift(hello,1.5)
I wasn't able to do what you wanted.
The error java.lang.ArrayIndexOutOfBoundsException has occured
Please check line 7 of /Users/guzdial/shift-broken.py
We need to prevent going past
the end of the sound
def shift(filename,factor):
source = makeSound(filename)
target = makeSound(filename)
sourceIndex = 1
for targetIndex in range(1, getLength( target)+1):
setSampleValueAt( target, targetIndex,
getSampleValueAt( source, int(sourceIndex)))
sourceIndex = sourceIndex + factor
if sourceIndex > getLength(source):
sourceIndex = 1
play(target)
return target
Now we have the basics of a
sampling synthesizer
For a desired frequency f we want a sampling interval like
this:
Useful exercise: Build a shift function that takes a
frequency as input.
Modules: Moving towards
movies and animations


The OS module offers a number of powerful
capabilities for dealing with files, e.g., renaming
files, finding out when a file was last modified,
and so on.
We start accessing the OS module by typing:


import os
The function that knows about directories is
listdir(), used as os.listdir()

listdir takes a path to a directory as input.
Using os.listdir
>>> import os
>>> print getMediaPath("barbara.jpg")
C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\barbara.jpg
>>> print getMediaPath("pics")
Note: There is no file at C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics
C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics
>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My
Documents\mediasources\pics")
['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg',
'students7.jpg', 'students8.jpg']
Writing a program to title pictures


We’ll input a directory
We’ll use os.listdir() to get each filename in the
directory



We’ll open the file as a picture.
We’ll title it.
We’ll save it out as “titled-” and the filename.
A Working Titling Program
import os
def titleDirectory(dir):
for file in os.listdir(dir):
print "Processing:",dir+"//"+file
picture = makePicture(dir+"//"+file)
addText(picture,10,10,"This is from CS1315 Spring 2003")
writePictureTo(picture,dir+"//"+"titled-"+file)
Showing it work
>>> titleDirectory("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")
Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students1.jpg
Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students2.jpg
Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students5.jpg
Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students6.jpg
Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students7.jpg
Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students8.jpg
>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")
['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg', 'titledstudents1.jpg', 'titled-students2.jpg', 'titled-students5.jpg', 'titled-students6.jpg', 'titled-students7.jpg',
'titled-students8.jpg']
Comparing before and after
What if you want to make sure
you’ve got JPEG files?
import os
def titleDirectory(dir):
for file in os.listdir(dir):
print "Processing:",dir+"//"+file
if file.endswith(".jpg"):
picture = makePicture(dir+"//"+file)
addText(picture,10,10,"This is from CS1315 Spring 2003")
writePictureTo(picture,dir+"//"+"titled-"+file)
Video Processing

Burst a movie into a series of JPEG frames


MediaTools or QuickTime Pro will do this for you.
Process each frame using os.listdir
Lightening every frame of a movie
def lightenMovie(folder):
import os
for file in os.listdir(folder):
picture=makePicture(folder+file)
for px in getPixels(picture):
color=getColor(px)
makeLighter(color)
setColor(px,color)
writePictureTo(picture,folder+“L"+file)
Creating animations


Simply draw on JPEG frames
And save each out as a consecutively numbered
picture
Simplest Animation
def AnimationSimple():
frames = 50
rx = 0
ry = 0
rw = 50
rh = 50
for f in range(1,frames+1):
pic=makePicture(getMediaPath("640x480.jpg"))
addRectFilled(pic,rx,ry,rw,rh,red)
if(f < 10):
writePictureTo(pic,'test0%d.jpg'%(f))
else:
writePictureTo(pic,'test0%d.jpg'%(f))
rx = 5 + rx
ry = 5 + ry