Transcript Image Scaling ~ Progress/Update

Image Scaling
Jackie Van Ryzin
http://compsci.snc.edu/cs460/vanrjl
Outline
• Project Description
–
–
–
–
Definition & Requirements
Solutions (what I did)
Exceptions (what I didn’t get to)
Methodology (how I did it)
• Demonstration – Zooming fun!
• Learning & Development Process
–
–
–
–
Strategies (what works for me)
Knowledge (useful CS concepts)
Extensions (where to from here)
Advice (for those yet to come)
• Q&A
Project Description
• Develop algorithms for real-time scaling
that maintains quality
• Study and implement smooth Bresenham
nearest neighbor interpolation method
• Look at ways to improve these methods
• Develop new algorithms or alter existing
• Implement a user interface that would
allow someone to zoom in and out as they
watch a streaming video
Overall Plan
• Read, research, understand existing algorithms
• Implement those algorithms using general
arrays
• Create program to read and alter images
• Combine image program and algorithms
• Develop interface to test and compare
algorithms and variations of existing one
• Alter programs & interface to scale real-time
streaming video in addition to still images
In the beginning…
• Thiadmer Riemersma was unhappy with:
– Quality of fast image scaling algorithms
– Slow processing for high quality scaling
routines
• He developed smooth Bresenham
algorithm (Dr. Dobb’s Journal, May 2002)
• Used concept of Bresenham’s line
algorithm to scale a horizontal line of
pixels
Background in Bresenham
• Determine which points of an n-dimensional
grid to plot to best approximate a straight line
between two given points (x0,y0) and (x1,y1)
• For each x between x0 and x1, choose row y that
most nearly matches fcn value to plot pixel (x,y)
To determine y
• General line formula is
( y1  y0 )
y  y0 
( x  x0 )
( x1  x0 )
• Know column x so solve for row y as
( y1  y0 )
(int)
( x  x0 )  y0
( x1  x0 )
• Can pre-calculate constant
( y1  y0 )
( x1  x0 )
• Since slope between 0 & 1 by assumption, after
rounding we either use same y or add 1 to it
Tracking Error
• User error value to track difference between current y
value and exact y value for current x
• As increment x++, increase error by slope
• When error > 0.5, line is closer to next y value rather
than current/previous
– y++ and error--
Bresenham for Images
• Pixels are picked up from discrete positions in
source image and placed at discrete locations in
the destination image
• Nearest neighbor sampling
– Magnifying: pixels duplicated
– Minifying: pixels dropped
– Both leave artifacts in destination image
• More accurate sampling requires reading pixels
from fractional positions in source image
• Techniques exist that incorporate weighted
averages – computation-intensive
A “Lightweight” Alternative
• Compromise between
linear interpolation &
coarse Bresenham
scaling
• Destination pixel set to:
Original
150%
75%
– Value of closest source
pixel
– Unweighted average of
neighbors
– Based on proximity to
source pixel
Smooth Bresenham (cont’d)
• Merit:
– Standard (unweighted) averages = simpler
calculations
– Increased performance speed – important for realtime applications
– Dropped pixels & excessive “jaggies” of fast nearestneighbor techniques (such as coarse Bresenham) are
absent or kept to a minimum
• Adjust this algorithm for 2D images
– Nested for loops
– Call horizontal line scaling algorithm
– Average scaled lines with each other
How we use this…
• OpenCV Platform
• Images in IplImage
format which has char*
pointing to data
• Data stored linearly as
bytes as shown
b (0,0)
g (0,0)
r (0,0)
b (1,0)
g (1,0)
r (1,0)
b (2,0)
g (2,0)
r (2,0)
:
:
b (0,1)
• Use CV’s split fcn to get 3 single-channel
images from a 3-channel image
• Run algorithm on each before remerging
with CV’s merge fcn
• CAREFUL with averaging!
g (0,1)
r (0,1)
b (1,1)
g (1,1)
r (1,1)
:
:
b (width-1,height-1)
g (width-1,height-1)
r (width-1,height-1)
Zooming!
• At first could capture keys (used 1 & 2) with
cvWaitKey for user to scale up or down
• Discovered cvTrackBar
– Attaches nicely to
window
– Auto updates the
variable (scale factor)
– Very user-friendly &
visually appealing!
Analysis
• First scale image using Bresenham, then “undo” the
scaling by using Bresenham on scaled image to get
back to original size
• Compare with original image (calculate average
difference between pixels of original and “un-scaled”
images)
• Do same with cvResize functions & compare results
I Wish I Could/Would Have...
• Been better able to convert bytes to ints and back
– Would have condensed processing time by calling
algorithm only once (on 3-channel image rather than
separately on 3 1-channel images)
• Been able to implement more alterations to the
algorithm for comparison and analysis
• Looked into MIP mapping as mentioned in DDJ
– Quick scaling by factor of 2
• Used actual calculations to determine speed
rather than simply visual
Methods
• Some mini-lectures/discussions with Dr.
Pankratz & Dr. McVey in beginning to
understand basic algorithm
• Test algorithms on arrays of ints first! (it was a
while before I tried images)
• Tried using VB at first because Professor Blahnik
provided code for reading from a webcam
– Converting algorithm into VB (pointers)
– DLL to leave it in C code
– Picture box didn’t resize when image did
Methods (cont’d)
• OpenCV
– Started with sample program from developers
– Strip it down to minimum with John’s help
– Add features and functions in baby steps
• Always had two versions
– Still images using LoadImage
– Webcam using Capture
• Wrote functions in small test program to run on
arrays of ints before adding to interfaces
Let’s Scale!
• Real-time scrolling from webcam
– HALLELUJAH!\scroll_webcam\scroll_webcam.dsw
• Compare with CV resizing using still image
– HALLELUJAH!\diff_cv\diff_cv.dsw
• How well does it “undo” the scaling?
– HALLELUJAH!\rescale_cmp\rescale_cmp.dsw
• Does a different threshold matter?
– HALLELUJAH!\cmp_threshold\cmp_threshold.dsw
Where did this stuff come from?
•
•
•
•
•
•
Dr. Dobb’s Journal
OpenCV manual
Dr. Pankratz & Dr. McVey
Forums and online tutorials for OpenCV
Practice, experimenting, trial-and-error
John, Ted, Ryan P
Important CS Concepts
• Data Structures
– Arrays, pointers, structures, chars vs. unsigned chars!
– Dynamic memory allocation & de-allocation
• Machine Org. & Assembly Language
– Fast calculations for dividing & masking
• Programming Languages
– Designing abstractly without a language
• All classes
– General learning, experimenting, researching
techniques
Where to From Here?
• Be able to convert to array of ints & back to
reduce calls to scaling algorithm
• Test & compare speed using actual clock
calculations
• Find a better mathematical/computational
method for comparing how “nice” a picture looks
scaled
• Alter algorithm in some way to improve quality
and/or speed
• Investigate MIP mapping and using a
combination of MIP & Bresenham
For Those Who Follow
• Meet with DCP and other profs OFTEN and right away
• Ask questions of anyone and everyone
– Someone may have a unique solution you wouldn’t think of
– Could make you discover a solution with a simple comment
about how chars in C aren’t fun
• When you get stuck, admit it and ask for help
• Write your journal! It keeps you accountable and
encourages progress.
• Paper can be useful! Draw images, diagrams, etc. It helps
to see it before you program it
• Baby steps…starting small is easier, and every little
milestone is rewarding. It also makes testing &
debugging easier.
Q&A
Thanks to Dr. Pankratz and Dr. McVey for
the guidance and ideas!
Thank you to classmates for ideas, answers,
and suggestions.
http://compsci.snc.edu/cs460/vanrjl