Transcript (slides 2MB)

CrowdFlow
Integrating Machine Learning
with Mechanical Turk for
Speed-Cost-Quality Flexibility
Alex Quinn, Ben Bederson, Tom Yeh, Jimmy Lin
Human Computation
Things
COMPUTERS
can do
Translation
Photo tagging
Face recognition
Human detection
Speech recognition
Text analysis
Planning
Things
HUMANS
can do
Human Computation
Things
COMPUTERS
can do
Translation
Photo tagging
Face recognition
Speech recognition
Human detection
Text analysis
Planning
Things
HUMANS
can do
Example: Human detection
Trade-off space
Speed, Affordability
Computers
Human
Computation
Human Workers
(traditional)
Quality
Trade-off space
Speed, Affordability
Computers
Human
Computation
Human Workers
(traditional)
Quality
Man-Computer Symbiosis
humans
computer
computer
humans
speed
cost
quality
speed
cost
quality
Supervised Automation
machine
with human
learning post-correction
Man-Computer Symbiosis
humans
computer
computer
computer
speed
cost
quality
humans
humans
speed
cost
quality
Supervised Automation
machine
with human
learning post-correction
speed
cost
quality
CrowdFlow
Mechanical Turk
Human Detection – Starting point
Human Detection – Task
Speed, Affordability
Human Detection – Results
60%
119 images took
3 hrs 50 mins
and cost $2.38
Quality
90%
Speed, Affordability
Human Detection – Scenarios
60%
1000 photos at 72% accuracy
would take 12 hrs 20 mins
and cost
$8.00
119 images took
3 hrs 50 mins
and cost $2.38
Quality
90%
Vision: Richer model
Input with computer results
Correct Validator
Incorrect
Fix
Fixer
Output
Start
Appraiser
over
Worker
Lessons Learned

Design for overall needs/constraints

Practical advice:




Pay consistently and reasonably
Reject only work that is definitely cheating
Build in fair cheating deterrence from the start
Keep instructions short, but always clear
Contact: Alex Quinn [email protected]