Transcript CrowdSearch (Ashok)

CrowdSearch: Exploiting Crowds for
Accurate Real-Time Image Search on
Mobile Phones
Original work by Tingxin Yan, Vikas Kumar, Deepak Ganesan
Presented by Ashok Kumar Jonnalagadda
Roadmap

Problem Description



System Architecture
The Crowd Search Algorithms





What is “crowdsourcing”?
Delay Prediction
Validation Prediction
Experimental Evaluation
Discussion/Criticism
Questions
The Perceived Problem

Text-based search is easy…
The Perceived Problem

Mobile-based search will become more important in the
future.

More than 70% of smart phone users perform searches.



Expected to be more mobile searches than non-mobile searches soon
Text-based mobile searches are easy as well…
Issues:



Small form-factor and resource limitations.
Typing on a phone is cumbersome
Scrolling through multiple search results.
multimedia searches requires significant
memory, storage, and computing resources.
Mobile: GPS and voice for search is becoming


more commonplace.
Image Search from Mobile.?

Image variations in







lighting
Texture
Type of features
image quality and many other factors.
Even Google Goggle doesn’t work with all categories.
Automated image search has limitations in terms of
Humans are naturally good at distinguishing images
The Perceived Problem

But how does a mobile phone user search for this?

No visible words/letters; too far away to know the address.
The Perceived Problem

Ways to find out what that building is:



Ask random people on the street
Travel to the building to see the address/sign
Take a picture of the building with your mobile device and send
to a search engine…

How easy is image searching on a mobile phone though?
The Perceived Problem


Image search is a non-trivial problem – have to deal with
variations in lighting, texture, image quality, etc.
Even when results are returned, scrolling through multiple
pages on a mobile device is cumbersome.


Search should be precise and return very few erroneous results.
Multimedia searches require significant



Memory
Storage
Computing resources
The Proposed Solution

CrowdSearch – Attempts to provide an accurate, image
search system for mobile devices by combining…


Automated image search and
Real-time human validation of search results

Leverage crowdsourcing through Amazon Mechanical Turk (AMT)
The Proposed Solution

Humans are good at comparing images

Could an automated search determine these two images
are of the same building?

Crowdsourcing increases search result accuracy.
System Architecture

Three main components:

Mobile Device




Remote Server



Initiates queries
Displays responses
Performs local image processing (maybe)
Performs automated image search
Triggers image validation tasks
Crowdsourcing System (AMT)

Validates image search results
Apple iPhone Mobile Client
System Operation Overview
System Operation Overview
System Operation Overview

How do we minimize delay and cost while maximizing accuracy?
System Architecture
Balancing Tradeoffs

Result delay


Result accuracy


Strive for high (i.e., ≥ 95%) accuracy
Monetary cost


Should minimize delay or at least keep it within a user-provided
bound
Low cost is better than high cost
Energy

Should consume minimal battery power
Accuracy Considerations

How many validations are required for 95% accuracy?

Requiring at least
three validations
out of five achieves
≥ 95% accuracy.
Optimizing Delay

Utilize parallel posting


Post all candidate images to the crowdsourcing system at the
same time.
But this approach increases cost!
5 cents
5 cents
5 cents
5 cents
= 20 cents
Optimizing Cost

Utilize serial posting


Post top-ranked candidate first, wait for responses, then post
next candidate if necessary.
This approach increases delay!
CrowdSearch Delay/Cost Optimization

Combine elements of parallel and serial posting

Prediction requires delay and validation models

Goal: want at least one verified result by the deadline.
CrowdSearch Delay/Cost Optimization
Delay Prediction Model

The delay of a single response is the combination of
acceptance delay and submission delay.


Both of these follow an exponential distribution with an offset.
Thus, overall delay is the convolution of these delays.
Delay Prediction Model Performance
Validation Model

Given a response set S, want to compute probability of
positive validation result.



Use training data to set these probabilities
If the probability of a positive
result is less than some
threshold, send the next
candidate to validation.
In this example, if the
threshold were set to < 76%,
the server would post the
next candidate image to AMT.
Power Considerations

Should some image processing occur on the local device
or should it be outsourced to the server?



It depends!
Use remote
processing when
WiFi is available.
Use local processing
when only 3G is
available

Extracting features
from query Image

(Scale Invariant feature transform)
Experimental Results

Any of the crowdsourcing schemes lead to better results!

Some types of images
are easier for
automated searches
to handle than others
Experimental Results

CrowdSearch leads to (given a long enough deadline)…


Behavior close to parallel posting for recall
Behavior close to serial posting for search cost
Thoughts/Criticism

The limited nature of the solution

Limitation to the four categories





Buildings
Books
Flowers
Faces
Only 1000 images in the backend database.

Would increasing the number of automated search images increase
total task time in a significant way?
Thoughts/Criticism

How useful is this anyway?


Are people willing to go through the trouble to set up a
payment account and pay 5-20 cents for a search?
How much effort would it usually take for someone to find out
what the object is through traditional means?


Especially for books!
Privacy concerns

People utilizing CrowdSearch must accept the fact that random
strangers know what they are looking at and searching for.


Additionally, their GPS information might be provided to the
CrowdSearch servers.
What about the privacy of the object of the search?

Undercover police officers
Questions?