CAPTCHA, THE ESP GAME, AND OTHER STUFF LUIS VON AHN

CARNEGIE MELLON UNIVERSITY

CAPTCHA

A PROGRAM THAT CAN TELL WHETHER ITS USER IS A HUMAN OR A COMPUTER

CAPTCHA

A PROGRAM THAT CAN GENERATE AND GRADE TESTS THAT: A. MOST HUMANS CAN PASS B. CURRENT COMPUTER PROGRAMS CANNOT PASS

EXAMPLE

PICKS RANDOM STRING OF LETTERS oamg RENDERS IT INTO A DISTORTED IMAGE

…AND GENERATES A TEST : TYPE THE CHARACTERS THAT APPEAR IN THE IMAGE

P STANDS FOR

PUBLIC

ALL CODE AND DATA USED BY A CA P TCHA SHOULD BE PUBLICLY AVAILABLE

COMPLETELY AUTOMATED PUBLIC TURING TEST

TO TELL COMPUTERS AND HUMANS APART

UTILIZING THE POWER OF

HUMAN CYCLES

CAPTCHA

APPLICATIONS EXAMPLES DIRTY HACKS

THE ESP GAME OTHER STUFF

APPLICATIONS

OF CAPTCHAS

QUICKVOTE Which is the best Computer Science Grad School in the US?

Berkeley CMU Cornell MIT Princeton Stanford vote (

FROM

WWW.SLASHDOT.ORG)

APPLICATIONS

FREE E-MAIL SERVICES DATA COLLECTION WORMS AND SPAM PREVENTING DICTIONARY ATTACKS (PINKAS AND SANDER ‘02)

APPLICATIONS

FREE E-MAIL SERVICES DATA COLLECTION WORMS AND SPAM PREVENTING DICTIONARY ATTACKS (PINKAS AND SANDER ‘02)

APPLICATIONS

FREE E-MAIL SERVICES DATA COLLECTION WORMS AND SPAM PREVENTING DICTIONARY ATTACKS (PINKAS AND SANDER ‘02)

APPLICATIONS

FREE E-MAIL SERVICES DATA COLLECTION WORMS AND SPAM PREVENTING DICTIONARY ATTACKS (PINKAS AND SANDER ‘02)

EXAMPLES OF

CAPTCHAS

PIX

WHAT ARE THESE PICTURES OF?

PIX

WHAT ARE THESE PICTURES OF?

THE IMAGES NEED TO BE RANDOMLY DISTORTED

PIX

WHAT ARE THESE PICTURES OF?

THE IMAGES NEED TO BE RANDOMLY DISTORTED

PIX

WHAT ARE THESE PICTURES OF?

THE IMAGES NEED TO BE RANDOMLY DISTORTED

WHAT ARE THESE PICTURES OF?

BONGO

VISUAL ANALOGY PROBLEMS (BONGARD 1951)

SOUND-BASED

CAPTCHA

HUMANS ARE BETTER THAN COMPUTERS AT UNDERSTANDING SPOKEN LANGUAGE

OPEN

PROBLEM

CREATE A CAPTCHA BASED ON LANGUAGE UNDERSTANDING

DIRTY

HACKS

CAPTCHA

SWEAT SHOPS

SPAM COMPANIES HIRE HUMANS TO SOLVE CAPTCHAS ALL DAY LONG $2.50 PER HOUR FOR EACH HUMAN 720 CAPTCHAS PER HOUR PER HUMAN 1/3 CENT PER ACCOUNT

Free Nude Photos > NEXT

Free Nude Photos Type the word in the box if you want to see the next picture

UTILIZING THE POWER OF

HUMAN CYCLES

CAPTCHA

APPLICATIONS EXAMPLES DIRTY HACKS

THE ESP GAME OTHER STUFF

LABELING IMAGES

WITH WORDS

MARTHA STEWART FLOWERS SUPER EVIL STILL AN OPEN PROBLEM

IMAGE SEARCH

ON THE WEB

USES FILENAMES AND HTML TEXT

ACCESSIBILITY

LESS THAN 10% OF THE WEB IS ACCESSIBLE TO THE VISUALLY IMPAIRED REASON: MOST IMAGES DON’T HAVE PROPER CAPTIONS

DESIDERATA

A METHOD THAT CAN LABEL ALL IMAGES ON THE WEB FAST AND CHEAP

USING HUMANS

CLEVERLY

COULD LABEL ALL IMAGES ON THE WEB IN 30 DAYS!

THE

ESP GAME

TWO-PLAYER ONLINE GAME PARTNERS DON’T KNOW EACH OTHER AND CAN’T COMMUNICATE OBJECT OF THE GAME: TYPE THE SAME WORD THE ONLY THING IN COMMON IS AN IMAGE

THE

PLAYER 1

ESP GAME

PLAYER 2 GUESSING: CAR GUESSING: HAT GUESSING: KID SUCCESS!

YOU AGREE ON CAR GUESSING: BOY GUESSING: CAR SUCCESS!

YOU AGREE ON CAR

© 2004 Carnegie Mellon University, all rights reserved. Patent Pending.

THE ESP GAME

IS FUN

4.1 MILLION LABELS WITH 23,000 PLAYERS THERE ARE MANY PEOPLE THAT PLAY OVER 20 HOURS A WEEK

LABELING THE

ENTIRE WEB

5000 PEOPLE PLAYING SIMULTANEOUSLY CAN LABEL ALL IMAGES ON GOOGLE IN 30 DAYS !

INDIVIDUAL GAMES IN YAHOO! AND MSN AVERAGE OVER 5,000 PLAYERS AT A TIME

A FEW

MILLION LABELS

CAN IMPROVE IMAGE SEARCH CAN BE USED TO IMPROVE COMPUTER VISION

SINGLE PLAYER

GAME

A SINGLE PERSON CAN PLAY WITH PRE RECORDED ACTIONS AS THEIR PARTNER WE EMULATE PARTNER BY PLAYING PRE RECORDED MOVES (0:12) CAR (0:15) HAT (0:21) KID (0:08) BOY (0:23) CAR WHEN 2 PEOPLE PLAY, WE RECORD EVERY ACTION WITH TIMING INFORMATION NOTICE THAT THIS DOESN’T STOP THE LABELING PROCESS!

WHAT ABOUT

CHEATING?

IF A PAIR PLAYS TOO FAST, WE DON’T RECORD THE WORDS THEY AGREE ON

WHAT ABOUT

CHEATING?

WE GIVE PLAYERS TEST IMAGES FOR WHICH WE KNOW ALL THE COMMON LABELS: WE ONLY STORE A PLAYER’S GUESSES IF THEY SUCCESSFULLY LABEL THE TEST IMAGES

SEARCH

RESULTS

SAMPLE

LABELS

BEACH CHAIRS SEA PEOPLE MAN WOMAN PLANT OCEAN TALKING WATER PORCH

SAMPLE

LABELS

SADDAM MR. WILSON MAN MEET YOUR MOUSTACHE THROUGH THE GAME!

BUSH PRESIDENT DUMB

WHY DO PEOPLE LIKE

THE ESP GAME?

“ ON THE OTHER HAND, THE TWO OF YOU ARE BRINGING YOUR MINDS TOGETHER IN A WAY THAT LOVERS WOULD ENVY. ”

“ “ STRANGELY ADDICTIVE ” IT’S SO MUCH FUN TRYNG TO GUESS WHAT OTHERS THINK. YOU HAVE TO “ “ STEP OUTSIDE OF YOURSELF TO MATCH ” IT’S FAST-PACED ” HELPS ME LEARN ENGLISH ”

BLOGS

“ IN OTHER CROWNING ACHEIVEMENTS, I HAVEN'T PLAYED IN AT LEAST A WEEK (PROBABLY LONGER), AND I AM STILL ON THE TOP TEN LIST OF PLAYERS ON THE ESP GAME. NUMBER EIGHT IF I REMEMBER CORRECTLY. GOD, MY LIFE IS SAD.

”

“ THERE ARE MANY FASCINATING THINGS ABOUT THE ESP GAME: HOW IT USES A GAME STRUCTURE TO BUILD AN INDEX OF IMAGES ON THE WEB WHAT WORDS BECOME TABOO FROM FREQUENT USE HOW INTERESTING SOME OF THE IMAGES PULLED RANDOMLY FROM THE WEB ARE BUT BY FAR THE MOST INTRIGUING ASPECT OF THE GAME IS HOW OFTEN YOUR RANDOM, UNKNOWN PARTNER IS A COMPLETE IDIOT.

”

OTHER

GAMES

LOCATING OBJECTS

IN IMAGES

THE ESP GAME TELLS US IF AN IMAGE CONTAINS A SPECIFIC OBJECT, BUT DOESN’T SAY WHERE IN THE IMAGE THE OBJECT IS SUCH INFORMATION WOULD BE EXTREMELY USEFUL FOR COMPUTER VISION RESEARCH

PAINTBALL

GAME

PLAYERS SHOOT AT OBJECTS ON THE IMAGE SHOOT THE: CAR WE GIVE POINTS AND CHECK ACCURACY BY GIVING PLAYERS IMAGES FOR WHICH WE ALREADY KNOW WHERE THE OBJECT IS

PAINTBALL

GAME

X% OF IMAGES (100-X)% OF IMAGES DON’T KNOW

BALL

REVEALING

IMAGES

GUESSER REVEALER GUESS CAR PARTNER’S GUESS

MEDIUM-TERM

GOAL

USE THE ESP GAME, ALONG WITH THE PAINT BALL GAME AND THE REVEALING GAME TO CREATE THE MOST ACCURATE COMPUTER VISION ALGORITHMS TO RECOGNIZE A WIDE VARIETY OF OBJECT CATEGORIES

X OR

NOT

CAN BE USED TO CLASSIFY ALL IMAGES ON THE WEB AS HAVING PROPERTY X OR NOT WE GIVE PEOPLE 40% OF IMAGES WITH PROPERTY X 40% OF IMAGES WITHOUT PROPERTY X 20% OF UNKNOWN IMAGES IF PEOPLE LIKE IMAGES WITH PROPERTY X, WE CAN OFFER AN INCENTIVE : THE MORE ACCURATE A PLAYER IS, THE MORE IMAGES WITH PROPERTY X THEY GET

THE POWER OF

HUMAN CYCLES

CAPTCHA OPEN PROBLEM GAME

OUR WORK ON CAPTCHA HAS APPEARED IN OVER 100

NEWSPAPERS AROUND THE WORLD

THE ESP GAME HAS APPEARED IN OVER 50

NEWSPAPERS AROUND THE WORLD

Name That Image

Computers excel at sifting information, but they have trouble distinguishing a picture of a tree from one of a turtle.

So researchers at Carnegie Mellon University knew they needed human volunteers to successfully label millions of unmarked images on the Web.

Luis von Ahn

, a graduate student in computer science, had the inspired idea of turning the laborious process into a two-player contest called the ESP Game (espgame.org). Each time you play, you are randomly paired with another anonymous player. You can't communicate with your partner, although you both see the same image. The goal is to guess the descriptive word your partner is typing for the image. Once you both type the same word, you see a new image.

Since October the game has attracted 15,000 registered players, and they have created more than 1.5 million labels (a label is a single descriptive word; most photos require multiple labels). The site draws from a database of 200,000 Web images; another 200,000 are to be added soon. Mr. von Ahn said that properly labeling Web images would allow for more efficient image searching, improve the screen readers used by the visually impaired and help users block inappropriate images.

WWW.

CAPTCHA.NET

WWW.

ESPGAME.ORG

MONI NAOR

HAIRSTYLES

RONI ROSENFELD DAVID STORK LATANYA SWEENEY SHINGO UCHIHASHI MANUELA VELOSO THE ALADDIN CENTER MICROSOFT RESEARCH MY MOM

SPEAKER’S ENTERTAINMENT

JOHN LANGFORD