Transcript Qualtrics and Mturk Workshop

Amazon Mechanical Turk (Mturk)
• What is MTurk?
– Crowdsourcing Internet marketplace that utilizes
human intelligence to perform tasks that
computers are unable to do
– Mturk samples are…
•
•
•
•
Fast
Cheap
Reliable
Diverse
First Things First…
• Go to www.mturk.com
Click here
First Things First…
• Sign in with existing Amazon.com account or
create one
Getting Started
You are called
a requester
Design
• The space that hosts your Qualtrics survey link
is called a HIT
• Your HIT is published to the Mturk site for
workers to see
• The first step is to design the layout of your
HIT
Properties
The title and description are seen by workers
Keywords categorize your HIT and make it
searchable
Properties
Set time restrictions
Chose Mturk Masters option
Set qualifications
Choose qualification requirement
Properties
Choose payment amount per HIT
Choose number of participants
The next step will be to design the layout
of the HIT
Layout
• Uses html code
• Give instructions and brief description of
study to workers
• Provide survey link
Layout
Template name should identify your study.
This is not displayed to workers.
Keep any of this you want and delete the
rest.
Layout
Code word and study instructions
Qualtrics link
Confirmation code
User comments
Layout
• Use a code word to prevent workers from repeating similar
surveys
• Lets say you create a survey about prejudice and use code
word ALPHA for the HIT
• Inform workers before they begin that the HIT uses code
word ALPHA
• After data collection you would like to re-run the study
after making some changes on Qualtrics
• You don’t want workers who completed the first round to
do the second round also
• Again, inform workers that the HIT uses code word ALPHA
and that if they had taken an ALPHA HIT before, they
should not continue
Layout
• Other ways of controlling for repeats
– In Qualtrics, create an item at the beginning of the
survey that asks the participant if they have
completed a HIT from this requester before
• If no, okay
• If yes, ask if the HIT they’ve done uses undesired code word
(in this case, its ALPHA)
– If yes, send participant to end of survey message
– If no, okay
– Ask at end of survey if participant had completed a
similar survey in the past
• Yes, no, I don’t remember
Layout
Confirmation code
Layout
• Confirmation Code
– Create a space for workers to enter the
confirmation code you provide at the end of
Qualtrics survey
– Code can be made up
– Used to check that workers completed entire
survey
Layout
Layout
Space
Link
Space
Layout
Space
Instructions
Text box
Layout
Space
Instructions
Text box
Layout – html code
<p><a href="http://kuclas.qualtrics.com/SE/?SID=SV_5pfmnKdhILnpOfi"
target="_blank">Click here to take the survey.</a></p> !survey link
<p>&nbsp;</p> !space
<p>At the end of the survey, you will be provided with a confirmation code.
Please enter the code below to verify that you have completed the task:</p>
<p><textarea rows=“1" cols="80" name=“Code"></textarea></p> !code box
<p>&nbsp;</p> !space
<p>Please provide any comments you may have below, we appreciate your
input!</p>
<p><textarea rows="3" cols="80" name="comment"></textarea></p> !comment
box
Preview and Finish
Publish
Name the batch
Negative balance
Publish
Publish HIT
Publish
• Note:
– If your Amazon account is funded, the HIT will be
published right away
– If not, you will be notified of this and be able to
click a link which will allow you to fund your
account using a variety of methods
Manage
• Watch completion rate
• Accept or reject work
• Cancel batch
Manage
Click here to accept
work (pay participants)
Manage
Approved 1 HIT out of 1
Confirmation code
for this worker
Marking worker and clicking approve
will initiate payment
Clicking worker ID allows you to block
worker and pay bonuses
MTurk – Things to consider
•
•
•
•
Blocking workers is BAD
Average pay anywhere from $.10 to $1.00
Command line tools
Professionalism
– Turkernation.com
• SPSP Seminar on MTurk
Thank you!
• Matt Baldwin: [email protected]
• Lucas Keefer: [email protected]