in All Posts, SEO/SEM, Tools

Setting up Amazon Mechanical Turk with SERP Turkey

SERP Turkey logoThis post details how to setup and use Amazon’s Mechanical Turk with the SERP Turkey tool. If you’re not familiar with SERP Turkey, or want instructions on that tool then you should first read the SERP Turkey introduction.

If you’ve never heard of mTurk then there are good introductions here and here. Basically mTurk provides a huge online pool of workers – people at home who get paid anywhere between a few cents to a few dollars to complete simple tasks online. Anyone can join the workforce as a ‘worker’, and people in the US can become ‘requesters’ that set tasks. Here I describe how to become a requester and how to setup tasks that bring workers to your SERP Turkey tests. Don’t despair if you are not in the US – there is a method for you to get Amazon Turk workers with an extra step – see the section below.

Once you’ve registered as a requester (which may take a day for your first payment to go through), and created your template (10 mins) you’ll be able to create new SERP Turkey tests and get the live in SERP Turkey in less than 5 minutes.

Registering as a requester

So – lets get started. Head over to the mTurk request start page, and fill in your email and press the button to proceed. You’ll be asked for your name, email confirmation and to create a password. Next you’ll be asked for your company name and US address and phone number. Check the box to agree to the terms and proceed to the next screen. So far so good!

Ok, if you’d like to take the tour you’re offered now, go for it and I’ll see you on the over side. Otherwise just press ‘Get Started’ and start setting up your first task.

But some credit

mTurk works on a prepayment system and you have to buy credit to start. This is simple – just goto ‘Account Settings’ and then click the ‘Prepay for Mechanical Turk HITs’ link.

You’ll need a US registered credit card (though I’ve heard some people got through without one) to make the payment. be aware that your first payment might take 24-36 hours to get approved.

Creating a task (HIT) for SERP Turkey

mTurk tasks are called HITs (Human Intelligence Task) and are generally created using re-usable templates. Templates will define:

  • HIT Title, Description & Keywords
  • Payment per task (per worker)
  • How many workers you want
  • Time allotted per assignment
  • How long before the task expires?
  • Worker required qualifications

Templates will also define how you wish the question of the task to be presented to the user.

So, lets get going… You’ll be asked what sort of work you want the user to do:

categories of mTurk work

Select ‘Categorization’ and you’ll be presented with a list of pre-built template types:

Template types

Select the Search Relevance template, which we’ll customise.

Here are the settings I’ve been using:

Title: Click the best result from a set of Search Results
Description: Simply click on the result that you’d select yourself for that search.
Keywords: search, evaluate, opinion, google, quick

Time allotted per assignment: 2 minutes
HIT expires in: 7 days (generally I’ve not been fussed about needing the results before a deadline – at $0.05 per HIT you should allow 24-36 hours per 100 users)

HIT Approval rate: >=90% or >=95% depending how quickly I wanted the results. Obviously >=95% gets slightly improved results but a little slower.
Number of HITS approved: >100 or >500, once again depending on speed vs. quality

Reward per assignment: $0.05 – I experimented with $0.10 but didn’t see any real benefit in speed or quality. I’d rather twice the number of users!
Number of assignments per hit: 200 – this is entirely up to you and your budget and the number of variants you want. The more the better! I’d say ~100 per variant is an absolute minimum. If you’re experiment has subtle changes only between the variants then you’ll need even more. Note that each worker can only do a HIT once.

If you want to read more about your configuration options then look at page 10 of this mTurk guide.

Once you’ve chosen your settings press the ‘Design Layout’ button to proceed. We’ll overwrite the template provided – press the “Edit HTML Source” button you see at the top of the text area. Paste in the code here:

Click the ‘Edit HTML source’ button again to toggle back to display mode and you should see something like this (click to enlarge):

mTurk Template

The ${term} and ${result_page} markers you see will be filled in when you create a task from this template. This makes the template re-usable for multiple SERP Turkey tasks. We ask the user to paste the response code they get back from mTurk as we have to ask them a question (mTurk rules), and this response code encodes whether their click was recorded or not (if they timed out by taking longer than 5 minutes, or fiddled with the page somehow they’ll get a failure code). Only you will know which codes are a success for a test (shown in your SERP Turkey dashboard).

Hit the “Preview and Finish” button and you’re template will be saved and ready to use. Woo!

Creating a SERP Turkey task in mTurk

Now you have a template, you need to create a HIT when you want to run a SERP Turkey test.

Step 1: Create your test in SERP Turkey. If you’re not sure how, then read the SERP Turkey introduction blog post I wrote on SEOmoz.

Step 2: On the SERP Turkey Dashboard page you’ll find a download link for an mTurk input file; download it. This input file will fill in the ${term} and ${result_page} variables we created in our template.

Step 3: Login to mTurk and hit the chubby “Publish” button at the top. Select whichever template you want (you probably just have the one we just created if this is your first time).

Step 4: Upload the input file you downloaded (or select a previously uploaded one).

Step 5: Now you’ll see a preview of how the HIT will look to workers. If everything looks in order press “Next” to see the final screen.

Step 6: You can now give this batch of HITs a name, review how much it is likely to cost etc. If everything looks good then hit “Publish”.

Now you’re test is live and out there in the world!

Monitoring results

You can now monitor (and download) the results in realtime via the SERP Turkey interface. You can also view the “Manage” tab in mTurk to see how many workers have recorded completing the job on your end.

You should periodically review the results in mTurk and approve the workers work, so they get paid (you want to establish a good relationship as a requester so try to pay within a couple of days – you can automate payment in your template settings). If you’re so inclined you can see what response code each worker entered and compare this to the ‘success’ response code in SERP Turkey’s dashboard to see whether their click was counted or not – they are only not counted currently if they took longer than 5 minutes from viewing the SERPs to clicking a result. You should be able to safely ignore this and just pay all the workers.

Cancelling a job

If you’re monitoring the results and realise that the test isn’t going how you planned, there is a problem or you think you already have enough data then you can cancel a job at any time in the Manage page of mTurk.

Outside of the US?

If you are outside of the US and cannot access mTurk directly, worry not. There is another solution: Smartsheet Crowdsourcing actually leverages Amazon’s Mechanical Turk, but you don’t need to be in the US to use it. You do have to pay a $30 monthly subscription but then you can leverage Turk. If someone wants to write a Smartsheet SERP Turkey post I’ll happily add a link in here and on the SERP Turkey page!

Future

IF SERP Turkey proves popular I will look at adding more features, and either way I will soon make it so you can download all the raw click data for your tests to do your own more detailed analysis. I just haven’t had time!

Wrap up

That’s pretty much all there is to it! ๐Ÿ™‚

If you have any feedback or suggestions then contact me by email at [email protected] or via twitter at @TomAnthonySEO, or leave a comment here.

Happy Turkeying!

Write a Comment

Comment

  1. Hey Tom,
    came here from SEOMoz and just wanted to let you know that I really loved your SERP Turkey test. Maybe I’ll set up a study in german, but I’m not sure about the quality of the outcome (if any) because of the language distribution in mTurk.

    Anyways, great article ๐Ÿ™‚

    Cheers
    Pascal

    • Hey Pascal!

      Thanks – I’m glad you like it.

      Yes, I am really interested in how it would work out in other languages, and also am not sure of the mTurk distribution. There will be less workers but if people are searching for jobs with German keywords then I imagine there will be competition in terms of the number of HITs they can choose from.

      Let me know if you try it and what you find out.

      Deutsch ist die nur andere Sprach dass ich ein BiรŸchen sprechen kann so es wรคre gut zu wissen. Not sure how many mistakes are in there! ๐Ÿ™‚

      Thanks!

  2. Hi Tom,
    I also came here from SEOMoz too, and I also liked your SERP Turkey test.
    I was thinking like Pascal that maybe Iโ€™ll try your experiment in Polish?However this might be tricky…
    Thanks for sharing your findings, really useful stuff.

    • Hi SEOgirl!

      Thanks a lot! Yes – finding testers will be hard maybe. However, please let me know how you get on. If you get it working and want to write a blog post then drop me an email so I can make sure I link over to it. ๐Ÿ™‚

      Thanks & have fun with it! ๐Ÿ™‚

  3. Hi Tom,

    I came here from the new Moz.

    Thanks so much for your SERP Turkey test! I’m currently trying out a few example and using my co-workers as testers. ๐Ÿ˜‰

    Great articles too!

    All the best,

    Amanda

  4. Hi Tom,

    I am attempting to set up a search results test using SERP Turkey and Mechanical Turk, but am having trouble following the instructions for setting up the test. I was able to navigate SERP Turkey to create the variations, download the CVS, etc., but cannot seem to get the mTurk part right.

    When I select “Categorization” as the project type, I do not see any templates. Furthermore, I am not presented with the option to edit the “Design Layout” and paste in the HTML provided above. I can upload the CSV file and provide a link to it, but do not see how I can encourage a Worker to fulfill the task.

    I wonder if the interface has changed since you created this guide. Do you have another example that I can reference or any updates to the guide above?

    Thanks in advance for any help and for creating this awesome tool!

    – Charlie

  5. Follow up to my previous questions: Looks like you can use the format above (including HTML design) using the “Survey” project design in mTurk. Hope that helps anyone else looking to use this awesome service.

Webmentions

  • Optimize Your SERP Snippet Click-Through Rate May 1, 2014

    […] This can be done manually, but is set up to be the optimal way to use SERP Turkey with Amazon’s Mechanical Turk. You can learn about setting up SERP Turkey with mTurk here. […]

  • Optimize Your SERP Snippet Click-Through Rate | Content Generator May 1, 2014

    […] This can be done manually, but is set up to be the optimal way to use SERP Turkey with Amazon’s Mechanical Turk. You can learn about setting up SERP Turkey with mTurk here. […]

  • Introducing SERP Turkey: A Free Tool to Split-Test and Gather CTR Analytics of SERP Entries « Business Leads « Get Business Leads May 1, 2014

    […] Setting up Amazon Mechanical Turk with SERP Turkey […]