Setting up Amazon Mechanical Turk with SERP Turkey

SERP Turkey logoThis post details how to setup and use Amazon’s Mechanical Turk with the SERP Turkey tool. If you’re not familiar with SERP Turkey, or want instructions on that tool then you should first read the SERP Turkey introduction.

If you’ve never heard of mTurk then there are good introductions here and here. Basically mTurk provides a huge online pool of workers – people at home who get paid anywhere between a few cents to a few dollars to complete simple tasks online. Anyone can join the workforce as a ‘worker’, and people in the US can become ‘requesters’ that set tasks. Here I describe how to become a requester and how to setup tasks that bring workers to your SERP Turkey tests. Don’t despair if you are not in the US – there is a method for you to get Amazon Turk workers with an extra step – see the section below.

Once you’ve registered as a requester (which may take a day for your first payment to go through), and created your template (10 mins) you’ll be able to create new SERP Turkey tests and get the live in SERP Turkey in less than 5 minutes.

Registering as a requester

So – lets get started. Head over to the mTurk request start page, and fill in your email and press the button to proceed. You’ll be asked for your name, email confirmation and to create a password. Next you’ll be asked for your company name and US address and phone number. Check the box to agree to the terms and proceed to the next screen. So far so good!

Ok, if you’d like to take the tour you’re offered now, go for it and I’ll see you on the over side. Otherwise just press ‘Get Started’ and start setting up your first task.

But some credit

mTurk works on a prepayment system and you have to buy credit to start. This is simple – just goto ‘Account Settings’ and then click the ‘Prepay for Mechanical Turk HITs’ link.

You’ll need a US registered credit card (though I’ve heard some people got through without one) to make the payment. be aware that your first payment might take 24-36 hours to get approved.

Creating a task (HIT) for SERP Turkey

mTurk tasks are called HITs (Human Intelligence Task) and are generally created using re-usable templates. Templates will define:

  • HIT Title, Description & Keywords
  • Payment per task (per worker)
  • How many workers you want
  • Time allotted per assignment
  • How long before the task expires?
  • Worker required qualifications

Templates will also define how you wish the question of the task to be presented to the user.

So, lets get going… You’ll be asked what sort of work you want the user to do:

categories of mTurk work

Select ‘Categorization’ and you’ll be presented with a list of pre-built template types:

Template types

Select the Search Relevance template, which we’ll customise.

Here are the settings I’ve been using:

Title: Click the best result from a set of Search Results
Description: Simply click on the result that you’d select yourself for that search.
Keywords: search, evaluate, opinion, google, quick

Time allotted per assignment: 2 minutes
HIT expires in: 7 days (generally I’ve not been fussed about needing the results before a deadline – at $0.05 per HIT you should allow 24-36 hours per 100 users)

HIT Approval rate: >=90% or >=95% depending how quickly I wanted the results. Obviously >=95% gets slightly improved results but a little slower.
Number of HITS approved: >100 or >500, once again depending on speed vs. quality

Reward per assignment: $0.05 – I experimented with $0.10 but didn’t see any real benefit in speed or quality. I’d rather twice the number of users!
Number of assignments per hit: 200 – this is entirely up to you and your budget and the number of variants you want. The more the better! I’d say ~100 per variant is an absolute minimum. If you’re experiment has subtle changes only between the variants then you’ll need even more. Note that each worker can only do a HIT once.

If you want to read more about your configuration options then look at page 10 of this mTurk guide.

Once you’ve chosen your settings press the ‘Design Layout’ button to proceed. We’ll overwrite the template provided – press the “Edit HTML Source” button you see at the top of the text area. Paste in the code here:

Click the ‘Edit HTML source’ button again to toggle back to display mode and you should see something like this (click to enlarge):

mTurk Template

The ${term} and ${result_page} markers you see will be filled in when you create a task from this template. This makes the template re-usable for multiple SERP Turkey tasks. We ask the user to paste the response code they get back from mTurk as we have to ask them a question (mTurk rules), and this response code encodes whether their click was recorded or not (if they timed out by taking longer than 5 minutes, or fiddled with the page somehow they’ll get a failure code). Only you will know which codes are a success for a test (shown in your SERP Turkey dashboard).

Hit the “Preview and Finish” button and you’re template will be saved and ready to use. Woo!

Creating a SERP Turkey task in mTurk

Now you have a template, you need to create a HIT when you want to run a SERP Turkey test.

Step 1: Create your test in SERP Turkey. If you’re not sure how, then read the SERP Turkey introduction blog post I wrote on SEOmoz.

Step 2: On the SERP Turkey Dashboard page you’ll find a download link for an mTurk input file; download it. This input file will fill in the ${term} and ${result_page} variables we created in our template.

Step 3: Login to mTurk and hit the chubby “Publish” button at the top. Select whichever template you want (you probably just have the one we just created if this is your first time).

Step 4: Upload the input file you downloaded (or select a previously uploaded one).

Step 5: Now you’ll see a preview of how the HIT will look to workers. If everything looks in order press “Next” to see the final screen.

Step 6: You can now give this batch of HITs a name, review how much it is likely to cost etc. If everything looks good then hit “Publish”.

Now you’re test is live and out there in the world!

Monitoring results

You can now monitor (and download) the results in realtime via the SERP Turkey interface. You can also view the “Manage” tab in mTurk to see how many workers have recorded completing the job on your end.

You should periodically review the results in mTurk and approve the workers work, so they get paid (you want to establish a good relationship as a requester so try to pay within a couple of days – you can automate payment in your template settings). If you’re so inclined you can see what response code each worker entered and compare this to the ‘success’ response code in SERP Turkey’s dashboard to see whether their click was counted or not – they are only not counted currently if they took longer than 5 minutes from viewing the SERPs to clicking a result. You should be able to safely ignore this and just pay all the workers.

Cancelling a job

If you’re monitoring the results and realise that the test isn’t going how you planned, there is a problem or you think you already have enough data then you can cancel a job at any time in the Manage page of mTurk.

Outside of the US?

If you are outside of the US and cannot access mTurk directly, worry not. There is another solution: Smartsheet Crowdsourcing actually leverages Amazon’s Mechanical Turk, but you don’t need to be in the US to use it. You do have to pay a $30 monthly subscription but then you can leverage Turk. If someone wants to write a Smartsheet SERP Turkey post I’ll happily add a link in here and on the SERP Turkey page!


IF SERP Turkey proves popular I will look at adding more features, and either way I will soon make it so you can download all the raw click data for your tests to do your own more detailed analysis. I just haven’t had time!

Wrap up

That’s pretty much all there is to it! 🙂

If you have any feedback or suggestions then contact me by email at [email protected] or via twitter at @TomAnthonySEO, or leave a comment here.

Happy Turkeying!

Introducing SERP Turkey: A Free Tool to Split-Test and Gather CTR Analytics of SERP Entries (on SEOmoz)

Measuring CTR data in search engine results is notoriously difficult, and with Google’s recent move to HTTPS for logged in users it is probably going to keep on getting harder. What I wanted was a simple way to measure the change in CTR for a given search query’s results when I adjusted entries, but nothing existed…. so I built the SERP Turkey tool.

Google +1 Button SEO – Getting Bulk Counts with the hidden API

Pile of +1 Buttons Now that the Google +1 button is out the Social SEO battle will inevitably be stepped up a gear. We know the search engines are using social shares and likes to impact the rankings, and Google’s +1 button is their way of ensuring at least some of this data is directly in their hands.

So obviously, tracking this data is very important for SEOs, and both Facebook and Twitter have APIs for pulling in Likes and Tweets, but currently we are left without a public API for doing this with Google’s +1 button. With a little digging behind the scenes I managed to find the API they use for the button. So now us SEOs can use it to pull out the Google +1 counts for lists of URLs (if you are interested in the technical details then see the optional section below).

I’ve put together a public Google Spreadsheet which can do all of this automatically for you – you just need to enter a list of URLs and the spreadsheet will pull in the counts for you. To get started, open the spreadsheet (download it right here) and select “Make a copy” from the File menu (you’ll need to be logged into a Google account to do this). Enter the URLs in the the A column (ensuring you pay attention to trailing slashed and they include http:// at the start), and the counts will appear in column B. You should see something like this:

Google Spreadsheet showing Google +1 Counts

Technical Bits

If you want to write your own solution, you can rip the code from the spreadsheet. The URL for the service is:

It is a JSON-RPC setup, meaning you send a JSON formatted request and get a JSON formatted response. The request should be via POST in the following format:

  1. [{"method":"pos.plusones.get","id":"p","params":{"nolog":true,"id":"","source":"widget","userId":"@viewer","groupId":"@self"},"jsonrpc":"2.0","key":"p","apiVersion":"v1"}]

The request will be returned like so:

  1. [{"result": { "kind": "pos#plusones", "id": "", "isSetByViewer": false, "metadata": {"type": "URL", "globalCounts": {"count": 3097.0} } } "id": "p"}]

Example PHP Code

If you want to implement this server side, here is some example PHP code to get you going:

  2. <?php
  4.  $url = "";
  6.  $ch = curl_init();  
  7.  curl_setopt($ch, CURLOPT_URL, "");
  8.  curl_setopt($ch, CURLOPT_POST, 1);
  9.  curl_setopt($ch, CURLOPT_POSTFIELDS, '[{"method":"pos.plusones.get","id":"p","params":{"nolog":true,"id":"' . $url . '","source":"widget","userId":"@viewer","groupId":"@self"},"jsonrpc":"2.0","key":"p","apiVersion":"v1"}]');
  10.  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  11.  curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-type: application/json'));
  14.  $curl_results = curl_exec ($ch);
  15.  curl_close ($ch);
  17.  $parsed_results = json_decode($curl_results, true);
  19.  echo $parsed_results[0]['result']['metadata']['globalCounts']['count'];
  21. ?>

There is an interactive demo here: Google +1 Button API demo.

Wrap Up

If you want to track social shares and likes and aren’t already setup with a Google Doc for the Twitter/Facebook parts you can head over to this post on Distilled which will help you get going. Combined with the above and you can build a complete system for tracking social search signals. Have fun!

Competitive Analysis in Under 60 Seconds Using Google Docs (on SEOmoz)

SERP reportFaced with a new client, and having established a list of keywords they need to target, you want to evaluate the competition to find out what sites are dominating the SERPs for these keywords. However… being an SEO you’re a busy guy (or gal), and you need it done right now. I’ve built a Google Docs tool to automagically do exactly that and this post will walk you through it.