advertising, commentary, economics, incentives, ratings, spam

The seedy side of Amazon's Mechanical Turk

25 Comments

I mostly side with Lukas and Panos on the fantastic potential of Amazon’s Mechanical Turk, a crowdsourcing service specializing in tiny payments for simple tasks that require human brainpower, like labeling images. Within the field of computer science alone, this type of service will revolutionize how empirical research is done in communities from CHI to SIGIR, powering unprecedented speed and scale at low cost (here are two examples). My guess is that the impact will be even larger in the social sciences; already, a number of folks in Yahoo’s Social Dynamics research group have started running studies on mturk. (A side question is how university review boards will react.)

However there is a seedier side to mturk, and I’m of two minds about it. Some people use the service to hire sockpuppets to enter bogus ratings and reviews about their products and engage in other forms of spam. (Actually this appears to violate mturk’s stated policies.)

For example, Samuel Deskin is offering up to ten cents to turkers willing to promote his new personalized start page samfind.

EARN TEN CENTS WITH THE BONUS – EASY MONEY – JUST VOTE FOR US AND COMMENT ABOUT US

EARN FOUR CENTS IF YOU:

1. Set up an anoymous email account likke gmail or yahoo so you can register on #2 anonymously

2. Visit http://thesearchrace.com/signup.php and sign up for an account – using your anonymous email account.

3. Visit http://www.thesearchrace.com/recent.php and vote for:

samfind

By clcking “Pick”

SIX CENTS BONUS:

4. Visit the COMMENTS Page on The Search Race, it is the Button Right Next to “Picks” on this page: http://www.thesearchrace.com/recent.php and

5. Say something awesome about samfind (http://samfind.com) on The Search Race’s Comments page.

Make sure to:

1. Tell us that you Picked us.
2. Copy and Paste the Comment you typed on The Search Race’s Comment page here so we know you wrote it and we will give you the bonus!

In fact, Deskin is currently offering bounties on mturk for a number of different spammy activities to promote his site. On the other hand, what Deskin is doing is not illegal and is arguably not all that different than paying PRWEB to publish his rah-rah press release (Start-up, samfind, Launches Customizable Startpage to Compete with Google, Yahoo & MSN, Los Angeles, California (PRWEB) August 4, 2008). And I have to at least give him credit for offering the money under his own name.

Another type of task on mturk involves taking a piece of text and paraphrasing it so that the words are different but the meaning remains the same. Here is an example:

Paraphrase This Paragraph

Here’s the original paragraph:

You’re probably wondering how to apply a wrinkle filler to your skin. The good news is that it’s easy! There are a number of different products on the market for anti aging skin care. Each one comes with its own special application instructions, which you should always make sure to read and carefully follow. In general, however, most anti aging skin care products are simply applied to the skin and left to soak in.

Requirements:
1. Use the same writing style as much as possible.
2. Vary at least 50% of the words and phrases – but keep the same concepts. Use obviously different sentences! Your paragraph should not be just a copy of the first with a few word replacements.
3. Any keywords listed in bold in the above paragraph must be included in your paraphrase.
4. The above paragraph contains 75 words… yours must contain at least 64 words and not more than 101 words.
5. Write using American English.
6. No obvious spelling or grammar mistakes. Please use a spell-checker before submitting. A free online spell checker can be found at www.spellcheck.net.

If you find it easier to paraphrase sentence-by-sentence, then do that. Please do not enter anything in the textbox other than your written paragraph. Thanks!

I have no direct evidence, but I imagine such a task is used to create splogs (I once found what seems like such a “paraphrasing splog”), ad traps, email spam, or other plagiarized content.

It’s possible that paid spam is hitting my blog (either that or I’m overly paranoid). I’m beginning to receive comments that are almost surely coming from humans, both because they clearly reference the content of the post and because they pass the re-captcha test. However, the author’s URL seems to point to an ad trap. I wonder if these commenters (who are particularly hard to catch — you have to bother to click on the author URL) are paid workers of some crowdsourcing service?

Can and should Amazon try to filter away these kinds dubious uses of Mechanical Turk? Or is it better to have this inevitable form of economic activity out in the open? One could argue that at least systems like mturk impose a tax on pollution and spam, something long argued as an economic force to reduce spam.

My main objection to these activities is the lack of disclosure. Advertisements and press releases are paid for, but everyone knows it, and usually the funding source is known. However, the ratings, reviews, and paraphrased text coming out of mturk masquerade as authentic opinions and original content. I absolutely want mturk to succeed — it’s an innovative service of tremendous value, one of many to come out of Amazon recently — but I believe Amazon is risking a minor PR backlash by allowing these activities to flow through its servers and by profiting from them.

25 thoughts on “The seedy side of Amazon's Mechanical Turk”

  1. Hello David,

    I see you found our HIT on mturk. We do use mturk for marketing purposes and have found it a great way to get people to take the first step of visiting our website and then to write something about it. People may do one, but to get both done (sometimes) requires asking for their assistance and providing an incentive – no matter how small.

    Frankly, we have had issues with traction and using mturk has also given us access to an audience that is not unlike our core users – people who want easy access to the websites they use – many of the mturk users are now users of samfind because of their introduction to our website on mturk.

    It has been a great resource to us and I think it is a completely legitimate way to get people to take that extra step to help promote a company.

    BTW – have you tried out samfind??

    Take care,

    Sam

  2. Sam, thanks for commenting here so quickly — impressive web monitoring. I agree there is a blurry line between legitimate and illegitimate marketing; I still think this crosses a line but that’s my personal opinion.

    It’s also true that such marketing tactics are unlikely to succeed unless you have a good product.

  3. what amazes me about MT is not that it is used for spamming (and I don’t think it’s possible to stop it, except in very explicit spamming cases), it’s that people are willing to do a job that takes at least a few minutes for a few cents. that’s way below the minimum wage!

  6. I would imagine that in some areas of the world those would be fine wages.
    Just as DiggIt was subjected to organized groups of supposedly “independent” clickers and commentators, MTurk will be used to organize supposedly disinterested opinion-givers who spread a particular viewpoint in various blogs and forums.
    We have scripted “news” and various “independent” public interest groups and I guess we are saddled with organized spamlike blog commentators busily grinding away on their axe.

  7. “Some people use the service to hire sockpuppets to enter bogus ratings and reviews about their products and engage in other forms of spam. (Actually this appears to violate mturk’s stated policies.)”

    Given that Amazon regularly removes negative reviews at the author’s request, how is this any worse?

  8. Interesting post. At Dolores Labs, we get a large number of inquiries from people hoping to do this type of splogging/grey-area marketing.

    I’m not sure this works so well. We’ve never done anything quite this brazen but as soon as we start to hit a grey-area (I believe just asking someone to sign up for anything technically violates Amazon.com’s terms of service) we find it very hard to recruit people. The Turker community seems to take a very negative view of these kinds of tasks, and companies doing this type of thing seem to very quickly get a bad reputation.

    I’ve seen far more questionable stuff on Turk. From time to time there are companies offering cash to people willing to scan in their credit card statements/phone bills, etc. (I can see how this could be a useful data for many reasonable applications, but it seems quite likely to also be used for more nefarious purposes).

  10. David – interesting question, but I don’t have a very strong opinion. The tasks might be less of a problem than you would expect from browsing AMT because they tend to pay the highest and often show up when you sort by price.

    I think most high volume turkers skip over them. AMT could probably use their own community to filter them quite easily. If you look on the turk message boards you sometimes see the turkers discussing the fact that they’ve reported offending tasks to Amazon, but lamenting the fact that Amazon takes no action. I think it would be in Amazon’s interest to remove them, but I’m not sure they should be blamed for having bigger issues to deal with.

  12. MTurk is indeed a fun little ecosystem to study and a great accessible tool for someone if their need fits with what turks will do.

    I’ve done numerous studies on Turks attempting to find out why they do all these odd ball tasks and whats most interesting is how much turk/task behavior mimics everyday web surfing in general less about making money.

    Before you ask, why do that for 3 cents, ask yourself, why do you participate in blogs, social networks and everything else on the web. What do you get out of it?

    In my Valley of Turks post, I think its clear that not all turks do it for the money. Its a challenge, its a game. The real challenge isn’t the work they do its finding cheap easy to do work thats an easy cent or buck or two.

    http://www.floozyspeak.com/blog/archives/2008/08/valley_of_the_t.html

    Ya gotta remember this is like a backdoor don’t really care about experiment from Amazon. It arrived on the scene in 2006, and it went through various stages of good PR mostly bad PR back then and only now have they started to look at it again and attempt to spruce it up. Microsoft and Google have had like systems, Microsofts is pretty much gone, Google’s is history but I hear its back again in Russia somewhere.

    I agree with Lukas in that the Turk community sniffs out scams faster than you and I would any day. They live on Turk scouring it daily for golden gems of work in the ruff of it all.

  15. Update: Someone uncovered a name-brand company (Belkin) apparently buying positive reviews (ironically on Amazon.com) using Mechanical Turk.

    http://www.thedailybackground.com/2009/01/16/exclusive-belkins-development-rep-is-hiring-people-to-write-fake-positive-amazon-reviews/

    http://www.crunchgear.com/2009/01/17/belkin-paying-65-cents-for-good-reviews-on-newegg-and-amazon/

    Everyone is blaming Belkin, who certainly deserves the lion’s share of blame, but I still believe Amazon should do a better job of policing mturk for these spam tasks that violate their stated policy anyway.

  20. Sam Deskin says:

    August 14, 2008 at 12:58 pm

    That is why I really believe they have to want to do these things … out of boredom.
    —-

    Sort of.

    1) For filler during TV time (during commercials, stc.)

    2) To truly help science (surveys, etc.)

  22. hi….i was working in amazon(mtruk).i made around $1000 they paid one time but the next time when i requested for 2nd payment and i had balance of $240 then they suspended my account they didnt pay any amount ,my account approval rate was 98% .they took all the money ..they are thief …dnt trust them any time they can suspend your account without any reason .my amazon worker id-A2LG0SJYIUIRNC..I lost total 400+240=640$

  23. Hi,
    I’ve done various reviews on Turks endeavoring to discover why they do all these odd ball assignments and whats most intriguing is how much turk/errand conduct impersonates regular web surfing by and large less about profiting.

    Before you ask, why do that for 3 pennies, ask yourself, why do you partake in online journals, informal communities and everything else on the web. What do you receive in return?

    In my Valley of Turks post, I think plainly not all turks do it for the cash. Its a test, its an amusement. The genuine test isn’t the work they do its finding shoddy simple to do work that is a simple penny or buck or two.

Comments are closed.