The seedy side of Amazon's Mechanical Turk

I mostly side with Lukas and Panos on the fantastic potential of Amazon’s Mechanical Turk, a crowdsourcing service specializing in tiny payments for simple tasks that require human brainpower, like labeling images. Within the field of computer science alone, this type of service will revolutionize how empirical research is done in communities from CHI to SIGIR, powering unprecedented speed and scale at low cost (here are two examples). My guess is that the impact will be even larger in the social sciences; already, a number of folks in Yahoo’s Social Dynamics research group have started running studies on mturk. (A side question is how university review boards will react.)

However there is a seedier side to mturk, and I’m of two minds about it. Some people use the service to hire sockpuppets to enter bogus ratings and reviews about their products and engage in other forms of spam. (Actually this appears to violate mturk’s stated policies.)

For example, Samuel Deskin is offering up to ten cents to turkers willing to promote his new personalized start page samfind.

EARN TEN CENTS WITH THE BONUS – EASY MONEY – JUST VOTE FOR US AND COMMENT ABOUT US

EARN FOUR CENTS IF YOU:

1. Set up an anoymous email account likke gmail or yahoo so you can register on #2 anonymously

2. Visit http://thesearchrace.com/signup.php and sign up for an account – using your anonymous email account.

3. Visit http://www.thesearchrace.com/recent.php and vote for:

samfind

By clcking “Pick”

SIX CENTS BONUS:

4. Visit the COMMENTS Page on The Search Race, it is the Button Right Next to “Picks” on this page: http://www.thesearchrace.com/recent.php and

5. Say something awesome about samfind (http://samfind.com) on The Search Race’s Comments page.

Make sure to:

1. Tell us that you Picked us.
2. Copy and Paste the Comment you typed on The Search Race’s Comment page here so we know you wrote it and we will give you the bonus!

In fact, Deskin is currently offering bounties on mturk for a number of different spammy activities to promote his site. On the other hand, what Deskin is doing is not illegal and is arguably not all that different than paying PRWEB to publish his rah-rah press release (Start-up, samfind, Launches Customizable Startpage to Compete with Google, Yahoo & MSN, Los Angeles, California (PRWEB) August 4, 2008). And I have to at least give him credit for offering the money under his own name.

Another type of task on mturk involves taking a piece of text and paraphrasing it so that the words are different but the meaning remains the same. Here is an example:

Paraphrase This Paragraph

Here’s the original paragraph:

You’re probably wondering how to apply a wrinkle filler to your skin. The good news is that it’s easy! There are a number of different products on the market for anti aging skin care. Each one comes with its own special application instructions, which you should always make sure to read and carefully follow. In general, however, most anti aging skin care products are simply applied to the skin and left to soak in.

Requirements:
1. Use the same writing style as much as possible.
2. Vary at least 50% of the words and phrases – but keep the same concepts. Use obviously different sentences! Your paragraph should not be just a copy of the first with a few word replacements.
3. Any keywords listed in bold in the above paragraph must be included in your paraphrase.
4. The above paragraph contains 75 words… yours must contain at least 64 words and not more than 101 words.
5. Write using American English.
6. No obvious spelling or grammar mistakes. Please use a spell-checker before submitting. A free online spell checker can be found at www.spellcheck.net.

If you find it easier to paraphrase sentence-by-sentence, then do that. Please do not enter anything in the textbox other than your written paragraph. Thanks!

I have no direct evidence, but I imagine such a task is used to create splogs (I once found what seems like such a “paraphrasing splog”), ad traps, email spam, or other plagiarized content.

It’s possible that paid spam is hitting my blog (either that or I’m overly paranoid). I’m beginning to receive comments that are almost surely coming from humans, both because they clearly reference the content of the post and because they pass the re-captcha test. However, the author’s URL seems to point to an ad trap. I wonder if these commenters (who are particularly hard to catch — you have to bother to click on the author URL) are paid workers of some crowdsourcing service?

Can and should Amazon try to filter away these kinds dubious uses of Mechanical Turk? Or is it better to have this inevitable form of economic activity out in the open? One could argue that at least systems like mturk impose a tax on pollution and spam, something long argued as an economic force to reduce spam.

My main objection to these activities is the lack of disclosure. Advertisements and press releases are paid for, but everyone knows it, and usually the funding source is known. However, the ratings, reviews, and paraphrased text coming out of mturk masquerade as authentic opinions and original content. I absolutely want mturk to succeed — it’s an innovative service of tremendous value, one of many to come out of Amazon recently — but I believe Amazon is risking a minor PR backlash by allowing these activities to flow through its servers and by profiting from them.

34 thoughts on “The seedy side of Amazon's Mechanical Turk”

  1. Hello David,

    I see you found our HIT on mturk. We do use mturk for marketing purposes and have found it a great way to get people to take the first step of visiting our website and then to write something about it. People may do one, but to get both done (sometimes) requires asking for their assistance and providing an incentive – no matter how small.

    Frankly, we have had issues with traction and using mturk has also given us access to an audience that is not unlike our core users – people who want easy access to the websites they use – many of the mturk users are now users of samfind because of their introduction to our website on mturk.

    It has been a great resource to us and I think it is a completely legitimate way to get people to take that extra step to help promote a company.

    BTW – have you tried out samfind??

    Take care,

    Sam

  2. Sam, thanks for commenting here so quickly — impressive web monitoring. I agree there is a blurry line between legitimate and illegitimate marketing; I still think this crosses a line but that’s my personal opinion.

    It’s also true that such marketing tactics are unlikely to succeed unless you have a good product.

  3. what amazes me about MT is not that it is used for spamming (and I don’t think it’s possible to stop it, except in very explicit spamming cases), it’s that people are willing to do a job that takes at least a few minutes for a few cents. that’s way below the minimum wage!

  4. I would imagine that in some areas of the world those would be fine wages.
    Just as DiggIt was subjected to organized groups of supposedly “independent” clickers and commentators, MTurk will be used to organize supposedly disinterested opinion-givers who spread a particular viewpoint in various blogs and forums.
    We have scripted “news” and various “independent” public interest groups and I guess we are saddled with organized spamlike blog commentators busily grinding away on their axe.

  5. “Some people use the service to hire sockpuppets to enter bogus ratings and reviews about their products and engage in other forms of spam. (Actually this appears to violate mturk’s stated policies.)”

    Given that Amazon regularly removes negative reviews at the author’s request, how is this any worse?

  6. Interesting post. At Dolores Labs, we get a large number of inquiries from people hoping to do this type of splogging/grey-area marketing.

    I’m not sure this works so well. We’ve never done anything quite this brazen but as soon as we start to hit a grey-area (I believe just asking someone to sign up for anything technically violates Amazon.com’s terms of service) we find it very hard to recruit people. The Turker community seems to take a very negative view of these kinds of tasks, and companies doing this type of thing seem to very quickly get a bad reputation.

    I’ve seen far more questionable stuff on Turk. From time to time there are companies offering cash to people willing to scan in their credit card statements/phone bills, etc. (I can see how this could be a useful data for many reasonable applications, but it seems quite likely to also be used for more nefarious purposes).

  7. David – interesting question, but I don’t have a very strong opinion. The tasks might be less of a problem than you would expect from browsing AMT because they tend to pay the highest and often show up when you sort by price.

    I think most high volume turkers skip over them. AMT could probably use their own community to filter them quite easily. If you look on the turk message boards you sometimes see the turkers discussing the fact that they’ve reported offending tasks to Amazon, but lamenting the fact that Amazon takes no action. I think it would be in Amazon’s interest to remove them, but I’m not sure they should be blamed for having bigger issues to deal with.

  8. MTurk is indeed a fun little ecosystem to study and a great accessible tool for someone if their need fits with what turks will do.

    I’ve done numerous studies on Turks attempting to find out why they do all these odd ball tasks and whats most interesting is how much turk/task behavior mimics everyday web surfing in general less about making money.

    Before you ask, why do that for 3 cents, ask yourself, why do you participate in blogs, social networks and everything else on the web. What do you get out of it?

    In my Valley of Turks post, I think its clear that not all turks do it for the money. Its a challenge, its a game. The real challenge isn’t the work they do its finding cheap easy to do work thats an easy cent or buck or two.

    http://www.floozyspeak.com/blog/archives/2008/08/valley_of_the_t.html

    Ya gotta remember this is like a backdoor don’t really care about experiment from Amazon. It arrived on the scene in 2006, and it went through various stages of good PR mostly bad PR back then and only now have they started to look at it again and attempt to spruce it up. Microsoft and Google have had like systems, Microsofts is pretty much gone, Google’s is history but I hear its back again in Russia somewhere.

    I agree with Lukas in that the Turk community sniffs out scams faster than you and I would any day. They live on Turk scouring it daily for golden gems of work in the ruff of it all.

  9. Update: Someone uncovered a name-brand company (Belkin) apparently buying positive reviews (ironically on Amazon.com) using Mechanical Turk.

    http://www.thedailybackground.com/2009/01/16/exclusive-belkins-development-rep-is-hiring-people-to-write-fake-positive-amazon-reviews/

    http://www.crunchgear.com/2009/01/17/belkin-paying-65-cents-for-good-reviews-on-newegg-and-amazon/

    Everyone is blaming Belkin, who certainly deserves the lion’s share of blame, but I still believe Amazon should do a better job of policing mturk for these spam tasks that violate their stated policy anyway.

  10. Sam Deskin says:

    August 14, 2008 at 12:58 pm

    That is why I really believe they have to want to do these things … out of boredom.
    —-

    Sort of.

    1) For filler during TV time (during commercials, stc.)

    2) To truly help science (surveys, etc.)

  11. hi….i was working in amazon(mtruk).i made around $1000 they paid one time but the next time when i requested for 2nd payment and i had balance of $240 then they suspended my account they didnt pay any amount ,my account approval rate was 98% .they took all the money ..they are thief …dnt trust them any time they can suspend your account without any reason .my amazon worker id-A2LG0SJYIUIRNC..I lost total 400+240=640$

  12. Hi,
    I’ve done various reviews on Turks endeavoring to discover why they do all these odd ball assignments and whats most intriguing is how much turk/errand conduct impersonates regular web surfing by and large less about profiting.

    Before you ask, why do that for 3 pennies, ask yourself, why do you partake in online journals, informal communities and everything else on the web. What do you receive in return?

    In my Valley of Turks post, I think plainly not all turks do it for the cash. Its a test, its an amusement. The genuine test isn’t the work they do its finding shoddy simple to do work that is a simple penny or buck or two.

  13. Thanks Lukas. Very interesting.

    What’s your view on whether Amazon should actively de-list these kinds of tasks that violate their policies?

  14. In order to hookup having a female on the web, you’ll discover that there are various solutions available directory that enable you to hunt for and contact a woman. You can find the sort of internet dating partner you’re looking for and commence exploring information for that perfect match. Among the most popular websites are affairhookup.com and eharmony.com. These sites give guidance for both women and men. Additionally, they provide information about how and where to fulfill a unique girl.

    Some web sites will likely let you talk with a woman that you’re interested in. These free hookup websites permit end users to browse through thousands of user profiles and send good friend demands to the people they’re considering. This way, you can make your choice dependent by yourself choices. A few of these websites will even have online video talk choices for you to see their looks. These facilities are a fun way to hookup with women on the internet.

    If you’re trying to find a location to chitchat and hookup with a lady, you can test internet sites that supply free of charge top quality balances. The main advantage of these internet sites is simply because they are anonymous and have high accomplishment costs. Many of them have reside cam features and a 24-hr hanging around period. They’re also great for locating a particular date within your area. If you’re seeking a woman in a rush, you may pick a courting website that may be handy for you.

    There are various reasons to hookup with females online. If you’re searching for a companion who shares the same intimate choices when you do, these internet websites are ideal for you. Many of them even offer you sex recommendations as well as other information and facts that can help you will be making your pursuit simpler. No matter if you’re looking for a companion or simply to have entertaining, you’ll find the excellent lover on the internet. It’s as easy as that.

    There are a selection of ways to satisfy women on-line. Start by looking for single ladies in your area who would like to try a similar things when you. These internet sites are simple to use and sometimes offer individuality assessments. If you’re a male, you should look at a free of charge internet dating website. Most totally free websites are only frauds. There’s a variety of diverse sites where you may connect with other individuals. Once you find the correct girl, it’s time and energy to chitchat!

    A lot of men have realized that meeting females online is increasingly simple. You can even talk with a woman who’s interested in your interests, without the need of endangering your security. A female may well be a excellent in shape for you if she’s considering males with the exact same pursuits as you may. The best thing about online dating is that you could fulfill women that are prepared to particular date you. It’s a wonderful way to become familiar with women that reveal the same desired goals and ideals.

  15. If you wish to hookup with a lady online, you’ll realize that there are numerous services offered Learn More Here that enable you to seek out and contact a female. You can find the level of courting lover you’re searching for and initiate surfing around user profiles for that perfect match. Among the most well-known internet sites are affairhookup.com and eharmony.com. These sites give suggestions for men and women. Additionally, they provide facts about how and where to fulfill an exclusive young lady.

    Some web sites will likely permit you to speak to a female that you’re enthusiastic about. These totally free hookup websites let users to browse through thousands of information and give close friend needs to the people they’re interested in. In this way, you can make the choice centered on your own choices. Some of these internet sites will even have video chitchat options that you can see their appearance. These types of services are an easy way to hookup with women on the web.

    If you’re searching for a spot to chitchat and hookup with a woman, you can consider internet sites that provide totally free premium balances. The main advantage of these websites is that they are anonymous and have high good results charges. Most of them have reside cam characteristics and a 24-hr hanging around period of time. They’re also perfect for getting a particular date in your town. If you’re seeking a woman in a big hurry, you may go with a courting website that is handy for you personally.

    There are many excellent reasons to hookup with females on-line. If you’re trying to find a lover who shares exactly the same intimate tastes when you do, these internet websites are perfect for you. Many of them even supply sexual intercourse tips and other information which can help you will make your pursuit less difficult. Whether or not you’re trying to find a friend or perhaps to obtain enjoyable, you’ll find the perfect spouse on-line. It’s as easy as that.

    There are a variety of ways in order to meet women on the web. You can begin by trying to find individual women in the area who would like to try exactly the same stuff as you may. These internet websites are simple to use and frequently provide character checks. If you’re a person, you should think of a no cost internet dating internet site. Most free websites are only frauds. There’s an array of different internet sites where you may link up with other individuals. Once you find the best female, it’s time for you to talk!

    A lot of men have found that conference females on the internet is incredibly easy. You can also speak to a girl who’s thinking about your interests, with out taking a chance on your basic safety. A girl may well be a good in shape for you if she’s enthusiastic about males using the same passions as you. The greatest thing about online dating services is you can meet up with ladies who are willing to particular date you. It’s a terrific way to familiarize yourself with ladies who talk about a similar desired goals and ideals.

  16. If you wish to hookup having a girl on-line, you’ll learn that there are numerous services available directory that allow you to search for and contact a lady. You can find the sort of online dating partner you’re looking for and begin exploring profiles for that ideal match. Probably the most popular websites are affairhookup.com and eharmony.com. These sites offer guidance for women and men alike. Additionally they supply facts about how and where in order to meet a particular young lady.

    Some web sites will likely enable you to speak to a lady that you’re considering. These free hookup websites allow customers to read through a huge number of user profiles and send out friend demands to those they’re considering. Using this method, you could make your choice centered all by yourself tastes. Some of these web sites may even have video chat alternatives so that you can see their appears. These services are a great way to hookup with women on the internet.

    If you’re searching for a spot to talk and hookup having a woman, you can test web sites that provide cost-free top quality credit accounts. The benefit of these websites is simply because they are anonymous and have high accomplishment costs. Most of them have are living cam features and a 24-60 minutes holding out period of time. They’re also great for finding a day within your area. If you’re looking for a female in a big hurry, it is possible to choose a dating website that may be handy to suit your needs.

    There are many excellent reasons to hookup with women on the web. If you’re looking for a companion who shares a similar sexual choices as you may do, these websites are fantastic for you. Some of them even supply sexual activity suggestions as well as other info which can help you make your research simpler. Whether or not you’re trying to find a companion or just to obtain fun, you’ll discover the perfect companion on the internet. It’s as elementary as that.

    There are lots of ways to fulfill ladies on-line. You could start by trying to find single women in your neighborhood who have an interest in a similar things as you may. These websites are simple to use and often offer character tests. If you’re a male, you should look at a no cost dating online site. Most totally free websites are only ripoffs. There’s a variety of different internet sites where you may hook up with other individuals. Once you discover the proper woman, it’s time and energy to talk!

    Countless men have found that conference women on the web is increasingly simple. Also you can talk to a female who’s thinking about your likes and dislikes, with out taking a chance on your security. A lady generally is a great in shape to suit your needs if she’s interested in gentlemen with the same passions as you may. An important feature about online dating is you can meet women who are willing to time you. It’s a great way to get acquainted with ladies who talk about a similar goals and ideals.

  17. If you would like hookup by using a girl on the web, you’ll find that there are many different providers offered YOURURL.com that enable you to look for and make contact with a lady. You can get the kind of online dating partner you’re searching for and commence surfing around user profiles to the perfect match. Probably the most well-known internet sites are affairhookup.com and eharmony.com. These sites provide guidance for men and women. Additionally they offer facts about where and how to satisfy an exclusive young lady.

    Some internet sites will even allow you to talk to a woman that you’re thinking about. These cost-free hookup websites enable users to search through thousands of profiles and send close friend needs to people they’re enthusiastic about. In this way, you possibly can make your choice centered by yourself choices. Some of these internet sites may also have video conversation choices that you should watch their appearance. These facilities are a great way to hookup with females on the internet.

    If you’re looking for a place to chat and hookup with a woman, you can try sites offering free high quality profiles. The benefit of these internet sites is simply because they are anonymous and have higher success prices. A lot of them have live cam functions plus a 24-hr waiting around period of time. They’re also great for locating a day within your city. If you’re trying to find a female very quickly, it is possible to choose a online dating website that may be hassle-free for yourself.

    There are various good reasons to hookup with women on the web. If you’re looking for a spouse who reveals a similar intimate preferences while you do, these internet sites are perfect for you. A number of them even provide gender suggestions along with other information and facts that can help you will be making your pursuit much easier. Regardless of whether you’re trying to find a companion or perhaps to get entertaining, you’ll get the excellent companion on-line. It’s as elementary as that.

    There are a selection of ways in order to meet women on the web. You can start by seeking single females in your town who are interested in exactly the same points while you. These websites are simple to use and frequently offer personality checks. If you’re a person, you should look at a no cost dating online web site. Most free websites are merely scams. There’s a variety of diverse internet sites where you may hook up with other people. Once you see the proper female, it’s time for you to talk!

    Lots of men have realized that reaching ladies on the web is increasingly simple. You may also talk with a female who’s interested in your likes and dislikes, without the need of taking a chance on your security. A female might be a very good match for yourself if she’s considering guys using the same passions while you. An important feature about online dating is that you could fulfill ladies who are able to particular date you. It’s a terrific way to become familiar with women who share the same desired goals and ideals.

Leave a Reply

Your email address will not be published. Required fields are marked *