Comments on: The seedy side of Amazon's Mechanical Turk http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/ Musings of a computer scientist and Yahoo on prediction markets, gambling, and estimating the odds of everything Fri, 12 Mar 2010 21:58:30 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Web Tasarim Adana http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-2001 Web Tasarim Adana Tue, 09 Mar 2010 16:05:08 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-2001 This article helped for me. Thank you, web tasarim adana This article helped for me. Thank you, web tasarim adana

]]> By: Manigandan K http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-343 Manigandan K Sat, 14 Mar 2009 11:24:27 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-343 Crowdsourcing relives most online community from PTC/PTR scams sites. Amazon's mturk.com provides work to Crowds is appreciable. thanks Crowdsourcing relives most online community from PTC/PTR scams sites. Amazon’s mturk.com provides
work to Crowds is appreciable.
thanks

]]> By: David Pennock http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-342 David Pennock Wed, 18 Feb 2009 14:02:18 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-342 Thanks Ed, congrats, looks great. Thanks Ed, congrats, looks great.

]]> By: Ed Chi http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-341 Ed Chi Sat, 14 Feb 2009 08:45:42 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-341 We published a paper in CHI2008 about how to use mturk for HCI research: Aniket Kittur, Ed H. Chi, Bongwon Suh. <a href="http://www-users.cs.umn.edu/~echi/papers/2008-CHI2008/2008-02-mech-turk-online-experiments-chi1049-kittur.pdf" rel="nofollow">Crowdsourcing User Studies With Mechanical Turk</a>. In Proceedings of the ACM Conference on Human-factors in Computing Systems (CHI2008), pp.453-456. ACM Press, 2008. Florence, Italy. We published a paper in CHI2008 about how to use mturk for HCI research:

Aniket Kittur, Ed H. Chi, Bongwon Suh. Crowdsourcing User Studies With Mechanical Turk. In Proceedings of the ACM Conference on Human-factors in Computing Systems (CHI2008), pp.453-456. ACM Press, 2008. Florence, Italy.

]]> By: David Pennock http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-340 David Pennock Sun, 18 Jan 2009 18:41:50 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-340 Update: Someone uncovered a name-brand company (Belkin) apparently buying positive reviews (ironically on Amazon.com) using Mechanical Turk. http://www.thedailybackground.com/2009/01/16/exclusive-belkins-development-rep-is-hiring-people-to-write-fake-positive-amazon-reviews/ http://www.crunchgear.com/2009/01/17/belkin-paying-65-cents-for-good-reviews-on-newegg-and-amazon/ Everyone is blaming Belkin, who certainly deserves the lion's share of blame, but I still believe Amazon should do a better job of policing mturk for these spam tasks that violate their stated policy anyway. Update: Someone uncovered a name-brand company (Belkin) apparently buying positive reviews (ironically on Amazon.com) using Mechanical Turk.

http://www.thedailybackground.com/2009/01/16/exclusive-belkins-development-rep-is-hiring-people-to-write-fake-positive-amazon-reviews/

http://www.crunchgear.com/2009/01/17/belkin-paying-65-cents-for-good-reviews-on-newegg-and-amazon/

Everyone is blaming Belkin, who certainly deserves the lion’s share of blame, but I still believe Amazon should do a better job of policing mturk for these spam tasks that violate their stated policy anyway.

]]> By: David Pennock http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-339 David Pennock Wed, 10 Sep 2008 13:02:09 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-339 A similar in spirit article on ReadWriteWeb on Aug 29 suggests that an "mturk spam" meme is in the air: <a href="http://www.readwriteweb.com/archives/amazons_mechanical_turk_used_for_fraud.php" rel="nofollow">The Dark Side to Mechanical Turk</a> A similar in spirit article on ReadWriteWeb on Aug 29 suggests that an “mturk spam” meme is in the air:

The Dark Side to Mechanical Turk

]]> By: David Pennock http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-338 David Pennock Tue, 09 Sep 2008 20:16:27 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-338 Related development: Brendan O’Connor (Dolores Labs), together with Rion Snow, Dan Jurafsky, and Andrew Ng (Stanford) published a paper showing how mturk can be used to quickly and cheaply obtain high quality machine learning data: http://blog.doloreslabs.com/2008/09/amt-fast-cheap-good-machine-learning/ Related development:

Brendan O’Connor (Dolores Labs), together with Rion Snow, Dan Jurafsky, and Andrew Ng (Stanford) published a paper showing how mturk can be used to quickly and cheaply obtain high quality machine learning data:

http://blog.doloreslabs.com/2008/09/amt-fast-cheap-good-machine-learning/

]]> By: Dan http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-337 Dan Wed, 03 Sep 2008 17:33:15 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-337 MTurk is indeed a fun little ecosystem to study and a great accessible tool for someone if their need fits with what turks will do. I've done numerous studies on Turks attempting to find out why they do all these odd ball tasks and whats most interesting is how much turk/task behavior mimics everyday web surfing in general less about making money. Before you ask, why do that for 3 cents, ask yourself, why do you participate in blogs, social networks and everything else on the web. What do you get out of it? In my Valley of Turks post, I think its clear that not all turks do it for the money. Its a challenge, its a game. The real challenge isn't the work they do its finding cheap easy to do work thats an easy cent or buck or two. http://www.floozyspeak.com/blog/archives/2008/08/valley_of_the_t.html Ya gotta remember this is like a backdoor don't really care about experiment from Amazon. It arrived on the scene in 2006, and it went through various stages of good PR mostly bad PR back then and only now have they started to look at it again and attempt to spruce it up. Microsoft and Google have had like systems, Microsofts is pretty much gone, Google's is history but I hear its back again in Russia somewhere. I agree with Lukas in that the Turk community sniffs out scams faster than you and I would any day. They live on Turk scouring it daily for golden gems of work in the ruff of it all. MTurk is indeed a fun little ecosystem to study and a great accessible tool for someone if their need fits with what turks will do.

I’ve done numerous studies on Turks attempting to find out why they do all these odd ball tasks and whats most interesting is how much turk/task behavior mimics everyday web surfing in general less about making money.

Before you ask, why do that for 3 cents, ask yourself, why do you participate in blogs, social networks and everything else on the web. What do you get out of it?

In my Valley of Turks post, I think its clear that not all turks do it for the money. Its a challenge, its a game. The real challenge isn’t the work they do its finding cheap easy to do work thats an easy cent or buck or two.

http://www.floozyspeak.com/blog/archives/2008/08/valley_of_the_t.html

Ya gotta remember this is like a backdoor don’t really care about experiment from Amazon. It arrived on the scene in 2006, and it went through various stages of good PR mostly bad PR back then and only now have they started to look at it again and attempt to spruce it up. Microsoft and Google have had like systems, Microsofts is pretty much gone, Google’s is history but I hear its back again in Russia somewhere.

I agree with Lukas in that the Turk community sniffs out scams faster than you and I would any day. They live on Turk scouring it daily for golden gems of work in the ruff of it all.

]]> By: David Pennock http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-336 David Pennock Fri, 29 Aug 2008 18:00:13 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-336 Thanks Lukas. Thanks Lukas.

]]> By: Lukas Biewald http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/comment-page-1/#comment-335 Lukas Biewald Thu, 28 Aug 2008 21:40:34 +0000 http://blog.oddhead.com/2008/08/13/the-seedy-side-of-amazons-mechanical-turk/#comment-335 David - interesting question, but I don't have a very strong opinion. The tasks might be less of a problem than you would expect from browsing AMT because they tend to pay the highest and often show up when you sort by price. I think most high volume turkers skip over them. AMT could probably use their own community to filter them quite easily. If you look on the turk message boards you sometimes see the turkers discussing the fact that they've reported offending tasks to Amazon, but lamenting the fact that Amazon takes no action. I think it would be in Amazon's interest to remove them, but I'm not sure they should be blamed for having bigger issues to deal with. David – interesting question, but I don’t have a very strong opinion. The tasks might be less of a problem than you would expect from browsing AMT because they tend to pay the highest and often show up when you sort by price.

I think most high volume turkers skip over them. AMT could probably use their own community to filter them quite easily. If you look on the turk message boards you sometimes see the turkers discussing the fact that they’ve reported offending tasks to Amazon, but lamenting the fact that Amazon takes no action. I think it would be in Amazon’s interest to remove them, but I’m not sure they should be blamed for having bigger issues to deal with.

]]>