August 20, 2010, Sunnyvale, CA

The last decade has witnessed the emergence of enormous scale artifacts resulting from the independent action of hundreds of millions of people; for example, web repositories, social networks, mobile communication patterns, and consumption in “limitless” stores… the first Research and Analysis of Tail phenomena Symposium (RATS)… will explore the different computational, statistical, and modeling problems related to tail phenomena… We are particularly encouraging summer interns in any of the Bay Area research centers to join us in the event.
We will start with a video welcome by Chris Anderson (Wired), followed by a series of invited talks by Michael Mitzenmacher (Harvard), Aaron Clauset (Univ. of Colorado), Neel Sundaresan (eBay), Sharad Goel (Yahoo! Research, NY) and Michael Schwarz (Yahoo! Research, CA).

We invite proposals for short (20 minute) talks from students and researchers working in the area.

CrowdCof2010: 1st Annual Conference on the Future of Distributed Work

October 4, 2010, San Francisco, CA

Were you crowdsourcing before it was cool? We want to hear about your projects.

We are inviting submissions on all topics regarding crowdsourcing, including:

• Past, present, and future of crowdsourcing
• Quality assurance and metrics
• Social and economic implications of crowdsourcing
• Task design/Worker incentives
• Innovative projects, experiments, and applications
• Submission Guidelines

Deadline: Sept. 1

CrowdConf will bring together researchers, technologists, outsourcing entrepreneurs, legal scholars, and artists for the first time to discuss how crowdsourcing is transforming human computation and the future of work.

Confirmed Speakers:
Sharon Chirella: Vice President, Amazon Mechanical Turk
Tim Ferriss : Author, The 4-Hour Work Week
David Alan Grier: Author, When Computers Were Human
Barney Pell: Partner, Search Strategist, and Evangelist, Microsoft
Maynard Webb: CEO, LiveOps
Jonathan Zittrain: Professor of Law and Computer Science, Harvard

Computational Social Science and the Wisdom of Crowds Workshop at NIPS 2010

December 10th or 11th, 2010, Whistler, Canada

We welcome contributions on theoretical models, empirical work, and everything in between, including but not limited to:

• Automatic aggregation of opinions or knowledge
• Prediction markets / information markets
• Incentives in social computation (e.g., games with a purpose)
• Studies of events and trends (e.g., in politics)
• Analysis of and experiments on distributed collaboration and consensus-building, including crowdsourcing (e.g., Mechanical Turk) and peer-production systems (e.g., Wikipedia and Yahoo! Answers)
• Group dynamics and decision-making
• Modeling network interaction content (e.g., text analysis of blog posts, tweets, emails, chats, etc.)
• Social networks

[Covers] computational social science… [and] social computing… with an emphasis on the role of
machine learning…

Deadline for submissions: Friday October 8, 2010

Scott Aaronson is a skeptic, to say the least. He made an amazing public bet to demonstrate his confidence. He pledged that if Deolalikar wins the $1 million prize, Aaronson will top it off with $200,000 of own money. Even more amazing: Aaronson made the bet without even reading the proof. [Update: I should have said "without reading the proof in detail": see comments] (Perhaps more amazing still: a PC World journalist characterized Aaronson’s stance as “noncommittal” without a drip of sarcasm.) [Hat tip to Dan Reeves.]

As Aaronson explains:

The point is this: I really, really doubt that Deolalikar’s proof will stand. And while I haven’t studied his long, interesting paper and pinpointed the irreparable flaw… I have a way of stating my prediction that no reasonable person could hold against me: I’ve literally bet my house on it.

Aaronson is effectively offering infinite odds [Update: actually more like 2000/1 odds: see comments] that the question “P=NP?” will not be resolved in the near future. Kevin McCurley and Ron Fagin made a different (conditional) bet: Fagin offered 5/1 odds (at much lower stakes) that if the question is resolved in 2010, the answer will be P≠NP. Bill Gasarch says that he, like Aaronson, would bet that the proof is wrong… if only he were a betting man. Richard Lipton recounts a discussion about the odds of P=NP with Ken Steiglitz.

But beyond a few one-off bets and declarations, where is the central market where I can bet on P=NP? I don’t even necessarily want in on the action, I just want the odds. (Really!)

My first thought was the Foresight Exchange. It does list one related contract — Good 3SAT Algorithm by 2020 — which should presumably go to zero if Deolalikar’s proof is correct. It hasn’t budged much, consistent with skepticism (or with apathy). My second thought was the PopSci Predictions Exchange (PPX), though sadly it has retired. InklingMarkets has a poll about whether P=NP will be resolved before the other Clay Institute prize questions, but not about how it will be resolved or the odds of it happening. (The poll is one of several markets sponsored by the Woodrow Wilson Center’s Science and Technology Innovation Program — hat tip to Vince Conitzer.) I don’t see anything at longbets, and anyway longbets doesn’t provide odds despite it’s name.

In 1990 Robin Hanson provocatively asked: Could gambling save science?. That question and his thoughtful answers inspired a number of people, including me, to study prediction markets. Indeed, the Foresight Exchange was built largely in his image. P=NP seems one of the most natural claims for any scitech prediction market.

All these years later, when I really need my fix, I can’t seem to get it!

If it wasn’t clear in my original post, I found Aaronson’s bet incredibly useful and I am thrilled he did it. I believe he should be commended: his bet was exactly what more scientists should do. Scientists should express their opinion, and betting is a clear, credible, and quantitative way to express it. It would be as shame if some of the negative reactions caused him or others not to make similar bets in the future.

I just wish there were a central place to make bets on scientific claims and follow the odds in the vision of Robin Hanson, rather than every scientist having to declare their bet on their own individual blogs.

In the eyes of a casino, a sharp is indistinguishable from a cheat. More than boring, this seems fundamentally unfair and unsustainable, inviting disguise. It also turns off a wealthy, influential, and game-loving segment of the population.

What I want: A casino by geeks for geeks that celebrates innovation, encourages cleverness, welcomes gadgets and wifi, and fosters hacking, outwit, and outplay.

At least one casino has seen the light. The M Casino in Las Vegas is parterning with Cantor Fitzgerald to support in-game sports betting with few rules and caps, inviting sharps to, more or less, “bring it on”.

…gamblers can bet on the game even during play, accepting ever-changing point-spreads and odds. They can invest money on a Knicks foul shot going through the hoop or a Dodger getting to first base — contending with ever-evolving odds.

More critically, bettors can create hedges while jumping in and out of positions. But instead of buying into the fast-breaking moves of Microsoft, they’re betting on the Mariners’ impending fortunes.

This form of wagering is new to Las Vegas but old-hat in other markets.

…unlike in most other casinos, laptop computers are welcome.

…”The M wants [sharp bettors] to be there,” believes [professional sports-bettor Steve] Fezzik. “They want your information, and that’s a progressive attitude.”

Kudos to M for taking a chance on a more interesting future and to Cantor for making it happen.

What Cantor is debuting may not be a whole lot more than betfair indoors, but it’s a long overdue start. Here’s to hoping we see even more innovation, including smarter and more expressive markets.

Dan and I built a second prototype in PHP. It was even uglier but about twice as fast and somewhat useable on a small scale (at least by user willing/able to formulate their own propositions in PHP). Yet it still wasn’t good enough to serve thousands of users accustomed to simplicity and speed.

The final live version of Predictalot was not only pleasing to the eye — thanks to Sudar, Navneet, and Tom — but pleasingly fast, due almost entirely to the heroic efforts of Mridul M who wrote a mini PHP parser inside of java and baked in a number of datbase and caching optimizations.

It seems that high-level programming languages haven’t climbed high enough. To field a fairly constrained web app that looks good and works well, we benefit greatly from having at least three specialists, for the app front end, the app back end, and the platform back end (apache, security, etc.).

Here’s a challenge to the programming language community: anything I can whip up in Mathematica I should be able to run at web scale. Math majors should be able to create Predictalot. Dan and I can mock up the basic idea of Predictalot but it still takes tremendous talent, time, and effort to turn it into a professional looking and well behaved system.

The core market math of Predictalot — a combinatorial version of Hanson’s LMSR market maker — involves summing thousands of e^x terms. Here we are in the second decade of the new millenium and in order for a sum of exponentials to execute quickly and without numeric overflow, we had to work out a transformation to conduct all our summations in log space. In other words, programming still requires me to think about how my machine represents my number. That shouldn’t qualify as “high level” thinking in 2010.

I realize I may be naively asking too much. Solving the challenge fully is AI-complete. Still, while we’re making impressive strides in artificial intelligence, programming feels much the same today as it did twenty years ago. It still requires learning specialized tricks, arcane domain knowledge, and optimizations honed only over years of experience, and the most computationally intensive applications still require that extra compilation step (i.e., it’s still often necessary to use C or Java over PHP, Perl, Python, or Ruby).

Some developments hardly seem like progress. Straightforward HTML markup like border=2 has given way to unweildy CSS like style=”border:2px solid black”. In some ways the need for specialized domain knowledge has gone up, not down.

Visual programming is an oft-tried, though so far largely unsuccessful way to lower the barrier to programming. Pipes was a great effort, but YQL proved more useful and popular. Google just announced new visual developer tools for Android in an attempt to bring mobile app creation to the masses. Content management systems are getting better and broader every day, allowing more and more complex websites to be built with less time touching source code.

I look forward to the day that computational thinking can suffice to create the majority of computational objects. I suspect that day is still fifteen to twenty years away.

Here’s an illustration why. Abe Othman recently alerted me to intrade’s market on where basketball free agent LeBron James will sign, at the time a featured market. Take a look at this screenshot taken 2010/07/07:

The market says there’s between a 42 and 70% chance James will sign with Cleveland, between a 5 and 40% chance he’ll sign with Chicago, etc.

In other words, it doesn’t say much. The spreads between the best bid and ask prices are wide and so its predictions are not terribly useful. We can occasionally tighten these ranges by being smarter about handling multiple outcomes, but in the end low liquidity takes the prediction out of markets.

Even if someone does have information, they may not be able trade on it so may simply go away. (Actually, the problem goes beyond apathy. Placing a limit order is a risk — whoever accepts it will have a time advantage — and reveals information. If there is little chance the order will be accepted, the costs may outweigh any potential gain.)

Enter automated market makers. An automated market maker always stands ready to buy and sell every outcome at some price, adjusting along the way to bound its risk. The market maker injects liquidity, reducing the bid-ask spread and pinpointing the market’s prediction to a single number, say 61%, or at least a tight range, say 60-63%. From an information acquisition point of view, precision is important. For traders, the ability to trade any contract at any time is satisfying and self-reinforcing.

For combinatorial prediction markets like Predictalot with trillions or more outcomes, I simply can’t imagine them working at all without a market maker.

Abe Othman, Dan Reeves, Tuomas Sandholm, and I published a paper in EC 2010 on a new automated market maker algorithm. It’s a variation on Robin Hanson‘s popular market maker called the logarithmic market scoring rule (LMSR) market maker.

Almost anyone who implements LMSR, especially for play money, wonders how to set the liquidity parameter b. I’ve been asked this at least a dozen times. The answer is I don’t know. It’s more art than science. If b is too large, prices will hardly move. If b is too small, prices will bounce around wildly. Moreover, no matter what b is set to, prices will be exactly as responsive to the first dollar as the million and first dollar, counter to intuition.

Our market maker automatically adjusts its level of liquidity depending on trading volume. Prices start off very responsive and, as volume increases, liquidity grows, obviating the need to somehow guess the “right” level before trading even starts.

A side effect is that predictions take the form of ranges, like 60-63%, rather than exact point estimates. We prove that this is a necessary trade off. Any market maker that is path independent and sensitive to liquidity must give up on providing point estimates. In a way, our market maker works more like real bookies who maintain a vig or spread for every outcome.

The market maker algorithm is theoretically elegant and seems more pratical than LMSR in many ways. However I’ve learned many times than nothing can replace implementing and testing a theory with real traders. Final word awaits such a trial. Stay tuned.

More people are playing Predictalot today than Mafia Wars or Zynga Poker… On Yahoo!, that is.

In fact, Predictalot is the #1 game app on Yahoo! Apps by daily count. By monthly count, we are 5th and rising.

A prediction is being made about every three minutes.

Come join the fun.

Yahoo! Lab’s own Predictalot market is now live and waiting for you to place almost any prediction your heart desires about the World Cup in South Africa.

Here are some terribly useful things you can learn this time around. All numbers are subject to change, and that’s kind of the point:

• There’s a 37% chance Brazil and Spain will both make it to the final game; there’s only a 15% chance that neither of them will make it
• There’s is a 1 in 25 chance Portugal will win the cup; 1 in 50 for Argentina
• 42.92% chance that a country that has never won before will win
• 19.07% chance that Australia will advance further than England
• 65.71% chance that Denmark, Italy, Mexico and United States all will not advance to Semifinals
• Follow Predictalot on twitter for more

If you think these odds are wrong, place your virtual wager and earn some intangible bragging rights. You can sell your prediction any time for points, even in the middle of a match, just like the stock market.

There are millions of predictions available, yet I really believe ours is the simplest prediction market interface to date. (Do you disagree, Leslie?) We have an excellent conversion rate, or percent of people who visit the site who go on to place at least one prediction — for March Madness, that rate was about 1 in 5. One of our main goals was to hide the underlying complexity and make the app fast, easy, and fun to use. I personally am thrilled with the result, but please go judge for yourself and tell us what you think.

In the first version of Predictalot, people went well beyond picking the obvious like who will win. For example, they created almost 4,000 “three-dimensional” predictions that compared one team against two others, like “Butler will advance further than Kentucky and Purdue”.

If you’re not sure what to predict, you can now check out the streaming updates of what other people are predicting in your social circle and around the world:

Also new this time, you can join a group and challenge your friends. You can track how you stack up in each of your groups and across the globe. We now provide live match updates right within the app for your convenience.

If you have the Yahoo! Toolbar (if not, try the World Cup toolbar), you can play Predictalot directly from the toolbar without leaving the webpage you’re on, even if it’s Google.

Bringing Predictalot to life has been a truly interdisciplinary effort. On our team we have computer scientists and economists to work out the market math, and engineers to turn those equations into something real that is fast and easy to use. Predictalot is built on the Yahoo! Application Platform, an invaluable service (open to any developer) that makes it easy to make engaging and social apps for a huge audience with built-in distribution. And we owe a great deal to promotion from well-established Yahoo! properties like Fantasy Sports and Games.

We’re excited about this second iteration of Predictalot and hope you join us as the matches continue in South Africa. We invite everyone to join, though please do keep in mind that the game is in beta, or experimental, mode. (If you prefer a more polished experience, check out the official Yahoo! Fantasy Sports World Soccer game.) We hope it’s both fun to play and helps us learn something scientifically interesting.

Read more here, here, and here.

Or watch a screencast of how to play:

1. The Hollywood Stock Exchange is the latest example of life imitating art imitating life. The venerable HSX, the second oldest play-money prediction market (and oldest that anyone has actually heard of) is getting real. You’ll soon be able bet real money on box office returns and deposit your winnings in First Life. The move has been a long time coming — Cantor Fitzgerald purchased HSX in 2001 with the explicit goal of converting it into a real-money exchange — but Cantor was decimated in the September 11 WTC attacks, and the road to regulatory approval has been slow. The real HSX will of course say goodbye to the virtual specialist and the opening weekend adjust, two facets of the game that make it fun to play, but that create significant amounts of (virtual) wealth out of thin air. The Cantor Gaming group is engaged in other interesting initiatives. They are taking over a sportsbook in Las Vegas and turning it into more of a derivatives exchange with live in-game betting, a step toward my dream of a geek-friendly casino. Interestingly, another company called Veriana Networks is close to launching a competing Hollywood derivatives market called the Trend Exchange.
2. I’ve recently seen ads in the elevator in my building for Nadex, the new incarnation of HedgeStreet that was acquired, redesigned, and rebranded as the North American Derivatives Exchange. I haven’t checked in on the market since the rebranding — in fact I can’t remember if my HedgeStreet account transfered over. I wonder how widespread their ad campaign is and how Nadex is doing?
3. Predictalot update: 50,000 people have logged in and 11,000 have made at least one prediction.* They’ve made 91,000 total predictions of 5,500 different kinds. We’ve been written up in NYTimes, VentureBeat, L’Atelier (“C’est ce que fait Yahoo!, avec Predictalot.”), and Wired. People are saying everything from “wicked fun”, “great idea”, and “love the game” to “disaster”. Either way, people care. It’s been a ton of fun and its popularity has wildly surpassed my expectations. The final four is this weekend. Then not much time to get in as many improvements as we can before the World Cup.
4. If you’re not aware, there is a new prediction market mailing list that strives to be open, transparent, objective, and independent under some simple ground rules. I encourage you to join it. I abandoned the old PM list owned by John Maloney for a number of reasons. The old list clearly operates in a payola style — sponsors of Maloney’s events receive prominent billing. That might be ok, except Maloney bills his list and his events as open, non-profit affairs, yet he charges quite a bit and is evasive about financial details (actually he says he doesn’t keep records). Over the years, Maloney has asked me/Yahoo! to sponsor his events several times, occasionally berating me when the answer is no. He’s made some questionable wikipedia edits to highlight his group. Still, I’ve gone to some of his events and find that very good people go there and give interesting talks, and Maloney is very reasonable and personable face to face. My final straw came when Maloney censored me on his list for criticizing him, admittedly in a snarky way.** So I joined with Emile, Bo, Forrest, Justin, Jed, Adam, and others to start afresh.
5. The 2010 ACM Conference on Electronic Commerce to be held June 7-11 at Harvard features at least six papers on prediction markets:
   • L. Jian and R. Sami, Aggregation and Manipulation in Prediction Markets: Effects of Trading Mechanism and Information Distribution
   • K. Iyer, R. Johari and C. Moallemi, Information Aggregation in Smooth Markets
   • A. Othman, D. Pennock, D. Reeves and T. Sandholm, A Practical Liquidity-Sensitive Automated Market Maker
   • S. Goel, D. Reeves, D. Watts and D. Pennock, Prediction Without Markets
   • A. Othman and T. Sandholm, Automated Market-Making in the Large: The Gates Hillman Prediction Market
   • Y. Chen and J. Wortman Vaughan, A New Understanding of Prediction Markets Via No-Regret Learning

__________
* I believe one in five logged in users placing a prediction is actually a high conversion rate for a prediction market and a testament to our user interface design. Mike Speiser told me that Bix had a tough time converting their users — who loved to vote — into traders in their American Idol prediction market, one reason why they abandoned the experiment.
** The exchange went like this: Maloney’s assistant Jennifer Hewitt announced that “Crowdcast, the leading provider of prediction market solutions for collective forecasting” was joining Maloney’s latest event. Emile asked “leading… Based on what metric, exactly?”. Then I quipped “leading recent contributor to John Maloney”, which Maloney censored. In Maloney’s defense, the “leading provider” language actually came from Crowdcast’s own website (their meta description tag) and so it appears in the snippet when you search for “crowdcast” in Google.

Now Abe created a fascinating video showing the evolution of prices over time in his market. You can see qualitatively that the thing actually worked, zeroing in closer and closer to the actual opening day as the market progressed.

Figure 3 on page 7 of Abe’s paper with Tuomas Sandholm in the 2010 ACM Conference on Electronic Commerce conveys similar information.

Despite plenty of precedent, and despite increasing evidence that non-market methods do surprisingly well too,* I still find it astonishing to see a bunch of people play a subtle betting game for nothing but bragging rights or a small prize and end up with something reasonably intelligent.

By implementing a working market used by over a hundred CMU students, Abe learned a great deal about practical yet important details, from the difficulty of crisply defining ground truth (when exactly is a building officially “open”?) to the black art of choosing the liquidity parameter of Hanson’s market maker.** Abe independently created an intuitive interval betting interface similar, and in some ways superior, to our own Yoopick interface and Leslie Fine’s Crowdcast interface. Abe went so far as to interview his top traders in great detail to learn about their strategies, which ran the gamut from building automated statistical arbitrage agents to calling construction crew members to learn inside information. Abe observed that interval betting using Hanson’s market maker leads to very “spiky” prices. Starting from this informal observation, Abe was able to actually prove an impossibility result of sorts that any price function with otherwise reasonable properties must be spiky in a formal sense. See Abe and Tuomas’s paper for the details.

__________
* Our paper “Prediction without markets”, by Sharad Goel, Daniel Reeves, Duncan Watts, and me, will be published in the 2010 ACM Conference on Electronic Commerce.
** Abe has now developed a flexible market maker that automatically adjusts liquidity to match trader activity. The paper, by Abe, Tuomas, Daniel Reeves, and me, will also be published in the 2010 ACM Conference on Electronic Commerce.

1. The Sixth Workshop on Ad Auctions is soliciting papers. The workshop will be held June 8, 2010, in Cambridge, MA, in conjunction with the ACM Conference on Electronic Commerce (EC’10). There is a terrific organizing committee this year spanning industry and academia, CS and business schools.
2. The EC’10 list of accepted papers is out and looks great.
3. The first-ever Behavioral and Quantitative Game Theory Conference on Future Directions will be held May 14-16 in Newport Beach, CA. The program looks fantastic.
4. Last fall, the University of Pennsylvania announced the first-ever undergraduate degree program in Market and Social Systems Engineering. Kudos to UPenn: the move shows impressive vision and leadership.
5. The NSF is funding research in the CS-Econ area. They support efforts to “explore the emerging interface between computer science and economics, including algorithmic game theory, automated mechanism design, computational tractability of basic economic problems, and the role of information, trust, and reputation in markets” (page 7).
6. The NBER Market Design working group is soliciting papers for a workshop October 8-9, 2010 in Cambridge, MA.
7. We are now reviewing some amazing submissions to Yahoo!’s 2010 Key Scientific Challenges program. Read the challenges for the area we call Algorithmic Economics.
8. Members of Yahoo! Labs can submit proposals to fund collaborative research with academic colleagues through the Yahoo! Faculty Research and Engagement program. If you’re interested, contact a Yahoo! Labs employee.

What should be the name? CS ∩ Econ is accurate but cryptic. At Yahoo!, we call it Algorithmic Economics. At Google, they call it Market Algorithms. The ACM Special Interest Group in this area calls it Electronic Commerce, causing complaints every year. I’ve heard people suggest Economics and Computation. The name Algorithmic Game Theory has emerged as something of a standard within the CS theory community. [Update: Noam suggests Algorithmic Game Theory and Economics and even renamed his blog accordingly.] The phrase Computational Economics makes sense but is already in use by a different field. A fun suggestion is Economatics (or Autonomics), meant to invoke a mashup of economics and automation.

Prediction markets had a similar naming/identity crisis. They’ve been called information markets, idea markets, securities markets, event markets, binary options, market in uncertainty, and more. But now almost everyone has settled on prediction markets. I’ve come to like the name and I think it’s helped establish the field in it’s own right. I hope we can settle on a good name for CS ∩ Econ in part so we can create the Journal of PerfectNameForCSEcon, an outlet sorely missing from the field.