Category Archives: research

Four free registrations to EC’11 for students

Thanks to a generous donation from Google, we are offering four free registrations for students to attend the 2011 ACM Conference on Electronic Commerce (EC’11) in San Jose.

To apply, please email David Pennock and Yoav Shoham by Wednesday May 11, 2011, with subject “YourLastName: EC’11 student registration award application” and include:

  1. Your name, university, personal homepage, and current student status (e.g., 2nd year Ph.D. student)
  2. Whether you are a member of ACM SIGecom
  3. Any papers at EC’11 for which you are an author or co-author
  4. Any papers at an EC’11 affiliated workshop (or under review) for which you are an author or co-author
  5. Please also arrange for your academic advisor to email verification of your student status in good standing to the same two email addresses with your last name in the subject.

Applications must be submitted by Wednesday May 11, 2011. We will award the four free registrations by Friday May 13, prior to the early registration deadline of May 16.

David Pennock, Chair ACM SIGecom
Yoav Shoham, General Chair, EC’11

P.S. This was announced on April 14 on the mailing list for the ACM Special Interest Group on Electronic Commerce (SIGecom). If you missed it, you should join! 🙂

CFP: Auctions, Market Mechanisms, and their Applications

From Peter Coles:

There is [less than] one week left to submit papers to AMMA, [The Second Conference on Auctions, Market Mechanisms and Their Applications], a market design conference that will be held in NYC this August. The conference brings together economists, computer scientists and practitioners who are interested in the use of market mechanisms to solve problems.

The best way to decide whether to submit to a conference you haven’t heard of is to look at the organizers and program committee. In this case, they’re superb.

Predictopus in the Times of India

Today, Yahoo! placed two full-page ads on the back cover of the Times of India, the largest English-language daily in the world, to promote Yahoo! Cricket, a site that reaches 13.4 percent of everyone online in India and serves as the official website of the ICC Cricket World Cup.

Take a look at the middle right of the second page: it says “Play exciting games and win big” and features… Predictopus! That’s the Indian spinoff of Predictalot, the combinatorial prediction game I helped invent.

Page 1 of two full-page Yahoo! Cricket ads in the Times of India, p. 31, 2011/03/30Predictopus on Page 2 of two full-page Yahoo! Cricket ads in the Times of India, p. 32, 2011/03/30

Predictopus has nearly 70,000 users and counting, and this ad certainly won’t hurt.


BTW, I grabbed these images from an amazing site called Press Display, which I discovered via the New York Public Library.

Times of India Mumbai edition
30 Mar 2011

Times of India Mumbai edition
30 Mar 2011

Also, congrats India, and thanks! I nearly doubled my virtual bet with the victory:

Dave's Predictopus prediction: India will advance further than Pakistan, 3/2011

Workshops @ACM Electronic Commerce: Ad Auctions, Social Computing, June 5, 2011

The 2011 ACM Conference on Electronic Commerce will be held June 5-9 in San Jose as part of the ACM Federated Computing Research Conference. FCRC is a collection of seventeen computer science conferences with joint plenary speakers, this year featuring David A. Ferrucci, head of IBM’s Watson project, CMU professor and GWAP founder Luis von Ahn, and 2011 Turing Award winner Leslie Valiant. I’d love to someday see a true unified computer science conference in the style of the math or economics national meetings. Barring that, FCRC is the next-best thing. I hope more conferences will join.

The EC’11 list of accepted papers is out and the program looks great (including six papers from Yahoo! authors). And it’s not too late to submit a paper to one of the associated workshops. Two of particular interest, both on June 5, 2011, are:

Workshop on Social Computing and User Generated Content

The workshop will bring together researchers and practitioners from a variety of relevant fields, including economics, computer science, and social psychology, in both academia and industry, to discuss the state of the art today, and the challenges and prospects for tomorrow in the field of social computing and user generated content.

Social computing systems are now ubiquitous on the web– Wikipedia is perhaps the most well-known peer production system, and there are many platforms for crowdsourcing tasks to online users, including Games with a Purpose, Amazon’s Mechanical Turk, the TopCoder competitions for software development, and many online Q&A forums such as Yahoo! Answers. Meanwhile, the user-created product reviews on Amazon generate value to other users looking to buy or choose amongst products, while Yelp’s value comes from user reviews about listed services…

SUBMISSIONS DUE April 15, 2011, 5pm EDT

Seventh Ad Auctions Workshop

In the past decade we’ve seen a rapid trend toward automation in advertising, not only in how ads are delivered and measured, but also in how ads are sold… The rapid emergence of new modes for selling and delivering ads is fertile ground for research from both economic and computational perspectives…

We solicit contributions of two types: (1) research contributions, and (2) position statements…

Submission deadline: April 15th, 2011 (midnight Hawaii Time)

We’re baaack: Predictalot is here for March Madness 2011

March Madness is upon us and Predictalot, the crazy game that I and others at Yahoo! Labs invented, is live again and taking your (virtual) bets. Filling out brackets is so 2009. On Predictalot, you can compose your own wild prediction, like there will be exactly seven upsets in the opening round, or neither Duke, Kentucky, Kansas, nor Pittsburgh will make the Final Four. You’ll want your laptop out and ready as you watch the games — you can buy and sell your predictions anytime, like stocks, as the on-court action moves for or against you.

Predictalot v0.3 is easier to play. We whittled down the ‘Make Prediction’ process from four steps to just two. Even if you don’t want to wager, with one click come check out the projected odds of nearly any crazy eventuality you can dream up.

Please connect to facebook and/or twitter to share your prediction prowess with your friends and followers. You’ll earn bonus points and my eternal gratitude.

The odds start off at our own prior estimate based on seeds and (new this year) the current scores of ongoing games, but ultimately settle to values set by “the crowd” — that means you — as predictions are bought and sold.

Yahoo! Labs Predictalot version 0.3 overview tab screenshot

For the math geeks, Predictalot is a combinatorial prediction market with over 9 quintillion outcomes. Prices are computed using an importance sampling approximation of a #P-hard problem.

What kind of information can we collect that a standard prediction market cannot? A standard market will say that Texas A&M is unlikely to win the tournament. Our market can say more. Yes, A&M is unlikely to reach the Final Four and even more unlikely to win apriori, but given that they somehow make it to the semifinals in Houston, less than a two hour drive from A&M’s campus, their relative odds may increase due to a home court advantage.

Here’s another advantage of the combinatorial setup. A standard bookmaker would never dare to offer the same millions of bets as Predictalot — they would face nearly unlimited possible losses because, by tradition, each bet is managed independently. By combining every bet into a single unified marketplace, we are able to limit the worst-case (virtual) loss of our market maker to a known fixed constant.

Yahoo! Key Scientific Challenges: Applications due March 11

Applications for Yahoo!’s third annual Key Scientific Challenges Program are due March 11. Our goal is to support students working in areas we feel represent the future of the Internet. If you’re a Ph.D. student working in one of the areas below, please apply!

We are thrilled to announce Yahoo!’s third annual Key Scientific Challenges Program. This is your chance to get an inside look at — and help tackle — the big challenges that Yahoo! and the entire Internet industry are facing today. As part of the Key Scientific Challenges Program you’ll gain access to Yahoo!’s world-class scientists, some of the richest and largest data repositories in the world, and have the potential to make a huge impact on the future of the Internet while driving your research forward.


– Search Experiences
– Machine Learning
– Data Management
– Information Extraction
– Economics
– Statistics
– Multimedia
– Computational Advertising
– Social Sciences
– Green Computing
– Security
– Privacy


– $5,000 unrestricted research seed funding which can be used for conference fees and travel, lab materials, professional society membership dues, etc.

– Access to select Yahoo! datasets

– The unique opportunity to collaborate with our industry-leading scientists

– An invitation to this summer’s exclusive Key Scientific Challenges Graduate Student Summit where you’ll join the top minds in academia and industry to present your work, discuss research trends and jointly develop revolutionary approaches to fundamental problems

CRITERIA: To be eligible, you must be currently enrolled in a PhD program at any accredited institution.

We’re accepting applications from January 24th – March 11th, 2011 and winners will be announced by mid April 2011.

To learn more about the program and how to apply, visit

International Conference on Prediction and Information Markets: Who’s going?

Abstracts are due today for the Third International Conference on Prediction and Information Markets. It will be held in Nottingham, England April 3-5 along with two related conferences: the International Conference on Gambling Studies and the International Conference on Money, Investment and Risk. (Abstracts also due today for both of these conferences.)

I’ve been considering submitting some thing(s): I’m curious if anyone else is planning to submit or attend?

Date: 3 – 5 April 2011
Event: Third International Conference on Prediction and Information Markets
Location: Nottingham Conference Centre
Organiser: Nottingham Business School


The Third International Conference on Prediction and Information Markets will be held in association with Economic Issues and the Journal of Prediction Markets.

Prediction / information markets offer a way of harnessing the wisdom of crowds. They have been used to aggregate information in order to provide forecasts of a wide range of events…

In recent years, a number of companies have employed these markets as a means of aggregating the information dispersed widely among their employees and customers…

Call for papers and presentations
Please send an abstract (maximum of 200 words) to Professor Leighton Vaughan Williams by email, by the closing date for receipt of abstracts of Monday 21 February 2011. Confirmation of receipt of abstracts will be sent within a maximum of five working days.

Predictalot goes East: Introducing Predictopus for the ICC Cricket World Cup

Yahoo! India Predictopus logo

I’m thrilled to report that Predictalot had an Indian makeover, launching as Predictopus* for the ICC Cricket World Cup. The Yahoo! India team did an incredible job, leveraging the idea and some of the code base of Predictalot, yet making it their own. Predictopus is not a YAP — it lives right on the Yahoo! Cricket website, the official homepage for the ICC Cricket World Cup. They’re also giving away Rs 10 lakhs — or about $22,000 if my calculations are correct — in prizes. Everything is bigger in India, including the crowds and the wisdom thereof. It will be great to see the game played out on a scale that dwarfs our college basketball silliness in the US.

The Y! India team reused some of the backend code but redid the frontend almost entirely. To adapt the game to cricket, among other chores, we had to modify our simulation code to estimate the starting probabilities that any team would win against any other team, even in the middle of a game. (How likely is it for India to come back at home from down 100 runs with 10 overs left and 5 wickets lost? About 25%, we think.) These starting probabilities are then refined further by the game-playing crowds.

It’s great to see an experiment from Labs grow into a full-fledged product run by a real product team in Yahoo!, a prime example of technology transfer at its best. In the meantime, we (Labs) are still gunning for a relaunch of Predictalot itself for March Madness 2011, the second year in a row. Stay tuned.

2011/02/24 Update: An eye-catching India-wide ad campaign for predictopus is live, including homepage, finance, movies, OMG, answers, mail, everywhere! Oh, and one of the prizes is a Hyundai.

predictopus ad on Yahoo! India homepage 2011/02/24

* Yes, that’s a reference to legendary Paul the Octopus, RIP. NSF and IARPA funding for collective intelligence

The US National Science Foundation’s Small Business Innovation Research program provides grants to to small businesses to fund “state-of-the-art, high-risk, high-potential innovation research proposals”.

In their current call for proposals, they explicitly ask for “I2b. Tools for facilitating collective intelligence”.

These are grants of up to US$150,000 with opportunity for more later I believe. The deadline is December 3, 2010! Good luck and (not so) happy Thanksgiving to anyone working on one of these proposals. I’m glad to help if I can.

The deadline for another US government program has passed, but should yield interesting results and may lead to future opportunities. In August, the Intelligence Advanced Research Projects Activity (IARPA, the intelligence community’s DARPA), which “invests in high-risk/high-payoff research programs” in military intelligence, solicited proposals for Aggregative Contingent Estimation, or what might be called wisdom-of-crowds methods for prediction:

The ACE Program seeks technical innovations in the following areas:

  • Efficient elicitation of probabilistic judgments, including conditional probabilities for contingent events.
  • Mathematical aggregation of judgments by many individuals, based on factors that may include past performance, expertise, cognitive style, metaknowledge, and other attributes predictive of accuracy.
  • Effective representation of aggregated probabilistic forecasts and their distributions.

The full announcement is clear, detailed, and well thought out. I was impressed with the solicitors’ grasp of research in the field, an impression no doubt bolstered by the fact that some of my own papers are cited 😉 . Huge hat tip to Dan Goldstein for collating these excerpts:

The accuracy of two such methods, unweighted linear opinion pools and conventional prediction markets, has proven difficult to beat across a variety of domains.2 However, recent research suggests that it is possible to outperform these methods by using data about forecasters to weight their judgments. Some methods that have shown promise include weighting forecasters’ judgments by their level of risk aversion, cognitive style, variance in judgment, past performance, and predictions of other forecasters’ knowledge.3 Other data about forecasters may be predictive of aggregate accuracy, such as their education, experience, and cognitive diversity. To date, however, no research has optimized aggregation methods using detailed data about large numbers of forecasters and their judgments. In addition, little research has tested methods for generating conditional forecasts.

2 See, e.g., Tetlock PE, Expert Political Judgment (Princeton, NJ: Princeton University Press, 2005), 164-88; Armstrong JS, “Combining Forecasts,” in JS Armstrong, ed., Principles of Forecasting (Norwell, MA: Kluwer, 2001), 417-39; Arrow KJ, et al., “The Promise of Prediction Markets,” Science 2008; 320: 877-8; Chen Y, et al., “Information Markets Vs. Opinion Pools: An Empirical Comparison,” Proceedings of the 6th ACM Conference on Electronic Commerce, Vancouver BC, Canada, 2005.

3 See, e.g., Dani V, et al., “An empirical comparison of algorithms for aggregating expert predictions,” Proc. 22nd Conference on Uncertainty in Artificial Intelligence, UAI, 2006; Cooke RM, ElSaadany S, Huang X, “On the performance of social network and likelihood-based expert weighting schemes,” Reliability Engineering and System Safety 2008; 93:745-756; Ranjan R, Gneiting T, “Combining probability forecasts,” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2010; 72(1): 71-91.


  • Will the incumbent party win the next presidential election in Country X?
  • Will the incumbent party win the next presidential election in Country X?
  • When will Country X hold its next parliamentary elections?
  • How many cell phones will be in use globally by 12/31/11?
  • By how much will the GDP of Country X increase from 1/1/11 to 12/31/11?
  • Will Country X default on its sovereign debt in 2011?
  • If Country X defaults on its sovereign debt in 2011, what will be the growth rate in the Eurozone in 2012?

Elicitation – Advances Sought
The ACE Program seeks methods to elicit judgments from individual forecasters on:

  • Whether an event will or will not occur
  • When an event will occur
  • The magnitude of an event
  • All of the above, conditioned on another set of events or actions
  • The confidence or likelihood a forecaster assigns to his or her judgment
  • The forecaster’s rationale for his or her judgment, as well as links to background information or evidence, expressed in no more than a couple of lines of text
  • The forecaster’s updated judgments and rationale

The elicitation methods should allow prioritization of elicitations, continuous updating of forecaster judgments and rationales, and asynchronous elicitation of judgments from more than 1,000 geographically-dispersed forecasters. While aggregation methods, detailed below, should be capable of generating probabilities, the judgments elicited from forecasters can but need not include probabilities.

Challenges include:

  • Some forecasters will be unaccustomed to providing probabilistic judgments
  • There has been virtually no research on methods to elicit conditional forecasts
  • Elicitation should require a minimum of time and effort from forecasters; elicitation should require no more than a few minutes per elicitation per forecaster
  • Training time for forecasters will be limited, and all training must be delivered within the software
  • Rewards for participation, accuracy, and reasoning must be non-monetary and of negligible face value (e.g., certificates, medals, pins)

Book of Odds is serious fun

In the Book of Odds, you can find everything from the odds an astronaut is divorced (1 in 15.54) to the odds of dying in a freak vending machine accident (1 in 112,000,000).

Book of Odds is, in their own words, “the missing dictionary, one filled not with words, but with numbers – the odds of everyday life.”

I use their words because, frankly I can’t say it better. The creators are serious wordsmiths. Their name itself is no exception. “Book of Odds” strikes the perfect chord: memorable and descriptive with a balance of authority and levity. On the site you can find plenty of amusing odds about sex, sports, and death, but also odds about health and life that make you think, as you compare the relative odds of various outcomes. Serious yet fun, in the grand tradition of the web.

I love their mission statement. They seek both to change the world — by establishing a reliable, trustworthy, and enduring new reference source — and to improve the world — by educating the public about probability, uncertainty, and decision making.

By “odds”, they do not mean predictions.

Book of Odds is not in the business of predicting the future. We are far too humble for that…

Odds Statements are based on recorded past occurrences among a large group of people. They do not pretend to describe the specific risk to a particular individual, and as such cannot be used to make personal predictions.

In other words, they report how often some property occurs among a group of people, for example the fraction all deaths caused by vending machines, not how likely you, or anyone in particular, are to die at the hands of a vending machine. Presumably if you don’t grow enraged at uncooperative vending machines or shake them wildly, you’re safer than the 1 in 112,000,000 stated odds. A less ambiguous (but clunky) name for the site would be “Book of Frequencies”.

Sometimes the site’s original articles are careful about this distinction between frequencies and predictions but other times less so. For example, this article says that your odds of becoming the next American Idol are 1 in 103,000. But of course the raw frequency (1/number-of-contestants) isn’t the right measure: your true odds depend on whether you can sing.

Their statement of What Book of Odds isn’t is refreshing:

Book of Odds is not a search-engine, decision-engine, knowledge-engine, or any other kind of engine
so please don’t compare us to Googleℱ. We did consider the term “probability engine” for about 25 seconds, before coming to our senses…

Book of Odds is never finished. Every day new questions are asked that we cannot yet answer…

A major question is whether consumers want frequencies, or if they want predictions. If I had to guess, I’d (predictably) say predictions — witness Nate Silver and Paul the Octopus. (I’ve mused about using * to aggregate predictions from around the web.)

The site seems in need of some SEO. The odds landing pages, like this one, don’t seem to be comprehensively indexed in Bing or Google. I believe this is because there is no natural way for users (and thus spiders) to browse (crawl) them. (Is this is a conscious choice to protect their data? I don’t think so: the landing pages have great SEO-friendly URLs and titles.) The problem is exacerbated because Book of Odds own custom search is respectable but, inevitably, weaker than what we’ve become accustomed to from the major search engines.

Book of Odds launched in 2009 with a group of talented and well pedigreed founders and a surprisingly large staff. They’ve made impressive strides since, adding polls, a Yahoo! Application, an iGoogle gadget, regular original content, and a cool visual browser that, like all visual browsers, is fun but not terribly useful. They’ve won a number of awards already, including “most likely company to be a household name in five years”. That’s a low-frequency event, though Book of Odds may beat the odds. Or have some serious fun trying.