Oddhead Blog

In 2003, we wrote a paper titled 1 billion pages = 1 million dollars? Mining the web to play Who Wants to be a Millionaire?. We trained a computer to answer questions from the then-hit game show by querying Google. We combined words from the questions with words from each answer in mildly clever ways, picking the question-answer pair with the most search results. For the most part (see below), it worked.

It was a classic example of “big data, shallow reasoning” and a sign of the times. Call it Google’s Law. With enough data nothing fancy can be done, but more importantly nothing fancy need be done: even simple algorithms can look brilliant. When in comes to, say, identifying synonyms, simple pattern matching across an enormous corpus of sentences beats the most sophisticated language models developed meticulously over decades of research.

Our Millionaire player was great at answering obscure and specific questions: the high-dollar questions toward the end of the show that people find difficult. It failed mostly on the warm-up questions that people find easy — the truly trivial trivia. The reason is simple. Factual answers like the year that Mozart was born appear all over web. Statements capturing common sense for the most part do not. Big data can only go so far.*

That was 2003.

In the paper, our clearest example of a question that we could not answer was How many legs does a fish have?. No one on the web would actually bother to write down the answer to that. Or would they?

I was recently explaining all this to a colleague. To make my point, we Googled that question. Low and behold, there it was: asked and answered — verbatim — on Yahoo! Answers. How many legs does a fish have? Zero. Apparently Yahoo! Answers also knows the number of legs of a crayfish, rabbit, dog, starfish, mosquito, caterpillar, crab, mealworm, and “about 133,000″ more.

Today, there are way more than 1 billion web pages: maybe closer to 1 trillion.

What’s the new lesson? Given enough time, everything will be on the web, including the fact that hungry poets blink (✓). Ok, not everything, but far more than anyone ever imagined.

It would be fun to try our Millionaire experiment again now that the web is bigger and search engines are smarter. Is there some kind of Moore’s Law for artificial intelligence as the web grows? Can sentience be far behind?

__________
* Lance agreed, predicting that IBM’s quest to build a Jeopardy-playing computer would succeed but not tell us much.

I’m thrilled to announce the launch of Predictalot, a combinatorial prediction market for the NCAA Men’s Basketball playoffs. Predict almost anything you can think of, like Duke will advance further than UNC, or Every final four team name will start with U. Check the odds and invest points on your favorites. Sell your predictions anytime, even as you follow the basketball games live.

The basic game play is simple: select a prediction type, customize it, and invest points on it. Yet you’ll never run out of odds to explore: there are hundreds of millions of predictions you can make. The odds on each update continuously based on other players’ predictions and the on-court action.

Predictalot is a Yahoo! App, so you can play it at apps.yahoo.com or you can add it to your Yahoo! home page. I have to admit, it’s an incredible feeling to play a game I helped design right on the Yahoo! home page.

That’s all you need to get started. If you’re curious and would like a peek under the hood, read on: there’s some interesting technology hidden in the engine.

Background and Details

Predictalot is a true combinatorial prediction market of the sort academics like us and Robin Hanson have been dreaming about since early in the decade. We built the first version during an internal Yahoo! Hack Day. Finally, we leveraged the Yahoo! Application Platform to quickly build a public version of the game. (Note that anyone can develop a YAP app that’s visible to millions — there’s good sample code, it supports YUI and OpenSocial, and it’s easy to get started.) After many fits and starts, late nights, and eventually all nights, we’re proud and excited to go live with Predictalot version 1.0. I can’t rave enough about the talent and dedication of the research engineers who gave the game a professional look and feel and production speed, turning a pie-in-the-sky idea into reality. We have many features and upgrades in mind for future versions, but the core functionality is in place and we hope you enjoy the game.

In the tournament, after the play-in game, the 64 top college basketball teams play 63 games in a single elimination tournament. So there are 2 to the power 63 or 9.2 quintillion total possible outcomes, or ways the entire tournament can unfold. Predictalot implicitly keeps track of the odds for them all. To put this in perspective, it’s estimated that there are about 10 quintillion individual insects on Earth. Of course, for all practical purposes, we can’t store 9.2 quintillion numbers, even with today’s computers. Instead, we compute the odds for any outcome on the fly by scanning through the predictions placed so far.

A prediction is a statement, like Duke will win in the first round, that will be either true or false in the final outcome. In this case, the prediction is true in exactly half, or 2 to the power 62 outcomes. (Note this does not mean the odds are 50% — remember the outcomes themselves are not all equally likely.) In theory, Predictalot can support predictions on any set of outcomes. That’s 2 to the power 2 to the power 63, or more than a googol predictions. For now, we restrict you to “only” hundreds of millions of predictions categorized into thirteen types. Computing the odds of a prediction precisely is too slow. Technically, the problem is #P-hard: as hard as counting SAT and harder than the travelling salesman problem. So we must resort to approximating the odds by randomly sampling the outcome space. Sampling is a tricky business — equal parts art and science — and we’re still actively exploring ways to increase the speed, stability, and accuracy of our sampling.

Because we track all possible outcomes, the predictions are automatically interconnected in ways you would expect. A large play on Duke to win the tournament instantly and automatically increases the odds of Duke winning in the first round; after all, Duke can’t win the whole thing without getting past the first round.

With 9.2 quintillion outcomes, Predictalot is to our knowledge the largest prediction market built, testing the limits of what the wisdom of crowds can produce. Predictalot is a game, and we hope it’s fun to play. We’d also like to pave the way for serious use of combinatorial prediction market technology.

Why did Yahoo! build this? Predictalot is a smarter market, letting humans and computers each do what they do best. People enter predictions in simple terms they understand like how one team fares against another. The computer handles the massive yet methodical number crunching needed to combine all the pieces together into a coherent overall prediction of a complex event. Markets like Predictalot, WeatherBill, CombineNet, and Internet advertising systems, to name a few, represent the evolution of markets in the digital age, empowering users with extreme customization. More and more, matching buyers with sellers — the core function of markets — requires sophisticated algorithms, including machine learning and optimization. Predictalot attempts to illustrate this trend in an entertaining way.

David Pennock
Mani Abrol, Janet George, Tom Gulik, Mridul Muralidharan, Sudar Muthu, Navneet Nair, Abe Othman, Daniel Reeves, Pras Sarkar

The Kelly criteria is a money management strategy for gamblers and investors. The strategy says that, when faced with a positive-expectation bet, you should invest a fraction of your budget that is proportional to your expected profit. The more your expect to gain, the more you should risk, but you never risk your entire budget.

The Kelly strategy is optimal in several senses: (1) it minimizes your “doubling time”, or the time it takes to go from having X dollars to having 2X dollars; (2) it minimizes the time it takes to achieve any given level of wealth; (3) it maximizes your long-run wealth.

(It turns out that the Kelly strategy is equivalent to maximizing a logarithmic utility function.)

A key reason the Kelly strategy is optimal is that it is very careful to never take you completely bankrupt: you spend only a fraction of your money, always reserving a bit for tomorrow, however small. This is sound advice when dealing with real money. (Aside: this all assumes you have a strict budget cap, which is not entirely realistic: you can almost always borrow at least some amount, even in today’s economy.)

But what about maximizing your virtual “wealth” inside a play-money game like NewsFutures, InklingMarkets, HubDub, or MediaPredict? The problem is not quite the same, precisely because you cannot really go bankrupt. Almost every game offers an option to “recharge” your account if you go bust. Even if the option is not explicit, you can always just abandon your account and start a new one with a fresh initial bankroll they typically give to new players.

So what is the Kelly criteria for play money? What is the optimal strategy that minimizes your doubling time when you’re always allowed to recharge back to a fixed starting value any time you go bankrupt? The answer is not obvious to me, so I’m crowdsourcing the problem: can readers derive the right rule?

My only conjecture is that it might become optimal to go “all in” on every single bet. But I’m not sure. [Update: I've convinced myself this is not optimal. Imagine two sequential bets, the first with minuscule expected profit and the second with huge expected profit: surely you should not go "all in" on the first.]

Note that finding the optimal solution may not just help you win more bragging rights in online games. There is a fascinating sports betting site called CentSports that gives everyone ten real cents to start with. If you can turn that ten cents into twenty dollars, they’ll cut you a check. Moreover, if you ever go to zero, they’ll restore you right back to ten cents. In other words, the system works just like play-money games except the potential for profit is real. So another way to phrase the challenge question is: what strategy in CentSports minimizes the time it takes you to go from ten cents to twenty dollars?

I’m happy to announce the public beta launch of Yoopick, a sports prediction contest with a twist.

You pick any range you think the score difference or point spread of the game will fall into, for example you might pick Pittsburgh wins by between 2 and 11 points.

The more your prediction is viewed as unlikely by others, and the more you’re willing to stake on your prediction, the more you stand to gain. Of course it’s all for fun: you win and lose bragging rights only.

You can play with and against your friends on Facebook.

You can settle a pick even before the game is over, much like selling a stock in the stock market. Depending on what other players have done in the interim, you may be left with a gain or loss. You gain if you were one of the first to pick a popular outcome.

If you run out of credit, you can “work off your debt” by helping to digitize old books via the recaptcha project.

Those are the highlights if you want to go play the game. If you’re interested in more details, read on…

Motivation, Design, and Research Goals

There are a great many sources of sports predictions, including expert communities, statistical number crunchers, bookmakers, and betting exchanges. Many of these sources are highly accurate, however they typically focus on predicting the outright or spread-adjusted winner of the game. Our goal is to obtain more information about the final score, including the relative likelihood of each point spread. For example, if our system is working, on average there should be more weight put on point spreads of 3 and 7 in NFL games than on 2,4,6, or 8.

We chose sports as a test domain to tap into the avid fan base and the armies of arm chair (and Aeron chair) prognosticators out there. However, the same approach should translate well to any situation where you’d like to predict a number, for example, the vote share of a politician or the volume of sales of your company’s widget. In addition to giving you the expected value of the number, our approach gives you the confidence or variance of the prediction — in fact, it gives you the entire probability distribution, or the likelihood of every possible value of the number.

Underneath the hood, Yoopick is a type of combinatorial prediction market where the possible outcomes are the values of the point spread, and each pick is a purchase of a bundle of outcomes in a given interval. We use Hanson’s logarithmic market scoring rules market maker to price the picks — that is, to set the risk/reward ratio. This pricing mechanism also determines the gain or loss when picks are settled early.

Wins and losses on Yoopick are measured in milliyootles, a social currency useful for expressing thanks.

Our market maker can — and we expect will — lose yootles on average. Stated another way, we expect players as a whole to gain on average. At the same time, we actively work to improve our market maker to limit its losses to control inflation in the game.

Because the outcomes of a game are tied together in a unified market, picks in one region automatically affect the price of picks in other regions in a logically consistent way. Players have considerable flexibility in how and what information they can inject into the market. In particular, players can replicate the standard picks like outright winner and spread-adjusted winner if they want, or they can go beyond to pick any interval of the point spread. No matter the form of the pick, all the information flows into a single market that aggregates everything in a unified prediction. In contrast, at venues from Wall Street to Churchill Downs to High Street to Las Vegas Boulevard, markets with many outcomes are usually split into independent one-dimensional markets.

Our goal is to test whether our market design is indeed able to elicit more information than traditional methods. We hope you have fun playing in our Petri dish.

Sharad Goel
David Pennock
Daniel Reeves
Prasenjit Sarkar
Cong Yu