Oddhead Blog

If you haven’t played around with Yahoo! Pipes, I highly recommend it. It’s a usable and useful service that brings web mashups to the masses, making this favorite hacker pastime as easy as dragging objects around on the screen.

For example, it took me probably about ten minutes as a first-time user to create a map mashup showing Barack Obama’s upcoming campaign stops. I “piped” the output of Washington Post’s RSS feed to a location-extractor module that identifies and geo-codes place names and renders them on a map. Here’s a screenshot of the output:

The easiest way to get started is to find an existing Pipe, clone it, and modify it as your own. Using this feature, I cloned my Obama map and in about one minute had a McCain map too.

Pipes uses a visual programming interface. The idea of “programming by picture” (I recall playing with one in the 1980s) never took hold as a mainstream tool. However, as a metaphor for mashups, where to goal is to chain together a number of sources and services, the visual approach seems exactly right. The implementation in a browser is a feat of ajaxian magic that I still find remarkable, even as Yahoo! and others are commoditizing the art. I imagine that even non-programmers should have little trouble constructing their own Pipes. Here is a screenshot of the source “code” for my Obama map:

Pipes has dozens of useful modules, including user input, Yahoo! Search, Flickr, and regular expressions.

You can embed the Pipe on your own website with a single line of javascript. I did this with my Obama and McCain campaign travel maps here. Or you can grab the output as an XML feed to use however you wish.

Pipes allows you to create human-readable URLs (e.g., http://pipes.yahoo.com/oddhead/obamatravelmap), a nice touch.

The icing on the cake for me is how Pipes — unlike so many other web sites, including some on Yahoo! — treats me and my Opera browser like adults:

(BTW, Pipes seems to work fine on Opera).

Unfortunately, Daniel Raffel, one of the key founders of Yahoo! Pipes, left Yahoo!. However, the team seems to be strong and continues to innovate, so I’m hopeful this fantastic service will continue to improve and thrive.

We just added a new feature to Yoopick designed especially for Frenchmen Chris and Emile and citizens of nineteen other countries to place their swagor* on how many Olympic medals they think their country will win.

We’ve argued that the Yoopick interface is useful for predicting almost any kind of number, and since medal count is indeed a number, we thought we’d give it a try.

Besides, Lance told us it would be a good idea.

Sign up, play, enjoy, and don’t forget to tell us what you think!

Thanks,
Sharad Goel
David Pennock
Dan Reeves

* Scientific wild-ass guess, on record

I’m happy to announce the public beta launch of Yoopick, a sports prediction contest with a twist.

You pick any range you think the score difference or point spread of the game will fall into, for example you might pick Pittsburgh wins by between 2 and 11 points.

The more your prediction is viewed as unlikely by others, and the more you’re willing to stake on your prediction, the more you stand to gain. Of course it’s all for fun: you win and lose bragging rights only.

You can play with and against your friends on Facebook.

You can settle a pick even before the game is over, much like selling a stock in the stock market. Depending on what other players have done in the interim, you may be left with a gain or loss. You gain if you were one of the first to pick a popular outcome.

If you run out of credit, you can “work off your debt” by helping to digitize old books via the recaptcha project.

Those are the highlights if you want to go play the game. If you’re interested in more details, read on…

Motivation, Design, and Research Goals

There are a great many sources of sports predictions, including expert communities, statistical number crunchers, bookmakers, and betting exchanges. Many of these sources are highly accurate, however they typically focus on predicting the outright or spread-adjusted winner of the game. Our goal is to obtain more information about the final score, including the relative likelihood of each point spread. For example, if our system is working, on average there should be more weight put on point spreads of 3 and 7 in NFL games than on 2,4,6, or 8.

We chose sports as a test domain to tap into the avid fan base and the armies of arm chair (and Aeron chair) prognosticators out there. However, the same approach should translate well to any situation where you’d like to predict a number, for example, the vote share of a politician or the volume of sales of your company’s widget. In addition to giving you the expected value of the number, our approach gives you the confidence or variance of the prediction — in fact, it gives you the entire probability distribution, or the likelihood of every possible value of the number.

Underneath the hood, Yoopick is a type of combinatorial prediction market where the possible outcomes are the values of the point spread, and each pick is a purchase of a bundle of outcomes in a given interval. We use Hanson’s logarithmic market scoring rules market maker to price the picks — that is, to set the risk/reward ratio. This pricing mechanism also determines the gain or loss when picks are settled early.

Wins and losses on Yoopick are measured in milliyootles, a social currency useful for expressing thanks.

Our market maker can — and we expect will — lose yootles on average. Stated another way, we expect players as a whole to gain on average. At the same time, we actively work to improve our market maker to limit its losses to control inflation in the game.

Because the outcomes of a game are tied together in a unified market, picks in one region automatically affect the price of picks in other regions in a logically consistent way. Players have considerable flexibility in how and what information they can inject into the market. In particular, players can replicate the standard picks like outright winner and spread-adjusted winner if they want, or they can go beyond to pick any interval of the point spread. No matter the form of the pick, all the information flows into a single market that aggregates everything in a unified prediction. In contrast, at venues from Wall Street to Churchill Downs to High Street to Las Vegas Boulevard, markets with many outcomes are usually split into independent one-dimensional markets.

Our goal is to test whether our market design is indeed able to elicit more information than traditional methods. We hope you have fun playing in our Petri dish.

Sharad Goel
David Pennock
Daniel Reeves
Prasenjit Sarkar
Cong Yu

Two New York Post video reporters came to Yahoo!’s midtown NYC office last Friday to interview me for a piece they were producing on intrade’s political prediction markets. The video is now up on NYPOST.COM and in the embedded player below. The reporters were friendly and professional — thankfully they cut out most of my word-fumbling moments — and the end result is an entertaining, polished, informative video geared toward newbies. My own role came out at least not terrible.

If you look carefully, you’ll see subtle product placement of the Yahoo! Election Dashboard, which aggregates a ton of election numbers including intrade prices. You can also see short clips of the conference room, whiteboard scribbles, ylogo, and cubes at our Y! Research NYC office.

See also: Chris Masse’s comments

The absurdity of gambling laws in the US leads to such silliness as:

• In 2007, Google, Microsoft, and Yahoo! paid millions in penalties for placing gambling ads, something they haven’t done since they were told to stop in 2004.
• Yahoo! can quote prices from intrade, but can’t link to intrade.
• Google can’t advertise for intrade/tradesports, but can place AdSense ads on intrade.com and tradesports.com. In other words, Google can’t sell eyeballs to gambling sites, but can sell eyeballs on gambling sites.

I am happy to report that on my suggestion intrade has listed futures contracts for 2008 search engine market share.

Here is how they work:

A contract will expire according to the percentage share of internet searches conducted in the United States in 2008. For example, if 53.5% of searches conducted in the United States in 2008 are made using Google then the contract listed for Google will expire at 53.5…

…Expiry will be based on the United States search share rankings published by Nielson Online.

I think this could be a fascinating market because:

• Search engine market share is very important to these major companies, with dramatic effects on their share prices.
• Search engine market share is fluid, so far with Google growing inexorably. However, Microsoft has cash, determination, Internet Explorer, and the willingness to experiment. Ask.com has erasers, 3D, ad budgets, and The Algorithm. Yahoo!, second in market share, often tests equal or better than Google, and new features like Search Assist are impressive.
• The media loves to write about it.
• A major search company might use the market to hedge. Well, this seems far-fetched but you never know. Certainly, from an economic risk management standpoint it would seem to make a great deal of sense. (Here, as always on this blog, I speak on behalf of myself and not my company.)

Finally, I have to comment on how refreshingly easy the process was in working with intrade. They went from suggestion to implementation in a matter of days. It’s a shame that US-based companies are in contrast stuck in stultifying legal and regulatory mud.

Addendum 2008/01/26: Here are links to some market research reports:
Nielsen | ComScore | HitWise | Compete

(It seems that Nielsen Netratings homepage is down; getting 404 error at the moment)

Addendum 2008/03/07: If you prefer, you can now also bet on search share just for fun with virtual currency at play.intrade.com.

(Nielsen Netratings homepage is still down, now for over a month. It’s even more ridiculous given that their own Nielsen Online website points to this page.)

I’m happy to report the launch of the Yahoo! Election ‘08 Political Dashboard. Using the dashboard, you can navigate through a wealth of election-related data, including prediction market data from intrade.com, polling data from Real Clear Politics, search buzz data from Yahoo!, and financial contributions data, regional demographic data, and historical voting records from AP. You can view the election landscape from the national level or dive in deeper to investigate trends state by state.

Yiling, Tej, Lance, and I played supporting roles among a cast that includes fantastic teams at Yahoo! News, UI/Design, Ops, Q&A, and more.

We’ve come a long way since 2006.

See also coverage from TechCrunch and the Yahoo! corporate blog.

Earlier this month, I spent a few days in lovely Banff, Alberta, Canada, at WWW2007, the 16th International World Wide Web Conference. Here are my thoughts from the event. [See also: Yahoo! Research's writeup.]

It’s becoming clear that other sciences beyond computer science, including economics and sociology, are necessary for understanding the web and realizing its full potential. This theme ran through both Tim Berners-Lee’s and Prabhakar Raghavan’s plenary talks. For every new advance in the web, once it reaches critical mass, the economic incentives to manipulate the system inevitably emerge. Email led to spam. Altavista led to keyword spam. Google led to link spam. Blogs led to comment and trackback spam. Folksonomies led to tag spam. Recommender systems and aggregators (e.g., Digg) led to shilling. It’s clear that a better understanding of incentives, game theory, and system equilibrium is needed, beyond just cool engineering feats. The University of Michigan calls this incentive-centered design and has a world-class research team exploring the topic; see Jeff MacKie-Mason’s blog ICD Stuff for an interesting and accessible discussion. Yahoo! Research is also betting on the importance of human incentives, building a group of economists and sociologists to complement our contingent of computer scientists.

Among conference events, nowhere was the convergence of economics and computer science more clear than at the Third Workshop on Sponsored Search Auctions. The workshop is a rare venue where terms like Nash equilibrium and NP-complete can coexist in harmony. The workshop explored the intricacies of web search advertising, a multi-billion dollar industry experiencing rapid growth. Contributions included new designs for auctioning off advertising space, new analyses of the systems currently used by search engines, new tools to help advertisers, and empirical studies of the industry. Participants included representatives from both academia and industry, including economists, computer scientists, search engine employees (including representatives from the “big three”: Google, Microsoft, and Yahoo!), and search engine marketers. Yahoo! had a large presence at the workshop: Yahoo! scientists (including me) served on the organizing committee, Yahoo! employees and interns presented six of the fourteen peer-reviewed papers, and many Yahoos attended, contributing to their voice to the discussion of this emerging field.

Bradley Horowitz’s talk also emphasized the new web order, where artists are needed as much as technologists: artists who can envision, create, and orchestrate online communities can be the difference between mass adoption and a flop.

An interesting addition to the WWW program was the Web History track and the Web History Center. Some of the talks were fascinating. Hermann Maurer recounted stories of interactive TV products that proliferated in Europe in the 1970’s and that mirrored almost everything that is done on the Web today in a primitive form. [Some keywords to search for if you're interested: PRESTEL, Teletel/Minitel (France), MUPID (Austria).] For example, one massive multiplayer game, which involved social exploration of 64 million virtual planets, each with a hidden secret, was so wildly popular that it crashed the network. The apparent winner of the contest returned his prize, admitting that he didn’t actually solve for the secrets, but rather hacked into the system and reverse engineered the code. This pre-Internet system even featured some things I’m still waiting for on today’s web, like micropayments.

Same speculation. Same media and market frenzy. On CNNMoney, they used the almost identical title, subtitle, and URL. Exactly one year apart, almost to the day. Bizarre.

http://money.cnn.com/2006/05/03/technology/microsoft_yahoo/

http://money.cnn.com/2007/05/04/technology/microsoft_yahoo/

P.S. I have no special knowledge of the situation. The latest reports I’ve seen say no deal.

I felt bad about this. So I made sure to answer this. And whaddya know?: they like me, they really like me!

——– Original Message ——–
Subject: Yahoo! Answers: Your answer has been chosen as the best answer
Date: 24 Apr 2007 09:44:59 -0700
From: Yahoo! Answers
To: pennockd

	Hey, Dave, look what you got! Congratulations, you’ve got a best answer and 10 extra points! Your answer to the following question really hit the spot and has been chosen as the best answer: Who will win in 2008 and why (real answers)? Go ahead, do your victory dance. Celebrate a little. Brag a little. Then come back and answer a few more questions! Thanks for sharing what you know and making someone’s day. The Yahoo! Answers Team
Get the Yahoo! Toolbar for one-click access to Yahoo! Answers.

So now I’m 1 for 1! I can see how this gets addictive.