All posts by David Pennock

Not the lesser of two evils

Every election, many voters rationalize their choice as the “lesser of two evils”.1

However, for me, this year’s election is not about the lesser of evils.

In fact, for the first time I can remember, I actually like both major candidates in the US Presidential election.

I like Obama more and I voted for him2 — I think he’s smarter, inspires optimism, and has better policies and people surrounding him. But I like many aspects of McCain including how he denounced Pat Robertson, Jerry Falwell, and the extreme religious right they represent.3

If the party of less government could ever manage to stop legislating morality, I might actually consider voting for them. By the opposite logic, I imagine some evangelicals actually hope that Obama wins, thus strengthening their argument that Republicans can’t win without them.

On a related note, I received an email chain letter from a Snopes-averse source4 warning that McCain’s campaign is sending out erroneous absentee ballot applications to Obama supporters in an attempt to disqualify voters. Initially I dismissed it as conspiracy theory. Then, a few days ago, I received an absentee ballot application in the mail myself, even though I had just finished voting! For a moment, I thought I was a target of the scam with the evidence right in my hand. I could feel the blograge composing in my head.

So I investigated. (Read: conducted a few web searches.)

The Wisconsin State Journal (in)concludes that McCain either meddled or messed up, with benefit of the doubt going to the latter. Blackboxvoting.com (not affiliated with Bev Harris’s more cited blackboxvoting.org) paints a picture of more widespread fraud and malicious intent.

And it seems that the application I received was a legitimate and well intentioned mailing from the League of Conservation Voters Education Fund, a left-leaning environmental organization. The application’s return address had one line missing and an incorrect zip code by one digit, but the address was “correct” in the sense that it would almost surely end up at the right place, so I believe this was not part of any intentional plot to mislead.5 Still, the whole ordeal got me thinking that perhaps all unsolicited applications for absentee ballots should be outlawed — there’s just too much room for error, both malicious and inadvertent.

1Likely more a testament to the effectiveness of attack ads than anything else, and one of the many maddening features of a duop-racy.
2If the choice had been between Clinton and McCain, I think I would have had a harder decision.
3I also like the fact that he defied Bush on torture and held firm on the Iraq surge, a strategy that seems to have helped, despite the political consequences. On the other hand, I cringe at the thought of a President Palin, an outcome with a better than 1 in 7 chance of happening if elected, according to one estimate.
4My mom! 🙂
5You decide: The return address on the application is: Middlesex County Clerk, PO Box 1110, New Brunswick, NJ 08903-1110. The correct address is County Clerk, Hon. Elaine Flynn, P.O. Box 1110, 75 Bayard Street, 4th Floor, New Brunswick, NJ 08901-1110.

NYCE Day: Thanks and thoughts

NYCE Day 2008 went very well, with over 100 attendees, great talks, and valuable discussion. Many thanks to the four plenary speakers — Costis, Asim, Susan, and Tuomas — and ten rump session speakers who came in from various NYC suburbs like Boston, Pittsburgh, and Palo Alto.

At dinner the night before,1 the organizers agreed that we were nervous because we weren’t at all nervous. Karin and Renee from the New York Academy of Sciences had taken care of almost everything, leaving little for us to fret about. It turned out we were right to not worry and wrong to worry about not worrying: indeed Karin, Renee, and NYAS were absolutely fantastic, orchestrating every detail of the event flawlessly, from technology to catered breaks. The venue itself is gorgeous — a well laid-out space in a modern building in the World Trade Center complex with stunning views2 and a number of nice touches, from an alcove with a computer station to check email to a subtle gradient in the wallpaper that slowly pixilates as your gaze moves from the center toward the side of the room. I came away incredibly impressed with NYAS and delighted to become a member.

Muthu provides an excellent summary of the event, divided into before and after lunch. Read that first and then come back here for my additional thoughts/notes:

  1. Costis gave us mostly bad news. He summarized some of his award winning work with Christos Papadimitriou and Paul Goldberg proving that computing equilibrium behavior in almost any moderately complex game may be beyond the reach of our computers,3 let alone our brains. As a saying goes, “if your laptop can’t find it, then neither can the market” [attribution: Kamal Jain?]. Still, all may not be lost. These results, as is the nature of computational complexity results, say only that some games are extremely hard to solve, not all games or even most games. Since nature is not adversarial (Murphy’s Law aside), it may be the case that among games that arise in the real world that we care about, a number of them can be solved for equilibrium. The problem is defining what “realistic” means in this context: an almost impossibly fuzzy task. Costis did end with some positive results, showing that anonymous games can be solved efficiently. Anonymous games crop up in realistic situations, for example in analyzing traffic, where only the quantity of cars near you matters and not the identity of the drivers inside.
  2. Asim described a sophisticated Bayesian model well suited for social network data that handles non-existant links — meaning the lack of connection between two people, by far the most common situation — much better than previous approaches. The approach is good for digging deeply into a small data set but at least for now has difficulty with moderately large amounts of data. (To get results in a reasonable amount of time, Asim had to down sample his already fairly modest sized corpus.) The talk didn’t help me overcome my bias that Bayesian methods ala UAI often don’t work well at Internet scales without modification.
  3. Susan gave a fantastic and energetic talk. She advocates economic models of online advertising that include more sophisticated users, as opposed to typical models that assume users scan from the top of the page down in a precise sequence. She went further to claim that users may actually choose their search engine based on the quality of the ads. Personally, I’m a bit skeptical about that, though I do agree that there is an indirect effect: search engines with better paying ads can afford to buy more traffic and improve their algorithmic search more. Susan highlighted the enormous shift in mindset required between economic theory and practice when just computing the mean of a data stream can take weeks (though this is changing with tools like Hadoop that can bring such computations down to hours or minutes as Sebastien confirms).
  4. Someone asked Tuomas why his expressive commerce company CombineNet uses first-price auctions instead of VCG pricing. He listed four of what he said were dozens of reasons on top of Rothkopf’s thirteen and Ausubel and Milgrom’s list. In fact he went further to say that as far as he knew no real auction anywhere in the world has ever used true VCG pricing for anything more complicated than selling a single good at a time.
  5. For those not familiar, a rump session is open to anyone to speak briefly on any relevant topic. As it turns out, in part because brevity forces clarity, and in part because editorial filtering overweights mediocrity, the rump session is often the most interesting part of a conference. The “NYCE rump” session was no exception, with topics spanning ad auctions, reputation, Internet routing, and user generated content. Ivy Li proposed a clever scheme whereby eBay sellers are motivated to reward buyers for honest feedback. Sebastien presented work with Sihem and I on an expressive bidding language for online advertising with fast allocation and pricing algorithms, with the goal of moving the industry toward an open standard. Sampath Kannan on leave at NSF had encouraging news on the funding front, laying out his vision for CS theory funding with an explicit call for proposals at the boundary of CS and economics.
  6. I think we did a good job of attracting a diversity of speakers and participants, with talks ranging from computational complexity to Bayesian models of social networks, with academia and industry represented, and with CS, economics, and business backgrounds represented.
1We had dinner at Gobo, a fantastic restaurant Muthu recommended that truly opened my eyes in terms of the tastes and textures possible with a vegetarian menu. Delicious.
2Speaking of views, I had a stunning and fascinating one from my hotel the night before, looking straight down onto ground zero of the World Trade Center complex from a relatively high floor of the Millenium Hilton (apparently intentionally misspelled). I booked the room for $185 on Hotwire, and then found out why. Though the WTC site still looks nearly empty, builders appear to be making up for lost time with round the clock construction. Put it this way: the hotel kindly provided complementary earplugs. All in all though the room and view were well worth the cost in dollars and sounds.
3Specifically, computing Nash equilibrium is PPAD-complete for most games. In terms of complexity classes, PPAD is a superset of P and a subset of NP. Almost surely there is no polynomial time algorithm, though the problem is not quite as hard as the classic NP-complete problems like traveling salesman.

Even the stock market doesn't know how to report prices

The debate over how to report prediction market prices may seem to arise only because so many of the markets have low liquidity. If prediction markets were more liquid, the logic goes, then it wouldn’t matter if observers reported the last trade price, the average of the last several prices, or the bid-ask spread: they’d all be roughly the same. Indeed, in the extreme case of an “infinite liquidity” automated market maker, they are all the same.

Lance encountered the problem when a few rogue trades caused the colors on our electoral markets map to swing in what would seem to be irrational ways, briefly painting California red for McCain, for example.

However yesterday proved that even one of the most traded stocks on one of the largest volume financial exchanges in the world can suffer from bizarre trading oddities that make reporting meaningful prices an exercise in ad hockery.

It seems that Google’s stock gyrated wildly near the close of trading in entirely inexplicable ways that seem impossible to rationalize, and all this despite enormous volume traded. From SeekingAlpha:

[Here are] the official [NASDAQ] datapoints: share volume of 12 million shares (that’s about $5 billion), more than double the normal amount; an intraday high of $489, and — most improbable of all — an intraday low of just 1 cent per share.

What I found most incredible is that NASDAQ actually rewrote history in response:

Sep 30, 2008 17:01:02 ET Pursuant to Rule 11890(b) NASDAQ, on its own motion, has determined to cancel all trades in security Google Inc Cl – A “GOOG” at or above $425.29 and at or below $400.52 that were executed in NASDAQ between 15:57:00 and 16:02:00 ET. In addition, NASDAQ will be adjusting the NASDAQ Official Closing Cross (NOCP) and all trades executed in the cross to $400.52. This decision cannot be appealed. MarketWatch has coordinated this decision to break trades with other UTP Exchanges. NASDAQ will be canceling trades on the participant’s behalf.

I had no idea that stock exchanges canceled trades “just because”. Barring system error, this seems just plain wrong — certainly worthy of serious outrage from traders. If someone agrees to trade at a wildly irrational price that’s their own problem and they should have to live with it.

Apparently it’s not only possible, but common. On the same day NASDAQ canceled trades in ROH deemed out of bounds:

Sep 30, 2008 17:14:37 ET Pursuant to Rule 11890(b) NASDAQ, on its own motion, has determined to cancel all trades in security Rohm and Haas Company (ROH) at or above $73.20 and at or below $68.93 that were executed in NASDAQ between 15:57:00 and 16:02:00 ET. This decision cannot be appealed.

Suddenly our hack fix to the electoral markets map* and the various controversial revisions at intrade and betfair [1, 2] don’t seem quite so crazarbitrary in comparison.

*We now report the last trade price only if it falls between the current bid-ask spread, otherwise we report the bid or ask price, whichever is closer to the last price. After all, if the last price falls outside the bid-ask boundaries, it no longer reflects current market sentiment.

Political ads: Insuring your message gets across. Literally.

Centrist Messenger How It Works SnippetHere’s a brilliant idea: Centrist Messenger let’s you buy political ads with a money-back guarantee. You pay only if your preferred candidate wins. If the other candidate wins, you get your money back.

Centrist Messenger backs the guarantee with contracts purchased from intrade, in the same way that Priceline backs its “Sunshine Guarantee” with contracts from WeatherBill. (So presumably fully insured ads cost about twice as much as uninsured ads.)

In addition, the ads you buy can’t be too partisan:

Centrist Messages can … make strong advocacy of a position and candidate. However, this advocacy cannot demonize the other side, focus solely on personality, or make false representations of the candidates’ positions.

I’ll add Centrist Messenger to WeatherBill, Priceline, Yoonew, and FirstDIBZ (was TicketReserve) as companies fashioning creative ways to package and sell “markets in uncertainty” in the US amid a challenging legal and regulatory landscape.

What other useful and/or fun ways can you imagine re-packaging gambles as either insurance or contingent goods? Here are some of my own brainstorms:

  • Buy a ticket to a sporting event whose cost is refunded if your team loses.
  • Buy a “streak ticket”: entitles you to a ticket to the next game as long as your team keeps winning. (Variant: “K-loss ticket” entitles you to tickets until your team loses K times.)
  • Buy a “playoff run ticket” which gives you tickets, flights, hotel, etc. for the duration of your team’s playoff run. In other words, as long as your team keeps winning, you keep getting tickets, hotel, and flight to the next game. You may be able to buy this at the beginning of the season cheaply since it’s worth nothing if your team does not make the playoffs.
  • Buy “price drop” insurance: If that precious electronic gadget you just bought (read: iPhone) drops in price within N days, get K times your money back.

Yahoo! Open Hack Day Sunnyvale, Sept 12-13, 2008

Yahoo! Open Hack Day 2008 As Jed points out, “an idea is only the first step in innovation, and it’s by far the easiest step”.

Yahoo! Hack Day was created precisely to summon and celebrate the hard step of innovation: the build it step. The goal is simple: take an idea and make it real — in 24 hours. Spend all day and all night coding until a working, useable, if brittle prototype of your idea emerges. Then show it off.

Hack Day is a religion inside Yahoo!, but on September 12-13, 2008, Yahoo! will open up its Sunnyvale campus, inviting any developer who feels like it to join the geek-out frenzy. Sign up here.

Schedule

8am-6pm PT Friday: Over 20 workshops covering YUI and the newest API offerings from Yahoo!, including BOSS, SearchMonkey, Fire Eagle, and more, and previews of what’s next.

8pm Friday: A surprise musical guest takes the stage (it’s not 2006 guest Beck, but apparently the lyrics are “hacker-friendly” and “may not be appropriate for young children”). Hacking continues all night.

2pm Saturday: Judging, including a special hack-off for the winners of the University Hack Days.

Saturday evening: Awards.

History, thoughts, and notes

At the first Open Hack Day in 2006 in Sunnyvale (see photos), 400 developers fueled by 500 pizzas and a live Beck performance cranked out 54 hacks. At Open Hack Day London lightening struck twice and it rained indoors. Bangalore followed.

If you’re a student, Yahoo! Hack Week may be coming to a campus near you. We’ve held Hack Weeks at Georgia Tech, CMU, UIUC, UC Berkeley, and Stanford, and I believe Waterloo is next. Here’s a quote describing these Hack U events: “Computer science students fueled by fast food, ultra-caffeinated beverages, and alternative music, are free to let their imaginations run wild, tapping the Yahoo! library of APIs to create hacks that advance the Internet experience.”

Why Hack Day? Many an engineer join Jed in lamenting how the PowerPointy set co-opted the term innovation, rendering it almost meaningless. Hack Day was created in part to reclaim innovation for the makers.

Why does it work? Hack Day is to programmers what NaNoWriMo is to writers: a timed artistic challenge that on it’s face is a ludicrous and artificial pretense for accomplishing a goal. Yet somehow the exercise induces a psychological state perfect for making progress on the initial “80% phase” of creation. The punishing deadline forces all meta processing aside — no critic, no perfectionist, no planner, no lazy dreamer — and encourages the raw energy embodied by Nike’s Kirk-beats-Spock slogan “just do it”.

A number of Yahoo! products, services, and features were born on a Hack Day (here are two). Yoopick was too.

Why open?

Openness is one of only three overarching goals for Yahoo!. The other two goalsstarting point for users and must buy for advertisers — are in some sense incontrovertible, yet the openness goal reflects a riskier “if we build it they will come” stand that’s grounded in Yahoo!’s respect for and debt of gratitude to Internet culture. Open Hack Day, Hadoop, Pig, cloud computing, academic relations, publications, APIs, BOSS, SearchMonkey, YUI, Pipes, OpenSocial, and OpenID are among the many examples showing that Yahoo!’s commitment to openness is real. (Jeremy, RIP 2008, said it better.)

Update 2008/09/11: Blueprint mobile SDK and more Y! Open announcements (music, homepage, mail, ads)

Update 2008/09/26: CNET says: Yahoo Open: Finally, a real answer to Google. Also, Google spouse Kara Swisher gets defensive and rewrites history.

New Yahoo! News election dashboard

Cross-posted on midasoracle.org

The Yahoo! News Political Dashboard has re-launched for the general election stretch run of the 2008 US Presidential election.

Yahoo! News political dashboard for the 2008 US general Presidential election

From the main map you can see the status of the election in every state according to either polls or Intrade prediction market odds. Hover your mouse over a state to see current numbers or click on a state to see historical trends. On the side, help you can see search trends, price blogs, story news, and demographic breakdowns at national and state levels.

You can also “create your own scenario” by picking who will win in every state. You can save and share your prediction and compare against markets, polls, history, or celebrities. More on ycorpblog.

In the markets view, states are colored either bright red or bright blue, regardless of how close the race is in that state. To see a visualization that blends colors to reflect the tightness of the race, see electoralmarkets.com.

Yahoo! News also offers a candidate badge that you can display on your blog declaring your choice. The badge features national-level polls, prediction markets, search buzz, and money raised.

New York Computer Science and Economics Day (NYCE Day) October 3, 2008

We invite participants to the first New York Computer Science and Economics Day (NYCE Day), viagra 60mg October 3 2008, unhealthy at the New York Academy of Sciences, 7 World Trade Center.

NYCE Day is a gathering for people in the NYC metropolitan area with interests in auction algorithms, economics, game theory, e-commerce, marketing, and business to discuss common research problems and topics in a relaxed environment. The aim is to foster collaboration and the exchange of ideas.

The program features invited speakers Asim Ansari (Columbia), Susan Athey (Harvard), Constantinos Daskalakis (MIT), and Tuomas Sandholm (CMU), and a rump session with short contributed presentations.

You can indicate your interest in the event on upcoming.yahoo but official registration should go through NYAS.

Your participation and suggestions are greatly welcome. Please distribute this announcement to people and groups who may be interested.

Thanks,
NYCE Day Organizers
 Anindya Ghose, NYU
 S. Muthu Muthukrishnan, Google
 David Pennock, Yahoo!
 Sergei Vassilvitskii, Yahoo!

P.S. This is one week prior and in the same location as the Symposium on Machine Learning.

P.P.S. For those familiar, NYCE Day is inspired as a Right Coast version of BAGT.

P.P.P.S. The New York Academy of Sciences in a spectacular venue. See for yourself.

Pipes dream

If you haven’t played around with Yahoo! Pipes, I highly recommend it. It’s a usable and useful service that brings web mashups to the masses, making this favorite hacker pastime as easy as dragging objects around on the screen.

For example, it took me probably about ten minutes as a first-time user to create a map mashup showing Barack Obama’s upcoming campaign stops. I “piped” the output of Washington Post’s RSS feed to a location-extractor module that identifies and geo-codes place names and renders them on a map. Here’s a screenshot of the output:

Screen shot of Yahoo! Pipe: Barack Obama 2008 US Presidential Election Campaign Travel Map

The easiest way to get started is to find an existing Pipe, clone it, and modify it as your own. Using this feature, I cloned my Obama map and in about one minute had a McCain map too.

Pipes uses a visual programming interface. The idea of “programming by picture” (I recall playing with one in the 1980s) never took hold as a mainstream tool. However, as a metaphor for mashups, where to goal is to chain together a number of sources and services, the visual approach seems exactly right. The implementation in a browser is a feat of ajaxian magic that I still find remarkable, even as Yahoo! and others are commoditizing the art. I imagine that even non-programmers should have little trouble constructing their own Pipes. Here is a screenshot of the source “code” for my Obama map:

Source code of Yahoo! Pipe: Barack Obama 2008 US Presidential Election Campaign Travel Map

Pipes has dozens of useful modules, including user input, Yahoo! Search, Flickr, and regular expressions.

You can embed the Pipe on your own website with a single line of javascript. I did this with my Obama and McCain campaign travel maps here. Or you can grab the output as an XML feed to use however you wish.

Pipes allows you to create human-readable URLs (e.g., http://pipes.yahoo.com/oddhead/obamatravelmap), a nice touch.

The icing on the cake for me is how Pipes — unlike so many other web sites, including some on Yahoo! — treats me and my Opera browser like adults:

Yahoo! Pipes treats me and my Opera browser like adults

(BTW, Pipes seems to work fine on Opera).

Unfortunately, Daniel Raffel, one of the key founders of Yahoo! Pipes, left Yahoo!. However, the team seems to be strong and continues to innovate, so I’m hopeful this fantastic service will continue to improve and thrive.

Quantcast, Scribd, and the two-minute web service signup

I joined the quantcast audience measurement service. It took about two minutes to sign up and initiate tracking. I’m impressed with the ease of use, the utility, and the inroads the company has made in the year or so since former Yahoo Mike Speiser first showed it to me.

Looks like I’m getting about 1000 visitors a month, roughly 3/4 that of Chris, 1/6 of Robin, 1/10 of Lance, 0.00079% of my employer, and 0.00073% of my employer’s frenemy.

I also joined the scribd document hosting service (“Youtube for documents”) and used it to embed a PDF in my previous post. Again, from signup to service took a matter of minutes. (I think scribd could be great for hosting my publications which are in need of both a content and interface update.)

Probably there’s some sort of business axiom here, probably already blogged and book-ed: the two minute rule of successful web services.