Category Archives: hacking

Oddhead Blog hacked… for the third time

My blog has been hacked yet again. For those keeping track, that’s infection number three. This latest exploit is very similar to the previous one. To humans arriving via browser (e.g., me), the site appears perfectly normal and healthy. Even upon clicking ‘view source’, nothing untoward is revealed. The <title> of my blog is, as always, Oddhead Blog.

However, when Google’s or Bing’s crawlers arrive to index my corner of the web, they see a different <title> altogether — Buy Cheap Cialis Online  — and immediately roll their eyes. (Actually even if you run 'curl http://blog.oddhead.com', you’ll see the spam keywords.) The effect of the attack is a kind of reverse cloaking. Cloaking is the black-hat SEO practice of serving legitimate content to crawlers and spam content to people. Here, the spam content is shown to the crawlers and the legitimate content to the people.

Once the crawlers report this appalling information back to their respective mother ships, the search engines have no choice but to delist and demote my blog in their pagerankings. Right now, if you search for or within Oddhead Blog on Google, you’ll see how poorly the bots in Mountain View think of me:

Oddhead Blog hacked again: Spam titles in Google's cache 2012-04-27

You can hardly find any deep links into my blog by searching Google. For example, try searching for Bem+Wom, my invented term for “BEtter Mousetrap, Word of Mouth”. Even try “Bem+Wom oddhead blog”. You”ll find aggregators republishing my content, but no links to the original source, my blog, anywhere in sight. (Note to self: the Bing results for Bem+Wom are awful.)

Once again I am at a loss to understand my attacker’s motivation. Clearly it’s not to sell Cialis to my users, as they remain blissfully ignorant of any changes. The only benefit to anyone is to remove one relatively obscure blog from the search engine rankings and thus to move the attacker one slot up. Having a blog tangentially about gambling probably puts me into a shady neighborhood of the web, yet reverse-cloaking your competition (even if it can be somewhat automated and strike more than one competitor) seems like an awfully indirect way to improve one’s standing in Google. It’s also possible this is an act of pure vandalism.

So what should I do? Although I partly blame WordPress for writing insecure software, I may end up paying WordPress protection money to make this problem go away. I am seriously considering giving up on self hosting and moving my whole operation to worpress.com’s hosted service, where presumably security is tighter, or at least it’s not my responsibility any more. My web hosting service, DreamHost, may also be partly to blame, yet I like the company and have been quite happy with them in many respects. Any advice, dear reader? WordPress.com? Blogger? Try again and hope the fourth time is the charm? Should I be looking to ditch DreamHost as well?

We’re baaack: Predictalot is here for March Madness 2011

March Madness is upon us and Predictalot, the crazy game that I and others at Yahoo! Labs invented, is live again and taking your (virtual) bets. Filling out brackets is so 2009. On Predictalot, you can compose your own wild prediction, like there will be exactly seven upsets in the opening round, or neither Duke, Kentucky, Kansas, nor Pittsburgh will make the Final Four. You’ll want your laptop out and ready as you watch the games — you can buy and sell your predictions anytime, like stocks, as the on-court action moves for or against you.

Predictalot v0.3 is easier to play. We whittled down the ‘Make Prediction’ process from four steps to just two. Even if you don’t want to wager, with one click come check out the projected odds of nearly any crazy eventuality you can dream up.

Please connect to facebook and/or twitter to share your prediction prowess with your friends and followers. You’ll earn bonus points and my eternal gratitude.

The odds start off at our own prior estimate based on seeds and (new this year) the current scores of ongoing games, but ultimately settle to values set by “the crowd” — that means you — as predictions are bought and sold.

Yahoo! Labs Predictalot version 0.3 overview tab screenshot

For the math geeks, Predictalot is a combinatorial prediction market with over 9 quintillion outcomes. Prices are computed using an importance sampling approximation of a #P-hard problem.

What kind of information can we collect that a standard prediction market cannot? A standard market will say that Texas A&M is unlikely to win the tournament. Our market can say more. Yes, A&M is unlikely to reach the Final Four and even more unlikely to win apriori, but given that they somehow make it to the semifinals in Houston, less than a two hour drive from A&M’s campus, their relative odds may increase due to a home court advantage.

Here’s another advantage of the combinatorial setup. A standard bookmaker would never dare to offer the same millions of bets as Predictalot — they would face nearly unlimited possible losses because, by tradition, each bet is managed independently. By combining every bet into a single unified marketplace, we are able to limit the worst-case (virtual) loss of our market maker to a known fixed constant.

Predictalot goes East: Introducing Predictopus for the ICC Cricket World Cup

Yahoo! India Predictopus logo

I’m thrilled to report that Predictalot had an Indian makeover, launching as Predictopus* for the ICC Cricket World Cup. The Yahoo! India team did an incredible job, leveraging the idea and some of the code base of Predictalot, yet making it their own. Predictopus is not a YAP — it lives right on the Yahoo! Cricket website, the official homepage for the ICC Cricket World Cup. They’re also giving away Rs 10 lakhs — or about $22,000 if my calculations are correct — in prizes. Everything is bigger in India, including the crowds and the wisdom thereof. It will be great to see the game played out on a scale that dwarfs our college basketball silliness in the US.

The Y! India team reused some of the backend code but redid the frontend almost entirely. To adapt the game to cricket, among other chores, we had to modify our simulation code to estimate the starting probabilities that any team would win against any other team, even in the middle of a game. (How likely is it for India to come back at home from down 100 runs with 10 overs left and 5 wickets lost? About 25%, we think.) These starting probabilities are then refined further by the game-playing crowds.

It’s great to see an experiment from Labs grow into a full-fledged product run by a real product team in Yahoo!, a prime example of technology transfer at its best. In the meantime, we (Labs) are still gunning for a relaunch of Predictalot itself for March Madness 2011, the second year in a row. Stay tuned.

2011/02/24 Update: An eye-catching India-wide ad campaign for predictopus is live, including homepage, finance, movies, OMG, answers, mail, everywhere! Oh, and one of the prizes is a Hyundai.

predictopus ad on Yahoo! India homepage 2011/02/24


* Yes, that’s a reference to legendary Paul the Octopus, RIP.

It’s official: More people are playing Predictalot than Mafia Wars

It’s true.

More people are playing Predictalot today than Mafia Wars or Zynga Poker… On Yahoo!, that is.

In fact, Predictalot is the #1 game app on Yahoo! Apps by daily count. By monthly count, we are 5th and rising.

A prediction is being made about every three minutes.

Come join the fun.

predictalot most popular game app on yahoo 2010-06-12

Predictalot for World Cup: Millions of predictions, stock market action

I just left the 2010 ACM Conference on Electronic Commerce, where six (!) out of 45 papers were about prediction markets.

Yahoo! Lab’s own Predictalot market is now live and waiting for you to place almost any prediction your heart desires about the World Cup in South Africa.

Here are some terribly useful things you can learn this time around. All numbers are subject to change, and that’s kind of the point:

  • There’s a 37% chance Brazil and Spain will both make it to the final game; there’s only a 15% chance that neither of them will make it
  • There’s is a 1 in 25 chance Portugal will win the cup; 1 in 50 for Argentina
  • 42.92% chance that a country that has never won before will win
  • 19.07% chance that Australia will advance further than England
  • 65.71% chance that Denmark, Italy, Mexico and United States all will not advance to Semifinals
  • Follow Predictalot on twitter for more

If you think these odds are wrong, place your virtual wager and earn some intangible bragging rights. You can sell your prediction any time for points, even in the middle of a match, just like the stock market.

There are millions of predictions available, yet I really believe ours is the simplest prediction market interface to date. (Do you disagree, Leslie?) We have an excellent conversion rate, or percent of people who visit the site who go on to place at least one prediction — for March Madness, that rate was about 1 in 5. One of our main goals was to hide the underlying complexity and make the app fast, easy, and fun to use. I personally am thrilled with the result, but please go judge for yourself and tell us what you think.

In the first version of Predictalot, people went well beyond picking the obvious like who will win. For example, they created almost 4,000 “three-dimensional” predictions that compared one team against two others, like “Butler will advance further than Kentucky and Purdue”.

If you’re not sure what to predict, you can now check out the streaming updates of what other people are predicting in your social circle and around the world:

Predictalot recent activity screenshot 2010-06-11 18:45

Also new this time, you can join a group and challenge your friends. You can track how you stack up in each of your groups and across the globe. We now provide live match updates right within the app for your convenience.

If you have the Yahoo! Toolbar (if not, try the World Cup toolbar), you can play Predictalot directly from the toolbar without leaving the webpage you’re on, even if it’s Google. 😉

playing predictalot from the yahoo! toolbar

Bringing Predictalot to life has been a truly interdisciplinary effort. On our team we have computer scientists and economists to work out the market math, and engineers to turn those equations into something real that is fast and easy to use. Predictalot is built on the Yahoo! Application Platform, an invaluable service (open to any developer) that makes it easy to make engaging and social apps for a huge audience with built-in distribution. And we owe a great deal to promotion from well-established Yahoo! properties like Fantasy Sports and Games.

We’re excited about this second iteration of Predictalot and hope you join us as the matches continue in South Africa. We invite everyone to join, though please do keep in mind that the game is in beta, or experimental, mode. (If you prefer a more polished experience, check out the official Yahoo! Fantasy Sports World Soccer game.) We hope it’s both fun to play and helps us learn something scientifically interesting.

Read more here, here, and here.

Or watch a screencast of how to play:

Meet my maker, or Bernie Pennock and the blowtorch fountain

My dad is an original maker. When I didn’t want to pay $200 to replace a broken car key housing, he sent me this vice made out of quarters he fashioned and all the parts I needed to attach it to the key.

Using quarters as a vice to hold a key

Aa biomedical engineer, he led a study showing that a non-invasive mask can save people from respiratory failure as well as intubation. The technique is now common practice and, fittingly, the device helped saved his own life several years ago. He also invented a piezoelectric band to measure heart rate and breathing during sleep more comfortably than electrodes.

He did his Ph.D. dissertation on, in a sense, protien folding, in the days when cut and paste meant scissors and glue. I have an original copy of his dissertation and it’s a beautiful object to behold.

Bernie Pennock's Ph.D. dissertation 1Bernie Pennock's Ph.D. dissertation 2Bernie Pennock's Ph.D. dissertation 3

And what about that blowtorch fountain?

Bernie Pennock and the Blowtorch Fountain

Read about it in this profile of my dad by Maureen Simpson highlighting both his hacker and painter sides.

In his retirement, Bernie Pennock found a way to turn fire into water.
The former medical research scientist said it was just one of the many problems that needed solving in his home, where art has become the answer.
“It’s really the same idea as what I did as a career,” Pennock said of his hobby. “You see a problem and think of how to solve it. I think of what I want to do and how to do it, and then I do it and see if it works.”
Using old brass blowtorches he has collected over the years from antique shops and friends, Pennock constructed a fountain next to the pathway leading up to his front door…
Instead of spitting flames, Pennock’s structure spouts water. He mounted the old-fashioned tools to a sheet of copper and then rigged a water pump and pipes behind it… Pennock said his friends describe the work of art as “very Rube Goldberg.”…
Inside his home — on lampshades, along walls and attached to windows — guests can see numerous examples of the former scientist’s artistic experiments. His most recent obsession, apart from the fountain, has been working with stained glass.
“It all started with this window that looks out on the pool,” Pennock said. “I wanted something that let in light, but wouldn’t allow you to see into the bathroom. When I got an estimate to find out how much it would cost to have someone do a stained glass window, I decided to make my own.”
The multi colored scene is based on a photograph Pennock took of two people walking on the beach. Since then, he has made at least a dozen more windows that include a copy of a Monet painting, the Talmadge Bridge in Savannah and his interpretation of
12 stained glass windows designed by Marc Chagall at the synagogue of the Hadassah-Hebrew University Medical Center in Jerusalem, Israel.
Pennock said he usually buys the windows from a Habitat for Humanity store and gets his stained glass from a supplier in Charleston. The next project he plans to take on is a bamboo sculpture, because he’s running out of windows.
“I dabble in a lot of things,” Pennock said. “I like to invent. I just start from scratch, get ideas and see what happens.”
Among his rules for living, which Pennock painted on leftover floor tiles that hang next to the blowtorch fountain, is fittingly: “Pay attention.”

Oh, my brother and sister are makers too. And my mom a trailblazer. I’ll leave those for another day.

__________
This more personal post inspired because Robin says Tyler says it’s OK.

Let the madness begin

Sixty-five men’s college basketball teams have been selected. Tomorrow there will be sixty-four. Half of the remaining teams will be eliminated twice every weekend for the next three weekends until only one team remains.

On April 5th, we will know who is champion. In the meantime, it’s anybody’s guess: any of 9.2 quintillion things could in principle happen.

At Predictalot it’s your guess. Make almost any prediction you can think of, like Duke will win go further than both Kansas and Kentucky, or the Atlantic Coast will lose more games than the Big East. There’s even the alphabet challenge: you pick six letters that include among them the first letters of all four final-four teams.

Following Selection Sunday yesterday, the full range of prediction types are now enabled in Predictalot encompassing hundreds of millions of predictions about your favorite teams, conferences, and regions. Check it out. Place a prediction or just lurk to see whether the crowd thinks St. Mary’s is this year’s Cinderella.

Come join our mad science experiment where crowd wisdom meets basketball madness. We’ve had many ups and down already — for example sampling is way trickier than I naively assumed initially — and I’m sure there is more to come, but that’s part of what makes building things based on unsolved scientific questions fun. Read more about the technical details in my previous posts and on the Yahoo! Research website.

And for the best general-audience description of the game, see the Yahoo! corporate blog.

Update: Read about us on the New York Times and VentureBeat.

You can even get your fix on Safari on iPhone!

Dave playing Predictalot on iPhone

Below is a graph of our exponential user growth over the last couple days. Come join the stampede!

graph of YAP installs for Predictalot

Predictalot! (And we mean alot)

I’m thrilled to announce the launch of Predictalot, a combinatorial prediction market for the NCAA Men’s Basketball playoffs. Predict almost anything you can think of, like Duke will advance further than UNC, or Every final four team name will start with U. Check the odds and invest points on your favorites. Sell your predictions anytime, even as you follow the basketball games live.

The basic game play is simple: select a prediction type, customize it, and invest points on it. Yet you’ll never run out of odds to explore: there are hundreds of millions of predictions you can make. The odds on each update continuously based on other players’ predictions and the on-court action.

Predictalot is a Yahoo! App, so you can play it at apps.yahoo.com or you can add it to your Yahoo! home page. I have to admit, it’s an incredible feeling to play a game I helped design right on the Yahoo! home page.

Predicalot app on the Yahoo! home page

That’s all you need to get started. If you’re curious and would like a peek under the hood, read on: there’s some interesting technology hidden in the engine.

Background and Details

Predictalot is a true combinatorial prediction market of the sort academics like us and Robin Hanson have been dreaming about since early in the decade. We built the first version during an internal Yahoo! Hack Day. Finally, we leveraged the Yahoo! Application Platform to quickly build a public version of the game. (Note that anyone can develop a YAP app that’s visible to millions — there’s good sample code, it supports YUI and OpenSocial, and it’s easy to get started.) After many fits and starts, late nights, and eventually all nights, we’re proud and excited to go live with Predictalot version 1.0. I can’t rave enough about the talent and dedication of the research engineers who gave the game a professional look and feel and production speed, turning a pie-in-the-sky idea into reality. We have many features and upgrades in mind for future versions, but the core functionality is in place and we hope you enjoy the game.

In the tournament, after the play-in game, the 64 top college basketball teams play 63 games in a single elimination tournament. So there are 2 to the power 63 or 9.2 quintillion total possible outcomes, or ways the entire tournament can unfold. Predictalot implicitly keeps track of the odds for them all. To put this in perspective, it’s estimated that there are about 10 quintillion individual insects on Earth. Of course, for all practical purposes, we can’t store 9.2 quintillion numbers, even with today’s computers. Instead, we compute the odds for any outcome on the fly by scanning through the predictions placed so far.

A prediction is a statement, like Duke will win in the first round, that will be either true or false in the final outcome. In this case, the prediction is true in exactly half, or 2 to the power 62 outcomes. (Note this does not mean the odds are 50% — remember the outcomes themselves are not all equally likely.) In theory, Predictalot can support predictions on any set of outcomes. That’s 2 to the power 2 to the power 63, or more than a googol predictions. For now, we restrict you to “only” hundreds of millions of predictions categorized into thirteen types. Computing the odds of a prediction precisely is too slow. Technically, the problem is #P-hard: as hard as counting SAT and harder than the travelling salesman problem. So we must resort to approximating the odds by randomly sampling the outcome space. Sampling is a tricky business — equal parts art and science — and we’re still actively exploring ways to increase the speed, stability, and accuracy of our sampling.

Because we track all possible outcomes, the predictions are automatically interconnected in ways you would expect. A large play on Duke to win the tournament instantly and automatically increases the odds of Duke winning in the first round; after all, Duke can’t win the whole thing without getting past the first round.

With 9.2 quintillion outcomes, Predictalot is to our knowledge the largest prediction market built, testing the limits of what the wisdom of crowds can produce. Predictalot is a game, and we hope it’s fun to play. We’d also like to pave the way for serious use of combinatorial prediction market technology.

Why did Yahoo! build this? Predictalot is a smarter market, letting humans and computers each do what they do best. People enter predictions in simple terms they understand like how one team fares against another. The computer handles the massive yet methodical number crunching needed to combine all the pieces together into a coherent overall prediction of a complex event. Markets like Predictalot, WeatherBill, CombineNet, and Internet advertising systems, to name a few, represent the evolution of markets in the digital age, empowering users with extreme customization. More and more, matching buyers with sellers — the core function of markets — requires sophisticated algorithms, including machine learning and optimization. Predictalot attempts to illustrate this trend in an entertaining way.

David Pennock
Mani Abrol, Janet George, Tom Gulik, Mridul Muralidharan, Sudar Muthu, Navneet Nair, Abe Othman, Daniel Reeves, Pras Sarkar

Notes from Yahoo! Open Hack Day NYC

Here are my notes from Yahoo! Open Hack Day NYC. For other perspectives read New York Times open sourcerer Nick Thuesen or the Yahoo! devel blog. You can watch videos of some of the talks or browse pictures.

First off, I cheated. I went to sleep in a hotel room rather than hack all through the night. (Even in college I woke up at 4am rather than pull an all nighter.) Still, I made decent progress on some pet projects including combinatorial betting. Daniel, Sharad, and Winter from Yahoo! Research New York participated for real, working through the night. Returning in the morning showered and caffeinated to greet the sleepwalkers was a little surreal. A number of ex-Yahoos joined the festivities including David Yang, Mor Naaman, and Chad Dickerson. (Havi joked that Yahoo! is like finishing school for entrepreneurs. If you count Yahoo! capture and releases like Mark Cuban and Paul Graham, the spreading influence is enormous.)

Clay Shirky kicked off the event. He’s a fantastic speaker — watch his talk here. His punch line — that successful communities like facebook, twitter, flickr, and wikipedia start small and cohesive (as opposed to large and fragmented: see Yahoo! 360) — was aimed perfectly at the many founders and foundreamers in the audience. There were speakers from Mint and foursquare and tutorials on the Yahoo! Application Platform, Yahoo! Query Language (the most popular service), Yahoo! TV widgets, and more. There was a round of Ignite NYC, a barrage of twenty-slides-in-five-minutes talks, some educational (geek’s guide to patents), some charitable (aid to South America), some hilarious (spaceman from outerspace), some thought provoking (makerbot 3d printers), and many all of the above (meta mechanical turk; the Emoji translation of Moby Dick). Watch the Ignite talks here.

A bunch of small touches made the event memorable, including a steampunk-themed hacking hall complete with retroRed Victorian couches, portraits of hackers through history, funky tweet-streaming sculptures, chalk drawings of old patents, power cords dangling from hanging bird cages, and a guitarherofoosball corner. The food was tasty and at times eccentric, like the hot dog stand and toppings bar under a rainbow umbrella, ice cream cart, and old-fashioned popcorn machine. There was plenty of beer, coffee, red bull, sliders, and cookies, and even (gasp) vegan fare, salmon, and salad.

I give the event an A for style (decor, food) and content (talks, hacks, organization). The one sour note was the wireless — certainly a key ingredient for a good hack day — which began flaky and ended slow but acceptable.

I attended the YAP tutorial and created a rudimentary application. I was pleasantly surprised how simple the process was — the documentation and sample code are great. You can get the hello world app (complete with social hooks) running and add some ajax magic within minutes.

By far one of the coolest sights was the MakerBot Industries 3D printer in action. It sucks in plastic wire, melts it, and deposits it in perfect formation to produce coins, busts, parts for itself, or almost anything in the thingiverse. For Hack Day, the device printed news headlines in peanut butter on toast. We met an nyc resistor who was working on a conveyer belt mechanism for his own MakerBot printer, and he invited us to craft night at their shared hackspace in Brooklyn (a place that would be heaven for my dad and brother; Sharad, Jake, Daniel, and Bethany went to check it out).

I missed the tutorial on Yahoo! TV widgets but I’d like to learn more. They are now in most major TV brands including Sony, Samsung, and LG — millions of sets around the world in the coming months. (The Sony won editor’s choice in the Sept 2009 issue of Wired magazine; the Samsung and LG rated close behind. The sole TV reviewed without Yahoo! Widgets, a Panasonic, was ridiculed for is clunky Viera Cast online interface.) If you’re an internet video startup, like my friend, you need a widget channel. Personally, I’d love to see a sports game tracker that highlights pivotal moments by monitoring in-game betting odds.

Footnote: Two Yahoos made a humorous video (that’s both self-promotional and -deprecating) on what people in Times Square think ‘hacker’ means:

See Paul Tarjan and Christian Heilmann for real definitions.

Yahoo! Open Hack Day NYC, Oct 9-10, 2009

Yahoo! Open Hack Day NYC 2009Join us on October 9, 2009 at the Millennium Broadway Hotel in New York City for Yahoo! Open Hack Day NYC. Come to listen, learn, and meet, but mainly come to make. Your goal: in 24 hours hackmash something together for bragging rights and prizes. Speakers include Clay Shirky (NYU), Carrie Cronkey (Mint.com), Dennis Crowley (foursquare), and Rasmus Lerdorf (inventor PHP). Register here. It’s free.

The 24-Hour Hackathon begins Friday afternoon. We encourage you to play around with Yahoo!’s Open Platforms and APIs like YAP, YQL, YUI, TVWidgets, our Social APIs, and more. And of course, feel free to use other APIs, developer tools and whatever software/hardware floats your boat…

At the end of the 24 hours, the hackers will have the chance to debut their hack and winners will be awarded with some enviable prizes…

And of course we will keep you well fed and hydrated throughout the two days. There will also be sleeping areas in case you want to take a nap.

Previous: the what and why of Open Hack.