It seems that even D.I.Y. freakonomists aren’t sure how to judge probability forecasts.
In measuring precipitation accuracy, the study assumed that if a forecaster predicted a 50 percent or higher chance of precipitation, they were saying it was more likely to rain than not. Less than 50 percent meant it was more likely to not rain.
That prediction was then compared to whether or not it actually did rain…
Is this the data you were looking for?
Precip Prediction Actually Rained
0% 7.9% of the time
10% 5.3%
20% 10.8%
30% 19.2%
40% 26.5%
50% 27.8%
60% 46.2%
70% 58.0%
80% 58.1%
90% 63.6%
100% 66.7%
Give the guy a break. It sounds like a lot of work to do that project.
Have a great weekend!
Yes, that’s the main thing I was looking for, thanks Josh. (For readers, the study’s author, J.D. Eggleston, posted these calibration results in the comments.)
You’re right. Taking a step back, this “bug” is a relatively minor one in the midst of an interesting report that is the clear result of a lot of effort, care, and competence. I admire and applaud the work, and I would encourage other D.I.Y. -onomists -ologists -ientists and -icians to undertake similar studies despite nitpickers like me.
The calibration data is interesting for its inaccuracy. Not that I’m surprised; if I were an on-air personality, I’d forecast the most pessimistic rain forecast possible so that when it wasn’t so bad viewers would be generally happy!
Interesting observation, Jed. Makes sense. It would be interesting if under some simple non-linear utility model (e.g., viewers prefer PredictRain-NoRain to PredictNoRain-Rain) the forecasts are actually rational.