Chris Wilson reviews Nate Silver's new book, The Signal and the Noise:
Predictions fail for many reasons, but Silver contends that the most frequent cause is a flaw in the forecaster’s assumption, as opposed to an inherent shortcoming in the methodology. The now notorious ratings agencies that continued to stamp AAA ratings on collections of highly risky mortgages into 2008, he notes, would nonetheless present their findings with a full flourish of statistical exactitude, working out the odds of a security paying out to two decimal places. But the larger problem, of course, was with the foundational assumptions that shaped the universe of mortgage-backed securities—the supposition, for example, that the worst-case scenario was a manageable downturn in housing prices that was adequately provided for in their models.
Put another way:
You might call this species of cocked-up forecasting the tyranny of significant digits; more broadly, it is the cardinal mistake of dressing up uncertainty—an incalculable unknown—with risk, a highly calculable gamble with discrete odds. Risk is gambling on a flush in poker, knowing the odds are one in four of drawing the suit that you need; uncertainty is playing poker without a clear idea of the rules or the distribution of cards in the deck.
Previous coverage of the book here and here. On a related note, Alison Gopnik notes that children, like Silver, learn by Bayesian inference.
The Dish 