Don’t trust expert predictions

Last night I finished Philip Tetlock’s Expert Political Judgment:

…no matter how unequivocal the evidence that experts cannot outpredict chimps or extrapolation algorithms, we should expect business to unfold as usual: pundits will continue to warn us on talk shows and op-ed pages of what will happen unless we dutifully follow their policy prescriptions.  We — the consumers of expert pronouncements — are in thrall to experts for the same reasons that our ancestors submitted to shamans and oracles: our uncontrollable need to believe in a controllable world and our flawed understanding of the laws of chance.  We lack the willpower and good sense to resist the snake oil products on offer.  Who wants to believe that, on the big questions, we could do as well tossing a coin as by consulting accredited experts?

Daniel Drezner has two excellent posts on the book, here and here.  Here is Louis Menand’s glowing review from The New Yorker.  Here is Tetlock’s home page.  Here is a sample book chapter.

And yes Tetlock has data, drawing upon twenty years of observation of 82,361 forecasts.  Tetlock also finds that "foxes" forecast better than "hedgehogs" and that only the forecasts of foxes have positive value.

This is one of the (few) must-read social science books of 2005.

My caveat: Assume that the experts are usually wrong in their novel predictions.  The consensus views of a science still might be worth listening to.  Economists cannot forecast business cycles very well, but you should listen when they tell you that a deflationary shock is bad news.  Each new forecast or new theory is an example of individual hubris and in expected value terms it is stupid.  But the body of experts as a whole, over time, absorbs what is correct.  A large number of predictions creates a Hayekian discovery process with increasing returns to scale.  Social knowledge still comes out ahead, and in part because of the self-deceiving vanities put forward every day.  You can find that point in Jonathan Swift’s Gulliver’s Travels.

Comments are open, most of all if you have read the book or other work by Tetlock.