Marginal Revolution: Don't trust expert predictions

July
S	M	T	W	Th	F	St
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31

« Sonic paternalism, Australian style | Main | Cato Unbound »

Don't trust expert predictions

Last night I finished Philip Tetlock's Expert Political Judgment:

...no matter how unequivocal the evidence that experts cannot outpredict chimps or extrapolation algorithms, we should expect business to unfold as usual: pundits will continue to warn us on talk shows and op-ed pages of what will happen unless we dutifully follow their policy prescriptions. We -- the consumers of expert pronouncements -- are in thrall to experts for the same reasons that our ancestors submitted to shamans and oracles: our uncontrollable need to believe in a controllable world and our flawed understanding of the laws of chance. We lack the willpower and good sense to resist the snake oil products on offer. Who wants to believe that, on the big questions, we could do as well tossing a coin as by consulting accredited experts?

Daniel Drezner has two excellent posts on the book, here and here. Here is Louis Menand's glowing review from The New Yorker. Here is Tetlock's home page. Here is a sample book chapter.

And yes Tetlock has data, drawing upon twenty years of observation of 82,361 forecasts. Tetlock also finds that "foxes" forecast better than "hedgehogs" and that only the forecasts of foxes have positive value.

This is one of the (few) must-read social science books of 2005.

My caveat: Assume that the experts are usually wrong in their novel predictions. The consensus views of a science still might be worth listening to. Economists cannot forecast business cycles very well, but you should listen when they tell you that a deflationary shock is bad news. Each new forecast or new theory is an example of individual hubris and in expected value terms it is stupid. But the body of experts as a whole, over time, absorbs what is correct. A large number of predictions creates a Hayekian discovery process with increasing returns to scale. Social knowledge still comes out ahead, and in part because of the self-deceiving vanities put forward every day. You can find that point in Jonathan Swift's Gulliver's Travels.

Comments are open, most of all if you have read the book or other work by Tetlock.

Posted by Tyler Cowen on December 5, 2005 at 07:49 AM in Books | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/t/trackback/3576/3792167

Listed below are links to weblogs that reference Don't trust expert predictions:

Comments

I've always thought that experts had some value at helping you to see the universe of possible futures, although not at telling you which possible future would occur. In this respect, they help to remedy two human behavioral/psychological limitations: (1) people tend to think in self-reinforcing patterns and to have difficulty challenging (or even articulating) their own assumptions and (2) people find it much easier to reason about concrete, semi-plausible stories rather than vague abstractions or unknown unknowns.

For example, many businesses suffer from myopia, focusing on one happy imagined future. If an expert with lots of inflated credentials can convince a business manager to consider the possibility of even one alternative future, that expert will have created positive value.

An example of part (2) would be the use of expert forecasts to train emergency-response teams -- if you are running a corporate information security department, or a financial risk management office, or a local fire department, you may use expert-generated scenarios in your training and planning exercises. It is 100% certain that the real emergency will not follow what you planned for, but, you still need something concrete for your planning exercise.

Posted by: DK at Dec 5, 2005 8:25:56 AM

"The consensus views of a science still might be worth listening to."

That's just Condorcet's law, right?

Posted by: Gary Leff at Dec 5, 2005 9:11:05 AM

Experts say you can't trust experts. Studies show that most studies are wrong.

Posted by: joshg at Dec 5, 2005 9:48:42 AM

Tetlock is asking the wrong question. Any prediction at a particular moment of time is always going to be subject to an enormous uncertainty factor. Rather, the best experts are the people who can adjust to new information on the fly.

Posted by: Mike Mandel at Dec 5, 2005 10:52:20 AM

How about experts USING extrapolation algorithms? Do the experts worsen the result of the extrapolation?

Posted by: Oskar Shapley at Dec 5, 2005 1:20:37 PM

And index funds beat managed funds in about 2/3rds of the cases. You can't make money betting favorites, but if you don't hang out in the jockeys dressing room you don't know enough to bet on anything else.

Posted by: Robert Schwartz at Dec 5, 2005 1:34:35 PM

The Wisdom of Crowds of Experts?! Seems counterintuitive.

Posted by: nnyhav at Dec 5, 2005 2:50:23 PM

Where do you get the statistic that "index funds beat managed funds in about 2/3rd of cases?" It doesn't sound right to me, and there is a bias in counting by number of cases.

By definition, an index fund should earn (before fees) precisely the market-cap weighted average of the before-fees performance of all of the actively managed funds in the same basket of securities, including both mutual funds, hedge funds, grandma, insiders, etc (all of whom are "managing" their funds.) That's b/c most indices are market-cap weighted averages. Thus, if you could select a random money manager in a market-cap weighted way, you would have a 50% chance of outperforming the index (before fees) and a 50% chance of underperforming the index.

Thus, the argument for index funds is based on performance, but on risk-reduction and fee-reduction -- you get identical average before-fees return, with lower risk/variance, and lower fees.

The bias in looking at this by # of cases is that fees + transaction costs are much worse for smaller pools of money. So if you are comparing after-fee performance, the average small money manager will underperform both larger money managers and larger index funds, simply b/c of fees.

Posted by: DK at Dec 5, 2005 3:25:10 PM

Most of the things that people are interested in hearing forecasts about are inherently hard to predict. That's why we are interested in them. Being able to predict when the sun comes up tomorrow is a highly useful skill, but it won't get you on TV to argue with other experts about it. In contrast, consistently predicting who is going to beat the point spread in NFL games is practically impossible because the point spreads are established in self-correcting markets, but because it's so hard to do, there is much demand for the opinions of people who can go on TV and act like they know what they are talking about.

What's amusing is how little interest there is in being able to model the past correctly. For example, eEconomist Steven D. Levitt's abortion-cut-crime didn't predict the past accurately, as an hour of checking Google showed, but it still helped make him rich and famous. Now we know his theory was based on simple incompetence on his part, but it was pretty obvious it failed to predict the past when it was unveiled in 1999.

Posted by: Steve Sailer at Dec 5, 2005 6:27:44 PM

This is from memory, and I don't yet have citations, but: some time ago, anthropologists arrived at the conclusion that genuinely-random oracles can be more useful than anyone's best judgement.

One example: When prey animals have learned the habits of the tribe's hunters, using an oracle to decide where to hunt next works better than best judgement -- it can't be predicted.

Note: I suspect that "civilized" oracles are less useful because they're likely to be used by specialists in interpretation. So the chicken oracle (feed a nonstandardized dose of poison to a chicken, see where it dies) is probably more useful than the I Ching or expert analysis by economists.

Posted by: Dan Goodman at Dec 5, 2005 7:42:47 PM

Levitt's counterattack is on his website and the jury is still out on
his thesis. We'll see the counter counter attack, etc, for a while.
As for Tetlock, a minor quibble on his posted chapter. From Soviet and
American archives we now know that Kennedy backed down, that the Soviet
demand for the US to withdraw US missiles from Turkey in exchange for
Soviet withdrawel of Soviet missiles from Cuba was met. Kennedy pushed
back the Soviets in American mythology, but not in the real world.
Khrushchev and Kennedy both lost power shortly after this crisis. They
made too many people nervous.

Posted by: wkwillis at Dec 6, 2005 5:19:30 AM

Mr. Sailer,

You might be right, but by this point we're all well aware of your contentions. There is no need to shoehorn it into every single thread of eight different weblogs. It makes you seem like a loony.

Posted by: joshg at Dec 6, 2005 9:28:10 AM

I've not had a chance to look at Tetlock's book, but I have read Klein's "Sources of Power" (which was about decision making). My opinion is that while experts make far better decisions than a non-expert, many experts have been co-opted by people with money. Like when the administration hired James Guckert/Gannon, Armstrong Williams, Dave Smith, and Karen Ryan (among others) to advertise for them. The book "Trust Us, We're Experts" is also about "scientists" for sale and the pseudoscience that gets bandied about in the media. Remember the brokerages that had to pay large fines for pushing stocks that they were being bribed to push?

"By definition, an index fund should earn (before fees) precisely the market-cap weighted average of the before-fees performance of all of the actively managed funds in the same basket of securities"
No. An index fund will carry the basket of securities that make up the S&P500, or Russell 2000 index, or whatever index the fund is based on. They are NOT averages of other funds. They are based on a single list that changes semi-annually (in the case of some of the Russell indices), or less frequently (in the case of the Dow Jones Industrial Average). When the word "core" appears as part of the name of the fund, it will be predominantly the index with a few stocks missing. Index funds usually have lower management fees since there is no need to manage them, nor do any form of analysis. And it is the management/12b-1 fees that can make a lot of difference in a fund's performance: which is why they tend to outperform managed funds. Very few indices are "based on fund managers" and those are indices with "growth" or "value" in their names, as G&V are subjective judgements of the fund managers (we think X is under priced, so that makes X a "value" stock).
http://www.russell.com/US/Indexes/US/methodology.asp

Large funds sometimes close themselves to new investors to keep their performance up. If they take in too much money, there aren't usually enough investments to be able to keep their performance up. It isn't the smaller funds that underperform, sometimes it is the large ones, as they grow too big for effective management, and have too much money to be able to find good invesments for it.

Small cap funds were popular for some time, and it looks like folks are trying to pitch Large cap funds as the next "big thing." I used to work for a company that ran websites for mutual fund companies, so I'd see the performance numbers all the time. The remark that an index would outperform 2/3 of funds is quite accurate, although the funds it would outperform might differ each year. What surprised me was that some funds would underperform the index every year, year after year, and would still have investors.

Posted by: Peter at Dec 7, 2005 12:37:04 AM

avontuurlijk lesbisch ^^^ bangelijk jong ^^^ asiatiche azione in cucina ^^^ allievo merda in pace ^^^ plus chaud poupee fait pipi ^^^ amour serveuse baise ^^^ sexy quarante cinq ^^^ maman trente ^^^ goodly infermiera urinate ^^^ attraente infermiera ubriache ^^^ kallare brudar kon ^^^ himmelsk sjukskoterska dildo ^^^ assurdo fighette masturbate ^^^ goodly allievo inculate ^^^ koselig filmrull ^^^ begjare ibenholt ^^^ varmest uhyggelig jomfru ^^^ varmere snillere mindrearig ^^^ evarestos neos agapi ^^^ oreo nosokoma striptiz ^^^ mman vampire ^^^ chaud maman mpeg ^^^

Posted by: levan at Sep 5, 2006 7:29:24 AM

Tetloff, Expert Political Judgment, notes by JZ in RTF

Some Notes by John Zube to
Philip Tetloff: "Expert Political Judgment"
(www.amazon.com) Posts: www.danieldrezner.com - www.marginalrevolution.com - glowing review in THE NEW YORKER: www.newyorker.com - Sample chapter 1 from the book: www.pupress.princeton.edu -

Note that I haven't read the whole book but just its first chapter, put on line and a few of the short reviews of it:

Can one judge a whole book merely by these?
Only if all of them do share a common premise and if that premise is wrong.

The common and false premise of the author and the reviewers is, in my opinion:
territorial organization, decision-making, constitutionalism, legalism, jurisdiction, administration, policies and "measures".
They are all taken for granted, considered to be rightful or unavoidable, as natural or as the given reality, not as a wrongful, irrational, intolerant and despotic system (even when under democratic or republican camouflage).

From it follows centralized and monopolized decision-making for large collectives, made up not only of followers but also of numerous victims, opponents and dissenters of various kinds.

Herbert Spencer called it "the great machine". F. A. Hayek, in "The Road to Serfdom", pointed out why the worst do get to the top in it.

The "great machine", a vast destructive bulldozer, an extermination machine, equipped with the most modern mass murder and destruction "weapons" - in charge of madmen:
Power addicts, habitual liars and cheats, religious gurus and dogmatists, true believers - and be it only in majoritarian territorial "democracies" or pretended "representation", e.g. "Christians" involved in a new "Crusade" or "Muslims" involved in a new world conquest attempt, driven and supported by the most popular prejudices, errors, myths, fads and flawed hypotheses, e.g.: religious intolerance, racism, the myths of the "chosen people", overpopulation notions, hatreds and fears of foreigners, belief in security through brute strength, or in more or less permanent or frequently repeated indiscriminate warfare, even in "ethnic cleansing", i.e., open mass murders.

In which direction will a madman jump next? What spleen will next grab his imagination and drive him into insane actions?
Where will a great arsonist light his fires next?
Where will he (or she) employ "his" great destructive bulldozer next?
Who and where will be the next victims of a mass murderer with WMDs at his disposal? - Or with conventional "weapons", which allow him to produce similar hundred-thousands of mass-murder victims?
Whom will he put next into his or her concentration camps or exterminate with his or her "modern", "scientific" and cheap ABC extermination camp "packages".

That, indeed, becomes unpredictable. E.g.:

Will people reserving to themselves the right to determine interest rates or other prices, either raise or lower them or keep them for a while?
Will central and territorial administrators of population policies take coercive steps to increase or reduce or to stabilize populations, with whatever coercive means that take their fancy?
Will central note-issuing banks, issuing legal tender money, increase or decrease inflation or deflation or continue a stagflation?
Will education ministers increase or decrease their imposed or subsidized mis-education systems.
Will "defence" ministers employ more or less inherently aggressive and mass murderous weapons and warfare methods?
Will transport ministers block or hinder transport more or less by their monopolies, licences and regulations?
Will health ministers prevent more or less new cures and persist more or less with old health fads?
Will immigration ministers put more or less illegal immigrants into concentration camps or deport them or prevent their arrival; will they increase or decrease "authorized" immigration?
Will ministers for postal services and communication restrict them more or less and increase their prices, while suppressing free competition?
What lies will the various propaganda ministries resort to next?
What kind of new taxes, prohibitions and obstacles and licences will "our representatives" and "great leaders" introduce next? Will they increase or decrease taxes and "concessions" and "exemptions", will they simplify taxation or further complicate it?
What kind of "new deal", "co-prosperity sphere" or "great society" or utopia will next grab their fancy?
Will they increase or reduce the "war against drugs" or just keep it going indefinitely?
Will they abdicate, however unlikely and rarely that ever happened?

Do most of the "freedom fighters", resistance movements, military insurrectionists, revolutionaries and liberators have really something much better in mind, or, often, even something still worse?

Are any of our territorial mis-rulers really aware and appreciative of all individual rights and liberties or even interested in them?
Are their opponents sufficiently aware of or interested in all fundamental rights and liberties?

In this kind of world-wide madhouse almost anything can happen, anywhere, anytime, to anyone.
Under this kind of condition reliable predictions are, indeed, close to impossible.

So, what has to be fundamentally changed in order to make scientific predictions in the "social sciences" possible?

Let all the different movements, sects, believers, ideologues, reformers, revolutionaries, liberators, dogmatists etc. sort themselves out, by individual decision-making, including individual secessionism and voluntary reorganization into freely competing and peacefully coexisting communities, societies, governments, utopias and States, that are all only exterritorially autonomous and that apply their personal laws, constitutions, jurisdictions and other institutions only to their own and voluntary members, wherever on the Earth's surface they may live or work.

Establish merely an international law between them that is based upon respect for all individual rights and liberties, to the extent that any of them are claimed by the members of any of these voluntary communities of like-minded people.
Let them be as free or unfree among themselves, always at their own risk and expense, as they want to be.

Outlaw only any aggressive actions and preparations for them.
And establish much more rightful and rational institutions for upholding basic rights and liberties than territorial States ever provided so far.(A vast topic on its own, with quite insufficient precedents in the past. It is partly discussed in my first peace book.)

Then you will become able to predict results of particular communities of volunteers or competing governments, by their own known principles, practices and institutions, all unanimously supported as well and as long as they can be.

E.g., the results for consistent free traders and for consistent protectionists can be safely predicted.
Likewise the results of continued monetary despotism in some voluntary communities and those of full monetary and financial freedom in others.

Each community of volunteers, exterritorially autonomous, a free experiment, among volunteers only, can then be judged upon its own merits or demerits, under optimal conditions for it. Then it will not suffer under any internal or external threats or trends caused by dissenters, opponents and non-conformists or by leading power mongers and power addicts (except those, unanimously approved of by their voluntary members as their "leaders" or gurus or "prophets").

Moreover, then it can also be predicted, for the long run, how these different societies and communities will tend to influence each other, without coercion, invasions, occupations, interferences, blockades, propaganda campaigns and "foreign policies", in the same way as really and fully introduced religious tolerance can be predicted in its results, or full freedom in the sphere of education and enlightenment, with all technological options being fully utilized.

In short: Successful experiments of this kind will tend to grow, by more and more people joining them or setting up similar experiments.
At the same time, the unsuccessful experiments will shrink, naturally, by more and more people seceding or deserting from them.
Finally, the latter will almost disappear or become reduced to insignificant and no longer dangerous sectarian groups.

This panarchistic or polyarchic method and model would introduce experimental freedom in the social sciences, as far as humanly possible and desired - by their voluntary participants.

From their experiments and the theories their observed facts would lead to, a new kind of political social and economic science would arise, one much more based on observable and recorded facts than are the present social sciences.

As far as all the associated and continued argumentation is concerned, I do recommend the recently developed computerized "argument mapping" of Paul Monk and others, which offers hope to finally settle theoretical questions that have remained unsettled for centuries because of the conventional methods used for their discussion.

PIOT, John Zube, jzube@acenet.com.au
7. 9. 2006

(PIOT: Panarchy In Our Time or: To each the government or non-governmental society of his or her choice. - www.panarchism.info, www.panarchy.org - The effects upon war and peace have been described by me in two books: www.butterbach.net/epinfo/peace.htm & www.butterbach.net/epinfo/abc.htm
A long alphabetized compilation: Pan A-Z, and, is available from me in zipped email attachments.
My literature list, for micro-fiched editions, is on: www.butterbach.net/lmp
Some of my favorite monetary freedom books are on www.reinventingmoney.com and the beginnings of a list of URLs to libertarian texts, still flawed and very incomplete, but growing, soon, is on www.panarchy.org/zube/list.index.html - With your assistance, this list could come to grow into a catalog for an electronic libertarian library. It is a work in progress and comes in its current stage, in my own version, to ca. 3.5 Mbs in RTF, accessible through me as a zipped but still very long email attachment. Its online version is soon to be updated by Gian Piero de Bellis.)

Posted by: John Zube at Sep 6, 2006 9:03:43 PM

FLV converter
Mac DVD Ripper

Posted by: aaa at Jun 4, 2007 5:52:43 AM

google排名
 wow power leveling
wow power leveling
wow powerleveling
wow powerleveling
wow power level
wow power level
hongkong hotel
hongkong hotel
beijing tour
beijing tour
翻译公司
 翻译公司
 上海翻译公司
 上海翻译公司
 北京翻译公司
 北京翻译公司
 门禁
 wow gold
wow gold
powerleveling
powerleveling
wow powerleveling
wow powerleveling
wow power leveling
wow power leveling
power leveling
power leveling
wow power level
wow power level
窃听器
 福彩3D
搬家公司
 香港六合彩