« Can computer failure cause a bank run? | Main | Mandatory health insurance »
Oomph v. Statistical Significance
Spurred perhaps by recent posts about "statistical significance", "real significance" and "oomph" in economics, the always-excellent Andrew Gelman joins in. Back story: the debate comes from an exchange between McCloskey & Ziliak v. Hoover & Siegel about the excessive influence given by economists to assessments of statistical significance ("asterisk econometrics") relative to than quantitative importance ("oomph").
Read Gelman's full discussion here. [HT: Mark Thoma]. Or just skip to Gelman's conclusion:
Finally, what does this say about the practice of statistics (or econometrics)? Does it matter at all, or should we just be amused by the gradually escalating verbal fireworks of the McCloskey/Ziliak/Hoover/Siegler exchange? ... I do think that statistical significance is often misinterpreted but I agree with Hoover and Siegler's attitude that statistical significance tells you about your uncertainty of your inferences. The biggest problem I see in all this discussion is the restriction to simple methods such as least squares. When uncertainty is an issue, I think you can gain a lot from Bayesian inference and also from expanding models to include treatment interactions.
My $0.02: Three observations:
- A time series fact: It is surely the case that the meaningless statements of statistical rather than economic significance are declining. At least they are in the seminars I attend.
- A cross-section fact: There remains enormous heterogeneity across fields in the emphasis given to "oomph". Macroeconomists have been taking magnitudes seriously for years, perhaps driven by the calibration movement. Perhaps we have our first field to have overshot, as calibration has driven out estimation, and with it, a strong sense of the uncertainty of the inferences being made.
- A hope: We have all sinned. May we sin less in the future.
UPDATE: More from Andrew Gelman here.
For those after the full debate, see:
- McCloskey and Ziliak, "The Standard Error of Regressions," Journal of Economic Literature 1996.
- Ziliak and McCloskey, "Size Matters: The Standard Error of Regressions in the American Economic Review," Journal of Socio-Economics 2004.
- Hoover and Siegler, "Sound and Fury: McCloskey and Significance Testing in Economics," Journal of Economic Methodology, 2008.
- McCloskey and Ziliak, "Signifying Nothing: Reply to Hoover and Siegler."
McCloskey has been making this point for some time, and a longer list of papers is available here, including several shorter (yet unsurprisingly persuasive) pieces.
Posted by Justin Wolfers on October 5, 2007 at 05:42 PM in Economics | Permalink
Comments
You know, if you have a high liquidity preference, that $0.02 of your would be worth more like $0.06 in today's market
Posted by: Bill at Oct 5, 2007 7:22:17 PM
General methodology points are boring, like whether you're a Popperian or a Kuhnian or a Feyerabending. Interesting on some level, but after a while, you don't care about that, just whether a hypothesis is new, true, and important, or a new thread has potential.
Posted by: eric at Oct 5, 2007 9:15:52 PM
Don't you think that the last response from McCloskey and Ziliak very emotional rather than rational?
And what is their view and foundation on time series? Why they take a time series as one observation?
Posted by: passerby at Oct 5, 2007 10:27:43 PM
Speaking from an epidemiology persepctive, McCloskey is correct, and most people misunderstand significance testing most of the time. They misunderstand it as a kind of 'truth machine'.
What has happened is that p < 0.05 on frequentist stats has become a standard sieve to be applied to data, before anyone really starts thinking about it. The advantage of this coarse sieve is that it allows much wider participation in science, since people don't really need to understand what they are doing in order to generate probably-useful material. This is a kind of efficiency argument.
The proof of the pudding is in the eating. When a scientific discipline is growing, and new workers are successfully making predictions on the basis of past work, and people are using this to make sucessful interventions in the real world - fine. This standard method of analysis can be accepted.
The dangers are when a field is not really moving forward but instead becoming more and more rigorous at seeking statistical significance, without ever building on it or testing it in real world interventions. Or when statsitical significance is used to check statistical significance in a large-scale process which is internally consistent but is just circular (this may have happened in medicine, over the past couple of decades - where there are ever larger clinical trials chasing ever smaller effect sizes; until the effect size is swamped by the intrinsic sampling biases of massive trials).
In the end I think all statistical sciences will need to grapple with the kind of issues raised by McCloskey, and will need to revisit signficance testing. But maybe this won't happen until there is another, better, but still *standardized* method with which to replace it.
Real scientific insight is rare and difficult to communicate - which is why McCloskey finds it hard to convince critics. They simply don't understand.
Posted by: Bruce G Charlton at Oct 6, 2007 1:56:01 AM
Bruce is right in his assessment of many people taking the test as a "deus ex SAS," as I've often said. What they also don't realize is that you can exhaust a dataset. That is, the power of the significance test declines with each run of the programs. Roughly, this means that if you start with 10 degrees of freedom for one's first analysis, say a mean, then when you run a regression you really only have about 9 degrees remaining.
And, as an aside, the number of degrees of freedom for the significance test depends not only on the number of observations, but on the survey design as well. It may be that this additional complication is what drives the retreat from significance testing.
Posted by: Paul McMahon at Oct 6, 2007 8:05:24 AM
Why the link to "gradually escalating verbal fireworks" is linked to a matter with Heckman and sociologists??
Posted by: passerby at Oct 6, 2007 9:51:33 AM
It appears that much work in economics has at bottom a desire for the researcher to "prove" some preconceived notion.
This leads to all sorts of mischief. The most important distortions come from the selective use of data. Applying questionable statistical techniques is just the icing on the cake.
Perhaps things are getting better (if they really are) because the feedback mechanism of critical evaluation is working better. In the past debates would stretch over years as one published paper was countered by another. Now papers are put online and available much more quickly. In addition the audience is larger and the rise of the blogs give many others to comment as well.
This discussion thread is a perfect example of the evolution.
There is also (at least in the US) much more push back to the prevailing macro-economic ideas, especially those having to do with free-market ideology and marginal tax rates. The increasing amount of criticism these schools of thought are receiving may have altered the behavior of those publishing in this field. Either they may be publishing less, or they may have trimmed their sails.
Posted by: robertdfeinman at Oct 6, 2007 11:08:26 AM
"There is also (at least in the US) much more push back to the prevailing macro-economic ideas"
Have you any evidence? With error bars and everything?
Posted by: Eric H at Oct 6, 2007 12:06:37 PM
Eric:
Yes, just consult any of the web sites that track blog traffic. You'll find that those critical of the status quo (which generally means left leaning) have been growing rapidly over the past four or five years, while those which take the opposite point of view are doing less well.
Then there are all the conservatives who have started to speak out against their own party and some of its economic assumptions.
Posted by: robertdfeinman at Oct 6, 2007 1:23:06 PM
Robert, your comments are getting so difficult to unpack that I don't know where to begin. Could you define "conservatives"? What is "their party"? Do you perhaps mean Republicans? Would you concede that both parties are filled with a variety of subgroups with their own agendas and that there has always been internal dissent? Isn't an alternative interpretation of these events that the republicans are more tolerant of dissent and that the left has been overtaken by intellectual sclerosis or totalitarianism? I don't fully believe the latter, but it is a possible interpretation of your observations. Similarly, it could be the case that left leaning blogs are fastest growing because they are smaller while conservative blogs are larger. And since libertarians are dissatisfied with the status quo, you may interpret their criticisms as being left-leaning (right-wingers certainly do), but you'll probably throw them right back under that neo-con bus as soon as you detect anything critical of the left in their thought.
Is this a left leaning blog? Is it critical of the status quo? (hint: what is the blog's tagline?) Is it successful? Just because we read blogs doesn't mean a significant portion of the population does. I think your confirmation bias is showing.
Posted by: Eric H at Oct 6, 2007 5:19:11 PM
OK, Eric here goes (my definitions, your mileage may vary).
The "conservative" view has historically been connected to support of the status quo. Since the status quo usually favors the privileged the effect is that conservatives are the party of the rich. The specifics of what policies this entails changes over time and place.
The "liberal" view favors the interests of the common man. The few times the "liberals" have gotten into power have usually been as a result of a revolution, and their control has usually been short-lived. Class solidarity from outside forces usually works towards restoration of the status quo ante.
The "libertarian" view is of fairly recent origin and mostly exists in the US. It has a focus on personal liberty which implies less government.
At this moment in time, in the US, I see the following:
Social conservatives - concerned with issues of morality, usually as determined within a religious framework.
Foreign policy conservatives - the so-called neo-cons. They are interested in promoting US power abroad, their motives are in dispute.
Social liberals - concerned with issues of personal liberty, which sometimes overlaps the libertarian position. They also think that government should play a role in regulating business.
Foreign policy liberals - A, mostly theoretical category. Those who object to our current foreign policy failures are mostly concerned with poor implementation, not misplaced goals. True peaceniks are few and far between.
Libertarians - The focus is on freedom of action especially as it concerns property and risk. They claim they want smaller government, but require a strong legal/police structure to enforce property rights. Since this model has never existed in a functioning government they are always in opposition to the existing administration in some way or other.
This is a libertarian site. Judging by some of your comments I assume you agree with most of the precepts of the owners.
I used to have a quip to the effect that conservatives want to tell people what to do while liberals want to tell institutions what to do. Or, to put it another way, conservatives don't trust people and liberals don't trust institutions.
Libertarians don't fit into the left-right model. In the past 40 years they have been allied with the conservatives because of the stated claim that the conservatives were in favor of "smaller government". Now that they have seen that this hasn't been true in practice I think they are somewhat adrift.
As for political parties, I don't see too much difference between them. The GOP will favor less regulation and lower taxes while the Dems will favor more social spending and more regulation of business. This is just a matter of degree, despite all the noise they generate. Both parties believe in the applicability of capitalism to all circumstances and they also believe in a strong military and police structure.
Given the institutional inertia present in the government there isn't much change in overall spending patterns when the majority party changes. If you look at the historic CBO data you'll see what I mean:
http://cbo.gov/budget/historical.pdf
Posted by: robertdfeinman at Oct 6, 2007 8:28:31 PM
A curious observation I made a long time ago that is reflected in this discussion is that for
some reason, econometricians can be far more polemical, and far more personalistic in their polemics,
than even ideologically inclined (and hence "naturally polemical") economists. So, I remember a long
time ago first going to meetings and going to sessions by Austrians or Marxists or whatever, where
people were reasonably polite to each other, even if they used some strong language about ideas.
But then I would go to econometrics sessions, and people would be getting up and saying to each other,
"You do not know what you are talking about; you are ignorant!" This sort of thing has quieted down
some, but somehow these battles over time series versus structural equations, classicals versus Bayesians,
and this bit over stat significance, all seem to involve more polemics than one might imagine is reasonable.
Each side has to put the other one down in a personalistic way intellectually.
Posted by: Barkley Rosser at Oct 8, 2007 5:56:10 PM
Posted by: 鑽石 at Apr 2, 2008 10:22:01 PM
Dear Barkley, and some others who have complained about the sharp tone in Ziliak's and my reply to Hoover and Siegler,
I plead innocent on grounds of self-defense. The "personalistic" tone was introduced by Hoover and Siegler. Their use of a French masculine gender to describe me is the least of their personalistic assaults on our integrity, intelligence, and character.
But surely, dear, you are not going to decide a scientific question by who is nicest, are you? That would be a big scientific mistake, wouldn't it? (In any case your vote on that score would have to go to sweet little me!) Perhaps you don't realize how important the issue is. 90 percent of the empirical papers in economics depend on it. As our epidemiologist colleague notes (and as many others in many other fields note, as we record in our book The Cult of Statistical Significance), most economists think Fisherian "significance" means "important," and that "insignificant" means "droppable, unimportant." Neither is true---and remarkably Hoover and Siegler say they agree. Understanding the point would mean that all econometric testing-without-loss-functions would have to stop, today. Big issue. Hot tone.
Regards,
Deirdre
Posted by: Deirdre McCloskey at Apr 6, 2008 9:11:55 PM
Dear Barkley, and some others who have complained about the sharp tone in Ziliak's and my reply to Hoover and Siegler,
I plead innocent on grounds of self-defense. The "personalistic" tone was introduced by Hoover and Siegler. Their use of a French masculine gender to describe me is the least of their personalistic assaults on our integrity, intelligence, and character.
But surely, dear, you are not going to decide a scientific question by who is nicest, are you? That would be a big scientific mistake, wouldn't it? (In any case your vote on that score would have to go to sweet little me!) Perhaps you don't realize how important the issue is. 90 percent of the empirical papers in economics depend on it. As our epidemiologist colleague notes (and as many others in many other fields note, as we record in our book The Cult of Statistical Significance), most economists think Fisherian "significance" means "important," and that "insignificant" means "droppable, unimportant." Neither is true---and remarkably Hoover and Siegler say they agree. Understanding the point would mean that all econometric testing-without-loss-functions would have to stop, today. Big issue. Hot tone.
Regards,
Deirdre
Posted by: Deirdre McCloskey at Apr 6, 2008 9:12:25 PM
Dear Barkley, and some others who have complained about the sharp tone in Ziliak's and my reply to Hoover and Siegler,
I plead innocent on grounds of self-defense. The "personalistic" tone was introduced by Hoover and Siegler. Their use of a French masculine gender to describe me is the least of their personalistic assaults on our integrity, intelligence, and character.
But surely, dear, you are not going to decide a scientific question by who is nicest, are you? That would be a big scientific mistake, wouldn't it? (In any case your vote on that score would have to go to sweet little me!) Perhaps you don't realize how important the issue is. 90 percent of the empirical papers in economics depend on it. As our epidemiologist colleague notes (and as many others in many other fields note, as we record in our book The Cult of Statistical Significance), most economists think Fisherian "significance" means "important," and that "insignificant" means "droppable, unimportant." Neither is true---and remarkably Hoover and Siegler say they agree. Understanding the point would mean that all econometric testing-without-loss-functions would have to stop, today. Big issue. Hot tone.
Regards,
Deirdre
Posted by: Deirdre McCloskey at Apr 6, 2008 9:13:38 PM
Dear Barkley, and some others who have complained about the sharp tone in Ziliak's and my reply to Hoover and Siegler,
I plead innocent on grounds of self-defense. The "personalistic" tone was introduced by Hoover and Siegler. Their use of a French masculine gender to describe me is the least of their personalistic assaults on our integrity, intelligence, and character.
But surely, dear, you are not going to decide a scientific question by who is nicest, are you? That would be a big scientific mistake, wouldn't it? (In any case your vote on that score would have to go to sweet little me!) Perhaps you don't realize how important the issue is. 90 percent of the empirical papers in economics depend on it. As our epidemiologist colleague notes (and as many others in many other fields note, as we record in our book The Cult of Statistical Significance), most economists think Fisherian "significance" means "important," and that "insignificant" means "droppable, unimportant." Neither is true---and remarkably Hoover and Siegler say they agree. Understanding the point would mean that all econometric testing-without-loss-functions would have to stop, today. Big issue. Hot tone.
Regards,
Deirdre
Posted by: Deirdre McCloskey at Apr 6, 2008 9:14:03 PM
花蓮,旅遊租車,花蓮旅遊,花蓮租車,花東旅遊,花蓮租車,花蓮租車,花蓮旅遊,租車,花蓮旅行社,花蓮旅遊景點,花蓮旅遊行程,花蓮旅遊地圖,花蓮一日遊,花蓮租車,花蓮租車旅遊網,花蓮租車,花蓮租車,花蓮租車,花東旅遊景點,租車,花蓮旅遊,花東旅遊行程,花東旅遊地圖,花蓮租車公司,花蓮租車,花蓮旅遊租車,花蓮租車,花蓮旅遊,花蓮賞鯨,花蓮旅遊,花蓮旅遊,租車,花蓮租車,花蓮租車 ,花蓮 租車,花蓮,花蓮旅遊網,花蓮租車網,花蓮,租車,花東 旅遊,花蓮 租車,花蓮,旅遊,租車公司,花蓮,花蓮旅遊,花東旅遊,花蓮地圖,包車,花蓮,旅遊租車,花蓮 租車,租車,花蓮租車資訊網,花蓮旅遊,租車,花東,花東地圖,租車公司,租車網,花蓮租車旅遊,租車,花蓮,賞鯨,花蓮旅遊租車,花東旅遊,租車網,花蓮海洋公園,租車 ,花蓮 租車,花蓮,花蓮旅遊,花蓮租車公司,租車花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮包車,花蓮租車網,花蓮旅遊,花蓮租車,花蓮旅行社,花東旅遊,花蓮包車,租車,花蓮旅遊,花蓮租車,花蓮一日遊,租車服務,花蓮租車公司,花蓮包車,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮包車,花蓮租車網,花蓮旅遊,花蓮租車,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮租車,租車網,花蓮租車公司,花蓮旅遊,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮租車公司,花蓮一日遊,租車,租車服務,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮旅遊,花蓮賞鯨,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮包車,花蓮租車網,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,租車花蓮,花蓮租車,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,租車花蓮,花蓮租車,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮包車,花蓮,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮包車,花蓮租車,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮包車,花蓮租車,花蓮旅遊,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮包車,花蓮租車網,租車公司,花蓮租車,花蓮租車公司,花蓮一日遊,花蓮旅遊,花蓮旅遊租車,花蓮租車網,花蓮租車,花蓮一日遊,租車花蓮,花蓮租車,花蓮旅遊租車,花蓮租車,花蓮租車旅遊,花蓮租車,花蓮旅遊,花蓮旅遊,花蓮包車,花蓮溯溪,花蓮泛舟,花蓮溯溪,花蓮旅遊,花蓮旅遊,花蓮租車,租車公司,花蓮旅遊租車,花蓮租車,租車,花蓮旅遊,花蓮租車,花東旅遊,花蓮賞鯨,花蓮旅遊,花蓮泛舟,花蓮賞鯨,花蓮溯溪,花蓮泛舟,花蓮泛舟,花蓮溯溪,花蓮旅遊,花蓮旅遊,花蓮租車,花東旅遊,花蓮,花東,花蓮旅遊,花東旅遊,花蓮租車,花蓮,花東,花蓮旅遊,花蓮租車,花東旅遊,花蓮旅遊,花蓮租車,租車,花蓮旅遊,花蓮租車,花蓮旅遊租車,花蓮旅遊,花蓮租車,花蓮,花東旅遊萬事通,花蓮旅遊,租車,花蓮旅遊,花蓮租車,花蓮旅遊,花蓮租車,花蓮租車,花蓮租車,花蓮旅遊,花蓮租車,花蓮旅遊,花蓮租車,花蓮旅遊,花蓮租車,花蓮旅遊,花蓮租車,花蓮租車,花蓮包車,花蓮旅遊,花蓮租車,花蓮太魯閣,花蓮包車,花東旅遊,花蓮旅遊行程,花蓮旅遊,花蓮 租車,花蓮租車,花蓮租車旅遊,花蓮旅遊租車,租車,花蓮旅遊推薦,花蓮旅遊包車,花蓮租車,花蓮,花蓮租車,花蓮地圖,花蓮旅遊,花蓮旅遊資訊網,花蓮旅遊景點,賞鯨,花蓮旅遊行程,花蓮旅遊,花蓮旅遊租車,花東旅遊景點,花東旅遊行程,花蓮旅遊,花蓮租車,租車,花東旅遊,花蓮旅遊,花蓮租車,花蓮,旅遊達人,旅遊達人blog,花蓮租車旅遊資訊網,花蓮,租車,花蓮,花東旅遊,地圖,租車,賞鯨泛舟溯溪,租車,[ 芝麻店家 ] 花蓮租車旅遊資訊網,花蓮租車-行易花蓮旅遊租車資訊網 ,旅遊網,旅遊景點,花蓮行程,花蓮,花東,旅遊租車,旅遊,花蓮,租車,花東旅遊,花蓮租車旅遊,行易旅遊民宿資訊網,花蓮,旅遊,花蓮,一日遊,花蓮好玩的地方,花蓮,,一日遊,花東,租車,旅遊,花蓮旅遊,花東旅遊,花蓮租車,花蓮租車,花蓮旅遊-花東旅遊萬事通
Posted by: 花蓮租車 at Jun 4, 2009 5:52:19 AM
