« So You Think You Can Be President? | Main | The J.K. Rowling effect »
The market for lemmas
It's not that big:
Of the sixty articles in AER, EJ, JPE, and QJE that had been cited more than 500 times, only one article contained an author written lemma.
Here is the paper, and thanks to Johan Almenberg for the pointer. On p.6 you see a good graph, "Lemma frequency by decade." Note also that a top paper in Econometrica is much more likely to have a lemma.
Posted by Tyler Cowen on August 22, 2007 at 11:24 AM in Economics | Permalink
Comments
Lemmas are for lemmings.
Posted by: Keith at Aug 22, 2007 11:51:35 AM
Cute. If anyone can get a Nobel Prize out of the market for lemmas (ala the one for lemons), my hat will be off!
Posted by: zlguocius at Aug 22, 2007 12:16:40 PM
Well yeah, but according to your link there were only ca 17 journal articles per year with Lemmas 1970-1990, the period most of the cited papers are from (35 per year 1990-2000).
How many articles were published per year in those journals? 4-500? So you have 3-4% of articles with lemmas, and 1,7% of the top 60 citations? The under representation may be statistically significant, but not very impressive, especially in such a small sample. One more would have made Lemmas more likely to be cited…
Anyway I am not sure what the point is, except the obvious fact that there is a life cycle with important new theories. The first idea is more likely to be expressed verbally, subsequent authors write more rigorous versions.
Unless you are Stigler or Kahneman there is not much this tells you.
Posted by: Scipio at Aug 22, 2007 1:08:40 PM
Pedro Romero and I wrote a related critique of Journal of Economic Theory:
http://www.econjournalwatch.org/pdf/KleinRomeroEconomicsInPracticeMay2007.pdf
We relate our findings to the Coelho & McClure piece.
Posted by: Daniel Klein at Aug 22, 2007 2:09:15 PM
Seems to be more proof that theoretical researchers don't talk to academic empirical researchers (and do either dialogue much with practitioners?)
So, if a lemma never gets empirically tested, does it make a sound?
Posted by: zbicyclist at Aug 22, 2007 2:30:08 PM
Shouldn't this post be called 'Lemmas for Markets'?
Posted by: Independent George at Aug 22, 2007 4:07:01 PM
Why in the world do people ever double space their papers? It always makes me think of freshman term papers. How painful.
Posted by: James at Aug 23, 2007 4:12:57 AM
Building on Scipio's point, the various ways the authors seek to "test" their hypothesis (that more lemmas -> less useful) seem a little odd. My prior is that they may well be right, but their "evidence" hasn't shifted that prior much yet.
(1) The small number of lemmas in the most cited articles may simply represent the fact that older papers (in the AER, EJ, JPE and QJE) were both less likely to have lemmas, and also more likely to have generated citations, because they have been around longer. I didn't see any attempt to control for this at all.
(2) They show that articles with lots (>=5) of lemmas from the 1980 JET generate little empirical literature. But they never compare this with the the articles with few (<5) lemmas. It may just be that the JET generates little empirical work regardless of it's complexity. (That's still interesting, but it risks misdiagnosing the nature of the problem.)
Posted by: conchis at Aug 23, 2007 10:04:00 AM
How about this: Papers without lemmas are accessible by many, many people, and papers with lemmas are accessible by a much narrower set of people. Akerlof's 1970 QJE on lemons markets is much more accessible to the layscholar than Charles Wilson's 1980 Bell Journal on equilibrium in markets with adverse selection. So the layscholars skip Wilson's piece because they can't/don't want to slog through it, but cite Akerlof's piece because it's an easier read.
In the econometrics literature, I wonder how many people have actually READ Hausman's 1978 Econometrica on specification tests, as opposed to simply citing it because "it's that specification test paper".
Posted by: AZ at Aug 23, 2007 10:21:59 AM
conchis,
You are correct in suggesting that supporting evidence/tests would make the case more persuasively (ceteris paribus, more evidence/testing is prefered). Happily, the posted paper is not "an island unto itself".
The posted paper's evidence is supported by and supportive of ALTERNATIVE tests of the Gordon hypothesis (that mathematical complexity is negatively related to operationalism in economic theory); see: "Theory versus Application: Does Complexity Crowd Out Evidence?" published in the Southern Economic Journal, January 2005; vol. 71, no. 3, pp. 556-565.
Posted by: indiana jim at Aug 23, 2007 11:32:59 AM
jim,
Thanks for the reference. As, I said I'm inclined to believe the result. But I'll stand by my claim that the "lemmas" paper shouldn't shift one's priors much, and to be honest, while the analysis in the SEJ paper is slightly better, it isn't exactly great either.
The SEJ paper claims to find two things:
(a) Papers with lemmas (PWLs) are themselves less likely to have "empirical content" than papers without lemmas. This seems entirely unsurprising. It's consistent with the Gordon hypothesis, but only in a way that makes it almost trivially true: complexity crowds out empirics because, by definition, you've spent your time generating lemmas rather than analysing data.
(b) PWLs have a lower proportion of citing articles with empirical content. This seems closer to what we want, but the paper unfortunately doesn't define it's comparison group adequately. PWLs could generate a smaller proportion of empirical papers either: (i) because the Gordon hypothesis is true and more complex theory papers generate fewer empirical citations than simpler theory papers; or (ii) simply because (per (a)) PWLs are more likely to be be theory papers rather than purely empirical papers, and theory papers tend to generate a higher proportion of theory papers than do purely empirical papers. It should have been pretty easy to rule out the second explanation, but they didn't try to.
(As an aside, I also suspect that the authors increase the significance of their results by artificially inflating the sample size (they count citing articles rather than cited articles) but I'm not sure that would affect the result much.)
Posted by: conchis at Aug 23, 2007 3:16:47 PM
Conchis,
The posted paper and the SEJ pub referenced provide an assortment of tests of the Gordon hypothesis. Quibble about the nuances of each, but as the Gordon hypothesis fails to be refuted by test after alternative test its merits become more and more clear.
Posted by: indiana jim at Aug 23, 2007 3:59:22 PM
jim, i don't think i'm quibbling with nuances. i'm saying that the studies don't adequately test the hypotheses they say they do. consequently, they fail to refute the null that the Gordon hypothesis is false. by your reasoning, this would seem to imply that "as the anti-Gordon hypothesis fails to be refuted by test after alternative test its merits are becoming more and more clear."
to put it more bluntly: a bunch of crap studies don't necessarily add up to convincing evidence. they may just add up to a bunch of crap.
i find this particularly annoying because, as i said, i'm a priori inclined to believe the hypothesis, and some really simple checks in both papers should have fixed the problems i've pointed out. when people don't bother to report really obvious checks like this, there are two potential reasons: (i) they didn't think to run some really obvious tests; or (ii) they actually ran the checks, but didn't the results they wanted. neither of these possibilities should exactly incline one to revise one's priors upwards.
if you want to argue that the Gordon hypothesis is probably true, i'll concur. if you want to argue that these studies contribute anything substantive to proving that, you'll find serious disagreement.
Posted by: conchis at Aug 23, 2007 7:12:30 PM
conchis,
you wrote: "to put it more bluntly: a bunch of crap studies don't necessarily add up to convincing evidence. they may just add up to a bunch of crap."
It appears that your judgement differs from that of the editors and
reviewers who have published these [sic] studies.
There are many possible explanation for an observation, but what tends
to win the day in science is when a particular one explains a variety
of observations. Gordon's hypothesis gains support from the evidence
presented in both the posted paper and the one published in the SEJ.
Obviously, you believe that you could provide even more convincing evidence
of Gordon's hypothesis. If you can, I would be happy if you would do
so, because, again, more evidence is, ceteris paribus, prefered.
Looking forward to YOUR working paper. . .
Posted by: indiana jim at Aug 23, 2007 8:27:58 PM
Jim, Explanations in science win out because they do a better job of predicting the evidence than alternative explanations. The problem with these papers is that they present evidence that one would place a high prior probability on observing even if Gordon's hypothesis is false. They consequently provide little reason for updating our beliefs about the hypothesis.
Feel free to keep handwaving about how this is a mere quibble, peer review is infallible, and that the fact that Gordon's hypothesis hasn't been contradicted means that it must be true. Repeating it won't make it any more correct. I think I've wasted more than enough time on this already, particularly given that you seem unwilling to engage with substantive argument.
Posted by: conchis at Aug 23, 2007 9:35:19 PM
conchis,
Of course evidence and its interpretation is judgemental; what to one is a quibble may be another's linch-pin. The evidence of the posted paper and the SEJ pub tell us the following: 1) complex mathematical articles in a sample of AER years were significantly less likely to contain empirics than a random sample of AER ariticles for the same years; 2) the papers that cited the sample of complex AER article were significantly less likely to be cited by articles containing empirics than was the AER random sample; 3) JET articles in the year 1980 containing at least 5 lemmas (highly complex)led to no operational statements that have been definitively tested in either those articles or in subsequent citing ones; 4)in a sample of "most frequently cited pubs" (500 or more cites) in top general interest journals in economics only 1 in 60 contained an author written lemma; 5) in most frequently cited articles in Econometrica and the Journal of the American Statistical Association, it was fairly common to observe author written lemmas; 6) a contingency table comparing 5 and 6 indicate a statistically significant difference between the appearance of author written lemmas in most cited general interest econ pubs (where Gordon's hypothesis applies) versus most cited econometrics pubs (where Gordon's hypothesis does not apply).
Your characterizations ("crap", "handwaving") seem silly.
Posted by: indiana jim at Aug 23, 2007 11:54:26 PM
indiana jim,
Why is it important that economic theories must lead to "operational statements" that can be tested? Haven't you read Rubinstein's 2006 Econometrica paper (http://arielrubinstein.tau.ac.il/papers/74.pdf)?
Posted by: MP at Aug 24, 2007 5:05:48 PM
MP,
It is important that economic theories lead to potentially refutable propositions only to the extent that one pursues economics as a social scientist. As noted in the paper, in physics Wolfgang Pauli famously dismissed statements that cannot be challenged as "not even wrong". Again, as noted in the posted paper, even refuted statement provide people with information about what not to do whereas statements that cannot be tested do not (they are as Pauli states "not even wrong").
Posted by: indiana jim at Aug 24, 2007 5:39:06 PM
Hi Best wishes。Allow me to offer my heartiest wishes.xicao loves-流水线娱乐博客常年提供高、中、低压锅炉钢管、流体钢管、结构钢管、化肥专用钢管、石油裂化钢管、地质钢管、液压支柱钢管及合金钢管-无缝管-无缝钢管等论文发表资讯/刊物信息，协助客户制定论文发表方案
Posted by: 无缝管 at Nov 13, 2007 8:05:00 PM
China Tours and Yangtze Cruises A professional China travel agency offering private & group package tours, tailor-made vacations for families, couples and individual travelers, Yangtze cruises, hotel & ticket booking services.
China Travel and Tours One-stop China travel service delivering China package tours, discount Yangtze River cruises and best rate hotels.
Tibet Train Travel A reliable tour operator specializing in Tibet tour packages, adventures, hotel and train ticket booking.
China Maps A free web-based source for China travel and tour map users
Voyage Chine, Tour de Chine, Circuit Chine - Un agence spécialiste voyage en Chine
Great Wall of China A website for the Great Wall of China.
Posted by: China tour at Nov 27, 2007 2:29:29 AM
Posted by: 翻译公司 at Feb 13, 2008 7:16:15 AM
Posted by: 鑽石 at Apr 2, 2008 10:01:13 PM
