« What I've been reading | Main | Where is the world's largest swimming pool? »
John DiNardo against Freakonomics
Here is John DiNardo's review of Freakonomics, and here, and here are all three reviews he has written; one version was just published in the Journal of Economic Literature. These reviews struck me as grumpy and unfair, ultimately citing few contrary facts and boiling down to the complaint that the authors have not disproven all competing theories, or that the authors do not have a complete account of what a good explanation would look like. Three separate reviews of one book? And a popular book at that? What is going on? DiNardo is not some guy in his pajamas, rather he is a tenured professor at the University of Michigan with presumably a high opportunity cost. I am reminded of Einstein's rejoinder. Upon being presented with a book of essays called something like "The critics against Einstein" (that is a paraphrase) he replied with something like "If they were right, one would have been enough." The point is not that Freakonomics is infallible (on most particular issues I genuinely do not know exactly how robust their results are and to know would take a good bit of study), but rather that factual criticism on a single point is usually the best way to make critical progress. Now I would like to read a philosophy of science critique of Stephen Hawking's A Brief History of Time.
Posted by Tyler Cowen on January 27, 2008 at 05:54 AM in Web/Tech | Permalink
Comments
DiNardo is jealous that he does not have a best selling book and that he does not have tenure at UofC.
Posted by: allbull27 at Jan 27, 2008 9:24:06 AM
The freakonomics folks tend to be quite careless. They really overstate things to make it fit with their narrative. For example, the claims about car seats for young kids was just wrong. THey easily could have done a little more research first. But instead, they wanted to race to the press with their 'surprising' findings.
Posted by: at Jan 27, 2008 11:00:18 AM
"the claims about car seats for young kids was just wrong"
Explain.
Posted by: Steve Miller at Jan 27, 2008 1:15:51 PM
This is similar to what happened to Krugman when he went over to the other side. The trouble with popular press is that non-economists do not care about the distribution of error terms (if they knew what it even meant). The trouble with academics is that they DO care, even when/if it doesn't matter. There's a tension between the two sets of concerns, and both sides offend each other (try to get a normal person to read ever an AER piece...) So... he's whining that Levitt is not being rigourous -- as if that's important. (If you think the popular press is rigorous or cares to be, Fox News will set you straight.)
Posted by: David Zetland at Jan 27, 2008 3:48:36 PM
Here is the link to one of the reviews in the American law journal. It strikes me as misleading to claim that DiNardo is concerned with "error terms" or whether any particular point in the paper is right or wrong. He discusses the question whether the authors of Freakonomics are actually interested in whether something is right or wrong in the first place.
DiNardo has a whole section on "what to expect and what not to expect," and he certainly does not expect a discussion of error terms in a popularization of science. It is worthwhile to read his (original) review just for this section.
http://www-personal.umich.edu/~jdinardo/Pubs/aler.pdf
Posted by: at Jan 27, 2008 4:22:51 PM
Shorter Tyler: The success of "Freakonomics" has opened up opportunities for us economists to achieve more celebrity and money, so let's not nitpick it over minor matters like whether or not what it says is true.
Posted by: Steve Sailer at Jan 27, 2008 7:02:48 PM
I haven't read DiNardo's reviews, but I had read in The Economist some time ago that the famous Donahue and Levitt paper on abortion and crime rates did contain some major errors
Posted by: enrique at Jan 27, 2008 7:10:04 PM
Right, and the Wall Street Journal also did a big article on Foote and Goetz's discovery that Levitt's most famous theory (legalized abortion cut crime) was based on two technical errors he had made. For some reason, however, the New York Times, where Levitt was a columnist and is now a blogger, has never seen fit to cover this story. I have some vague sense that an economist might be able to explain this discrepancy in coverage using economic concepts like "self-interest."
Posted by: Steve Sailer at Jan 27, 2008 7:40:34 PM
Here is he link to the economist article, which probably should not exist. But I am not quite sure that this has something to do with DiNardo's critique, which is not about some particular point being right or wrong. Mistakes happen.
http://www.hss.caltech.edu/~alan/law33/Oops-onomics.pdf
Posted by: at Jan 27, 2008 7:45:16 PM
Mistakes certainly do happen, but when a mistake becomes the conventional wisdom, as is true with Levitt's abortion-cut-crime theory, you might think that the Newspaper of Record would have an obligation to point out the mistake, even if the NYT is making money off the glamorous reputation of the man who made the mistake.
Hey, here's a way to rectify the situation! Tyler is a columnist for the NYT, so he could just write a column in the NYT about how two obscure economists showed that NYT's star blogger Steve Levitt messed up and inflicted a wrong idea on the world. What could be simpler?
Posted by: Steve Sailer at Jan 27, 2008 8:52:11 PM
I just looked at John DiNardo's webpage on freakanomics and he explains how he wrote a paper and then decided to divide into three parts that are conceptually different (only two of which are published). I think this is a perfectly reasonable thing to do. It would have been a bit more fair to John DiNardo if this explanation was presented in the post.
Posted by: Bhash Mazumder at Jan 27, 2008 11:12:38 PM
Part of the reason for the multiple versions is the review process at the JEL. What I call the "extended play" version is the earliest so far as I know.
I am surprised by the negative reaction here given that MR promotes Economic Journal Watch, which does more or less the same thing as DiNardo's review, but without (in general) knowing as much econometrics as DiNardo does. Knowing metrics helps in ways that go well beyond concerns about distributions of error terms - how 80s - especially when causal claims are at issue.
Also, if one knows DiNardo, the idea that he is jealous of Levitt's wealth is not very plausible. He is one of those folks for whom the marginal utility of wealth is pretty low; he is more concerned with other things.
Jeff
Posted by: Jeff Smith at Jan 27, 2008 11:51:39 PM
Freakonomics was such a popular book that nitpicky articles have value.
This part of the review was interesting:
The text goes on to discuss the Romanian abortion ban referring to both popular articles as
well as more scholarly publications. One surprising rendition of the originals includes a pair of
papers by Cristian Pop-Eleches (Pop-Eleches 2005b, Pop-Eleches 2002), which is summarized in
Freakonomics this way on page 118:
Ceausescu’s incentives produced the desired effect. Within one year of the abortion
ban, the Romanian birth rate had doubled. These babies were born into a country
where, unless you belonged to the Ceausescu clan or the Communist elite, life was
miserable. But these children would turn out to have particularly miserable lives.
Compared to Romanian children born just a year earlier, the cohort of children born
after the abortion ban would do worse in every measurable way: they would test lower
in school, they would have less success in the labor market, and they would also prove
much more likely to become criminals.
The curious reader who tracked down the relevant papers by Pop–Eleches would be very
surprised to learn that the description in Freakonomics is virtually the opposite of what is
actually claimed.18
On average, children born in 1967 just after abortions became illegal display better
educational and labor market achievements than children born just prior to the change.
This outcome can be explained by a change in the composition of women having
children: urban, educated women were more likely to have abortions prior to the policy
change, so a higher proportion of children were born into urban, educated households.
(Pop–Eleches (2002), page 34)
While Pop–Eleches relates suggestive evidence that conditional on the usual list of demographic
characteristics, a fetus born after than ban is more likely to engage in criminal behavior, Pop-
Eleches’ conclusion is that the effect is second order.
Posted by: lemmy cauion at Jan 28, 2008 2:05:18 AM
Freakonomics was such a popular book that nitpicky articles have value.
This part of the review was interesting:
The text goes on to discuss the Romanian abortion ban referring to both popular articles as
well as more scholarly publications. One surprising rendition of the originals includes a pair of
papers by Cristian Pop-Eleches (Pop-Eleches 2005b, Pop-Eleches 2002), which is summarized in
Freakonomics this way on page 118:
Ceausescu’s incentives produced the desired effect. Within one year of the abortion
ban, the Romanian birth rate had doubled. These babies were born into a country
where, unless you belonged to the Ceausescu clan or the Communist elite, life was
miserable. But these children would turn out to have particularly miserable lives.
Compared to Romanian children born just a year earlier, the cohort of children born
after the abortion ban would do worse in every measurable way: they would test lower
in school, they would have less success in the labor market, and they would also prove
much more likely to become criminals.
The curious reader who tracked down the relevant papers by Pop–Eleches would be very
surprised to learn that the description in Freakonomics is virtually the opposite of what is
actually claimed.18
On average, children born in 1967 just after abortions became illegal display better
educational and labor market achievements than children born just prior to the change.
This outcome can be explained by a change in the composition of women having
children: urban, educated women were more likely to have abortions prior to the policy
change, so a higher proportion of children were born into urban, educated households.
(Pop–Eleches (2002), page 34)
While Pop–Eleches relates suggestive evidence that conditional on the usual list of demographic
characteristics, a fetus born after than ban is more likely to engage in criminal behavior, Pop-
Eleches’ conclusion is that the effect is second order.
Posted by: lemmy cauion at Jan 28, 2008 2:05:58 AM
"Freakonomics" made economics sexy. And from someone who hates calling things sexy that aren't well, sexy, that's saying something.
I don't think "Freak" was intended to invoke Michael Oher or the elephant man. It had a great title. It was very easily consumed. It didn't have numbers or equations, because, well, math sure does kill a mood.
As I recall, Freakonomics basically had broad themes of
Don't believe the headlines, especially those from policians (i.e., interested parties).
Don't underestimate the power of incentives.
Beware information asymmetry.
Economics can actually be useful (although it seemed to me mostly about statistics informed by economic questions).
They sexed up the previous "A Mathematician Reads the Newspaper," a book apparently a few years too early and without a catchy title, 6th grade prose, or controversial conclusions.
I think it's important for us to critically analyze what we read (one of their themes by the way), so critiques of works like Freakonomics are important. Maybe they need additional criticism because they are ambassadors to the general public on behalf of an obscure sect.
But, I don't think it's necessary to go so far as to attack the premise. Think of all the books that aren't as good as Freakonomics and worthy of much greater criticism. It's fine to say, "hey guys, we need a little better representation here" but I don't think it's good (not even from a self-interested point of view) to imply that the alternatives are better when they are not.
People need to understand economics, for their good, not ours (but als ours), and the broad question should be, "does this book help or hurt?" On that score, I think it helps.
Their argument about abortion didn't convince me. Correlation does not prove causation. And I get the feeling that people who read the book and accept the theory with nary a gimlit eye, kind of missed the point.
As a critique of their economics, it's good. As a review of popular literature, it does miss the point.
It kind of reminds me of libertarians who attack the most libertarian candidate since Jefferson because he's not quite their idea of the libertarian ideal. That's fine, but when election time rolls around, it's time to recognize the difference between more libertarian versus less libertarian.
Posted by: Andrew at Jan 28, 2008 6:24:02 AM
Errata:
People need to understand economics, for their good, not ours (but ALSO FOR ours),
Addendum:
Their argument that women are in the best position to determine whether they can properly raise child made sense (it would be nice if they would choose birth control, might be cheaper for them too).
They did make the point that it would be ridiculous policy to try to prevent crime by promoting abortion due to the ham-fisted mathematics of the equation (a la, you may kill many fetuses to prevent one crime).
What wasn't convincing to me was the assertion that because crime began to drop some time after abortion was legalized that this constitutes proof.
My pet theory is that crime dropped because the economy began it's boom, it looks like violent crime peaked in 1992 (http://www.disastercenter.com/crime/uscrime.htm).
A lot of kids were home playing video games. Maybe even Micheal Jordan had an impact, inspiring kids to play on the playground rather than hanging out. I know it kept me out of some trouble. Maybe even the availability of drugs had people in their homes committing non-violent crimes that aren't reported instead of out in the street committing violent crimes. Washington D.C.'s resident population has dropped considerably since 1970. Is this due to abortion or due to urban exodus? Maybe the overall secular population drop in violent cities has something to do with it. We like to attribute step changes to singular events. But maybe there is a critical mass of population density that contributes to violent crime. I remember there being a lot of news stories about the violence in the 80s. Maybe statistics reporting just got better and then maybe news reporting just got more sensational. Maybe in response people picked up stakes and did what they deemed best to remove themselves from high victimization situations, sort of a positive feedback crime recession. I haven't thought this through that well. I don't get paid to do so.
Basically, my point is, there's a lot of questions to be asked. Did they ask them all? And just because they didn't disprove the abortion angle, that doesn't mean they proved it.
Posted by: Andrew at Jan 28, 2008 6:58:39 AM
DiNardo was one of my professors at Michigan. Great guy. A minor big-shot in econometrics. He always struck me as a bit of a fundamentalist on whether or not someone has shown causation (but I was a masters student, so what do I know?).
That said, he seemed to think deeply and carefully about econometrics, and who writes stuff like these reviews despite its negligible impact on his career.
Posted by: slog at Jan 28, 2008 7:56:35 AM
Personally, it seems as though a lot of the scholarly criticism of Freakonomics has hinged around the idea that it is a trivialization of the economic field, that we need "big picture" thinkers like Keynes more than we do application people. That being said, for all the factual controversies in the book, Levitt and Dubner are probably, on balance, far more accurate with what they have written than what the track record has been for Keynes...
There may also be some sort of a resistance to Levitt's book on the grounds of it making a rigorous academic discipline "too accessible", but those concerns tend to be summarily ignored.
Posted by: Michael Fisk at Jan 28, 2008 8:38:05 AM
Given the impact Freakonomics is having on popular understanding of economics in general and specific subject in particular, I believe the more criticism the better. By the way, the assertion that DiNardo is motivated by jealousy is absurd. He's tenured at Michigan, for cring out loud.
Posted by: Ted Craig at Jan 28, 2008 9:47:33 AM
The abortion-cut-crime theory is a case study in how wrong ideas can become conventional wisdom. I pointed out to Levitt in Slate in 1999 that he failed to do simple reality checks on his theory. For example, nationally, the first cohort born immediately after legalization committed murders at three times the rate of the last cohort born before legalization, but he had failed to look at the data closely enough.
Levitt replied that his complex analysis of state by state data supported his view. Well, six years later Christopher Foote and Christopher Goetz finally tried to reproduced his state-by-state analysis and discovered that it was caused by two technical errors Levitt had made. So, that demonstrates the value of simple reality checks.
Posted by: Steve Sailer at Jan 28, 2008 2:23:49 PM
Ted Joyce has a forthcoming ReSTAT (first entry in his online CV) that is a test of DL based on a similar logic to what Sailer just suggested, for those interested. The use of older (i.e., "untreated cohorts") is a good way to disentangle the abortion effects from any prevailing period effects. The difficulty in the abortion-crime hypothesis is you have a kind of Hail Mary thing going on - the treatment occurs twenty years earlier, and you're hoping that treatment in those states will not be correlated with anything else that shows up later. But in the case of abortion and crime, that's a lot to ask, which is why I like Joyce's (and Sailer's, to give him credit) recommendation to use older age cohorts as implicit control groups for state-specific unobservables that could confounding the regressions. After reading Joyce's two papers on this now (the 2004 JHR paper and the 2008 ReSTAT paper forthcoming) and the Foote and Goetz forthcoming QJE paper, it does seem like the abortion-crime evidence isn't nearly as strong as we used to think.
But, at the same time, there are corollary studies published that don't suffer from the same problems as Sailer says the original DL had. For instance, Stephens and Charles (2006) Journal of Law and Economics finds drug use to be lower in the repeal states among the treated cohort than the untreated cohort, using basically the triple difference method that Sailer is implicitly wanting with his "reality checks." Another by Ozbeklik (unpublished) using the same have found lower teen fertility, too - always comparing the ones born in the years that abortion was legalized to those born one year earlier. And then there is the original Gruber, Levine and Staiger (QJE 1999) on the "marginal child" and living standards of aborted children and an unpublished NBER working paper by Annant, Gruber, Levine and Staiger (2006) that replicates some of the abortion-crime results using state-year fixed effects (I'm not sure if they correct for the measurement problems that Foote and Goetz point to, though).
Sailer, what's your opinion of all these studies? Are you saying you think they are all dishonest scholars?
Posted by: jason voorhees at Jan 28, 2008 3:29:31 PM
My view when I was actively involved in the controversy in 1999 and 2005-2006 was always that it looked beyond the ability of state of the art social science to resolve. That may not be true forever.
That said, I'm distinctly dubious about using statistical techniques developed for measuring crop productivity on state level data regarding crime and abortion precisely because people, unlike crops, move over the course of two decades. Moreover, crime is one of the biggest reasons people move, as are social attitudes linked to abortion laws and abortion rates.
Posted by: Steve Sailer at Jan 28, 2008 11:24:53 PM
My position has always been that Levitt's theory should not be the conventional wisdom -- it might be true, it might not be, but we're a long way from knowing -- but that's what it has become -- in part because of the New York Times' obvious conflict of interest in both employing and promoting Levitt.
Posted by: Steve Sailer at Jan 29, 2008 2:01:44 AM
My position has always been that Levitt's theory should not be the conventional wisdom -- it might be true, it might not be, but we're a long way from knowing -- but that's what it has become -- in part because of the New York Times' obvious conflict of interest in both employing and promoting Levitt.
Posted by: Steve Sailer at Jan 29, 2008 2:03:51 AM
"Jason",
You surely know you are tilting at windmills with Sailer. He will cite econometric work when it suits his view and then throw out the whole enterprise as bunk in the next breath. All the while he spouts his own theories which he has neither the skills nor the data to rigorously test.
He is right D-L may or may not be right. But this is how science progresses. Theory-Hypothesis-Data-Test-Wash-Rinse-Repeat. It is easy for every professional blog commenter to throw stones at scientific work that flies in the face of their beliefs. However, they have no interest in actually moving the scientific debate forward by rigorously putting forth a theory and then empirically testing its implications.
Sometimes their criticisms are right even when they lack the ability to express them correctly. But more often than not they are wrong based on a complete and total ignorance of the tools being used.
You would think after almost 10 years Sailer would take the time to learn the basics of the techniques he has earned his infamy criticizing. Think how much misunderstanding could be avoided if once in his life Sailer worked through the proofs for consistency and unbiasedness of Ordinary Least Squares.
Posted by: GoodnessOfFit at Jan 29, 2008 10:47:59 AM
