« Hypotheses which are too simple to be true as stated | Main | Plan B, from Luigi Zingales »
Do researchers suffer from winner's curse?
OH NY GOD...READ THIS!!!!!!!!!!!!!!!!!
In economic theory the winner’s curse refers to the idea that someone who places the winning bid in an auction may have paid too much...The same thing may be happening in scientific publishing, according to a new analysis. With so many scientific papers chasing so few pages in the most prestigious journals, the winners could be the ones most likely to oversell themselves—to trumpet dramatic or important results that later turn out to be false. This would produce a distorted picture of scientific knowledge, with less dramatic (but more accurate) results either relegated to obscure journals or left unpublished.
My colleague Omar Al-Ubaydli was part of this work. Fortunately, it is mentioned on the very cover of the magazine. Nonetheless I believe it is true and it is most true for "hot" fields. The original paper is here.
Posted by Tyler Cowen on October 10, 2008 at 05:07 PM in Science | Permalink
Comments
This is a paradox in teaching medical research writing. We tell students to be accurate and cautious in their interpretation of data. But we then have a culture where those who "over-interpret" get the best press. Given that a few top papers can influence tenure and hiring decisions, I am concerned that incentives may be mis-aligned.
Posted by: JD at Oct 10, 2008 5:18:03 PM
You don't have to call it the winner's curse to know this is true. Every applied microeconomists knows that they can't get non-results published. That's also why everyone's obsessed with the treatment evaluation causality stuff. Ugh.
Posted by: jason voorhees at Oct 10, 2008 5:35:12 PM
Papers that attack the foundation of the logic(or lack thereof) of the public giving a private monoply to the federal reserve bank also tends not to get hyped in the media too much.
Economist who attack the bailout also tend not to do well in companies that are owned in part by GE, Ruppert Murdoch or Warren Buffet among others.
Posted by: Gabe at Oct 10, 2008 5:37:38 PM
In the scientific fields I've been involved in (math and CS), there is a strong countervailing prejudice against 'hot' and hyped results. Success in these fields requires the social skills to aim high while simultaneously signaling modesty and indifference. it's a weird kind of social skill, though, in that aspergers'-like behavior makes it easier, not harder, to show the right kind of indifference.
The most effective strategy, though, is to propose bold conjectures. A really great conjecture will make your reputation whether it is proved true, proved false, or best of all, unable to be proved for 300 years!
Posted by: DK at Oct 10, 2008 5:45:19 PM
Markets in everything?
http://www.jnrbm.com/
I propose the B.S. prize. Get a million bucks for calling B.S. on someone published in Science or Nature. $500k for JAMA and $2mil for a Nobel winner.
Posted by: Andrew at Oct 10, 2008 5:47:34 PM
Sounds remarkably similar to the one-upsmanship that's been at work among journalists and authors (and now bloggers) for ages. "Sure, everyone's talking about this topic, but WE'RE talking about it in ways that are even more shocking! Our take is that this is bigger and scarier and all around way worse than what they're saying!"
Posted by: Michael at Oct 10, 2008 6:00:35 PM
Can researchers on winner's curse suffer from winner's curse as well?
Posted by: kurt at Oct 10, 2008 6:01:52 PM
Dang, you beat me to the joke, kurt. I was about to type in, "I have tried for years to get a paper published that says the refereeing process tends to select papers with true hypotheses, but no journal was interested."
Posted by: Bob Murphy at Oct 10, 2008 6:35:52 PM
You mean Hepatitis B doesn't influence the ratio of women in China?!?!? Stunning...
Posted by: Dave at Oct 10, 2008 6:43:23 PM
"Dr Ioannidis based his earlier argument about incorrect research partly on a study of 49 papers in leading journals that had been cited by more than 1,000 other scientists. They were, in other words, well-regarded research. But he found that, within only a few years, almost a third of the papers had been refuted by other studies. For the idea of the winner’s curse to hold, papers published in less-well-known journals should be more reliable; but that has not yet been established."
This sounds to me like
1) We don't know how likely papers in other journals are to be refuted, so we don't have a sense that the ones in highly regarded journals are *unusually* likely to be refuted;
2) Even if they are unusually likely to be refuted, I'd like to see an argument that it's because they're *worse*, not because the extra attention paid to them due to their place of publication has led to more efforts to replicate/refute their results.
Posted by: Andromeda at Oct 10, 2008 6:54:45 PM
I thought the winner's curse says a person who wins an auction paid the most, therefore, they feel like they lost because they paid more than the next highest bidder, IE they got cheated out of a small amount of money.
Then logically, the winner's curse applied to papers would be that published papers present more information than was necessary to get published, IE they could have published 2 papers instead of the one.
Therefore, I refuse to read the paper because I disagree with the conclusion from the snippet Tyler posted... Why does the extra information have to be incorrect? Doesn't the Journal check the validity of the information before it is published? (And I'm not talking about the economist.)
Posted by: Brainwarped at Oct 10, 2008 7:05:52 PM
I think that publication bias is a much more serious problem than this supposed "winner's curse" in scientific publication. If there's a winner's curse problem it means that the journals and referees are not doing their job well. Publication bias is harder to correct. Besides the obvious stuff that people are more interested in "positive" results than in negative ones, publication bias has also affected by specification bias.
Posted by: Bruno at Oct 10, 2008 7:11:10 PM
http://www.boston.com/news/world/europe/articles/2008/10/10/betting_makes_nobel_literature_jury_suspect_leak/ <-- Check it out. Markets in Le Clezio.
Posted by: Andromeda at Oct 10, 2008 7:32:39 PM
I actually think that this comes back to a public choice problem, though I cannot pose it clearly. I say this based on my experience managing the Navy's nuclear research and what it looked like compared to the DOE's civilian nuclear research program. The Navy's program was high on technical merit, but very low on gee whiz. The civilian program was the opposite.
Why? My sense at the time was because the Navy was buying the research to solve real problems and cared very much about useful results. Civilian scientists on the other hand didn't care if they actually build reactors from their ideas, in fact the rational ones didn't expect to ever do so. Rather they just wanted the research dollars to keep flowing. So that means they had to appeal to the funding sponsors. What were their incentives? They needed to sell their research program to politicians to keep it going, and big ideas are more salable than important but unexciting details. Essentially analogous to why politicians spout terrible economics... because the people they're accountable to--the public--like it and buy it (i.e., vote for them).
I now work in other scientific endeavors. From my new position as a researcher I would say that the observation seems to generalize well. Big research scientists sell big ideas, even if their not really very good. My favorite example is a theory (I will not name it for fear of getting on the wrong side of funding sponsors), which after its publication in a bunch of papers and a book on it, was shown to be wrong at its core by researchers in Eastern Europe. Nevertheless, the originator continues to proselytize for it, expanding it in ever more big-thinking ways, without ever confronting or even acknowledging his detractors. Though wrong, it's really neat, so it continues. Recognize that this is in a physical science/engineering, where right and wrong answers are clear upon the evidence--that is, there is no doubt but that the original work on this idea is WRONG. I would expect the problem to be much worse where truth is harder to demonstrate.
Again, ultimately, I think this is because the NSF suffers from a similar disease, in having to account to congress. What research economist has not groaned at having to write the "Broader Impact" part of a grant proposal? Why is that there? Congress likes that in a research program... just like they like big ideas...
Posted by: Angus Hendrick at Oct 10, 2008 10:01:19 PM
Who Says Capitalism Is Dying?
Some good news for the weekend:
http://www.nytimes.com/2008/10/11/world/asia/11china.html?hp
"BEIJING — Chinese leaders are expected to allow peasants to buy or sell land-use rights for the first time, a step that could draw hundreds of millions of farmers more firmly into the city-centered market economy.
The new policy, which is being discussed this weekend by Communist Party leaders and could be announced within days, would be the biggest economic reform in many years and would mark another significant departure from the system of collective ownership and state control that China built after the 1949 revolution."
Read on.
Who says capitalism is dying?
Posted by: Don the libertarian Democrat at Oct 10, 2008 10:14:33 PM
I guess closure on the above thought as a public choice problem requires noting that basic research is a public good and that from that stems a government role in providing it efficiently. Essentially, the promotion of big ideas is lobbying.
Posted by: Angus Hendrick at Oct 10, 2008 10:15:17 PM
@Andrew at Oct 10, 2008 5:47:34 PM
>I propose the B.S. prize. Get a million bucks for calling B.S. on someone published in Science or Nature. $500k for JAMA and $2mil for a Nobel winner.
We now know who controls the $700 billion and what he plans to do with it ... :)
Posted by: AZ at Oct 10, 2008 10:32:16 PM
Emily Oster's work comes to mind. Here's a quick chronology:
1) Hepatitis B is responsible for the "missing women" in China.
2) Publish in top journal, get good job at Chicago.
3) Use same data to overturn result. (which suggests she wasn't very careful initially)
4) Publish paper overturning result
5) Shockingly get praised by Tyler Cowen for being so brave (albeit sloppy)
Posted by: vokuhila at Oct 10, 2008 11:23:40 PM
One way to get around this is to be a Bayesian and adopt informed priors on issues. That way, instead of looking for a "p value of such and such" you start demanding extraordinary evidence for extraordinary claims.
Posted by: Thorfinn at Oct 11, 2008 1:42:31 AM
> 3) Use same data to overturn result. (which suggests she wasn't very careful initially)
I could swear I've read in multiple articles that Oster used different data to overturn her initial paper.
Posted by: Michael at Oct 11, 2008 2:44:52 AM
"We now know who controls the $700 billion and what he plans to do with it ... :)"
Don't I wish. It would be used for just about the opposite of its current destiny. On that note, it seems like there ought to be some kind of do-over on the election now that the government is going to have vastly different powers. I mean, why elect a socialist or a warrior when you've already enacted socialism and there's no way in hell you'll have any money to fight wars.
Brain,
"Then logically, the winner's curse applied to papers would be that published papers present more information than was necessary to get published,"
Yes, and then the point is that the extra information is NOT TRUE, or at least sketchy, and if everyone agreed to not stretch, then the playing field would be even.
It's kind of like doping in sports where doping really helps.
So, I suppose the direct analogy to winner's curse as explained would be that auction winner's beat the next highest bidder by an amount paid in counterfeit money.
Posted by: Andrew at Oct 11, 2008 5:03:09 AM
Bob,
See: http://www.jnrbm.com/
Journal of Negative Results in Biomedicine
Apparently the economics profession needs its equivalent. Who better than you to start it?
Posted by: Andrew at Oct 11, 2008 5:05:33 AM
To clarify, in this case it is not the "winners" in the sense of the researchers who are cursed or suffer. It is knowledge or academia or the journals that do from "buying" faulty research.
I think it was Emily Ostrom, not Oster, whom Tyler praised, and Ostrom is certainly admirable, not a case of a winner's curse in regards to her research.
Econ Journal Watch may be the nascent economics journal of negative results.
Posted by: Barkley Rosser at Oct 11, 2008 7:22:53 AM
The first line of your post seems to be a typo? Is it "OH MY GOD!" or "OH NO GOD!"
I fear that you're saying both....
Posted by: ck at Oct 11, 2008 7:56:58 AM
Barkley - there are 7 past MR posts about Emily Oster, and zero about Elinor Ostrom (she has been mentioned in the comments). I tried to copy and paste the 7 posts that mention Oster but the TypePad spam filter thinks it's spam.
Posted by: AZ at Oct 11, 2008 8:04:01 AM
