« Fruitless Fall | Main | The Secret Scripture »
Facts about publication bias
In 1995, only 1 percent of all articles published in alternative medicine journals gave a negative result. The most recent figure is 5 percent negative.
That is from Ben Goldacre's excellent Bad Science, right now available only in the UK. This is one of the best books I have read on how to think like a scientist and how to critically evaluate evidence and also on why we don't have a better press corps when it comes to science.
I thank Michelle Dawson for the pointer to the book.
Posted by Tyler Cowen on October 12, 2008 at 10:21 AM in Books, Science | Permalink
Comments
Ben also has a blog at badscience.net
Posted by: Richard Gadsden at Oct 12, 2008 10:26:04 AM
I'd be interested to know how much the percentage in non-alternative medical journals differs.
Posted by: Dan at Oct 12, 2008 10:30:39 AM
Yes, an excellent book, an excellent blogsite and an excellent weekly article in the Guardian newspaper. In fact, in my opinion, it is simply the best weekly column in the British media.
And I know that US and UK have spelling differences, but Goldaker vs Goldacre shouldn't really be allowed :)
Posted by: Rana at Oct 12, 2008 10:45:50 AM
Thank you for the link, Tyler. Just ordered my copy. You have been an invaluable guide to lots of books ever since I came across your website late last year. Your wide-ranging reading lists are markedly unusual, especially for chalky academics like us. I hope your colleagues and the other posters in your threads appreciate just how rare your roaming interests happen to be.
-- Michael Gordon
Posted by: the buggy professor at Oct 12, 2008 11:01:12 AM
A small correction. His last name is actually spelled Goldacre.
Posted by: Craig at Oct 12, 2008 11:12:23 AM
I'd also suggest "The Undergrowth of Science" by Walter Gratzer. It covers the big pseudoscientific fads/frauds but is particularly interested in the external forces that acted upon the scientists involved to sway the needle (be it the Prussian-Franco animosity that lead to the N-Ray or the academic environment around cold fusion).
Posted by: John F at Oct 12, 2008 11:21:21 AM
I would also like to know the percentage of failure reports in conventional medical journals. I do know that it is low enough that academic medical practitioners are publicly concerned. They want the data from failed attempts made public, not just that from successes. The absence of failure data is seriously hindering the progress of research and evaluation of potential problems.
There are two major reasons that failures of traditional medicine do not get published. First, the research sponsors do not want to reveal their research directions, nor their embarassment from the universally low success rate of new drugs and techniques. Second, there is a long standing bias against publishing failure in all branches of science. Failures will not win a Nobel prize, and articles about failures will not get the article references needed for a good academic rating.
Posted by: rjh at Oct 12, 2008 12:09:35 PM
Writing up the failed experiment for publication is no less work than writing up the successful experiment. There is some public value in having the failed experiment available, but can you internalize enough of that value to the researcher who is going to write it up?
Posted by: Cyrus at Oct 12, 2008 12:25:08 PM
A good read for science laymen: "Trick or Treatment". Reviews the history of acupuncture, homeopathy, chiropractic treatment and herbal medicine, and explains the scientific evidence that has shown how essentially useless they are. Also out of the UK, although many of the studies they review are from the US.
Posted by: Richard Koffler at Oct 12, 2008 2:00:29 PM
I have changed the spelling, thanks for the correction...
Posted by: Tyler Cowen at Oct 12, 2008 3:11:14 PM
At least if you are getting acupuncture, you aren't in a hospital receiving a fatal medical mistake.
That said, I would really like to read a paper the equivalent of "I, procedure" that would go through in detail how a truly successful medical miracle came about. I have a feeling a strong involvement in most of them is a surgeon, internist, or orthopedist willing to go out on a limb.
Posted by: Andrew at Oct 12, 2008 3:38:17 PM
@rjh; You have the reasons in reverse, primarily it's down to journals not wanting to publish negative results. The 'embarrassment' factor only applies to a small, but critical, percentage of science and they typically aren't publishing anyway.
Also, it's important to remember that if you can't publish you negative data in a journal then using the negative data to inform subsequent reseach is an advantage as you 'may' know something others don't. However, others may know it too....
Posted by: MikeA at Oct 12, 2008 4:23:29 PM
Why would you bother announcing failure to the world? Take the time you'd spend writing it up and try the next thing until you get something worth sharing - something that works. Sounds like the alternative medicine people just need to wipe out that last 5%.
Posted by: Tom Hanna at Oct 12, 2008 5:01:58 PM
@ Tom Hanna; I hope you are kidding. You see what you see. We're not talking about technical flaws here were are talking about systemic under-reporting of valid and valuable 'negative' results. 5% is too low. Knowing that something has been tried and it didn't work is kind of useful don't you think?
Posted by: MikeA at Oct 12, 2008 5:23:31 PM
Data on unpublished negative studies are naturally hard to come by, but a few days ago Pfizer was sued for hiding negative studies on Neurontin. According to the New York Times, 10/8/08:
"Dr. Dickersin, the Johns Hopkins expert, said that of 21 studies she reviewed, five were positive and 16 negative, meaning they did not prove the drug was effective. Of the five positive studies, four were published in full journal articles, yet only six of the negative studies were published and, of those, two were published in abbreviated form."
Given that the vast majority of alternative medicine studies are small (due to lack of funding) and thus less likely to reach statistical significance, and pharma trials sponsored by industry are often much larger (a study with 1,000s of patients can find statistical significance even with the clinical significance is almost nil), one could question how many FDA approved drugs would be in the same research status as alternative medicine were it not for the millions of dollars thrown at them. Some studies have indicated that entire classes of drugs, such as antidepressants, are essentially placebos.
Although the standard estimate of the placebo effect is 30-40% positive results, studies have shown that when both the patient and doctor are convinced the treatment works (even when the treatment is later proven totally useless), positive results occur in 60-70% of patients.
I agree with Andrew (3:38 PM) that in conditions of uncertainty like this, using a 'placebo' with a low risk profile is preferable.
Posted by: SteveSC at Oct 12, 2008 5:38:08 PM
Steven Landsburg over at Slate magazine had a piece on this a while back, suggesting that articles that say that the minimum wage has essentially no impact on inflation don't get published. Thus, the published articles probably overstate the impact of MW, because with a 95% confidence interval, 1 in 20 articles will show a false result, and those are the ones that get published.
http://www.slate.com/id/2103486/
Posted by: liberalarts at Oct 12, 2008 5:53:53 PM
Its impossible to evaluate the significance of that figure without knowing the percentage of negative results that get written up in biomedical journals, and in scientific journals in general. Is this a failing of alternative medical journals, or the way in which we approach research in general?
When trying to find the answer to that question - yes negative results are underreported in biomedicine, and industry funding probably leads to bad studies - I came across the Journal for Negative Results in Biomedicine. http://www.jnrbm.com/home/
Not something that I'll be digging into any time soon, but good to know that its out there.
Posted by: Justaguy at Oct 12, 2008 6:18:50 PM
Another book that broaches this topic, though pointing more at conventional medicine is "Overdo$ed America" (aka Overdosed America). It is an excellent read and very insightful into the way that commercial interests affect research publications and their interpretations.
Posted by: Al at Oct 12, 2008 6:22:10 PM
For an interesting interview with Goldacre, listen to this episode of the Skeptic's Guide to the Universe podcast:
http://media.libsyn.com/media/skepticsguide/skepticast2008-09-17.mp3
The podcast is one of my favorites; it's an irreverent group discussion of science and skeptical topics in the news, by people who really know their stuff. To subscribe to it, go here:
http://www.theskepticsguide.org/
Posted by: Ronald Hayden at Oct 12, 2008 8:02:37 PM
I must be lucky. When a blood cancer came into my life, I was able to spend weeks sifting through the internet (blogs, medical and other web sites, journals, pubmed, etc.) for research into treatments for my cancer, and the overwhelming majority of that research was 'negative' or at least concluded no benefit to various 'newer' drugs or protocols or treatment options versus existing standards of care. There were endless discussions and arguments about theories of disease behavior and interpretation of data. It was hardly a calvacade of one breakthrough after another.
And while acupuncture may be harmless in and of itself, telling people it will solve significant medical issues is not, and when you're thrown into the waters of diseases like high-mortality cancers, the alternative medicine crowd is out in even greater and more visible force.
As for Neurontin, the drug clearly has efficacy for some patients in certain conditions, though perhaps not much in epilepsy. It is a valuable drug. It's better to HAVE it available to for use than to NOT have it. I cringe to hear the populist approach to drug research that is increasingly prevalent, and which often vastly oversimplifies the issue. I see patients in blood cancers who cannot get treatments that may help them, and in those cases I want BETTER not simply MORE regulation.
Posted by: MM at Oct 12, 2008 8:05:11 PM
The publication bias in empirical economics research is unsurprising when you consider that finding nothing often means insignificant results. It is difficult to differentiate between running regressions of the wrong functional form, data shortcomings, and the existence of no statistical relationship. Hence, except in the occasion of a precisely estimated zero, it's unlikely to see publications finding no effect.
Posted by: CG at Oct 12, 2008 9:17:49 PM
Some studies have indicated that entire classes of drugs, such as antidepressants, are essentially placebos.
It wasn't reported that the study length was inadequate to determine this. The amount of time expected for antidepressants to start working was longer than the study length.
When a blood cancer came into my life
I don't know about other fields, but from my work with blood cancers negative research does get reported. However, a lot of it is in rebuttal to previous positive research done. Considering that blood cancers cover such a wide variety of diseases, and a specific cancer can be broken down into a lot of sub-groups it's hard to determine the efficacy of drugs. Take the drug Gleevec, for a certain subset of patients it's a wildly successful drug. However, if you applied it to leukemia patients as a whole you'd hardly see a difference.
Posted by: JordanT at Oct 13, 2008 1:27:05 AM
I think the important point is that most things don't work, and most statistics are wrongly done, so the positive results are probably not true.
Back when I was reading about chemotherapy, I got the impression that it is not effective in a vast majority of patients, but it is still administered, it does cause neuropathy in many, and with hemotoxicity form the major dose-limiting conditions.
This is a special case because patients are dying and will try anything, but as an example, the situation doesn't really jibe with most peoples' assumption of how the world works.
I suppose all that's left is to blame these failures on the free market we have in medicine.
Posted by: Andrew at Oct 13, 2008 3:52:15 AM
I guess I should say 'some' of the positive results are not true.
When we read a paper and there is a picture, we assume that is the best picture they could get, maybe by pure luck. Who knows how many tries they had. In fact, the main thing keeping me from publishing a paper right now is the ability to get a pretty picture and statistical data.
The statistical hurdle may be a good thing, but am I tempted to game the statistics...of course not! I mean, if I don't publish, the only things is that the last 4 years of my life have been WASTED! I'm sure all other researchers are as objective and principled as I am, if not moreso.
I think this gives a skewed view of results. It made sense before we had the ability to point to a website where all the research could be archived and before there were a gazillion journals. But, that is extra work, and we still have the system that encourages splash and doesn't reward full disclosure.
Posted by: Andrew at Oct 13, 2008 4:04:20 AM
Oh, and one more, sorry, but it's just me up at 4 a.m.
I was told by my advisor I couldn't publish a result that didn't work. Later he told me I misunderstood him. Unfortunately, I still don't understand.
Another problem was that I couldn't get a picture in 63x so I asked what was acceptable and what was optimal. He said 63x was acceptable and 63x was optimal. He wasn't trying to be humorous.
Student-advisor communication issues are a separate issue, but I wonder if identical conversations have happened between submitters and reviewers. Committees would probably accept negative outcomes if the researcher put his foot down and said "that's all you get." But if you ask someone what they want, they will tell you what they prefer. I suspect they prefer the dramatic over the important.
Posted by: Andrew at Oct 13, 2008 4:07:14 AM
