« Words I wrote yesterday about fiscal policy | Main | The new iTunes pricing »
Cancer and Statistical Illusion
The cover of this month's Wired promises "The Truth About Cancer" but the article inside is a tissue of misleading statistics and faulty logic. The article begins with fancy graphics telling us "If we find cancer early, 90 percent survive" but "If we find cancer late, 10 percent survive." And this:
Find the disease early "and the odds of survival approach 90 percent...This reality would seem to make a plain case for shifting resources toward patients with a 90 percent, rather than a 10 or 20 percent, chance of survival."
Thus, the opening block of text commands, "Scientists should stop trying to cure cancer and start focusing on finding it early. It's the smart way to cheat death."
The fallacy in all of this is painfully easy to spot. If we measure survival, which these studies do, with a 5 or 10 year survival rate then obviously people whose cancers are detected early will survival longer than people whose cancers are detected late.
The key question is whether people who are treated early survive longer than people whose cancers are detected early but who are not treated. In Thomas Goetz's long article there is not a single piece of evidence which demonstrates that this is true. Indeed, quite the opposite. About 9 pages into the article, after the jump, we find this about CT scans for lung cancer:
As with the Action Project, these studies found that, yes, CT scans detected a huge number of early cancers—10 times as many as they would expect to find without scanning. In that regard, the scans did their job as a screening test. And as expected, the number of surgeries based on those diagnoses jumped. But when Bach looked at the resulting mortality rates, he found essentially no difference between those who received a CT scan and those who had not. Despite the additional surgeries, just as many people were dying as before.
Nowhere does the author mentions that this finding invalidates just about everything he has told us in the first eight pages.
Addendum 1 : Do note that I have nothing against early detection and I am not claiming that it never works. My problem is with misleading statistical analysis.
Addendum 2: Careful readers will note that this is an almost perfect example of the economicitis fallacy that I blogged about late last year.
Posted by Alex Tabarrok on January 7, 2009 at 07:22 AM in Economics, Medicine | Permalink
Comments
Let's also recall that "cancer" is not a homogeneous concept (i.e., "the disease").
You can easily survive late-stage colorectal cancer, so long as the tumor is still encapsulated. Lung cancer is still almost always a death sentence, no matter how early it is detected.
Something to keep in mind when Obama & Co. talk about "jobs" -- as if all "jobs" are the same.
Posted by: KipEsquire at Jan 7, 2009 7:34:48 AM
Lung cancer is still almost always a death sentence, no matter how early it is detected.
As is pancreatic and stomach cancer. I recently had a relative told its either pancreatic or a
form of lymphoma. Imagine being told is lymphoma and breathing a sigh of relief.
Prostate is so slow, usually something else gets you.
Posted by: Superheater at Jan 7, 2009 7:46:56 AM
Fine post. This is an important topic I wish I knew more about.
Related is the danger that comes from early diagnosis - unnecessary but risky treatment. There has been some publicity about that problem in Canada recently. Combine the two and the medical industry has some serious explaining to do.
Posted by: tomslee at Jan 7, 2009 7:53:26 AM
As a general rule, early detection is a good idea, especially if it's easy and cheap. Small tumors usually have a better prognosis than large tumors. This is not true for some of the really bad ones, such as lung or pancreatic cancer or glioblastoma. But I would still urge women to get their Pap smears and mammograms, and colonoscopy for those over 50 is probably a good idea. While early detection certainly sounds good, prevention is even better. Many cancers are lifestyle-related (e.g., smoking, drinking, diet, obesity, excessive sun exposure, etc).
Posted by: Ned at Jan 7, 2009 8:56:19 AM
Alex, this is a great observation; it's so obvious but it never occurred to me. Did you invent this, or did you read someone else making the point? Also, have you emailed any doctors about this? Probably not the author of the article, since he would be predisposed to dismiss it as nitpicking, but other people with no large stake in the outcome (besides saving lives!).
I don't know if you've heard of this one, but Mary Meyer (at U of Georgia I think) claims that the official statistics on how many lives airbags save, is wrong because of the way the NHTSA performed the analysis. (They looked at only crashes involving fatalities, rather than all crashes.) Here is a story about it. The NHTSA says she's nuts and that airbags do save lives on net. (Note that Meyer is saying they are dangerous for everybody, not just petite women and children.) If you like mistaken statistical arguments, you might check it out.
Posted by: Bob Murphy at Jan 7, 2009 9:01:37 AM
"The key question is whether people who are treated early survive longer than people whose cancers are detected early but who are not treated."
Unquestionably yes. Even after adjusting for age and life expectancy (sex, race, income), detecting almost any from of cancer in situ before metastases significantly increases five year survival rates.
I'm trying to find a non-concluted chart (like the one I have in my Cancer Facts and Figures 2008 booklet) on the national cancer institute's website, with no luck. Here are some highlights from the booklet:
Cancer Survival % (all stages) Local Regional Distant
Breast 89 98 84 26
Colon 64 90 68 10
liver 11 22 7 3
lung 15 49 15 3
Yeah, the article's statistics are faulty, and the way the writer keeps referring to The Cancer is annoying; but his meta point is true. On the other hand, the scientific community is already way ahead of him. All the new cutting edge treatments and research are ways of early detection, genetic tests, and targeted treatments (like gleevec).
Posted by: at Jan 7, 2009 9:21:50 AM
While I am an actual, honest to god subscriber to Wired, and I love the magazine, I would say that this article is par for the course with them. I would summarize their type of journalism as "aggressive ignorance". Get a small bit of knowledge, and then blow it all out of proportion by extrapolation.
Posted by: Buzzcut at Jan 7, 2009 9:36:28 AM
"but his meta point is true"
I take Alex's meta point to be that "five-year survival" is a misleading measure, because the clock starts ticking at different stages.
Example: if "early detection" causes me to find a tumor two years earlier than I would have otherwise, the relevant comparison would be no-early-detection-five-year-rate versus early-detection-seven-year-rate.
Posted by: Rich at Jan 7, 2009 9:37:42 AM
reminds me a stat in American football, whichever teams kneals down the most wins the game. :).
Posted by: Mr. Beefy at Jan 7, 2009 10:40:48 AM
It seems that actually we don't have enough information to judge the data. If the rate of the cancer's progress is typically very fast, and the survival rate tail flattens out after three years, then using a 10 year survival rate should still give meaningful information.
Of course, they didn't give these details. I'm just saying that it doesn't seem clear either way.
Posted by: Anon at Jan 7, 2009 10:56:46 AM
To the unnamed commenter at Jan 7, 2009 9:21:50 AM:
The idea is that we need to think about measuring something more fixed than "survival" -- something like "age at death" instead.
For example, if you die at age 40, then it doesn't matter that the cancer was detected at age 20 and you "survived" for 20 years, or whether it was detected at age 39 and you "survived" for 1. The point is that you still lived to be 40 in either case.
But if it can be shown that people with early detection live to be 41, whereas people with late (or no) detection live to be 39, then you can make the case that early detection extends life expectancy by 2 years.
This would be the more interesting result.
Posted by: J.V. at Jan 7, 2009 11:07:33 AM
Early detection is not the issue. It might be fine to detect something early, but the choice given to the individual after early detection is the issue. If the individual chooses not to undergo a risky treatment, his cost for health care can go up enough to make him/her think about the treatment in the first place. I think this as a form of monopoly where you don’t have a choice. As long as the medical profession and health care providers don’t offer all the choices to individuals we would be faced with factors mostly out of our control that decide how we live our life.
Posted by: JAK at Jan 7, 2009 11:12:16 AM
Early detection is not the issue. It might be fine to detect something early, but the choice given to the individual after early detection is the issue. If the individual chooses not to undergo a risky treatment, his cost for health care can go up enough to make him/her think about the treatment in the first place. I think this as a form of monopoly where you don’t have a choice. As long as the medical profession and health care providers don’t offer all the choices to individuals we would be faced with factors mostly out of our control that decide how we live our life.
Posted by: JAK at Jan 7, 2009 11:14:22 AM
One of the problems is definition. As alluded to with prostate cancer, not all cancers grow at the same rate and have the same martality potential. Detecting these benign cancers can result in increased mortality and morbidity due to hacking (aka surgery) and medical errors that can befall patients. The infamous Rand study that showed no detectable difference in health outcomes between people with health insurance and those without illistrate this.
Posted by: fresno dan at Jan 7, 2009 12:30:31 PM
SUGGESTED READING
http://cancertutor.com/
Posted by: RON FLECK at Jan 7, 2009 12:34:45 PM
There is no generic cancer. Early detection is important depending on the type of cancer one has. In my case it was Large B-cell Lymphoma. Very aggressive, has to caught early or survival rates plummet. However, if caught early it is one of the "curable" forms of cancer. Other types of Lymphoma allow you to live many years without to much discomfort until the end and there is at this time there is no cure, to that end.
The real issue with cancer is the approach to "killing" cancer. Chemo kills all your cells, it's a race to kill the cancer before the cure kills you. The question is not early detection but a new approach to treating cancers based on your own body's immune system. If "traditional" treatment does not work the patient is scheduled for a stem cell transplant from their own body, if possible or a matched donor. The ideal approach would be to use stem cells first not last, but that would require a reversal of 50 years of orthodox thinking that we first have to kill with chemicals and radiation.
So the basis of the article is faulty on many different levels.
Posted by: Georege Kalogridis at Jan 7, 2009 12:35:11 PM
I came across this same point in my cancer reading a couple years ago.
Every time I read the same critique I ask myself "can they really be this bad" and the more I read my answer is "yes."
If you would have told me that the world was like this when I was in high school I might have went ahead and killed myself, or chosen to major in economics.
Posted by: Andrew at Jan 7, 2009 2:00:50 PM
I think you're referring to what epidemiologists call "lead time" bias.
Posted by: Frannie at Jan 7, 2009 3:34:20 PM
To the poster concerned with alerting the medical mainstream: this concept - lead time bias - as well as the myriad other problems that can affect medical research are well understood and appreciated by epidemiologists and others in the medical community. Lead time bias, length time bias, selection bias... are all taught in biomedical statistics courses, medical school, and epidemiology courses. Quality research generally avoids or accounts for such biases. Though, to be sure, there is plenty of poor quality research being published!
Posted by: at Jan 7, 2009 6:23:09 PM
Hi Alex - Thomas Goetz here, author of the said article in Wired. I am a fan of your blog, and appreciate your spotting my failing to mention lead-time bias & length-time bias, as we call these statistical glitches in public health - though I am well aware of the phenomenon. But that omission - which I'll account for - hardly makes the piece "a tissue of misleading statistics and faulty logic," as you contend. On the contrary, the whole premise of the article is that early detection is, on the surface, an alluring and seemingly simple fix that, once you start dealing with the numbers, becomes a far more complicated task. Thus, the mission of the Canary Foundation (the focus of the story) is to use statistics and multiple-stage tests to narrow the screening pool only to those with a high risk of developing *lethal* disease, versus non-lethal disease. The problem with CT scans as a first-stage test is that they fail is to apply this statistical winnowing, that they fail to discern between lethal and non-lethal cancers. The reason there is "not a single piece of evidence" that demonstrates two-stage tests are effective is because, well, those tests are only now being developed. That's *what the story is about*, that very search for evidence (ie, an effective two-stage test), and to fault the story for failing to provide such evidence is a logical fallacy all its own.
As for the lead-time bias question, as Frannie notes, it's a well-documented phenomenon in epidemiology, which I am well aware of. No doubt there is some fraction of survival rates in certain cancers, breast cancer most obviously (due to mammography), where lead-time bias might account for some fraction of the recent uptick in survival rates. But here's the thing: even accounting for lead-time bias, screening has been shown to extend lives. And here's the bigger thing: lead-time bias is largely an academic point, anyway, because survival rates for cancer have *not significantly budged* in 30 years, since there are so few effective screening tests (the slight improvements that have manifested are due almost entirely to fewer smokers). In other words: I chose not to mention lead-time bias because it's effect is negligible, given that survival rates are so stagnant and so poor to begin with. You can contest the wisdom of that omission, but it was hardly done out of ignorance, nor should it be taken as evidence that I was trying to mislead readers. On the contrary, the issue is, in effect, so statistically insignificant that to introduce it, explain it, and account for it, would have been an unnecessary distraction and digression.
However, I will admit that flagging the lead-time issue may be an interesting curiousity to a quant crowd, such as your readers (a different audience than mine). But it is hardly an argument against early detection, nor, I can scarcely believe, a reason to impugne the entire substance of my story. Which in any case I hope your readers actually take the time to read and consider on its own merits: http://wired.com/medtech/health/magazine/17-01/ff_cancer
all best,
tg
Posted by: thomas goetz at Jan 7, 2009 7:42:41 PM
I understand the statistical illusion, but it's silly to imply there is a lack of evidence that early detection/treatment doesn't improve the odds of survival. Surgery, for example, can be very effective against cancer if the cancer is found early enough that you can cut it out without dying. If Bob Marley had had his cancerous toe amputated, he'd still be with us. Radiation can be effective against a limited spot, too, but it can't be used against cancer that has spread widely, which leaves only chemotherapy, so now you are down to one method.
Posted by: Steve Sailer at Jan 7, 2009 9:11:27 PM
"The key question is whether people who are treated early survive longer than people whose cancers are detected early but who are not treated"
Does it happen?
Posted by: s24 at Jan 8, 2009 3:52:33 AM
Thomas Goetz (and others) --
Alex claims that none of the studies you cite in your article support your thesis. Can you please give the citation for an epi study that appropriately adjusts for "lead-time" bias? Not being an epidemiologist, I'd like to be sure that what they view as lead-time bias is actually addressing the concern that Alex raised.
Roger
Posted by: RogerClemens at Jan 8, 2009 9:36:54 AM
I believe that there is also a TREATMENT BIAS here. Cancers found earlier might not be treated as aggressively.
I had testicular cancer (seminoma) and found it while it was still stage I. Not only is the survival rate said to be higher for this type of cancer if found early, but the treatment methods are also less radical.
Perhaps part of the problem with early detection is that cancers that are found earlier might not be treated as aggressively as late stage cancer, thereby increasing the risk of death.
Recurrence of a cancer MAY indicate that it wasn't treated aggressively enough. Of course, some cancers are just hard to beat.
Posted by: Jason R. at Jan 8, 2009 10:25:19 AM
There was a really nice discussion of this whole issue on Respectful Insolence (a blog written by a surgeon/medical researcher whose area of research is breast cancer). You can find the first part at
href="http://scienceblogs.com/insolence/2007/04/detecting_cancer_early_part_1_more_compl.php
The series of posts is quite good, and readable with some knowledge of statistics and little knowledge of medicine. (I speak from experience here.)
Posted by: albatross at Jan 8, 2009 1:38:49 PM
