« Passing down temptation to the next generation | Main | Chic Ironic Bitterness »
Was RAND wrong?
No, not Ayn Rand, the RAND experiment on health care. The RAND experiment randomly assigned people to different health plans and one of the big findings was that cost sharing reduced use of health care but had little effect on health outcomes. My colleague, Robin Hanson, likes to use this as a club to argue that we should cut medical spending in half.
Even randomized experiments have problems, however, and it turns out that there was a lot of attrition in the RAND experiment. A Healthy Blog quotes from a new paper in the October 2007 issue of the Journal of Health Politics, Policy and Law, by Dr. John Nyman of the University of Minnesota (alas not online).
Of the various responses to cost sharing that were observed in the participants of the RAND HIE, by far the strongest and most dramatic was in the relative number of RAND participants who voluntarily dropped out of the study over the course of the experiment. Of the 1,294 adult participants who were randomly assigned to the free plan, 5 participants (0.4 percent) left the experiment voluntarily during the observation period, while of the 2,664 who were assigned to any of the cost-sharing plans, 179 participants (6.7 percent) voluntarily left the experiment. This represented a greater than sixteenfold increase in the percentage of dropouts, a difference that was highly significant and a magnitude of response that was nowhere else duplicated in the experiment.
What explains this? The explanation that makes the most sense is that the dropouts were participants who had just been diagnosed with an illness that would require a costly hospital procedure. … If they dropped out, their coverage would automatically revert to their original insurance policies, which were likely to cover major medical expenses (such as hospitalizations) with no copayments … As a result of dropping out, these participants’ inpatient stays (and associated health care spending) did not register in the experiment, and it appeared as if participants in the cost-sharing group had a lower rate of inpatient use. … the cost-sharing participants who remained exhibited a lower rate of inpatient use than free FFS participants, not because they were responding to the higher coinsurance rate by forgoing frivolous hospital care but instead because they did not need as much hospital care, since many of those who became ill and needed hospital care had already dropped out of the experiment before their hospitalization occurred. …
Hat tip to The HealthCare Economist.
Posted by Alex Tabarrok on October 18, 2007 at 02:58 PM in Economics, Medicine | Permalink
Comments
shame on y'all for bringing this up
Posted by: R. Richard Schweitzer at Oct 18, 2007 3:13:10 PM
The randomized-experiment-as-Holy-Grail approach has its share of flaws, not least of which is that the tendency to participate or stay in a randomized experiment correlates with important unobservables. This study is a very fine example of that.
Posted by: Keith at Oct 18, 2007 3:41:33 PM
Could be possible. I am suprised RAND didn't check this for such an important study. I wonder how many more such studies tracking self-reported survey data have such problems.
Posted by: sa at Oct 18, 2007 4:13:09 PM
Not checking for something so obvious appears either dishonest or horribly incompetent.
Posted by: Yancey Ward at Oct 18, 2007 4:16:53 PM
I remember reading recently something old written about the experiment. They did mention the different drop-out rates and tried to account for why people left. They said it didn't make much of a difference (but I forget why)
Posted by: Rachel Soloveichik at Oct 18, 2007 4:46:11 PM
A pretty bad mistake... they may have just assumed that 6.7% couldn't make any significant difference to the final numbers, but frankly, that's just bad statistics.
Posted by: John at Oct 18, 2007 5:23:39 PM
"I remember reading recently something old written about the experiment. They did mention the different drop-out rates and tried to account for why people left. They said it didn't make much of a difference (but I forget why)"
To really evaluate this we'd have to know something about the outcomes of people who left. You can make "guesses" about this but the data support some rather interesting possibilities now. For, the difference in hospitalizations was only one third the size of the difference in attrition meaning that the data could plausibly support a case where these with co-pays had higher net hospitalization rates.
Given that the interpretation is equivalent health outcomes at lower costs, this could matter a lot.
Posted by: Joseph Delaney at Oct 18, 2007 5:37:31 PM
Hanson's haughty, I'm-so-damn-logical,-the-rest-of-you-fools-*must*-agree-with-me style often bugs me. He spends a lot of time talking in that article about how error prone and unscientific real world medical practice is. Now it turns out he was building his tightly reasoned, irrefutable economic prescriptions on an uncorroborated, faulty study. The irony is pretty sweet!
Posted by: dstevens at Oct 18, 2007 5:45:02 PM
In the original article in the New England Journal of Medicine, they discussed different retention rates and why they did not think it was too much of a problem. You can read their initial reasons why on page 14 in the pdf of the original article by using the following link:
http://www.rand.org/pubs/reports/2006/R3055.pdf
Hanson also linked directly to this pdf in his Overcomingbias post about the study:
http://www.overcomingbias.com/2007/05/rand_health_ins.html
Posted by: KapKool at Oct 18, 2007 6:41:30 PM
"To really evaluate this we'd have to know something about the outcomes of people who left"
The did gather data on people who left and used this to test their results for bias. Again, just read the original article by using the link that I posted above.
Posted by: KapKool at Oct 18, 2007 6:44:18 PM
The author of the anti-RAND article, John Nyman, has a huge personal investment in his view that health insurance is good and everyone who suspects that it causes overconsumption of health care is wrong. Trusting him on this issue would be like trusting a die-hard supply-sider's analysis of the Bush tax cuts.
Look at various analyses before drawing conclusions.
Posted by: Arnold Kling at Oct 18, 2007 7:14:47 PM
I checked KapKool's links. He seems to be right. RAND didn't ignore the drop-outs. They followed up by checking on the smoking habits and general health of the dropouts. This assertion is there on Pg14 and 18. So far so good. However, their follow-up of the drop outs wasn't 100%. In the case of volundtray drop outs it was 73%, for those who died it was 77%,etc. Now if a total of 95% people compete the plan(according to RAND) then out of 5% who didn't complete, a sub-set of those who dropped out voluntarily out which 27% didn't respond to the follow-up. Now this might distort the results for some trials where the sample is quite small but if we look at the table on pg 17 we see that the largest % of voluntary dr0p-outs was 10% (for teh catastrophic trial). Assuming the entire 27% that didn't follow up were counter-factuals we would get a error/sample of 27%*10%/90%(because of the drop outs, teh effective sample reduces) ~= 3%.
I don't think this is a big deal for the strength of results they got. My apologies to RAND.
Posted by: sa at Oct 18, 2007 7:22:43 PM
Yeah, I have been trying to trace the limited information follow-up. They clearly made a noble effort and in a modern trial they'd have a flow chart. But I am still not convinced that this removes the threat to validity of non-random censoring -- it just lowers the size of the possible bias.
It is a pity that nobody has ever considered replicating it.
Posted by: Joseph Delaney at Oct 18, 2007 7:49:58 PM
Yancey Ward said:
"Not checking for something so obvious appears either dishonest or horribly incompetent."
Are we not being a bit too negative, Nancy?
Posted by: John Pertz at Oct 18, 2007 10:08:24 PM
Arnold Kling>> Ok, so he got the results he "wanted". Something I think is a problem among many economists,
and social scientists in general. Instead of this strange attack on the study, can't you explain what is
incorrect in the study, rather than arguing that he has an agenda (and you don't, right?).
Posted by: Mikael S. at Oct 19, 2007 4:18:54 AM
Instead of this strange attack on the study, can't you explain what is
incorrect in the study, rather than arguing that he has an agenda (and you don't, right?).
Mikael, there’s nothing inherently objectionable in Arnold Kling's argumentative strategy. In assessing the soundness of an argument, what ultimately matters is whether the argument is sound or flawed. One reason for me to believe that there is a flaw in the argument is that I have been able to find one. But even if after looking for a flaw I find nothing, I may have other reasons to think the argument is flawed. One such reason is given by whether the agent who advances the argument has an interest in the argument’s conclusion. If the president of the Ford Motor Company advances an argument defending the claim that Fords are the best cars in the world, I may be reasonably sceptical about the truth of that claim even if I am unable to find a flaw in his argument. The fact that I’ve been unable to find a flaw in an argument after looking for it gives me a reason to believe the argument is sound, but the fact that the person who defends the argument is interested in the truth of its conclusion gives me a reason to believe that the argument is more likely to be flawed than I antecedently thought it might have been. Whether we should believe the argument is sound or faulty depends on how these (and other relevant) reasons balance against one another.
Posted by: Pablo Stafforini at Oct 19, 2007 6:02:40 AM
Sa,
If you assume the extreme counterfactual that they were all dead then the catastrophic plan had more than 300% higher mortality.
(11/1294 for the free plan versus 31/759 for the catastrophic plan)
Posted by: RobbL at Oct 19, 2007 9:21:20 AM
"If you assume the extreme counterfactual that they were all dead then the catastrophic plan had more than 300% higher mortality."
I think the issue is not "worst case imputation" so much as the potential error due to differential loss to follow-up is greater than the size of the effect (even after the attempt to interview drop-outs, although then only barely).
So you have two groups of drop outs:
1) Drop out with less information (who could plasubility skew cost data but are unlikely to affect the hard endpoints)
2) Complete loss to follow up.
The author applied imputation techniques (which is what I'd have done too) and so we can assume their reporting is in good faith (now that I have read the RCT, it looks okay on first glance even if I dislike their style).
SO the authors can say that it is probable and reasonable to accept the conclusions that they present. However, they are back into the realm of observational epidemiology where this is based on an unverifiable assumption.
It means that we can't rule, conclusively, any distribution among the complete loss to follow-up population. We can talk about likelihoods (I spend my life doing this) but we lose the cold, hard glare of certainty that we'd have if the loss to follow-up was much less than the size of the treatment effect.
It's not a fault of the investigators; they did everything that could reasonably have been expected to be done. Disliking their conclusion (my personal bias) does not make their work bad.
Still, we need to give the evidence the correct weight and it is less than I had expected reading Robin Hanson's blog (but way better than the paper linked above made em think). It's not bad research (au contraire, it's quite good) but even more than before it needs replication and validation.
Posted by: Joseph Delaney at Oct 19, 2007 11:19:45 AM
John Pertz,
You are right to chastise me for questioning the integrity and competence of their methods without actually reading the relevant section of their study first to see if the criticisms leveled at it were, in fact, legitimate. I just assumed that Alex Tabarrok had checked the original paper to see if the criticisms by Nyman were completely on target. It now appears that the Rand researchers did all they could reasonably be expected to do to account for the dropouts and their potential impact on the stated results.
Posted by: Yancey Ward at Oct 19, 2007 11:46:04 AM
Actually -- if this study were done today, I think everything possible to account for differential drop out rates would have included an Intent-to-Treat Analysis. Although, these have only become recently popular -- this probably would have changed the result.
Posted by: Jor at Oct 20, 2007 12:01:07 AM
"Intention to Treat" is able to handle poor treatment adherence and switching. It can't handle loss to follow-up without an assessment of the outcomes of the lost participants.
Posted by: Joseph Delaney at Oct 20, 2007 2:29:55 AM
Levitra (vardenafil HCl) is a prescription medicine that is indicated to treat erectile dysfunction (ED). Consistent with the effects of PDE5 inhibition, administration of Levitra with nitrates and nitric oxide donors is contraindicated. Caution is advised when PDE5 inhibitors, including Levitra, are used concomitantly with stable alpha-blocker therapy, because of the potential for lowering blood pressure. Levitra is not recommended for patients with uncontrolled hypertension (>170/110 mmHg).
Posted by: Serg at Dec 16, 2007 2:38:27 PM
In men for whom sexual activity is not recommended because of their underlying cardiovascular status, any treatment for erectile dysfunction, including Levitra,
generally should not be used. In patients taking certain CYP3A4 inhibitors (eg, ritonavir, indinavir, saquinavir, atazanavir, ketoconazole, itraconazole, erythromycin, and clarithromycin), lower doses of Levitra are recommended, and time between doses
of Levitra may need to be extended. See prescribing information for Levitra for
dosing guidance.In clinical trials, the most commonly reported adverse events with Levitra were headache, flushing, and rhinitis. Adverse events were generally transient.
Posted by: Serg at Dec 16, 2007 2:39:07 PM
Nonarteritic anterior ischemic optic neuropathy (NAION) has been reported rarely postmarketing in temporal relationship with the use of PDE5 inhibitors, including Levitra. Sudden loss of hearing, sometimes with tinnitus and dizziness,
also has been reported rarely in temporal association with the use of PDE5 inhibitors, including Levitra. It is not possible to determine if these events
are related to PDE5 inhibitors or to other factors. Physicians should advise patients to stop use of PDE5 inhibitors, including Levitra, and seek prompt
medical attention in the event of sudden loss of vision or hearing.
Posted by: Serg at Dec 16, 2007 2:39:55 PM
The recommended starting dose of Levitra is 10 mg. Titrate up to 20 mg or down to 5 mg based on efficacy and side effects. The maximum recommended dosing frequency is once daily. Levitra is available in 2.5-mg, 5-mg, 10-mg and 20-mg tablets. For Prescribing Information please visit New Online Pharmacy
Posted by: Serg at Dec 16, 2007 2:40:36 PM
