Editor’s Note: The following guest post by Dr. David Newman is reprinted with permission from his website and blog, Smartem.Org. Dr. Newman is an Emergency Physician and Director of Clinical Research at Mt. Sinai School of Medicine in the Department of Emergency Medicine. He is the author of the critically-acclaimed Hippocrates’ Shadow: Secrets From the House of Medicine. CardioBrief readers might also enjoy watching his TED talk, Truth That Lasts.
Data, Drugs, And Deception– A True Story
by Dr. David Newman
Last week The Lancet published a meta-analysis of 27 statin trials, an attempt to determine whether patients with no history of heart problems benefit from the drugs — true story. The topic is controversial, and no less than six conflicting meta-analyses have been performed — also a true story. But last week’s study claims to show, once and for all, that for these very low risk patients, statins save lives — true story.
Actual true story: the conclusions of this study are neither novel nor valid.
The Lancet meta-analysis, authored by the Cholesterol Treatment Trialists group, examines individual patient data from 27 statin studies. Their findings disagree with an analysis published in 2010 in theArchives of Internal Medicine, and with analyses from two equally respected publications, theTherapeutics Letter and the Cochrane Collaboration.* Despite this history of dueling data the authors of last week’s meta-analysis, in a remarkable break from scientific decorum, conclude their report with a directive for the writers of statin guidelines: the drugs should be broadly recommended based on the new analysis.
As an editorialist points out, if implemented, the CTT group recommendations in the United States would lead to 64 million people, more than half of the population over the age of 35, being started on statin therapy — true story.
Where is the magic, you ask, in this latest effort? What is different? In some ways, nothing. Indeed just a year and a half earlier The Lancet published a meta-analysis of 26 of the same 27 studies, with the same results, by the same authors (true story, and an odd choice on the part of the journal). So the findings aren’t new. They are, however, at odds with other meta-analyses. Why? It is the way they calculated their numbers. This meta-analysis, like the earlier one from the same group, reports outcomes per-cholesterol-reduction. The unit they use is a “1 mmol/L reduction in low density lipoprotein (LDL)”, in common U.S. terms, a roughly 40-point drop in LDL.
That’s the magic: each of the benefits reported in the paper refers to patients with a 40-point cholesterol drop. Voilá. One can immediately see why these numbers would look different than numbers from reviews that asked a more basic question: did people who took statins die less often than people taking a placebo? (The only important question.) Instead, they shifted the data so that their numbers corresponded precisely to patients whose cholesterol responded perfectly.
Patients whose cholesterol drops 40 points are different than others, and not just because their body had an ideal response to the drug. They may also be taking the drug more regularly, and more motivated. Or they may be exercising more, or eating right, and more health conscious than other patients. So it should be no surprise that this analysis comes up with different numbers than a simple comparison of statins versus placebo pills. Ultimately, then, this new information tells us little or nothing about the benefits someone might expect if they take a statin. Instead it tells us the average benefits among those who had a 40-point drop in LDL.
But LDL drop cannot be predicted. Some won’t drop at all, some will drop just a bit, and some may drop more. Therefore the numbers here tell an interesting story about certain patients who took statins, but they have no relevance to patients and doctors considering statins. And yet, the latter group is the target of the study’s concusions.
True story: in prior meta-analyses that found no mortality benefit the investigators simply looked at studies of patients without heart disease and compared mortality between the statin groups and the placebo groups. No machinations, no acrobatics, no per-unit-cholesterol. They took a Joe Friday approach (just the facts, ma’am), and found no mortality benefit.
Perhaps never has a statistical deception been so cleverly buried, in plain sight. The study answers this question: how much did the people who responded well to the drug benefit? This is, by definition, a circular and retrospective question: revisiting old data and re-tailoring the question to arrive at a conclusion. And to be fair they may have answered an interesting, and in some ways contributory, question. However the authors’ conclusions imply that they answered a different, much bigger question. And that is not a true story.
Guideline writers, doctors, patients, journalists, and policy makers will all have to pay close attention to avoid the trappings of deceptive data, dressed up as a true story.
——————–
*The Cochrane Collaboration analysis reports an overall mortality benefit with statins (RR=0.86), however their summary suggests that statins should be used for primary prevention “with caution.” In particular on p.12, after a discussion of the biases in many of the trials that led to their numerical finding, they clearly state that using statins for patients with anything less than a 2% per year risk of coronary events “is not supported by existing evidence.” This cutoff encompasses virtually all people that would be considered candidates for primary prevention.
thank you. Clearly, this important commentary raises questions about the integrity of Lancet as well as the authors of the study.
But what is most discouraging, and dangerous, is that this kind of deception occurs so frequently. Meanwhile, major news outlets, television channels, etc. have reported the results as the authors intended. And so the gravy train has left the station.
Lancet should publish a clarifying editorial and send it everywhere they can. They are an accessory to any harm that has resulted, to patients and to the credibility of medical science.
I disagree with this critique. The authors weighed each RCT by the LDL-C lowering to be able to provide a standardized comparison across the studies (it’s in the Methods of the paper). By doing so, they didn’t break the randomization, and shouldn’t have introduced confounding, as suggested by this commentary piece. And being able to say “for each 1 mmol/L LDL-C reduced, the RR of CHD is 0.79, 95% CI 0.77, 0.81” is very helpful metric.
I don’t claim to be an expert in meta-analysis but I’m not sure that Holmes is responding to the fundamental flaw in logic in the Lancet paper raised by Newman. The Lancet paper– correctly, as Newman acknowledges– identifies a benefit for statins in people who respond to statins, but it says nothing about the broader primary prevention population, some of whom will not respond to statins. The conclusion reached by the CTT authors, that statins should be more widely used in this primary population, is therefore unwarranted.
I am hopeful that Dr. Newman will respond in more detail and discuss the technical issues raised by Holmes.
Within any randomized group, some will do better than others. When targeting a lower LDL, those who achieve and maintain the target do better than those who do not. This phenomenon has been called dose targeting bias.
What is key to understand though is that an intervention may have NO benefit, yet this relationship of better LDL lowering to better outcomes can still be observed.
Therefore if the trials show no net benefit between statin vs placebo, it is not necessarily true that the subgroup analysis the Lancet group employs means some people benefited.
This is easiest to see by considering this point: If the statin-treated non-CAD patients had no overall reduction in mortality, and the subgroup with a large LDL drop did have lower mortality, then those without a large LDL drop must have been harmed by statins to account for the lack of overall effect compared to the placebo-treated patients.
If investigators really believe some benefit from statins accrues to those without apparent CAD provided LDL falls at least 40 pts, then the answer is to perform a trial where entry criteria include an ability to show a 40 pt drop in LDL, then randomize pts to statin vs no statin. That ain’t gonna happen because 1. few seriously believe that, and 2. this message is really a marketing message intended to promote use of statins in the general population.
“If the statin-treated non-CAD patients had no overall reduction in mortality, and the subgroup with a large LDL drop did have lower mortality, then those without a large LDL drop must have been harmed by statins to account for the lack of overall effect compared to the placebo-treated patients.”
I’m not sure that this is the case. It certainly would be if one had an infinite amount of data and the effect sizes were known precisely. But as a practical matter, if you mix enough patients with gram-(-) infections into a trial of a gram-(+)-specific antibiotic, you will not see a statistically significant effect. This does not mean the antibiotic caused harm to the gram-(-) patients, only that the effect was washed out by statistical noise.
In the CTT analysis the outcomes in the statin vs placebo groups are reported by per-cholesterol-reduction. As we know the placebo groups typically will not have any significant cholesterol reductions. Thus the statin groups will see greater cholesterol reductions than their placebo group counterparts, which is expected. But within the statin group the folks that have 1 mmol/L reductions, and 2, and 2.5, will be different people than the people who do not have these reductions. No way around that.
But the placebo folks are not subjects who are matched individually to the statin group subjects for characteristics such as diet, lifestyle, sociodemographics, exercise, or anything else. Thus by necessity the differences between those who dropped their LDL substantially and those who didn’t become confounders that destroy the equivalence created by randomizing to the statin and placebo groups. The comparison, instead of being ‘the average person taking statins’ compared to ‘the average person taking placebos’, becomes ‘statin-takers who dropped their cholesterol by 1 mmol/L’ to ‘the average person taking placebos’.
You can’t neutralize for this, even with “weighting each RCT by LDL reduction,” partly because of the unknown (lurking) confounders that can’t be adjusted for when setting the entire statin group to a major cholesterol reduction.
The utility of weighting trials according to LDL reduction is not that it maintains the equivalence of randomization, it is that it helps to neutralize the heterogeneity in the overall statin meta-group. The heterogeneity may theoretically be created by having different statins and different dosing regimens (and thus different cholesterol-reducing power), and weighting by LDL reduction is a way of trying to neutralize, or correct for, these differences.
Bottom line, an honest comparison would be statin group outcomes versus placebo group outcomes, regardless of cholesterol. If this showed no benefit (as it has repeatedly) then one should surrender that there is no benefit overall, and then ask this separate question: what if we examined the effect only among those with a major drop in cholesterol? The Lancet meta-analysis did this, and it is an interesting question, but it is not relevant to what people can expect if they decide to take a statin—unless they turn out one year down the line to be a person whose cholesterol dropped 1 mmol/L. This is NOT what guidelines address, but the authors’ pose their meta-analysis as a straightforward comparison of groups, and on this authority make conclusions about guidelines and low risk patients.
I strongly hesitate to insert myself in a debate about cardiology among cardiologists, but from the lay point of view, a few questions/comments.
First, I was mildly disappointed also to see the Cochrane study cited as supporting your position without mentioning that two of the four authors of the Cochrane study wrote an editorial accompanying the Lancet paper that was generally supportive of its conclusions. I realize this does not bear directly on the issue of the conclusions they reached from their own data analysis, but it seems pertinent, and it seems especially important to disclose all the facts when accusing others of being disingenuous.
The post also seems to imply that this was an article about patients who achieved a minimum LDL reduction of 40 mmol/L. My read of the paper was that this was proposed as the slope of a linear estimate.
I believe I’ve understood your point about the loss of randomization that occurs on “responder analysis”, but the approach does not seem intrinsically outrageous. LDL is an established biomarker of risk, and so it is intuitively reasonable that risk reduction should be a function of LDL lowering, with the greatest risk reduction being associated with the greatest LDL lowering. And we know that people on statins will generally see LDL lowering relative to the same patient treated with PBO. And we can measure LDL quite easily and not use the results of this study to put all patient on statins forever irrespective of response.
The more rigorous treatment of a binary division into “statin treated” and “not statin treated” cannot accomodate the fact that patients are not binary. Some are compliant, some respond, some are neither. Why would it be surprising in any way that looking for reduction in CV risk as a function of LDL lowering would be a more sensitive measure of benefit that averaging those who are compliant with those who are not?
Exciting that so many are interested. Feeling compelled to engage:
John – Good point about the author overlap. Was only two of the seven Cochrane review authors, and I figured it’s not really my business to guess at their perception of the validity of their own Cochrane review so I left it out. Certainly could have mentioned it, didn’t mean to obfuscate.
The fact that it’s intuitively reasonable to use cholesterol response as a reporting standard doesn’t make it scientifically sound. The answer to whether or not antihypertensives will be effective, and how effective they will be, in preventing MI and death is not to report analysis by drop in BP. Some people don’t drop at all, some people drop a great deal, and this can’t be predicted. Perhaps those who don’t drop are prevented from rising, and still see a benefit. Too many unknowables, and this is why this type of analysis is never used to determine who should take antihypertensives, although it is occasionally used to project benefit among those who have a certain drop.
Moreover, the cardiovascular effect of statins is famously unrelated to degree of cholesterol drop in virtually every examination (until this one), which is what led to theories about anti-inflammatory effect, vasodilatory effect, etc., all of which are proposed to help explain how and why they work—since cholesterol drop doesn’t seem to.
Finally, your point that patients are not binary (that there are many lurking variables we will never be able to account for) is EXCELLENT, and it is precisely the strongest argument for a simple comparison of outcomes between groups. Defining one group by their cholesterol drop muddies this completely, and destroys the equivalence of lurking variables that is created by randomization.
” LDL is an established biomarker of risk…”: if that is established by the usual sort of correlative study, then don’t we enter the world of circular argument here?
The conclusive argument is to ask whether, in suitably competent RCTs of defined populations, statins extend lives. It would seem that they don’t.
Thank you for your thoughtful response.
The studies I’ve seen suggest that only about half to two thirds of pts prescribed statins are still on therapy one year later. Is it possible that LDL lowering in the Lancet paper was largely a surrogate for compliance? Would blood draws provide meaningful data on this point, or is the liver extraction ratio of statins too high? The Truvada HIV prophylaxis pivotal trials used this sort of data to great effect.
With appropriate respect for the greater expertise of others, I think that more emphasis on subgroups might be more useful than continuing debate about the average effect in a large heterogeneous group. But I would appreciate your thoughts on this.
The entire debate seems bewildering to me. Essentially, as I understand it, the study recommends that pretty much everyone over 50 start taking statins. I do understand that some people have inherited problems with heart issues and absurdly high cholesterol and will benefit from statins. But statins are known to cause depletion of CoQ10 and in some cases muscle and/or memory problems; there are questions about the connection between cholesterol depletion and Alzheimer’s and Parkinson’s. Why in the world would any reputable doctor suggest to a healthy person that they take statins rather than exercising each day and eating lots of vegetables? And how crazy does a patient have to be to follow such a doctor’s orders?
Interesting to note that Steve Nissen has just signed on to a trial with very similar design.
http://www.forbes.com/sites/matthewherper/2012/06/07/orexigen-calls-dr-nissen/