As a major clinical trial in cardiology nears completion it has provoked a storm of criticism and controversy.
The brouhaha erupted in response to a late change to one of the most important— and already controversial— trials in cardiovascular medicine. The NIH-funded ISCHEMIA trial was designed back in 2011 to provide a definitive answer to the important and complicated question of whether stents provide something more than symptomatic relief to patients who have stable angina (i.e., who are not having a heart attack or other acute coronary event). The change to the trial, say the trial critics, means that the long and expensive trial won’t be able to deliver a definitive resolution to this question.
By way of background: it was long believed that squashing blockages in the coronary arteries would improve long term prognosis, but a decade ago the COURAGE trial found no clinical benefit for stenting over optimal medical therapy (OMT). COURAGE therefore appeared to confirm an emerging view at the time that heart attacks did not come from the big blockages seen on angiograms but instead were more likely to arise from any of a number of much smaller and widely prevalent blockages in the coronary arteries. COURAGE, in other words, appeared to suggest that stenting a stable blockage was nearly as futile as whack-a-mole.
COURAGE was enormously influential and helped change practice patterns in cardiology. Prior to COURAGE most stenting procedures were done in patients with stable angina. After COURAGE the balance shifted to acute procedures, in which the benefits were more certain, but many interventional cardiologists expressed concerns that COURAGE had not given stents a fair chance. The goal of ISCHEMIA was to resolve the lingering questions remaining after COURAGE. One important strength of ISCHEMIA touted by the trial leadership was the choice of a hard primary endpoint— cardiovascular death and MI (heart attacks)— that are felt to be less susceptible to manipulation or bias (whether conscious or unconscious) of “softer” endpoints like rehospitalization or revascularization.
But now a controversy has erupted over a recent, very late change in the design of ISCHEMIA. Last year, as the trial was ending its years-long effort to recruit patients to be in the trial, the ISCHEMIA investigators changed the primary endpoint of the trial to add the soft endpoints of hospitalization for unstable angina or heart failure to the hard endpoints of CV death and MI. The investigators did not make a public announcement of the change but in January of this year quietly updated the description of the trial on the official ClinicalTrial.Gov website.
The change was noticed by a group of UK investigators at Imperial College in London led by Darrel Francis (Imperial College, London). Their paper, with first author , was published last week in Circulation: Cardiovascular Quality and Outcomes and immediately provoked a firestorm. The paper contains a devastating criticism of the endpoint change. The article accuses the ISCHEMIA investigators of “moving the goalpost” and explains why, because ISCHEMIA is not a blinded trial, the new endpoints “will undermine the value of this well-designed study.” They recommend that “when the trial reports, we suggest that readers focus on the primary end point pre-specified, not any versions edited near trial end when the great majority of events will have already accumulated.”
Twitter Explosion
The online publication of the Francis article prompted an explosion on Twitter. One ISCHEMIA investigator, Sripal Bangalore (NYU), immediately dismissed the article for containing “factual errors,” but he declined to provide specific details. To date only one factual mistake has emerged. In response to the criticism Yves Rosenberg, the NIH project officer responsible for the trial, explained that the original January 2012 protocol allowed for and clearly described the process leading to the change in endpoint. In other words, the trial investigators had given permission back in 2012 to themselves to move the goalpost under prespecified conditions but they did not make this fact publicly known until now.
In many respects, then, the ISCHEMIA investigators have only themselves to blame for a big part of the controversy. If the investigators had issued a press release, or some other public explanation of the change, they might well have prevented the perception that they were trying to sneak the change in without anyone noticing.
It should be noted, however, that not one of the ISCHEMIA investigators or supporters has seriously argued that the change in the trial does not weaken the trial or that the original endpoint was not more desirable and reliable.
Rosenberg expressed “utter stupefaction and profound disappointment” with the Francis paper. Gregg Stone (Columbia University), one of the trial leaders, tweeted that the journal editors should not have published the “inflammatory/controversial” article without a rebuttal “unless the goal is to emulate the Inquirer.”
The ISCHEMIA investigators say that they plan to respond in detail to the Francis article. In an email Judith Hochman (NYU) declined to respond in detail to the article but pointed out that “the original dual endpoint of CV death and MI is a key secondary outcome that remains of great interest and that will be presented in reporting trial results.”
Ethan Weiss (UCSF), a cardiologist unconnected to the trial, said that “it is worth asking how and why the protocol was specified to be changed the way it was since the major mission of the trial was apparently to test the effect on hard endpoints. I would love to know (as a taxpayer) how the decision was made to do such a trial, market it as it was marketed, and then surprise the world with a last-minute change that greatly diminishes the value of the study without even a Friday afternoon press release. Put more simply, what will we learn from ISCHEMIA?”
The change in endpoint also calls into question the role of ClinicalTrials.Gov, the website that serves as the database of record for clinical trials. A major function of the site is keeping researchers honest by requiring them to prespecify the trial design and endpoints. When ISCHEMIA was first registered only the original hard endpoint was listed. The ISCHEMIA investigators have now posted the study protocols for the trial which include the information about the plan to change the endpoint if necessary, but there was no way for an outside observer to know about this plan. This appears to defeat the purpose of preregistration.
On his Twitter feed Francis responded to the criticisms with withering sarcasm. He quoted the trial investigators repeatedly citing the hard endpoints as a chief strength of the trial. He even posted a video taken from the ISCHEMIA trial website in which Stone repeatedly cited the hard endpoints as a key indicator of the trial’s strength and integrity. “As long as there’s no difference in hard outcomes, in death or myocardial infarction,” said Stone in the video, “I believe that revascularization for stable ischemic heart disease is going to be relegated to primarily patients who have failed an initial medical approach.”
The main reason for the change was the inability of the trial investigators to enroll as many patients as they had hoped. With their original goal of 8,000 patients they would have been able to accumulate enough of hard endpoints to achieve a robust finding. But because of difficulties in recruitment they were only able to enroll a little more than 5,000 patients and were therefore unable to accrue enough hard endpoints to achieve statistical or clinical significance. To maintain any hope of achieving a meaningful result they therefore adjusted the primary endpoint of the study to include the softer events.
The major obstacle to enrollment, as I wrote several years ago, is that cardiologists did not enroll sufficient number of their patients. Despite a widespread “official” consensus that the necessary equipoise for the trial existed, many interventional cardiologists remained convinced that stents conferred important benefits to their patients. They felt it was irresponsible or imprudent to withhold stents from patients with significant blockages.
This ambivalence of interventional cardiologists is best evidenced by Stone himself, since he is both a trial leader and perhaps the best known representative for the aggressive defense of interventional cardiology. Despite his vocal support for the trial in the video, Stone failed to lead by example. His own New York Presbyterian hospital, according to an article in the New York Times, had only enrolled 1 patient in the trial in its first years. NYU, by contrast, where Hochman and Bangalore are located, had enrolled 24 patients in the same period.
Important context to the debate over ISCHEMIA is the fierce debate that took place at the end of last year over Francis’s own ORBITA trial, which first raised many of the same issues now swirling in the ISCHEMIA debate. The much smaller ORBITA trial used sham procedures in the control group, thereby becoming the first true placebo-controlled trial of stents. Francis and colleagues were able to demonstrate that knowledge by either physician or patient about treatment assignment can have a huge impact on subsequent treatments, including the soft endpoints now included in ISCHEMIA. The implication, therefore, is that the results of ISCHEMIA, especially for the softer endpoints, may be unreliable since neither patients nor physicians are blinded to their treatment assignment.
I asked Brahmajee Nallamothu (University of Michigan), the editor-in-chief of Circulation: Cardiovascular Quality and Outcomes about the controversy, and in particular for his response to the ISCHEMIA investigators who said that it was irresponsible to publish the article. Here is his response:
Overall, I’m proud we published this piece. I continue to believe that the observations by Chris Rajkumar and his colleagues are quite novel and worthy of discussion. They are based on a careful read of the literature and raise fundamental issues about how the scientific community should assess endpoints in unblinded clinical trials – a topic that our journal cares deeply about and has published previously on.
As an editor, I see CQO – like many scientific journals – as a platform for debate in addition to being a vehicle for research dissemination. I am glad that the article is being discussed widely but we need to correct two things. First, we need to fix factual errors about the change in endpoints being post-specified rather than pre-specified. We are already working with the authors to do this after reviewing the ISCHEMIA trial protocol recently posted on clinicaltrials.gov following our official acceptance of the Rajkumar et al. article. I’m also excited that the ISCHEMIA investigators are preparing a detailed response, which I hope CQO can publish as well.
Second, in retrospect I can see the argument that the ISCHEMIA investigators should have been allowed an opportunity to respond to the Rajkumar et al. article at time of its original publication. Because ISCHEMIA is ongoing, there is concern that an asynchronous debate on this topic may undermine the trial. We had a vibrant discussion about this issue on our weekly editorial team call, and overall feelings across our editors were mixed. Ultimately, I take responsibility for these decisions but it was never my intent to be harmful to the conduct of the trial. I’m hopeful that allowing the ISCHEMIA investigators to respond in a sequential manner will not detract from this debate but only enrich it. Science takes the long-view.
Let me be clear: I recognize ISCHEMIA is a tough trial to perform. It addresses a critical question in our field, and Judith Hochman – the Study Chair of ISCHEMIA – is on my Mount Rushmore of clinical trial heroes. In the past, she figured out ways to complete large, ambitious, and highly relevant studies like SHOCK and OAT when no one else could. So I really admire her for putting together this amazing team for ISCHEMIA and tackling something that really matters. We published this with full appreciation of the challenge of doing studies like ISCHEMIA, including the effort of individuals and subjects as well as the public investment to date.
But that should not deter us from a debate. We live in a dichotomania world obsessed by up-or-down results from null-hypothesis significance testing and p-values. It is clear that the ISCHEMIA investigators were thoughtful and did not make their decision to change its primary endpoint lightly. But this is what makes Rajkumar et al.’s observations so important, regardless of whether this change was pre- or post-specified. I’m glad the ISCHEMIA investigators will be providing us with additional insights and also that we are having this debate now – even before its results are known.
Finally, I’m grateful that CQO can be a part of this debate.
See this update on the story:
Related Reading:
- Waiting For ISCHEMIA: Why Won’t Cardiologists Enroll Patients?
- What If PCI Is Just A Sham?
- ORBITA Trial Puts Interventional Cardiologists On The Defensive
- In Defense Of ORBITA
nice and detailed article, salute to whom who wrote it so deeply every point is covered
Do cardiologists realise that there may be laymen wondering whether the intellectual structure of their discipline is invalid, and that it maintains the appearance of being intact by behaviour that might easily be mistaken for corruption?
Clinical trials have always been a subject of intense debate and controversy but they are a necessary evil and must for something like cardiology where saving life is the primary objective.
As a relatively healthy candidate for a recommended PCI procedure with an awareness of how cardiologists reactted to theCourage Trial and now the contamination of this trial, I can only suspect findings aren’t turning out like they were expected. My conclusion is I will not have the PCI procedure.