Think about this: A new article reports that a major NIH-funded trial runs into trouble. The article raises all sorts of fundamental questions about our ability to perform meaningful clinical research. But instead of expressing concern about these legitimate problems, medical leaders ignore these questions and instead focus their ire and criticism on the article authors, saying it is improper to attack the trial without first checking with the trial leaders.
The ISCHEMIA trial was designed to answer fundamental questions about an important topic: the diagnosis and treatment of stable angina. As the trial was nearing completion last year the investigators changed the primary endpoint, which had been the relatively “hard” combination of cardiovascular death and MI. This hard endpoint had been a powerful selling point of the trial. But because ISCHEMIA had trouble recruiting patients— more on that later— they were only able to enroll 5,179 patients instead of the planned 8,000. Because of this change the investigators believed they would be unable to accumulate enough endpoint events to achieve a useful result. To salvage the trial they added the “soft” endpoints of hospitalization for unstable angina or heart failure to the hard endpoints of CV death and MI.
As I reported earlier, the change became public, and controversy ensued, with the publication of an article by in Circulation: Cardiovascular Quality and Outcomes. They accused the ISCHEMIA investigators of “moving the goalpost.” They pointed out that because patients and doctors in ISCHEMIA were aware of their treatment assignment their subsequent treatments— including things like the new hospitalization for unstable angina endpoint– were likely to have been influenced by that knowledge. This, they argued, meant that the trial would be much less likely to provide a definitive answer to an extremely relevant clinical problem.
The Francis paper was instantly attacked by the ISCHEMIA investigators and other members of the cardiology establishment. The NIH’s Yves Rosenberg expressed “utter stupefaction and profound disappointment” over “grave inaccuracies and mis-statements.” over the paper. One ISCHEMIA investigator, Sripal Bangalore (NYU), immediately dismissed the article for containing “factual errors,” but didn’t offer any details.
The attacks did not focus on the main points of the Francis paper. Instead the attacks focused exclusively on how the endpoint change was characterized in the Francis paper. “On January 17, 2018, over 99% of the way through the recruitment period, an amendment was made to the clinicaltrials.gov website,” the UK authors wrote.
“We must reflect on whether there is any meaning to the term primary end point if these goalposts can be moved at will after a trial has begun recruiting—and especially if a very prominent trial does so shortly before reporting its result,” the British authors wrote. It was this assertion, that the endpoint change had been made capriciously or irresponsibly, which the ISCHEMIA investigators and supporters objected to most.
In short order several important details about the change to ISCHEMIA emerged. The trial leaders publicly revealed for the first time that the endpoint change had been based on a plan outlined in the trial’s original protocol. An Independent Advisory Panel with access to aggregate outcome data, but not outcome data by treatment group, recommended the change, which was approved by the NHLBI in June of 2017. In January 2018 the ClinicalTrials.Gov website was updated to reflect the change. However, there was no public record or explanation for the change.
The ISCHEMIA supporters blame Francis et al for not fact checking their paper with them. “If we had been given the opportunity to vet the information about the trial included in the manuscript prior to the article’s publication, we might have prevented the dissemination of misinformation,” wrote ISCHEMIA trial leaders Judith Hochman (NYU) and David Maron (Stanford) in a published comment in response to the paper.
This of course is true, but it also represents an attempt to deflect responsibility and places an impossible burden on trial critics. In the interests of full transparency, the investigators— of a publicly funded trial, it should be emphasized– could and should have taken the proactive opportunity to announce and explain this major change in their trial. An explanation of the change on ClinicalTrials.Gov or a published statement or press release could have entirely prevented this mixup.
One typical response to the Francis article came from Ajay Kirtane (Columbia University). He expressed grave concern about the publication of the Francis letter, saying that “damage has already been done” and compared its publication, astonishingly, to “current day political affairs.” “We do need to show that we are better than this as a profession going forward,” he tweeted.
He echoed his Columbia colleague Gregg Stone(Columbia University), one of the trial leaders, who tweeted that the journal editors should not have published the “inflammatory/controversial” article without a rebuttal “unless the goal is to emulate the Inquirer.”
Kirtane and Stone were gravely disturbed by the perceived absence of respect paid to the ISCHEMIA leaders but they expressed no concern whatsoever about the larger questions raised by the fact that a major $84 million NIH trial had been weakened or compromised.
The entire episode can be seen as a perfect example of the establishment circling the wagons in a desperate attempt to defend itself from the threat of an attack from those perceived to challenge the status quo.
Francis fired back on Twitter with a devastating response, first noting that for all Kirtane’s and Stone’s insistence on paying due respect to the trial, their home institution, Columbia University, had only enrolled 3 patients in the trial. Kirtane and Stone, in other words, paid far more respect to ISCHEMIA through their words than through their actions.
Francis then went on to enumerate instances in which the trial investigators publicly cited the original hard endpoints as a chief strength of the trial. He even posted a video taken from the ISCHEMIA trial website in which Stone repeatedly cited the hard endpoints as a key indicator of the trial’s strength and integrity. (The video has now been removed without comment or explanation from the ISCHEMIA website.) Here are Stone’s own words in the video:
…as long as there is no difference in hard outcomes, death or myocardial infarction, I believe that revascularization for stable ischemic heart disease is going to be relegated to primarily patients who have failed an initial medical approach.
Francis also blasted critics demanding an immediate correction by pointing to the absence of any similar demand to the many publications from the ISCHEMIA leadership that highlighted the original primary endpoint as a key feature of the trial— without any mention of the alternative plan to expand the endpoint.
One thing needs to be clear: the attacks on the Francis paper deflect attention from the importance and significance of the change in primary endpoint of ISCHEMIA, a change which will significantly impact our perception of the trial and its ability to provide meaningful information. I don’t want to suggest that the trial will now be meaningless, but this change does significantly diminish the trial’s importance. And yet the investigators and defenders of the trial have chosen not to address or explain this key point, though the change was made back in June 2017.
The ISCHEMIA investigators have also chosen not to address a host of other important questions raised by this episode, including questions about transparency, preregistration and prespecification, and the role of ClinicalTrials.Gov.
The controversy should also bring attention to another little discussed but glaring problem, which is the commitment of the cardiology community to support research that raises legitimate questions about common clinical practices. This is, in the current lingo, an existential problem for all of cardiology, not just interventional cardiology, and yet there is very little public discussion about it. As this episode demonstrates, the leaders of cardiology are far more inclined to question the propriety of junior authors criticizing established investigators than they are to address fundamental questions about their profession.
Along these lines, it should be acknowledged that it has long been apparent to nearly everyone in the field that ISCHEMIA would have trouble enrolling patients. I wrote about this problem back in 2009, before the trial even started, and then again in 2015. These articles expose only the tip of a much bigger iceberg.
Regarding this part of the story the ISCHEMIA investigators can be seen as both sympathetic victims and guilty assailants. Hochman and others worked hard, using videos and social media and speaking engagements, to persuade the cardiology community to enroll patients in the trial. On the other hand, they were obviously unable to respond effectively to the resistance they encountered.
No where is this tension within the cardiology community more apparent than the fact, mentioned earlier, that Columbia University, the home institution of co-principal investigator Gregg Stone, only enrolled 3 patients in the trial. In their responses to the Francis paper the ISCHEMIA investigators portray themselves as victims. I am reminded of the old story about the defendant convicted of murdering his parents who then throws himself on the mercy of the court on the grounds that he is an orphan.
Related Reading:
- Cardiology World Erupts Into Controversy Over Change In Major Clinical Trial
- Waiting For ISCHEMIA: Why Won’t Cardiologists Enroll Patients?
- What If PCI Is Just A Sham?
- ORBITA Trial Puts Interventional Cardiologists On The Defensive
- In Defense Of ORBITA
We may, in fact, learn quite a bit from the ISCHEMIA trial.
For example, if they do get a statistically significant difference in hard outcomes—CVD deaths and/or myocardial infarctions—then elective revascularizations are saved. OK, because of the enrollment shortfall, if they even get a trend at p=10%, then some revascularizations in stable ischemic heart disease should at least get new life.
However—if it takes the addition of unblinded hospitalizations for unstable angina or heart failures in order to get a significant result, then that tells us two things:
One, we’ve been selling snake oil (and making a lot of money doing it), and
Two, that in all CVD trials henceforth we need to limit our analyses to MIs and mortality (—and to reconsider all past trials where unblinded revascularizations played a big role in MACE “results”).
Both, of course, are extremely important teachings.
(Since the Independent Advisory Panel that recommended the change in the primary endpoint had access to the aggregate outcome data so far, we can hypothesize which of these cases the results are likely to be…)
It will be interesting to see the “spin” in how the results are written up!
“Integrity Time,” guys and gals.
Larry, you are a national resource.
After your first article, I went and read the Rajkumar & Francis et al. piece. Seemed awfully reasonable (and respectful) to me. I urge everyone to read their arguments. http://circoutcomes.ahajournals.org/content/11/3/e004665?cpetoc=
(I am still waiting for an enumeration of their “grave inaccuracies and mis-statements” and “factual errors.” Still no list? And how germane are they?)
And a referenced video now removed from the ISCHEMIA website? Seems like an admission of guilt to me.
I agree that this looks an awful lot like “the establishment circling the wagons in a desperate attempt to defend itself from the threat of an attack from those perceived to challenge the status quo.”
And that “the leaders of cardiology are far more inclined to question the propriety of junior authors criticizing established investigators than they are to address fundamental questions about their profession.”
Ouch. But evidently true.
Some people might need to take a hard look in the mirror…
I echo your compliment to Mr Husten.
In your link, for which many thanks, I find “When the trial reports, we suggest that readers focus on the primary end point of cardiovascular death and nonfatal MI that was originally pre-specified”. Fair enough, but in my non-expert way I’d also look at total mortality.
P.S. Kudos to the editors of Circulation: Cardiovascular Quality and Outcomes for their decisions.
Agree completely with support for editors of CQO for publishing Francis paper. Thank you and keep up the good work. We need integrity in cardiology and a return to hard endpoints instead of hanging our hats on soft, fluffy, MACE. Otherwise we’re simply promoting the interests of corporate America instead of speaking on our patients’ behalf.
It’s been a month. If you are unwell, Mr Husten, best wishes and get well soon.
I am fine! Just taking a long needed break/sabbatical and cooking up more trouble for the future. Thanks for asking.