More Blasts Of Concern Over ROCKET-AF

The controversy over the big Xarelto trial does not appear to be going away.

A new investigation published in the BMJ raises more troubling questions about the ROCKET-AF trial, which compared the novel oral anticoagulant rivaroxaban (Xarelto, Johnson & Johnson) to warfarin in patients with atrial fibrillation. The controversy about the trial first unfolded last fall when it became known that the portable Alere devices used to monitor and calibrate warfarin usage in the trial were seriously defective. Since then investigators, industry, regulators, and critics have been trying to assess whether the defective devices compromised the findings of the trial.

The new development in the story is that shortly after the trial started J&J employees became aware that questions were being raised about the monitoring device in a small number of cases. The company launched a program to investigate these concerns but, according to BMJ associate editor Deborah Cohen, failed to inform either regulators or the trial’s data and safety monitoring board (DSMB) about the findings of the program.

The BMJ reports that 149 samples were sent to the company by trial investigators who had questions about the accuracy and reliability of the INR readings. 71 of the samples were from patients randomized to warfarin. (Fake INR readings were generated for patients who were randomized to rivaroxaban.) According to the company there were “16 instances where the value from the point of care device and lab were… inconsistent.” Thus in this small, nonrepresentative sample nearly one quarter of the portable readings used in the trial did not match standard laboratory readings.

The BMJ quotes a member of the trial’s DSMB, Peter Rothwell (Oxford University), who said he had “no memory of the board being told” about the findings. “Clearly, if the sponsor of the trial had concerns about the validity of the point-of-care testing of INR that would have been important for the board to be made aware of [them],” he told the BMJ. Neither did the company inform either the FDA or the EMA about the findings.

One ROCKET-AF investigator, Markku Kaste (Helskini University General Hospital) agreed that the findings should have been shared. “I now have some doubts about the validity of ROCKET AF trial. It possibly skews the data in favour of rivaroxaban,” he said.

Defending ROCKET-AF

J&J responded that the company behaved properly because there was no good reason for it to report this data. I spoke with Pete DiBattiste, the head of development for cardiovascular drugs Janssen (the pharmaceutical arm of J&J), who told me that the company “took great care and are highly confident that we submitted all relevant information not only to the FDA but to all health authorities around the globe.” He pointed out that the 149 samples submitted by investigators to the program represented less than one-twentieth of 1% of the approximately 366,000 INR measurements obtained in the trial.

The small number of samples sent by investigators “is itself a sign of reassurance,” said DiBattiste. The program “was put in place as a safety check” but “ended up yielding no meaningful issue” and so there was no reason to report the findings to either the DSMB or regulators, he said. He also pointed out that a certain amount of discordance would be expected, and that the FDA standard for approval of devices does not require perfect accordance.

J&J has also defended ROCKET-AF on the basis of analyses performed by Janssen and its European partner, Bayer, as well as separate analyses performed by the ROCKET-AF investigators. But these analyses have also been subject to criticism. In her article Cohen cites numerous questions that have been raised about these analyses.

“We need to find ways to reduce uncertainty and increase clarity about the balance of benefit to harm,” write Kamal Mahtani and Carl Heneghan in an accompanying BMJ editorial. Ideally they would like to see independent trials to replicate the results but they acknowledge that “this may take several years. In the mean time, making the data available for independent scrutiny should be a mandatory regulatory requirement, particularly when there are questions about trial rigour.” For now, however, “patients and clinicians must… live with the uncertainty left by the evidence currently available.”

In a second accompanying commentary, Vinay Rathi, Harlan Krumholz, and Joseph Ross discuss the larger regulatory issues involving diagnostic devices raised by this case. They note that diagnostic devices “rarely require prospective clinical studies for clearance” and that devices that have been recalled can be used as predicates for the approval of new devices. Even the FDA acknowledges that “there are few performance standards… based on clearly defined scientific parameters.”

Perspective From Sanjay Kaul

Sanjay Kaul (Cedars Sinai) was a member of the FDA panel that reviewed ROCKET-AF He told me that he thinks Janssen should have disclosed the data to either the DSMB or the regulators. “It was a mistake that continues to fuel the controversy and perpetuate the perception of malfeasance.” Kaul does not find strong evidence in the available data suggesting that the ROCKET-AF results are invalid, but he does support “an reassessment of benefit-risk by independent academics” to perhaps “dispel any residual doubt or adjudicate remaining uncertainties.”

Here is Kaul’s full statement:

“The saga continues!

“I find it unacceptable that Janssen did not disclose the results of the Covance recheck program to the DSMB or the regulators. It was a mistake that continues to fuel the controversy and perpetuate the perception of malfeasance.

“I am all for data transparency and independent replication as suggested by the editorialists. However, the key question here is whether there is sufficient evidence that the benefit-risk balance of rivaroxaban was materially impacted by the faulty INR readings to revoke the approved indication. The EMA conducted an ‘independent’ investigation and concluded the benefit-risk balance remained unchanged. The FDA is also conducting its own investigation and so far has not made any major pronouncements. I don’t want to second guess the FDA, but if there was an obvious ‘fatal flaw’ to render the trial data invalid, they would have acted sooner in the interest of public safety.

“Here is my own personal perspective.

Of the 767 values where lab INR>4, 172 (22%) device INR values were between 2 and 3 and 47 (6%) device INR values were less than 2. A device INR of <2 (a 2-INR category discordance) is likely to lead to dose adjustment (escalation). It is important to keep in mind that dose adjustments are seldom based on single INR values. Please note that the difference in bleeding in those with vs without discrepancy is 1.5/100 patient years to 2.0/100 patient years. This is a marginal difference in a relatively small subset (6% of cohort) which is not likely to result in a MATERIAL difference in the benefit-risk balance. Even if this discrepancy was observed in 100% of the cohort (worst-case scenario), I am not convinced this would result in an undesirable benefit-risk balance. It is reassuring to learn that in data sets external to ROCKET-AF trial (where the defective INR device was not utilized), warfarin-related bleeding rates were consistent with those observed in ROCKET-AF.

“What compounds the issue is that dose adjustment resulting from a clinically meaningful discrepancy in INR values would have expected to result in lower efficacy event rates (stroke and systemic embolism) and higher safety event rates (bleeding) with warfarin. While observed bleeding rates were aligned with the expected rates, observed stroke rates were the opposite of expected (higher rates). In addition, why would rivaroxaban (fixed doses were used) bleeding outcomes also be affected in a directionally similar manner as with warfarin by INR discrepancy but not stroke? These data appear to suggest that observed differences in warfarin event rates cannot be primarily attributed to the adjustments made in warfarin dose based on discrepancy-related device INRs.

“Bottom line, short of conducting a new trial, perhaps, an reassessment of benefit-risk by independent academics might be able to dispel any residual doubt or adjudicate remaining uncertainties amongst the skeptics.”

Previous coverage:

• Alere Withdraws Anticoagulation Monitoring System At Heart Of ROCKET AF Controversy (July 13, 2016)
• New Studies Seek To Allay Concerns Over The ROCKET-AF Trial (February 3, 2016)
• ROCKET-AF Investigators Seek To Calm Concerns About Trial Reliability (December 13, 2015)
• New Questions Raised About ROCKET AF, Pivotal Xarelto Trial Chaired By Califf (November 30, 2015)