Clot Extraction For Stroke: Beware The Hype

Clot extraction (thrombectomy) for stroke has been on a roll lately. After the failure of a series of trials a few years ago investigators and industry went back to the drawing board. The result of their efforts– a new and improved thrombectomy– has brought about a dramatic reversal of fortune for the technology. In the past year five newer trials have been stopped due to highly positive findings. Now a new meta-analysis published in JAMA confirms some of the good news, but it also offers reasons for caution.

Rory Spiegel, an emergency medicine resident who writes the EM Nerd blog, has been writing about the recent trials with terrific depth and insight fueled by a healthy dose of skepticism. I asked him for his perspective on the new meta-analysis. I’ve reprinted his meticulous and thoughtful analysis below. It’s worth reading in full, but here are a few of the key points.

Spiegel doesn’t argue that the recent thrombectomy trials are not valid or important. But he does argue powerfully that the broad application of the technology in clinical practice is not warranted by the data, which has significant limitations. In particular, Spiegel argues that there is no reason to believe that the current evidence supports a major national effort to bring thrombectomy to a large proportion of the population.

Spiegel says that the evidence is far too weak to provide a reliable estimate of the real effects of thrombectomy, so “it is difficult to assess whether this benefit justifies the resources required to support its implementation on a national level.” Despite an impressive improvement in functional outcome at 90 days in the meta-analysis, Spiegel points to important limitations of this data:

• higher revascularization rate with thrombectomy
• no difference in 90 day mortality of symptomatic intracranial hemorrhage
• no difference according to time-to-treatment, “once again calling into question the time is brain mantra so frequently proclaimed.”
• a likely exaggerated effect due to stopping the trials early

Most importantly, Spiegel points to substantial heterogeneity in the trials, severely limiting the value of a meta-analysis. “What we have is a heterogeneous data set, ultimately providing more questions than answers,” he concludes.

Here is Rory Spiegel’s complete analysis:

Meta-analyses function under the assumption that the summation of data from multiple sources is a more accurate estimate of the true effect size than any one individual trial. And yet sometimes such statistical endeavors serve only to add dirt to the already muddy water. Such is the case with the recent trials examining endovascular therapy for acute ischemic stroke. Prior to the publication of MR CLEAN and its’ band of statistically significant misfits, the data regarding endovascular therapy has been consistently negative. Over the past year five RCTs examining endovascular therapy for acute ischemic stroke have been published. In direct contrast to the three trials published in 2013, all of the recent trials were impressively positive (1,2,3,4,5). So much so that the AHA, in their recently updated guidelines, recommended the use of endovascular therapy in a population far broader than any of the studies examined. Even more concerning was their whole-hearted support of the development of a regionalized system capable of instituting the use of endovascular therapy at a national level (6). Stating:

Regional systems of stroke care should be developed. These should consist of:

 (a) Healthcare facilities that provide initial emergency care including administration of intravenous r-tPA, including primary stroke centers, comprehensive stroke centers, and other facilities.

(b) Centers capable of performing endovascular stroke treatment with comprehensive periprocedural care, including comprehensive stroke centers and other healthcare facilities, to which rapid transport can be arranged when appropriate (Class I; Level of Evidence A). (Revised from the 2013 guideline)

They go on to say:

It may be useful for primary stroke centers and other healthcare facilities that provide initial emergency care including administration of intravenous r-tPA to develop the capability of performing emergency noninvasive intracranial vascular imaging to most appropriately select patients for transfer for endovascular intervention and reduce time to endovascular treatment (Class IIb; Level of Evidence C). (Revised from the 2013 guideline)

Even the biggest cynics must concede there is a signal of benefit demonstrated throughout the recent trials examining endovascular therapy for acute ischemic stroke. How much of this is due to the true effect of the treatment in question, and how much is in fact due to statistical noise is far more difficult to discern. Due to a lack of both blinding and equipoise of the trialists and the premature stoppage of all four trials

following MR CLEAN, the data is likely to be a distortion of reality (7). And yet without a clear understanding of this effect size, it is difficult to assess whether this benefit justifies the resources required to support its implementation on a national level.

In an attempt to clarify this benefit, Badhiwala conducted a systematic review and meta-analysis of RCTs examining the efficacy of endovascular treatment for acute ischemic stroke (8). Published in JAMA, the authors included the five most recent positive trials, as well as the three negative trials published in the NEJM in February of 2013. The primary outcome was the frequently used and widely misunderstood ordinal analysis of functional status (mRS) at 90-days. The authors also examined a dichotomous outcome of the percent of patients alive and independent (mRS of 0-2) at 90 days, rate of revascularization at 24-hrs, symptomatic intracranial hemorrhage, and all cause 90-day mortality (8).

In their pooled analysis, the authors found a shift towards improved functional outcomes at 90-days in the patients randomized to receive endovascular therapy when compared to standard care. This difference demonstrated an odds ratio of 1.56 (1.14–2.13 p = 0.005), which translates into a 12% absolute increase in the amount of a patients alive and independent at 90-days (3.8%-20.3%; p = 0.005). Not surprisingly, patients randomized to the endovascular arm had a significantly higher rate of revascularization at 24-hours when compared to the patients in the standard care group (75.8% vs. 34.1%). No difference was observed in the 90-day mortality (15.8% vs. 17.8%) or the rate of symptomatic intracranial hemorrhage (5.7% vs. 5.1%) (8).

An interesting side note, one of the subgroups the authors examined in their secondary analysis was whether time to randomization had any effect on the efficacy of endovascular therapy. Specifically they looked at time from symptom onset to randomization. They examined the effect size of endovascular therapy as compared to standard care depending on whether patients were randomized before or after three hours from symptom onset. Temporality did not seem to affect outcomes (8). Once again calling into question the time is brain mantra so frequently proclaimed.

Although these results are positive, they are a far cry from the astonishing outcomes reported in the four trials released in the wake of the publication of MR CLEAN. Each of these trials was stopped prematurely because of an unplanned interim analysis provoked by the positive results found in MR CLEAN. And though it is unlikely that four trials would all demonstrate positive results due to random chance, it is well known that trials stopped early for benefit will demonstrate significantly more optimistic results than the intervention’s true effect size (6). In fact, the Badhiwala meta-analysis demonstrates a clear inverse association with the magnitude of benefit and the size of the sample at termination (8). And so the question becomes is it appropriate to pool this data at all?

To answer that question one has to ask, what is the goal of a meta-analysis such as this in the first place? The assumed benefit to performing a meta-analysis is that the summation of these data sets provides a more accurate description of the true effect size than each individual data set can provide. This supposition rests on the notion that all the studies included in the meta-analysis are examining the same study population, and that the variance of results is due to random errors in sampling. This is what is known as a “fixed effect” model. Unfortunately most data is not so homogeneous, and it is common for the variation observed between trials to be due to more than just random error, but to considerable differences in the populations being compared. In such cases, the results of a direct-pooled analysis will likely deviate from reality. Statistical models that attempt to account for these random deviations should be utilized. These are known as “random-effect” models (9).

In the Badhiwala et al meta-analysis the heterogeneity of the results between the trials examined was high and the authors correctly utilized a random-effect model. The authors used the I² index to assess the degree of variation between studies. I² describes the extent of variation across trials that cannot be explained by random chance. An I² score of 0.0 implies all of the variation observed between trials can be accounted for by random errors in sampling. Conversely if the I² is 75, only 25% of the variation can be accounted for by sampling error with the remaining variation (75%) due to heterogeneity between trials (10). In the Badhiwala et al meta-analysis the I² = 75.4, confirming that the authors are likely attempting to pool trials with very different populations. Whether this variation is due to methodological differences or the bias that accompanies premature stoppage is unclear. Applying a statistical value to this uncertainty does not legitimize it.

The question remains, who exactly should receive endovascular therapy for acute ischemic stroke? Unfortunately this analysis has not gotten us any closer to identifying the ideal patient. It does highlight that patient selection is likely the most important criteria in determining efficacy. Based on the most recent trials examining endovascular therapy for acute ischemic stroke, patients who demonstrate a small infarcted core with a large surrounding penumbra on perfusion imaging would be the model candidates for endovascular therapy. Until larger more methodologically rigorous data sets are published this is just an unproven hypothesis.

  Despite its methodologic rigor, Badhiwala et al’s meta-analysis brings us no closer to certitude. It serves to place an objective number on the current ambiguous state of the data concerning endovascular therapy for acute ischemic stroke. But the inherent value of its statistical manipulations in a pooled data set is unclear. This analysis provides little utility over our unstructured judgment of each respective trial’s importance, while validating our suspicion that these trials are examining very different populations. I am sure in the coming months and years this study will be cited in vain. Used to champion the fortification of endovascular therapy’s place in the ivory tower of medicine. What will be remembered are odds ratios and p-values, while the true meaning will be forgotten. That what we have is a heterogeneous data set, ultimately providing more questions than answers.

Sources Cited:

1. Berkhemer OA, Fransen PS, Beumer D, et al. A randomized trial of intraarterial treatment for acute ischemic stroke. N Engl J Med. 2015;372:(1)11-20.
2. Campbell BC, Mitchell PJ, Kleinig TJ, et al. Endovascular Therapy for Ischemic Stroke with Perfusion-Imaging Selection. N Engl J Med. 2015.
3. Goyal M, Demchuk AM, Menon BK, et al. Randomized Assessment of Rapid Endovascular Treatment of Ischemic Stroke. N Engl J Med. 2015.
4. Saver JL, Goyal M, Bonafe A, et al. Stent-Retriever Thrombectomy after Intravenous t-PA vs. t-PA Alone in Stroke. N Engl J Med. 2015
5. Jovin TG, Chamorro A, Cobo E, et al. Thrombectomy within 8 Hours after Symptom Onset in Ischemic Stroke. N Engl J Med. 2015;
6. Powers WJ, Derdeyn CP, Biller J, et al. 2015 American Heart Association/American Stroke Association Focused Update of the 2013 Guidelines for the Early Management of Patients With Acute Ischemic Stroke Regarding Endovascular Treatment: A Guideline for Healthcare Professionals From the American Heart Association/American Stroke Association. Stroke. 2015;46(10):3020-35.
7. Bassler D, Briel M, Montori VM, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010;303(12):1180-7.
8. Badhiwala, JH et al. Endovascular Thrombectomy for Acute Ischemic Stroke A Meta-analysisJAMA. 2015;314(17):1832-1843.
9. Cornell JE, Mulrow CD, Localio R, et al. Random-effects meta-analysis of inconsistent effects: a time for change. Ann Intern Med. 2014;160(4):267-70.
10. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-60.