How I Interpreted the Randomised Trials of Carotid Angioplasty/stenting versus Endarterectomy☆

Systematic reviews and meta-analyses 

It is appropriate to begin by reviewing the totality of evidence from all randomised controlled trials. This requires a systematic review of all trials and a meta-analysis of data from all trials.

A systematic review uses explicit scientific strategies to a) identify and include all trials that are relevant to the research question, b) reduce any bias (e.g. in trial selection and data extraction) in the estimate of the direction of the treatment effect (of carotid stenting vs endarterectomy) from using only selected trials, and c) increase the precision of the estimate of the treatment effect by examining a larger amount of data and thereby reducing random error.7

A meta-analysis is commonly performed using the inverse variance method in which the weight given to each study is the inverse of the variance of the effect estimate (i.e. one divided by the square of its standard error). Thus larger studies, which have smaller standard errors, are given more weight than smaller studies, which have larger standard errors. This choice of weight minimizes the imprecision (uncertainty) of the pooled effect estimate.7 A fixed-effect meta-analysis makes the assumption that the true effect of the intervention (e.g. CAS vs CEA), in both direction and magnitude, is the same value in every study (that is, it is fixed across studies). This assumption implies that any observed differences among study results are due solely to the play of chance, i.e. that there is no statistical heterogeneity.

However, observed differences among study results may also reflect heterogeneity resulting from clinical and/or methodological diversity among the trials. Heterogeneity is assessed using a Cochran Q statistic, and P-value obtained by comparing the Q statistic with a χ2 distribution with k−1 degrees of freedom, where k is the number of studies. Heterogeneity is also assessed using the I2 statistic, which describes the percentage of variation across studies that is due to heterogeneity (rather than chance) and is calculated as 100%×(Q−df)/Q where Q is the Cochran Q statistic.7, 8 I2 is an intuitive and simple expression of the inconsistency of studies' results because, unlike Q, it does not inherently depend on the number of studies considered.

If there is heterogeneity and this cannot be readily explained, one approach is to incorporate the data into a random-effects meta-analysis model. This involves the assumption that the effects being estimated in the different studies are not identical (i.e. fixed), but follow some sort of distribution. The random effects model represents our lack of knowledge about why the observed effects of the intervention differ by considering the differences as if they were random. The centre of this distribution describes the average of the effects, while its width describes the degree of heterogeneity. The conventional choice of distribution is a normal distribution. As it is difficult to establish the validity of any distributional assumption, random-effects meta-analyses are open to criticism.

If the variation in the results of different studies (heterogeneity) is due to biases associated with methodological diversity, the random-effects pooled estimate will only estimate the average treatment effect provided the biases are symmetrically distributed, leading to a mixture of over- and under-estimates of effect. However, this is unlikely to be the case. If the variation in effects (heterogeneity) is due to clinical diversity, the fixed-effect pooled estimate should be interpreted differently from the random-effects estimate since it relates to a completely different question. The fixed-effect estimate and its confidence interval addresses the question “what is the best estimate of the intervention effect?” The random-effects estimate and its confidence interval address the question “what is the average intervention effect?” The answers to these questions coincide either when no heterogeneity is present, or when the distribution of the intervention effects is roughly symmetrical. When the answers do not coincide, the random-effects estimate may not reflect the actual effect in any particular population being studied.

In this paper I will examine a recently updated Cochrane systematic review and meta-analysis of all randomised trials comparing CAS with CEA.9 This will provide an estimate of the overall treatment effect of stenting vs endarterectomy as well as an analysis of whether there is consistency or heterogeneity of the treatment effect among the different trials. If there is consistency, the next question to be addressed is whether there is sufficient statistical power to perform hypothesis-generating subgroup analyses, acknowledging the potential hazards of this type of analysis.7 However, if there is highly significant heterogeneity, the validity of combining the data (e.g. for a meta-analysis) may be doubtful. The validity of the systematic review and meta-analysis will also be limited if there is any bias. Examples of this include publication bias [studies which have concluded a “positive” or interesting result are more likely to be published, and therefore easier to locate, than studies which have produced a “negative” or neutral result], study quality bias [more methodologically robust trials tend to indicate that new treatments are less effective than do less reliable trials] and outcome recording bias [where there is a tendency for some trials to publish the results of their most impressive outcomes and not report the results of their least impressive outcomes]).7

Cochrane systematic review and meta-analysis of all available data from randomised trials 

The recent Cochrane systematic review9 identified 12 randomised trials that compared the safety and efficacy of endovascular treatment of carotid stenosis by means of percutaneous transluminal carotid angioplasty and/or stenting with CEA in a total of 3,227 patients.10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 Five trials, involving 2,286 patients, were stopped early.11, 12, 13, 14, 15

Five clinically relevant questions will be addressed (Table 3):

Table 3. Questions addressed in the trials of CAS vs CEA

	1.Is CAS associated with a lower risk of periprocedural death or stroke CEA? Preliminary evidence suggests that CAS and CEA incur a similar perioperative mortality rate (1.2%), but that CAS may have about a 40% (5 to 84%) higher relative risk and 2% higher absolute risk of 30-day stroke compared with CEA. 2.Does CAS incur fewer local periprocedural complications than CEA? CAS significantly reduces the relative risk of cranial nerve injury by 90% and by about 7% in absolute terms (7.2% endarterectomy vs 0.4% angioplasty/stent). 3.Is CAS as effective as CEA in reducing long-term stroke? Limited long-term follow-up data suggest that there was no significant difference between the treatments in preventing stroke or death, but the confidence intervals are wide and there is significant heterogeneity among the trials. 4.Is CAS with a cerebral protection device more effective than CAS without cerebral protection? Preliminary case series have suggested that protection devices do improve the safety of stenting, but there are no reliable data from trials in which patients were randomly assigned to the use of cerebral protection devices or not. 5.Is CAS as effective as CEA in reducing the 30-day risk of death/stroke in neurologically asymptomatic carotid stenosis? There are insufficient data from trials in which patients with neurologically asymptomatic carotid stenosis were randomly assigned to CAS vs CEA.

Eight trials involving 2,915 patients with carotid stenosis (about 90% symptomatic) reported the major outcome of “any stroke or death within 30 days of the procedure”.10, 11, 12, 13, 14, 15, 16, 17 A meta-analysis of the available data showed no significant heterogeneity among the eight trials (χ2 11.81, P=0.11, I2=41%) and a significant excess risk of death/stroke among patients randomly assigned CAS, compared with CEA (122/1464 [8.3%] stenting vs 89/1451 [6.1%] endarterectomy; fixed-effect odds ratio [OR]: 1.39, 95% confidence interval [CI]: 1.05 to 1.84, P=0.02).

This result may be viewed as a typical intervention effect from the studies included in the analysis and suggests a significant increase in the odds of procedural stroke or death among patients treated with CAS compared with CEA. However, the estimate of excess risk (about 39%) is imprecise and varies, with 95% confidence limits from as low as 5% to as high as 84%. When the random-effects model was used, the pooled estimate and confidence interval of the odds ratio for 30-day death/stroke among patients assigned to CAS compared with CEA was similar (OR: 1.44). However, the 95% confidence intervals were now even wider (95% CI: 0.91 to 2.26) and overlapped with an OR=1, indicating that the result was not statistically significant at the p=0.05 level.

It is important to note that the increased risk of 30-day death/stroke after CAS was not driven by an excess risk of death. Among the seven trials which recorded death within 30 days of the procedure10, 11, 12, 14, 15, 16, 17 (2,683 patients), there was no significant heterogeneity among the trials (p=0.69; I2=0%) and no significant difference between treatment groups (30-day death 16/1352 [1.2%] following stenting vs 16/1331 [1.2%] after endarterectomy). This corresponds to a fixed-effect OR 0.99 (95% CI: 0.50–1.97, P=0.98) and a random-effects OR 1.00 (95% CI 0.49–2.04, P=0.99). The identical results achieved by both the fixed-effect and random-effects methods (albeit with wide 95% confidence intervals and thus substantial imprecision of the estimates) suggest that there was no heterogeneity among the various studies for procedural death. Accordingly, the observed increase in procedural risk after CAS was driven by an increase in the odds of stroke (97/1357 [7.1%] after stenting vs 70/1341 [5.2%] after endarterectomy. Fixed-effect OR=1.40 (95% CI 1.02–1.91, P=0.04). There was substantial heterogeneity among the seven trials (χ2 10.65, P=0.06; I2=53%). The results from the individual larger trials varied from favouring stenting (SAPPHIRE14), equivalence (CAVATAS16 and SPACE15) to favouring endarterectomy (EVA-3S).11 However, the cause of the heterogeneity among the trials could not be adequately determined because insufficient individual patient data were available for the systematic review and meta-analysis, and there were too few trials to justify further subgroup analyses.

At first sight, it seems intuitive that the use of less sophisticated angioplasty/stenting techniques by inexperienced operators in trials comparing CAS with CEA might explain the higher rates of procedural stroke associated with angioplasty/stenting in the Leicester trial (OR 21.2; 95% CI: 1.01 to 445)12 and the EVA-3S trial (OR 3.5, 95% CI: 1.5 to 8.2),11 and therefore account for a substantial portion of the heterogeneity of treatment effects seen in the meta-analysis. However, there is no actual evidence to support this.9

First, a plot showing the stroke and death rate for each of the trials in the systematic review over time revealed no improvement in the 30-day event rate from trial to trial with time.9 Second, an extensive analysis of the various explanations for the higher risk of stroke among patients assigned to CAS in the EVA-3S trial (8.7%) failed to reveal any clear explanation regarding experience of the operators, baseline patient characteristics, centre enrolment or other obvious factors other than the intrinsic nature of the procedure. Paradoxically, the results for individual investigators in the EVA-3S study showed an inverse learning curve (ie higher rates of procedural stroke/death among more experienced operators).22 Interestingly, the main finding (for many) from the EVA-3S trial was that the risk of stroke among patients assigned to CEA was unusually low compared with outcomes from all of the other trials (2.7%), rather than the stroke rate being exceptionally high among patients assigned to CAS.11 Finally, the CAPTURE registry found that the rate of stroke, myocardial infarction or death within 30 days of stenting did not differ amongst three operator experience levels.6

Other potential biases in the comparison of CAS with CEA, which may have contributed to some of the heterogeneity of results observed among the trials, were important differences in (i) the baseline characteristics of the study population, (ii) the revascularisation technique among the trials and (iii) the duration of follow-up (Table 4).

Table 4. Differences among the trials comparing carotid angioplasty/stenting with endarterectomy

	Patients •Although most trials only included patients with neurologically symptomatic carotid stenosis, some included patients with neurologically asymptomatic carotid stenosis.14, 16 •Only patients at high surgical risk for carotid endarterectomy were included in the SAPPHIRE trial.14 Diagnosis •Diagnostic angiography was mandatory before randomisation in three trials (and its risk was added to the treatment) whereas patients could be randomised according to the results of non-invasive techniques (mainly duplex ultrasound) in other trials Revascularisation technique •The technique for carotid angioplasty/stenting has changed over time. •Use of an embolus protection device (EPD) was varied among trials11, 14, 15 Operator experience •Operator experience was different among the trials.10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 Follow-up •Data for death or any stroke during follow up were available from six trials, but events were given for different time points; two trials presented results at six months' after randomisation,11, 20 two trials presented outcome events at 12 months after randomisation,13, 14 and one trial each presented results at two years17 and three years16 of follow up.

The baseline characteristics of the randomised patients were not the same in all of the trials. Although most only included patients with neurologically symptomatic carotid stenoses, two included patients with neurologically asymptomatic carotid disease (70% of patients assigned to CAS in the SAPPHIRE trial14 and 12% assigned to CAS in the CAVATAS trial16 described no symptoms ipsilateral to the carotid artery stenosis within six months of randomisation). The SAPPHIRE trial also only recruited patients deemed “high surgical risk”.14 Although the CAVATAS protocol did not specify ‘high risk’ as an exclusion criterion, a subsequent analysis of the baseline characteristics suggested that CAVATAS also selected a higher proportion of patients at “high surgical risk” compared to earlier trials of CEA.1, 16

The revascularisation technique for performing CAS also differed among the trials. In the earliest studies,10, 12, 13, 16, 19 most patients assigned to endovascular therapy were treated with balloon angioplasty only and without cerebral protection (which had not been invented). For example, in the CAVATAS study,16 only 22% of patients underwent stenting, usually following unsuccessful balloon dilatation. Moreover, the stents were not designed for dedicated use in the carotid artery. In subsequent trials,11, 14, 15 most patients assigned to CAS received self-expanding thermal stents in conjunction with the use of one of the various cerebral protection devices.

Heterogeneity may also have arisen because the systematic review and meta-analysis included trials that had been prematurely terminated11, 12, 13, 14, 15 in addition to those that completed.10, 16, 17, 18, 19, 20, 21 These “early stopping trials”, particularly if the termination was data-dependent due to an increased hazard, are at risk of over-estimating the risk of the treatment.

The random-effects model for 30-day stroke produced a similar odds ratio (OR: 1.47) to the fixed-effects model, but the 95% confidence intervals were now wider (95% CI 0.81–2.67) and overlapped with an OR=1 (i.e. not statistically significant at the p=0.05 level).9 This pooled estimate and its confidence interval from the random-effects meta-analysis refers to the centre of the distribution of the effects of carotid stenting compared with endarterectomy. The confidence interval describes uncertainty in the location of the average of the systematically different effects in the different studies. It does not describe the width of the distribution. It also does not describe the degree of heterogeneity among studies as may be commonly believed. These findings from the random-effects model reduce the overall impact of the findings of the fixed-effect model and suggest that the data may not be robust or reliable. Furthermore, the results may be biased because three of the trials were stopped early because of an excess event rate in the endovascular treatment group. Finally, the results may not be externally valid (ie generalisable) beyond the centres which took part in the constituent trials (and who had a specific interest in carotid intervention and secondary prevention of stroke).

The rate of cranial nerve injury (CNI) within 30 days of the procedure was reported in six trials. There was no significant heterogeneity among the studies (chi squared 1.87, P=0.60; I2=0%). There was, however, a significant reduction in the rate of CNI in patients treated by CAS (3/758 [0.4%] after stenting vs 54/755 [7.2%] after endarterectomy; fixed-effect OR: 0.07, 95% CI: 0.03 to 0.20, random-effects OR: 0.09, 95% CI: 0.04 to 0.25).

Data for death/any stroke during follow up (including the initial 30-day post-treatment phase) are available from six trials, but (unfortunately) events are given for differing time points. Two trials reported six months' results,11, 20 two presented outcome events at 12 months after randomisation13, 14 and one trial each presented results after two years17 and three years16 of follow up. There was significant heterogeneity among the trials (chi squared 14.05, P=0.02; I2=64%). Overall, there was no significant difference in the rate of late death/stroke (81/882 [9.2%] after stenting vs 73/888 [8.2%] after endarterectomy; fixed-effect OR 1.13, 95% CI: 0.81 to 1.58). The random-effects OR was 1.18 (95% CI: 0.61 to 2.28).

Data for the risk of suffering ‘any stroke’ during follow up (excluding the initial 30-day post-treatment phase) were available from three trials.11, 14, 17 Heterogeneity among trials was not significant (chi squared 0.82, P=0.36; I2=0%). The risk of late ‘any’ stroke was similar (14/441 [3.2%] after stenting vs 14/439 [3.2%] after endarterectomy; fixed-effect OR 1.00, 95% CI: 0.47 to 2.14, random-effects OR 0.99, 95% CI: 0.46 to 2.14).

The rationale for using a cerebral protection device is that it catches any debris released during angioplasty and stent deployment. Although stenting maintains laminar flow across the stenosis and seals the site of dissection, thereby preventing a free intimal flap, the wire mesh of the stent may act as a “cheese grater” during deployment because the stent shortens during expanding and may slice debris off the atheromatous plaque into the cerebral circulation.

Only two trials reported data regarding the effects of cerebral protection devices on the 30-day risks of stroke/death.11, 15 The use of protection devices was not allocated randomly (e.g. 227 protection vs 20 no protection in EVA-3S11; 151 protection vs 416 no protection in SPACE15). This, therefore, introduces the possibility of selection bias compromising any assessment of the effect of protection devices. Furthermore, there was significant heterogeneity between two trials in the results (chi squared 4.53, P=0.03; I2=78%). When the results from SPACE and EVA-3S were pooled, no significant difference in the 30-day rate of death/stroke was observed in the fixed-effect model (29/378 [7.7%] where protection was used vs 33/436 [7.6%] where no protection was used; OR: 0.77, 95% CI: 0.41–1.46). In the random-effects model, the OR was 0.57, 95% CI: 0.14–2.33.

Only two trials have reported 30-day stroke/death rates in neurologically asymptomatic patients16, 19 and in one of these19 there were no periprocedural strokes or deaths at all. Accordingly, the test for heterogeneity is not applicable. The overall results are therefore derived from the CAVATAS trial16 and are extremely imprecise due to the very small number of outcome events. The fixed-effect model showed no significant difference in periprocedural stroke or death among patients assigned carotid angioplasty/stent (3/30 [10%]) compared with endarterectomy (2/21 [10%]). The fixed-effect and random-effects OR was 1.06 (95% CI: 0.16 to 6.94).