Article Text
PDF
Abstract
Background and purpose The Alberta Stroke Program Early CT Score (ASPECTS) is a commonly used scoring system to select patients with stroke for endovascular treatment (EVT). However, the inter- and intra-reader variability is high.
Objective To determine whether the inter- and intra-reader variability is different for various regions of the ASPECTS scoring system by evaluating the interobserver variability of ASPECTS between different readers in a per-region analysis.
Materials and methods All patients with acute ischemic stroke who proceeded to EVT in our institutions over a 4-year period were retrospectively identified from a prospectively maintained database. Images were reviewed by two experienced neuroradiologists, who recalculated the ASPECTS independently. We examined each region of the ASPECTS system to evaluate agreement between the raters in each area.
Results 375 patients were included. The median total ASPECTS was 9 (IQR 8–9). The most common region showing ischemic change was the insula, with the M6 region being least commonly affected. Overall interobserver agreement for ASPECTS using Cohen’s κ was 0.56 (95% CI 0.51 to 0.61). The region with the highest agreement was the insula (κ=0.56; 0.48 to 0.64). The region with the lowest agreement was M3 (κ=0.34; 0.12 to 0.56). Agreement was relatively good when ASPECTS were dichotomized into 0–5 versus 6–10 (κ=0.66; 0.49 to 0.84).
Conclusions Substantial interobserver variability is found when calculating ASPECTS. This variability is region dependent, and practitioners should take this into account when using ASPECTS for treatment decisions in patients with acute stroke.
- stroke
- CT
Statistics from Altmetric.com
Introduction
Most of the recent thrombectomy stroke trials used the Alberta Stroke Program Early CT Score (ASPECTS) to select patients who might benefit from endovascular treatment.1 ASPECTS is not a perfect tool on which to base such a decision. One factor limiting the usefulness of ASPECTS is its inter-rater agreement, with results varying in the literature.2 3 It has been suggested that this variation is not uniformly distributed throughout the components of the score: as the regions comprising ASPECTS vary in both size and normal Hounsfield unit values, it is possible that early ischemic changes are more challenging to detect in some areas than others, and agreement may thus have corresponding regional differences. For this reason, we aimed to characterize ASPECTS per-region agreement in a large cohort of patients with acute ischemic stroke.
Methods
Study subjects
Patients admitted to the Mayo Clinic (Rochester, Minnesota, USA) and the Toronto Western Hospital (Toronto, Ontario, USA) with a diagnosis of acute ischemic stroke secondary to large vessel occlusion of the internal carotid artery or middle cerebral artery were retrospectively evaluated for inclusion in the study. The inclusion criteria were:
clinical diagnosis of acute stroke in the anterior circulation;
age ≥18;
availability of pretreatment non-contrast (NCCT) images acquired within 24 hours of onset of symptoms.
Exclusion criteria included evidence of intracranial hemorrhage or other non-ischemic pathology on baseline imaging or severe artifacts in the NCCT images. The study was given ethical approval by the institutional review boards at both institutions.
CT acquisition
All CT scans were obtained using a multi-detector CT scanner. At one site, images were obtained using an Aquilion ONE 320-slice CT scanner (Canon Medical Systems), on which images were obtained in spiral mode, with a rotation time of 1.5 s, collimation of 128×0.6, pitch of 0.656, kVp of 120, and mAs of 200. At the second site, images were obtained using a Siemens F-128 CT scanner in spiral mode with a rotation time of 1.0 s, collimation of 128×0.6, pitch of 0.6, kVp of 120, and effective mAs of 350. Images at both sites were reconstructed in an oblique axial plane with slice thickness of 3–5 mm.
ASPECT scoring
ASPECTS was assessed by two neuroradiologists (NRAD1 and NRAD2). Both had >5 years’ experience in neuroradiology, and were experienced in the use of ASPECTS for the selection of patients with stroke for endovascular treatment. Both had also undergone appropriate training in assessment of ASPECTS (www.aspectsinstroke.com).4 5 Early ischemic change was defined as the presence of hypodensity and/or loss of grey-white differentiation, with or without cortical swelling. Assessors were blinded to clinical details other than the presence of unilateral anterior circulation ischemic stroke, without specifying the side. Assessors submitted scores of 0 or 1 for each of the 10 regions individually, from which a total score of 0–10 was calculated.
Statistics
Analyses were carried out using JMP Pro v. 14.1 (SAS Institute Inc, Cary, North Carolina, USA). A weighted κ was calculated as a measure of interobserver agreement for total ASPECTS and Cohen’s κ was calculated for individual regions. Values of κ are presented with 95% confidence intervals. A Bland-Altman analysis was performed to evaluate the differences between the two readers. Baseline demographics are presented as mean±SD for continuous variables and median with IQR for ordinal variables.
Results
A total of 375 patients were included in the final analysis. At baseline, the median age in our sample was 62 years (±11; range 22–96) and 202 patients (53.9%) were female. One hundred and eighty-five strokes (49.3%) were right-sided, and the median National Institutes of Health Stroke Scale score was 17 (IQR 11–17). The mean time from symptom onset to CT scan was 4.78 (range 0.8–28.8) hours. The median total ASPECTS was 9 (IQR 8–9).
The relative prevalence of ischemic change per region is shown in figure 1. The most common region showing early ischemic change was the insula for both NRAD1 and NRAD2; the least commonly affected was M6 for NRAD1 and M4 for NRAD2. Calculating a weighted κ value showed an overall agreement of 0.56 (95% CI 0.51 to 0.61). A Bland-Altman analysis showing differences between reviewer according to the ASPECTS is shown in (figure 2). The results for regional agreement are shown in table 1. The region with the highest agreement was the insula (κ=0.56; 0.48 to 0.64). The region with the lowest agreement was M3 (κ=0.34; 0.12 to 0.56). Overall agreement was relatively good when ASPECTS were dichotomized into 0–5 versus 6–10 (κ=0.66; 0.49 to 0.84). For patients who were imaged within 3 hours of symptom onset, weighted κ was 0.52 (95% CI 0.33 to 0.72), whereas for those patients imaged beyond 3 hours, weighted κ was 0.57 (95% CI 0.31 to 0.83).
Distribution of total aspects per region. ASPECTS, Alberta Stroke program Early CT Score; NRAD, neuroradiologist.
Results of the Bland-Altman analysis. ASPECTS, Alberta Stroke program Early CT Score.
Comparison of results per ASPECTS region*
Discussion
In this study, we quantified per-region agreement for ASPECTS on NCCT images from 375 patients with acute ischemic stroke, as assessed by two experienced neuroradiologists. In our dataset, the greatest agreement was found in the caudate, lentiform, and M5 regions, with the M3 and internal capsule regions having the lowest interobserver agreement. This is relevant because interventional treatment decisions are often based on the ASPECTS, and so decision makers using it need to be aware of the limitations of this system.
It should also be noted that no prospective randomized controlled trials have shown that thrombectomy is futile or unsafe in patients with ASPECTS <5. Several ongoing trials are examining the role of endovascular treatment (EVT) in this patient cohort. In the meantime, most centers will consider treating patients with low ASPECTS on a case-by-case basis. For this reason, we dichotomized ASPECTS into low (≤5) and higher (>5) subgroups.
Other groups have previously looked at this field of study. Gupta et al 3 investigated inter-rater agreement between two neuroradiologists in a cohort of 155 patients undergoing EVT who had a median ASPECTS of 8 (IQR 7–9). In their group with relatively modest ischemic change, they found a κ coefficient of 0.77 for total ASPECTS; per-region κ varied from 0.73 in the insula and 0.71 in the caudate to 0.28 in the internal capsule, although this was attributed to the very small number of internal capsule strokes in their sample.3 Similarly, Finlayson et al looked at agreement between four observers (two neurologists, two neuroradiologists) in 181 patients.6 They found that the intraclass correlation coefficient for total ASPECTS was 0.83; again, the caudate (0.79) had the highest correlation coefficient, with the internal capsule having the lowest (0.37). In comparison with previous investigations, ours is the largest study to date looking at per-region interobserver variability with ASPECT scoring. In addition, previous studies were single-center, whereas our study was carried out at several centers.
The underlying reasons why the internal capsule is most prone to disagreement are probably both biological and methodological. Different regions of the brain are known to have varying levels of sensitivity to ischemic changes.7 If the caudate and lentiform are infarcted, the anterior limb of the internal capsule is usually involved as well, owing to a shared blood supply; however, some assessors determined ischemic changes only using the posterior limb.5 The internal capsule is often primarily relatively hypoattenuated on CT in healthy subjects, potentially making it difficult to detect further hypoattenuation with early ischemia. Furthermore, the initial paper from the Calgary group, describing the ASPECTS score in 2001, reported disagreement between the six physicians (who developed ASPECTS) of what constituted the internal capsule for the purposes of scoring.5 Readers varied in whether they assessed both limbs of the internal capsule (and removed a point if any limb was affected) or the posterior limb only. With disagreement between the group who developed the score, it is not surprising that there would also be some disagreement among less experienced readers, especially since the supply to the internal capsule and its anterior and posterior limbs is highly variable. Finally, when we look at the distribution of ischemic changes by ASPECTS region, most were seen in the insula (48.8%), reflecting the particular vulnerability of this region to ischemic changes. Changes in prevalence can also affect κ values,8 9 and it is possible that this has had some effect here. The confidence intervals of all of our per-region κ values overlapped, implying that there was no statistically significant difference, although we may be underpowered to look for differences between subgroups.
Similar reasons can perhaps account for the much better agreement seen in the caudate. Although relatively small, the caudate is clearly outlined by the hypodense internal capsule posterolaterally and the lateral ventricles anteromedially, facilitating visual assessment. Unlike the ambiguity over the limbs of the internal capsule with some readers, there can be relatively little discussion about what constitutes the caudate. In addition, supply to the caudate is fairly uniform (ie, the medial lenticulostriate perforators). Finally, as can be seen in figure 1, the caudate was relatively prone to ischemic changes—it is therefore likely that this increased frequency of observation had a role in decreasing the interobserver variability.
This argument also extends to the other regions—agreement was substantially higher in those regions that were much more often involved on the ASPECTS (eg, insula, lentiform) than in the less frequently involved regions (eg, M3, M4).
The limitations of our study include not having a 'gold standard' measure of ischemia, ideally in the form of diffusion-weighted imaging. Patients with extensive infarcts (eg, ASPECTS ≤5) in our study were also relatively underrepresented (n=5, 1.3%). However, calculation of the ASPECTS in these patients is often more straightforward, and the calculation of more ‘subtle’ ASPECTS (eg, 7–9) is typically more challenging, particularly for less experienced observers. One other limitation is that we did not measure intraobserver variability, as we felt this was less clinically relevant. Another potential limitation is that our patient cohort did not include many 'late presenters'—the median time to scanning was 4.8 hours. However, there was no significant difference in weighted κ values when we compared those patients who were imaged at≤3 hours versus >3 hours (0.52 vs 0.57).
We believe that, despite the aforementioned limitations, our study is important since it demonstrates that ASPECTS is not a perfect tool, even for trained eyes. It is highlighted by the main finding of our study—namely, that there is a substantial interobserver variability which affects certain regions more than others, even when read by experienced fellowship-trained neuroradiologists. In view of the increasing number of non-radiologists using ASPECTS to triage patients with stroke to EVT, the relevance of our findings may be even more substantial as many studies have shown that the accuracy of ASPECTS increases with various training levels.10 11
Conclusions
In conclusion, we have shown that while overall, ASPECTS is a relatively reliable tool for assessment of ischemic changes on CT, there is a significant interobserver and—even in experienced eyes—a differential per-region variability.
References
Footnotes
Twitter @paddynicholson
Contributors Each author has made substantial contributions to the conception or design of the work, or the acquisition, analysis, or interpretation of data; drafting the work or revising it critically for important intellectual content; final approval of the version published; and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Funding SMS is supported by a postdoctoral fellowship grant from the AHA (award number 19POST34381067).
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement No data are available.