Article Text

Download PDFPDF

Original research
Comparison of three commonly used CT perfusion software packages in patients with acute ischemic stroke
  1. Miou S Koopman1,
  2. Olvert A Berkhemer1,2,3,
  3. Ralph R E G Geuskens4,
  4. Bart J Emmer1,
  5. Marianne A A van Walderveen5,
  6. Sjoerd F M Jenniskens6,
  7. Wim H van Zwam7,
  8. Robert J van Oostenbrugge8,
  9. Aad van der Lugt2,
  10. Diederik W J Dippel3,
  11. Ludo F Beenen1,
  12. Yvo B W E M Roos9,
  13. Henk A Marquering1,4,
  14. Charles B L M Majoie1
  15. on behalf of the MR CLEAN Trial Investigators
    1. 1 Department of Radiology and Nuclear Medicine, Amsterdam UMC, location AMC, Amsterdam, The Netherlands
    2. 2 Department of Radiology and Nuclear Medicine, Erasmus MC University Medical Center Rotterdam, Rotterdam, The Netherlands
    3. 3 Department of Neurology, Erasmus MC University Medical Center, Rotterdam, The Netherlands
    4. 4 Department of Biomedical Engineering and Physics, Amsterdam UMC, location AMC, Amsterdam, The Netherlands
    5. 5 Department of Radiology, Leiden University Medical Center, Leiden, The Netherlands
    6. 6 Department of Radiology, Radboud University Medical Center, Nijmegen, The Netherlands
    7. 7 Department of Radiology and CArdiovascular Research Institute Maastricht (CARIM), Maastricht University Medical Center, Maastricht, The Netherlands
    8. 8 Department of Neurology and CArdiovascular Research Institute Maastricht (CARIM), Maastricht University Medical Center, Maastricht, The Netherlands
    9. 9 Department of Neurology, Amsterdam UMC, location AMC, Amsterdam, The Netherlands
    1. Correspondence to Miou S Koopman, Radiology and Nuclear Medicine, Amsterdam UMC, location AMC, Amsterdam 1105 AZ, The Netherlands; m.s.koopman{at}amc.uva.nl

    Abstract

    Background and purpose CT perfusion (CTP) might support decision making in patients with acute ischemic stroke by providing perfusion maps of ischemic tissue. Currently, the reliability of CTP is hampered by varying results between different post-processing software packages. The purpose of this study is to compare ischemic core volumes estimated by IntelliSpace Portal (ISP) and syngo.via with core volumes as estimated by RAPID.

    Methods Thirty-five CTP datasets from patients in the MR CLEAN trial were post-processed. Core volumes were estimated with ISP using default settings and with syngo.via using three different settings: default settings (method A); additional smoothing filter (method B); and adjusted settings (method C). The results were compared with RAPID. Agreement between methods was assessed using Bland–Altman analysis and intraclass correlation coefficient (ICC). Accuracy for detecting volumes up to 25 mL, 50 mL, and 70 mL was assessed. Final infarct volumes were determined on follow-up non-contrast CT.

    Results Median core volume was 50 mL with ISP, 41 mL with syngo.via method A, 20 mL with method B, 36 mL with method C, and 11 mL with RAPID. Agreement ranged from poor (ISP: ICC 0.41; method A: ICC 0.23) to good (method B: ICC 0.83; method C: ICC 0.85). The bias (1.8 mL) and limits of agreement (−27, 31 mL) were the smallest with syngo.via with additional smoothing (method B). Agreement for detecting core volumes ≤25 mL with ISP was 54% and 57%, 85% and 74% for syngo.via methods A, B, and C, respectively.

    Conclusion Best agreement with RAPID software is provided by syngo.via default settings with additional smoothing. Moreover, this method has the highest agreement in categorizing patients with small core volumes.

    • stroke
    • brain ischemia
    • CT perfusion
    • post-processing software
    • ischemic core

    Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Endovascular thrombectomy (EVT) has been proved to be effective in seven randomized controlled trials for the treatment of patients with acute ischemic stroke if treatment is started within 6 hours.1–7 However, successful recanalization does not guarantee a good outcome after 90 days. Therefore, since resources are currently scarce, continuous interest in imaging to select patients who will most likely benefit from reperfusion therapies remains.8–10

    CT perfusion (CTP) is commonly used in acute stroke imaging because of its widespread availability and fast acquisition. CTP software can create brain perfusion maps indicating several parameters such as cerebral blood volume (CBV), cerebral blood flow (CBF), mean transit time (MTT), and time to maximum peak (Tmax). Based on these parameters, summary maps are created which allow visualization of irreversibly injured tissue (ischemic core) and functionally impaired tissue that is at risk of subsequent infarction (penumbra).

    Despite its potential, the applicability of CTP in acute ischemic stroke evaluation has been questioned. One reason is the lack of standardized acquisition and post-processing protocols. It has been shown in previous studies that software packages differ in underlying algorithms leading to varying results in quantification.11–15 No consensus on the most favorable algorithm has been reached so far. Accuracy and comparability between vendors is of great importance when calculating ischemic core volume, particularly when selecting patients for EVT based on these values.

    Current decision making in acute ischemic stroke is based on time domains, which do not take into account the biological variation between patients. Identification of potentially salvageable tissue with perfusion imaging may facilitate a more personalized approach in selection for EVT. Several trials used CTP to select patients for EVT within 6 hours from stroke onset. In a recently published pooled analysis of the HERMES data, an independent association of ischemic core and functional outcome was found; however, no modification of treatment benefit of EVT over standard care could be established.16

    Recently, the DAWN and DEFUSE3 trials showed a highly beneficial effect of EVT beyond the 6-hour window in patients selected with CTP or MRI.17 18 Patients were selected based on ischemic core volumes estimated with RAPID software (iSchemaView). In hospitals without access to RAPID, selection of patients for EVT presenting beyond the 6-hour window depends on comparability of perfusion imaging results with the selection criteria in those two trials.

    Therefore, we aimed to compare ischemic core volumes estimated by IntelliSpace Portal and syngo.via, both commonly used CT perfusion software packages with ischemic core volumes as estimated by RAPID.

    Methods

    Study design and patient selection

    In this post hoc analysis we included patients from the MR CLEAN trial.1 Patients who met the following inclusion criteria were eligible: clinical diagnosis of acute ischemic stroke, deficit on the National Institutes of Health Stroke Scale (NIHSS) of 2 points or more, no hemorrhage on CT or MRI, and intracranial large artery occlusion of anterior circulation confirmed by CT angiography. A subset of prospectively collected baseline CTP imaging data with >100 mm brain coverage of patients was used. Final infarct volume was determined on 5–7-day follow-up non-contrast CT (NCCT). If 5–7-day NCCT was not available, 24-hour follow-up NCCT was used. Exclusion criteria were severe motion artifacts and poor scan quality. This resulted in a study population of 35 patients.

    Ethics statement

    The MR CLEAN study protocol was approved by the Medical and Ethical Review Committee (Medisch Ethische Toetsings Commissie of Erasmus MC, Rotterdam, the Netherlands) and the research board of each participating center. A detailed description of the MR CLEAN trial protocol has been published previously.19 All patient records and images were anonymized before analysis, and written informed consent was obtained from all patients or their legal representatives as part of the original trial protocol.

    Imaging acquisition and post-processing

    CTP acquisition protocols differed per center. Table 1 shows detailed information about scanner type, scan protocol, and acquisition time. All datasets were post-processed using three fully automated software packages: RAPID (version 2017; iSchemaView, Menlo Park, California, USA), IntelliSpace Portal CT Brain Perfusion (version 7.0; Royal Philips Healthcare, Best, The Netherlands), and syngo.via CT Neuro Perfusion (version 2017; Siemens Healthcare, Erlangen, Germany) (table 2). All software packages use automated registration, segmentation, and motion correction.

    Table 1

    Detailed information of CT perfusion (CTP) scan protocols

    Table 2

    Software parameters

    RAPID is a fully automated software package that uses a delay-insensitive algorithm. Ischemic core was defined as reduction in CBF to <30% compared with the contralateral hemisphere.

    IntelliSpace Portal CT Brain Perfusion (ISP) uses a delay-sensitive algorithm. 3D motion correction and filtering was applied on all scans. Ischemic core was defined as a relative MTT >1.45 and a CBV <2.0 mL/100 mL.

    Syngo.via CT Neuro Perfusion relies on a deconvolution model with a delay-insensitive algorithm as well as on interhemispheric comparison. The side with the highest time to drain is automatically characterized as the lesion side and the contralateral side is used as a reference for relative values. Subsequently, summary maps were generated depicting ischemic core with corresponding volumes.

    With syngo.via, labeling of voxels as ischemic core was done with three different settings. With method A we generated maps based on default settings. Ischemic core volume was defined as an absolute CBV of <1.2 mL/100 mL. For method B we used the same thresholds with the application of an additional smoothing filter to minimize artifacts. With method C we used the same settings as those used with RAPID software; ischemic core was defined as a reduction of CBF below 30% compared with normal brain tissue. Additional smoothing did not change the results for method C and is therefore not presented separately.

    Follow-up non-contrast CT (NCCT) imaging

    Patients underwent follow-up NCCT 24 hours and 5–7 days after randomization. Final infarct volume was determined on 5–7-day follow-up NCCT. If 5–7-day NCCT was not available (n=2), 24-hour follow-up NCCT was used. Final infarct volume was determined using a semi-automated in-house developed method.20 Baseline CTP core measurements were compared with final infarct volumes as determined on NCCT.

    Statistical analysis

    Continuous variables were reported as median (IQR) or mean (SD) based on (non)-normality of the data. Categorical variables were reported as number (percentage). The Wilcoxon test for paired differences was used as a non-parametric test.

    Bland–Altman analyses with 95% limits of agreement and calculation of intraclass correlation coefficient (ICC) (two-way mixed model for absolute agreement, single measure) were performed to determine agreement between RAPID, ISP, and syngo.via for each setting. ICC values were interpreted by the proposed standards of Koo et al: <0.50 (poor), 0.50–0.75 (moderate), 0.75–0.90 (good), and >0.90 (excellent).21

    In order to analyze whether both ISP and syngo.via could accurately categorize patients according to RAPID-based core volumes, we determined diagnostic agreement for detecting ischemic core volumes of ≤25 mL, ≤50 mL, and ≤70 mL for ISP and each syngo.via method. Sensitivity, specificity, and overall diagnostic accuracy were calculated. The threshold of 25 mL was chosen based on upper interquartile range (p75) of the DEFUSE3 trial and thresholds of 50 mL and 70 mL were based on data (upper limits of inclusion) of DAWN, EXTEND-IA, and DEFUSE3 trials.3 17 18

    P values <0.05 were considered statistically significant. Statistical procedures were customized in R (R Core Team v3.5.0; R Foundation for Statistical Computing, Vienna, Austria, September 2018).

    Results

    The baseline characteristics of the study population are shown in table 3. The median age was 66 years and median baseline NIHSS score was 16. Median time from stroke onset to randomization was 160 min (IQR 140–235). Summary maps generated with ISP, syngo.via, and RAPID of two different patients with left hemispheric stroke are shown in figures 1 and 2.

    Table 3

    Clinical characteristics at baseline

    Figure 1

    Summary maps of a patient with a left hemispheric stroke. The area of abnormality of both the ischemic core (red/purple) and penumbra (green) is visually more or less similar in all three maps. Ischemic core volumes were 19 mL, 9 mL, 14 mL, and 19 mL with syngo.via method A (A), B (B), and C (C) and ISP (D), respectively. The ischemic core volume was 8 mL with RAPID (E,F).

    Figure 2

    Summary maps of a different patient with a left hemispheric stroke. Ischemic core (red/purple) volumes were 42 mL, 27 mL, 58 mL, and 197 mL with syngo.via method A (A), B (B) and C (C) and ISP (D), respectively. The ischemic core volume was 31 mL with RAPID (E,F).

    Ischemic core volume

    In 35 patients the median ischemic core volume was 11 mL (IQR 4.2–34) with RAPID. Median ischemic core volumes were 50 mL (IQR 31–131) with ISP and 41 mL (IQR 28–58), 20 mL (IQR 9.7–36), and 36 mL (IQR 14–58) with syngo.via methods A, B, and C, respectively. Ischemic core volumes estimated with syngo.via method C were similar with and without additional smoothing filter. The core volumes were significantly different between ISP and RAPID (p<0.001), syngo.via method A, and RAPID (p<0.001) and between syngo.via method C and RAPID (p<0.05). The difference was smaller and not significant between syngo.via method B and RAPID (p=0.18).

    The agreement between ischemic core volumes calculated by ISP, syngo.via, and RAPID is illustrated by scatterplots (figure 3) and the difference in relation to the size by Bland–Altman plots (figure 4). The mean (SD) difference in prediction errors between RAPID and syngo.via was 19 (19) mL, 1.8 (15) mL, and 22 (20) mL for methods A, B, and C, respectively. Comparison of ISP and RAPID showed a mean difference of 55 (50) mL. Limits of agreements for ischemic core area were smallest with method B (−27, 31 mL) and largest for ISP (−43, 154 mL). A poor agreement was found between syngo.via method A and RAPID (ICC 0.23; 95% CI −0.11 to 0.52) as well as between ISP and RAPID (ICC 0.41; 95% CI 0.10 to 0.65). The agreement between RAPID and syngo.via method B (ICC 0.83; 95% CI 0.69 to 0.91) and syngo.via method C (ICC 0.85; 95% CI 0.73 to 0.92) was good (table 4).

    Figure 3

    Scatterplots of ischemic core volumes estimated with ISP and syngo.via versus RAPID. The dotted oblique line represents the reference line (x=y).

    Figure 4

    Bland–Altman plots for volumetric agreement of core volumes with ISP, syngo.via methods A, B, and C and RAPID. Solid lines demonstrate the mean difference between ISP, syngo.via, and RAPID. Dotted lines represent 95% limits of agreement.

    Table 4

    Performance characteristics of ISP and syngo.via versus RAPID

    Threshold selection

    Agreement for detecting an ischemic core volume ≤25 mL with syngo.via was respectively 57%, 85% and 74% (table 5) and 54% with ISP. Method B demonstrated the best true positive prediction for ischemic core volume classified as ≤25 mL, ≤50 mL, and ≤70 mL. With a RAPID-based core estimate of 25 mL, syngo.via methods A, B, and C would incorrectly categorize 15/22 patients, 2/22, and 9/22 patients as having an ischemic core volume of ≤25 mL. When using syngo.via method B, overestimation would lead to the unnecessary exclusion of two of the 22 patients (9%) for treatment compared with a RAPID-based core estimate of <25 mL (table 5).

    Table 5

    Agreement of detecting ischemic core volumes with thresholds of ≤25 mL, ≤50 mL, and ≤70 mL

    Final infarct volume

    Median final infarct volume was 49 mL (IQR 25–90). A comparison of ischemic core volumes and final infarct volumes for each subject are shown in online supplementary data (table 6 and figure 5). With syngo.via 14/35 (40%), 5/35 (14%), and 14/35 (40%) of the cases had a final infarct volume that was smaller than the baseline ischemic core volume for methods A, B, and C, respectively. RAPID showed overestimation of baseline ischemic core volume in 2/35 cases (6%) and ISP in 17/35 cases (49%). Median core volume estimated with ISP was larger than final infarct volume, with a median difference of −12 mL (IQR −43–12).

    Discussion

    In this study we evaluated the agreement between ischemic core volumes as estimated by ISP and syngo.via (with three different settings) and volumes estimated by RAPID software. The agreement between RAPID and syngo.via ranged from poor to good, depending on the chosen settings. Standard settings of syngo.via software (method A) resulted in overall larger ischemic core volumes compared with RAPID. Agreement improved remarkably with an additional smoothing filter applied to standard settings (method B), especially within the smaller ischemic core volume range. This setting demonstrated the smallest prediction error and narrowest limits of agreement. We hypothesized that using similar thresholds (rCBF <30%) in both software packages would lead to better agreement between software packages. However, method C did show significantly larger core volumes compared with RAPID.

    Poor agreement was found between RAPID and ISP software using default thresholds. With ISP software we were not able to adjust the threshold or apply additional smoothing filters due to software restrictions. Overall, with increasing ischemic core volumes, all estimations with ISP and syngo.via showed increased deviation from RAPID-based ischemic core volumes.

    Threshold selection was based on data from EXTEND-IA and DAWN and DEFUSE3 trials.3 17 18 DAWN and DEFUSE3 showed benefit of EVT in an extended time window (>6 hours) in patients selected with CTP or MRI.17 18 For clinical decision making, both under- and overestimation may have important consequences. With one software package clinicians may select a patient for EVT whereas use of a different software package would exclude the same patient. At threshold volumes of 25 mL, 50 mL, and 70 mL, syngo.via method B showed the highest sensitivity with the lowest rate of overestimation for true ischemic core volume, as defined by RAPID. Syngo.via method B was superior to the other methods in categorizing patients based on a threshold of 25 and 50 mL. From a clinical standpoint, a method that is more restrictive would be preferred because this method would bias towards underestimating ischemic core. This would lower the rate of falsely identifying a patient as not eligible for reperfusion therapy based on a larger ischemic core volume.

    The results of our study are in line with previous software comparison studies where a large variation in quantitative CTP measurements have been described.11 13 15 22 Direct comparison of three software packages with final infarct volume found significant differences between the three software packages (ISP, syngo.via and RAPID).12 This study showed that RAPID provided the best agreement with final infarct volume after EVT in fully recanalized patients.

    Several factors are known to contribute to differences in CTP results, such as CT scan parameters (voltage, current, scan time), contrast delivery, type of post-processing algorithm, and selected thresholds.23 24 Moreover, differences in post-processing steps such as defining arterial input and venous output function, motion correction and smoothing will lead to varying results. It has been shown that automated software can improve the quality, reliability, and reproducibility of CTP compared with manual post-processing.25

    The variability in our results might be partially attributed to differences in the acquisition algorithm. ISP relies on a delay-sensitive algorithm whereas syngo.via and RAPID are based on delay-insensitive algorithms that compensate for the arrival delay of contrast agents.15 26 Kudo et al compared five different software packages in two groups, depending on the algorithm used. They found that overestimation of CBF and MTT values occurred in the delay-sensitive algorithms, possibly due to the tracer delay effect.15 Despite increasing use of CTP, there is no definite consensus between manufacturers as to which parameters are optimal to define infarct core. Bivard et al stated that, regardless of which perfusion algorithm was used, CBF was the most accurate parameter for defining ischemic core.13 In our results, despite choosing the same parameters and thresholds in syngo.via method C as in RAPID, significant differences were found for the calculated ischemic core volume between the two software packages.

    This study has several limitations. First, CTP imaging was obtained from two scanner brands and three different scanner types, leading to heterogeneity in our data. It is possible that imaging data derived from different scanners may not be processed adequately by the three tested software packages. However, this variability in scanner brands and acquisition methods is a reflection of daily clinical practice. Furthermore, our sample size is relatively small, limiting the power to detect differences in ischemic core volumes between the two software packages. Only a small number of patients had RAPID-based infarct volumes larger than 50 mL (n=5), therefore limiting the power to accurately test accuracy and agreement for thresholds of 50 and 70 mL.

    A third limitation is the lack of diffusion-weighted MRI (MRI-DWI) performed directly after CTP as reference standard. MRI-DWI is still considered to be the most sensitive and accurate method for identifying cerebral ischemia.27 Using RAPID as the comparison method has certain drawbacks, knowing that ischemic core estimated with RAPID might differ from what would be defined as true ischemic core with MRI-DWI. However, since RAPID was used in all major randomized clinical trials for selection, RAPID-based CTP-derived results remain clinically relevant. Moreover, the accuracy of RAPID with regard to DWI and final infarct volume has been well described in previous studies.12 28–31

    In our study we found that median CTP-derived ischemic core volumes were generally smaller than final infarct volumes with syngo.via and RAPID. ISP estimated ischemic core volumes on CTP larger than final infarct volumes in almost half of the cases. No conclusions can be drawn as to the extent of underestimation, since both time between CTP and recanalization as well as incomplete reperfusion might have led to infarct growth between baseline imaging and follow-up. Since our sample size was already small, we did not perform a subgroup analysis of subjects with complete endovascular reperfusion. Nevertheless, it is unlikely that the ischemic core would become smaller or disappear.

    Regardless, in our study we focused on the comparative agreement of results of CTP analyses and did not emphasize the validity of each method separately. We therefore refrain from drawing any conclusions as to which package is more accurate.

    Conclusions

    Ischemic core volume estimations vary greatly between different software packages. Using the same algorithm does not warrant optimal agreement. Best agreement between syngo.via and RAPID was found when applying an additional smoothing filter to the syngo.via standard settings and not when the RAPID algorithm was used in syngo.via. The differences should be acknowledged when selecting patients for EVT based on recent late window trial results. Further research is required to validate these findings across vendors in a larger group of patients.

    Acknowledgments

    The authors gratefully acknowledge the support of iSchemaView for the analysis with RAPID software.

    References

    Footnotes

    • Contributors The MR CLEAN trial was designed by WHZ, AvdL, RJvO, DWJD, YBWEMR, and CBLMM. OAB collected and prepared the data for the trial. OAB, RREGG, MAAvW, and SFJ collected the data for this study. MSK conducted the statistical analysis, interpreted the results, and drafted the paper. HAM and BJE assisted with the statistical analysis, interpretation of the results, and drafting the paper. OAB, MAAvW, SFJ, WHZ, RJvO, AvdL, DWJD, LFB, YBWEMR, and CBLMM critically revised the paper. All authors approved the version to be published.

    • Funding The MR CLEAN trial was partly funded by the Dutch Heart Foundation, grant no 2008T30, and by unrestricted grants from AngioCare BV, Medtronic/Covidien/EV3,MEDAC Gmbh/LAMEPRO, Penumbra, Stryker, and Top Medical/ Concentric.

    • Competing interests Amsterdam UMC, location AMC received funds from Stryker for consultations by CBLMM, YBWEMR, and OAB. CBLMM received research grants from CVON/Dutch Heart Foundation, European Commission and Twin Foundation. HAM, RREGG, CBLMM, and YBWEMR are shareholders of Nico.lab, a company that focuses on the use of artificial intelligence for medical image analysis. Erasmus MC received funds from Bracco Imaging for consultations by DWJD. Erasmus MC received funds from CVON/Dutch Heart Foundation, European Commission, Stryker, Penumbra, and Medtronic for the execution of stroke trials by DWJD and AvdL. Maastricht University Medical Center received funds from Stryker and Cerenovus for consultations by WHZ.

    • Ethics approval A central medical ethics committee and the research boards of all participating centers accepted the MR CLEAN trial.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement Because of the sensitive nature of the data collected for this study, requests to access the data set may be sent to the MR CLEAN executive committee (https://www.mrclean-trial.org/).

    • Patient consent for publication Obtained.