You are here

Article Text

Article menu

Download PDFPDF

Ischemic stroke
Original research
Viz LVO versus Rapid LVO in detection of large vessel occlusion on CT angiography for acute stroke
  1. Adam Delora1,
  2. Christopher Hadjialiakbari1,
  3. Eryn Percenti2,
  4. Jordan Torres2,
  5. Yazan J Alderazi3,
  6. Rime Ezzeldin4,
  7. Ameer E Hassan5,
  8. Mohamad Ezzeldin6,7
  1. 1 Emergency Medicine, HCA Houston, Kingwood, Texas, USA
  2. 2 Internal Medicine, HCA Houston, Kingwood, Texas, USA
  3. 3 HCA Gulf Coast Division, HCA Houston, Webster, Texas, USA
  4. 4 Medicine, Jordan University of Science and Technology, Irbid, Jordan
  5. 5 Department of Neurology, University of Texas Rio Grande Valley, Harlingen, Texas, USA
  6. 6 Department of Clinical Sciences, College of Medicine, University of Houston, Houston, Texas, USA
  7. 7 Neuroendovascular Surgery, HCA Houston, Houston, Texas, USA
  1. Correspondence to Dr Mohamad Ezzeldin, Department of Clinical Sciences, College of Medicine, University of Houston, Houston, TX, USA; mohamadezzeldin{at}hotmail.com

Abstract

Background Endovascular thrombectomy improves outcomes and reduces mortality for large vessel occlusion (LVO) and is time-sensitive. Computer automation may aid in the early detection of LVOs, but false values may lead to alarm desensitization. We compared Viz LVO and Rapid LVO for automated LVO detection.

Methods Data were retrospectively extracted from Rapid LVO and Viz LVO running concurrently from January 2022 to January 2023 on CT angiography (CTA) images compared with a radiologist interpretation. We calculated diagnostic accuracy measures and performed a McNemar test to look for a difference between the algorithms’ errors. We collected demographic data, comorbidities, ejection fraction (EF), and imaging features and performed a multiple logistic regression to determine if any of these variables predicted the incorrect classification of LVO on CTA.

Results 360 participants were included, with 47 large vessel occlusions. Viz LVO and Rapid LVO had a specificity of 0.96 and 0.85, a sensitivity of 0.87 and 0.87, a positive predictive value of 0.75 and 0.46, and a negative predictive value of 0.98 and 0.97, respectively. A McNemar test on correct and incorrect classifications showed a statistically significant difference between the two algorithms’ errors (P=0.00000031). A multiple logistic regression showed that low EF (Viz P=0.00125, Rapid P=0.0286) and Modified Woodcock Score >1 (Viz P=0.000198, Rapid P=0.000000975) were significant predictors of incorrect classification.

Conclusion Rapid LVO produced a significantly larger number of false positive values that may contribute to alarm desensitization, leading to missed alarms or delayed responses. EF and intracranial atherosclerosis were significant predictors of incorrect predictions.

  • CT Angiography
  • Technology
  • CT
  • Stroke
  • Thrombectomy

Data availability statement

Data are available upon reasonable request.

https://doi.org/10.1136/jnis-2023-020445

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    WHAT IS ALREADY KNOWN ON THIS TOPIC

    • Endovascular thrombectomy has been shown to decrease disability, improve functional outcomes, and reduce mortality over standard medical management for anterior large vessel occlusion (LVO) and is time-sensitive. Implementation of automated LVO detection software on CT angiography has been shown to decrease time to puncture and other workflow metrics. Automated system alarms with many false values have been shown to lead to alarm fatigue, delayed reads, and the ignoring of these systems altogether.

    WHAT THIS STUDY ADDS

    • This study compares the diagnostic accuracy of two commercially available LVO detection automation software packages, Viz LVO and Rapid LVO, implemented in two community-based comprehensive stroke centers. The study also looks for predictors of incorrect classifications like low ejection fraction or intracranial atherosclerosis to give algorithm developers insights to improve the accuracy of their systems to prevent undue burden to providers. The study also looks at how these algorithms are tuned to focus on reducing false negative values at the expense of false positive values and false positive values can negatively impact the practitioners that use these systems.

    HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

    • This study provides important insight into predictors of these false values for algorithm developers when improving the accuracy of their algorithms. The study also helps providers understand how these algorithms are tuned to properly use the results. Results of this study show that these two software packages favor preventing false negative values due to the risks associated with LVO at the expense of false positive values. Large numbers of false positive values can lead to alarm fatigue and overwhelm providers.

    Introduction

    Large vessel occlusions (LVOs) are responsible for the majority (90%) of mortality and severe disability in patients who experience ischemic strokes.1 There is some variation in terminology, but initial studies primarily defined LVOs as occlusions of the intracranial internal carotid artery (ICA) or the first segment of the middle cerebral artery (M1), and a subset of these studies also included the second segment of the middle cerebral artery (M2) and the first or second segment of the anterior cerebral artery (A1 or A2).2–4 Endovascular thrombectomy has been shown to decrease disability, improve functional outcomes, and decrease mortality over standard medical management for anterior LVOs and is time-sensitive.5 6 Previous studies have suggested worse outcomes with increased time to reperfusion and have even estimated the pace of neural circuitry loss in human ischemic stroke over time.7 8 LVOs can produce large infarcts if timely intervention is not performed to salvage ischemic brain tissue. Hence, finding LVOs early using CT angiography (CTA) is paramount.9–11 Given the time-sensitive nature of these interventions, computer automation may aid in the early detection of these LVOs, and multiple software packages have been released for the early detection of these occlusions.12 13 Previous studies have shown that these software systems can process an image in less than 8 min and can decrease the door-to-needle time by earlier notification of the neurointerventionalist.13–15 Also, implementation of these software systems has been shown to decrease workflow metric times such as CTA to stroke team notification time and CTA to angiosuite arrival time, and in a spoke and hub model it has been shown to decrease transfer times.16–18 Still, false values may lead to alarm fatigue, delayed reads, and the ignoring of these systems altogether.19 In this article, we evaluate two software packages that use different algorithms, Viz LVO and Rapid LVO, for the automated detection of anterior LVOs. The Rapid LVO algorithm performs elastic image registration for bone removal followed by multiscale vessel enhancement filtering and then compares the hemispheric density of the sum of all voxels.15 In simpler terms, the software warps the CTA onto a known image and removes bone tissue based on location and brightness of the voxels, and then runs an algorithm to detect the vessels and calculates their brightness and density as compared with the other side of the brain. Viz LVO utilizes a three-dimensional convolutional neural network to segment images, and the extracted features are fed into a random forest classifier to detect LVOs.20 Viz LVO removes scans with metal and poor bolus timing and removes any bone tissue, and uses an artificial intelligence to segment the images and extract dozens of different variables associated with the vessels, like the length and brightness of the vessel. These data are sent into another machine learning algorithm and outputs a classification. Viz LVO required a set of prelabeled CTAs to train it to detect LVOs. Both of these algorithms require the developer of the software or a user to set a cut-off based on the output to say whether or not it has detected an LVO. These packages have previously been evaluated independently, but to our knowledge this is the first direct comparison of these two software packages on the same dataset at two comprehensive stroke centers.15 21–28

    Methods

    Stroke data were reviewed from two HCA Houston Healthcare facilities, HCA Houston Kingwood and HCA Houston Clear Lake, from January 2022 to January 2023 from multimodal CT with thin-slice CTA for suspected LVO. Two automated LVO detection software, Rapid LVO version 5.2.2 and Viz LVO version 1.16, were used to analyze these images, and an attending diagnostic neuroradiologist or neurointerventional radiologist interpreted the scans. The software was running simultaneously due to overlap in the contracts with these two companies. We manually abstracted the data from the radiology reads from 371 scans. We removed 11 samples that were missing data resulting in a cohort of 360 patients. We calculated summary statistics using the reads from Rapid LVO and Viz LVO, with the neuroradiologist interpretation as the gold standard. We also ran a McNemar test, a widely accepted non-parametric statistical test for determining whether one machine learning algorithm makes different proportions of classification errors than another on a machine learning task.29 We formed a contingency table of the correct and incorrect classifications of Viz LVO and Rapid LVO and performed a McNemar test on the table, which looks at the marginal homogeneity of the contingency table. High-grade atherosclerosis was determined by radiologist interpretation, and we manually reviewed all images for a Modified Woodcock Score of 2 or greater. To calculate the Modified Woodcock Score, the segment of each ICA from the petrous apex to the ICA bifurcation was qualitatively evaluated based on the thickness and continuity of the calcifications. These visual scores have been shown to correlate with quantitative measures of intracranial ICA calcification measures.30 The ejection fraction (EF) was determined based on ultrasounds obtained from certified echocardiography technicians and calculated by board-certified cardiologists using their preferred method. We also looked for coils, clips or other metal and intracranial hemorrhage. We also abstracted demographic data and past medical history from the electronic medical record. Variables included age, self-identified sex and race, diabetes, hypertension, atrial fibrillation, hyperlipidemia, congestive heart failure, prior stroke, and smoking. These became the independent variables for a multiple logistic regression to determine which variables predicted incorrect classification by these algorithms.

    Results

    A total of 360 participants were included with a mean (SD) age of 65±16.5 years old, with 160 males and a total of 47 LVOs confirmed by diagnostic or neurointerventional radiologists. Rapid LVO had a specificity of 0.85 and a sensitivity of 0.87, with a positive predictive value (PPV) of 0.46 and a negative predictive value (NPV) of 0.97. Viz LVO had a specificity of 0.96 and a sensitivity of 0.87, with a PPV of 0.75 and an NPV of 0.98. The Rapid LVO and Viz LVO confusion matrices are shown in figure 1. The McNemar test performed on the correct and incorrect classification of the two types of software showed a statistically significant difference between classifications by the two algorithms (P=0.00000031). The contingency table is shown in figure 2. This indicates that the Rapid LVO and Viz LVO have a different relative proportion of errors. Rapid LVO and Viz LVO incorrectly classified radiologist-confirmed LVOs in six of the scans. The incorrectly classified occlusions were located in the ICA, M1, and M2 on the left and right sides, and four of the incorrectly classified occlusions were common between the two algorithms.

    We took the data manually extracted from the electronic health record relating to demographics, comorbidities, atherosclerosis on imaging, and EF and performed a multiple logistic regression on the incorrect predictions of the algorithms. Viz LVO results are shown in table 1 and Rapid LVO results are shown in table 2. On Viz LVO, low EF (p=0.00125) and Modified Woodcock Score >1 (P=0.000198) were significant predictors of incorrect classification. When using Rapid LVO, lower EF (P=0.0286) and Modified Woodcock Score >1 (P=0.000000975) were also significant predictors of incorrect classification. The hospital where the scans were taken was not a statistically significant predictor of incorrect classification in either software package. Comorbidities and medical history like diabetes, hypertension, atrial fibrillation, hyperlipidemia, congestive heart failure, prior stroke, and smoking were not significant predictors of incorrect classification in either of these two software packages.

    View this table:
    Table 1

    Viz LVO multiple logistic regression results on incorrect classification

    Figure 1
    Figure 1

    Rapid and Viz LVO confusion matrices showing numbers of correct and incorrect classifications compared with radiologist interpretation. LVO, large vessel occlusion.

    Figure 2
    Figure 2

    Contingency table showing numbers of correct and incorrect classification of Viz LVO and Rapid LVO. LVO, large vessel occlusion; ELVO, emergent large vessel occlusion.

    View this table:
    Table 2

    Rapid LVO multiple logistic regression results on incorrect classification

    Discussion

    Computer automation using different methods like deep learning, artificial intelligence, and other techniques is an emerging tool in the field of neuroradiology and can be helpful in the detection of LVOs. LVOs are responsible for a large portion of the mortality and severe disability in patients who experience ischemic strokes, and their detection is time sensitive.1 This study retrospectively evaluated the performance of two automated LVO detection tools for patients at two comprehensive stroke centers who underwent multimodal CT for a suspected acute ischemic stroke. Of these emerging software packages, both Rapid and Viz have shown promise in prior studies. Previous versions of Rapid LVO version 4.9 have been evaluated independently compared with physicians in previous studies that found that the software has a sensitivity of 0.92, a specificity of 0.81, a PPV of 0.58, and an NPV of 0.97.26 A previous version of the Viz LVO algorithm had a sensitivity of 0.81, a specificity of 0.96, a PPV of 0.65, and an NPV of 0.99.22 To our knowledge, our study is the first to compare two modern versions of these software systems directly on the same image set. Health systems typically have one software or the other, but due to a crossover in contracts at two comprehensive stroke centers in our hospital system, we had the opportunity to evaluate both Rapid and Viz capabilities.

    In our retrospective performance analysis, Rapid LVO had a specificity of 0.85 and a sensitivity of 0.87, with a PPV of 0.46 and an NPV of 0.97. Viz LVO had a specificity of 0.96 and a sensitivity of 0.87, with a PPV of 0.75 and an NPV of 0.98. In other terms, out of every 100 scans Rapid LVO and Viz LVO correctly detected 11 LVOs, and they both missed six LVOs, but Rapid LVO incorrectly identified 13 of the scans as LVOs while Viz LVO incorrectly identified four of the scans as LVOs. The values obtained are consistent with previous performance evaluations of these two algorithms.22 26 Of the 47 scans that were determined to have LVOs, both Rapid LVO and Viz LVO detected 41 (87.23%) of them. Rapid LVO had a larger number of false positives than Viz LVO. Repeated false positive alarms can desensitize the users to these alarms, and many users may shut them off altogether.19 Each of these algorithms requires a cut-off, and there is a trade-off between sensitivity, specificity, NPV, and PPV. Different algorithms will have a different combination of these values, but these systems select cut-offs to limit the number of false negatives because a missed LVO can be catastrophic.

    We also performed an analysis of potential predictors of incorrect classification. We found that a lower EF predicted incorrect classification in both software packages. We also found that intracranial atherosclerosis, as defined by a Modified Woodcock Score >1, predicted incorrect classification in these software packages. The other variables we evaluated, such as age, race, prior stroke, smoking, atrial fibrillation, etc, were not significant predictors of incorrect classification. The site where the scans were performed did not predict an incorrect classification.

    There are some limitations to this study. The PPV increases as the prevalence increases, and the NPV is inversely related. Institutions with different thresholds to perform CTAs on patients and variations in disease prevalence will cause the PPV and NPV to vary across institutions. This may make it difficult to generalize results to other institutions. There is some inherent variation in CTA reads of steno-occlusion by radiologists, but we believe the reads by diagnostic and interventional neuroradiologists provide an acceptable gold standard. Also, we did not collect data on which brand of CT scanner was used to obtain the images. Previous studies have looked at whether the sensitivity and specificity changed across CT scanner brands, and it does not appear that the type of scanner had any significant impact on algorithm performance, but future studies could look more closely at this.13 We did not find intracranial hemorrhage (ICH) or metal (clips, coils, etc) to be a significant predictor of incorrect classification, but the number of patients with ICH and metal was small (n=27), and a step in the Viz LVO algorithm is to remove these samples. Also, Rapid LVO will likely add this step to their algorithm in future versions. A future study with larger numbers of ICH and LVO may be able to determine whether ICH impacts the detection of LVOs by these algorithms. EF and intracranial atherosclerosis are predictors, but there may be some confounding variables that we did not measure in this study. Additionally, we were not able to assess for alarm fatigue during this study and were unable to determine thresholds for alarm fatigue among team members.

    Conclusion

    Rapid LVO produced a significantly larger number of false positive values than Viz LVO. False positive values can be a source of alarm desensitization, leading to missed alarms or delayed responses. We hope these data will be used to provide insight into the causes and predictors of incorrect classification in these algorithms, like EF and intracranial atherosclerosis, and will improve future versions of these algorithms.

    Data availability statement

    Data are available upon reasonable request.

    Ethics statements

    Patient consent for publication

    Ethics approval

    HCA Houston Healthcare Kingwood Institutional Review Board has determined this retrospective research activity to be exempt or excluded from Institutional Review Board (IRB) oversight in accordance with current regulations and institutional policy. Our internal reference number for this determination is 2022-1055. There was no direct patient contact in performing this study. In addition, our patients sign a data usage form at registration related to data collection and utilization of their data for research. The research was overseen in our research protocol submitted for IRB review and research committee who holds monthly ethics reviews.

    References

    1. ↵
      2. Malhotra K ,
      3. Gornbein J ,
      4. Saver JL
      . Ischemic strokes due to large-vessel occlusions contribute disproportionately to stroke-related dependence and death: a review. Front Neurol 2017;8:651. doi:10.3389/fneur.2017.00651
    2. ↵
      2. Goyal M ,
      3. Demchuk AM ,
      4. Menon BK , et al
      . Randomized assessment of rapid endovascular treatment of ischemic stroke. N Engl J Med 2015;372:1019–30. doi:10.1056/NEJMoa1414905
      OpenUrlCrossRefPubMed
    3. ↵
      2. Saver JL ,
      3. Goyal M ,
      4. Bonafe A , et al
      . Stent-retriever thrombectomy after intravenous t-PA vs. t-PA alone in stroke. N Engl J Med 2015;372:2285–95. doi:10.1056/NEJMoa1415061
      OpenUrlCrossRefPubMed
    4. ↵
      2. Campbell BCV ,
      3. Mitchell PJ ,
      4. Kleinig TJ , et al
      . Endovascular therapy for ischemic stroke with perfusion-imaging selection. N Engl J Med 2015;372:1009–18. doi:10.1056/NEJMoa1414792
      OpenUrlCrossRefPubMed
    5. ↵
      2. Goyal M ,
      3. Menon BK ,
      4. van Zwam WH , et al
      . Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. Lancet 2016;387:1723–31. doi:10.1016/S0140-6736(16)00163-X
      OpenUrlCrossRefPubMed
    6. ↵
      2. Lin Y ,
      3. Schulze V ,
      4. Brockmeyer M , et al
      . Endovascular thrombectomy as a means to improve survival in acute ischemic stroke: a meta-analysis. JAMA Neurol 2019;76:850–4. doi:10.1001/jamaneurol.2019.0525
      OpenUrl
    7. ↵
      2. Saver JL
      . Time is brain--quantified. Stroke 2006;37:263–6. doi:10.1161/01.STR.0000196957.55928.ab
      OpenUrlAbstract/FREE Full Text
    8. ↵
      2. Khatri P ,
      3. Yeatts SD ,
      4. Mazighi M , et al
      . Time to angiographic reperfusion and clinical outcome after acute ischaemic stroke: an analysis of data from the Interventional Management of Stroke (IMS III) phase 3 trial. Lancet Neurol 2014;13:567–74. doi:10.1016/S1474-4422(14)70066-3
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      2. Powers WJ ,
      3. Rabinstein AA ,
      4. Ackerson T , et al
      . Guidelines for the early management of patients with acute ischemic stroke: 2019 update to the 2018 guidelines for the early management of acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2019;50:e344–418. doi:10.1161/STR.0000000000000211
      OpenUrlCrossRefPubMed
    10. ↵
      2. Sanossian N ,
      3. Fu KA ,
      4. Liebeskind DS , et al
      . Utilization of emergent neuroimaging for thrombolysis-eligible stroke patients. J Neuroimaging 2017;27:59–64. doi:10.1111/jon.12369
      OpenUrl
    11. ↵
      2. Vagal A ,
      3. Meganathan K ,
      4. Kleindorfer DO , et al
      . Increasing use of computed tomographic perfusion and computed tomographic angiograms in acute ischemic stroke from 2006 to 2010. Stroke 2014;45:1029–34. doi:10.1161/STROKEAHA.113.004332
      OpenUrlAbstract/FREE Full Text
    12. ↵
      2. Weyland CS ,
      3. Papanagiotou P ,
      4. Schmitt N , et al
      . Hyperdense artery sign in patients with acute ischemic stroke – automated detection with artificial intelligence-driven software. Front Neurol 2022;13:807145. doi:10.3389/fneur.2022.807145
    13. ↵
      2. Dehkharghani S ,
      3. Lansberg M ,
      4. Venkatsubramanian C , et al
      . High-performance automated anterior circulation CT angiographic clot detection in acute stroke: a multireader comparison. Radiology 2021;298:665–70. doi:10.1148/radiol.2021202734
      OpenUrlPubMed
    14. ↵
      2. Hassan AE ,
      3. Ringheanu VM ,
      4. Preston L , et al
      . Artificial intelligence – parallel stroke workflow tool improves reperfusion rates and door‐in to puncture interval. SVIN 2022;2. doi:10.1161/SVIN.121.000224
    15. ↵
      2. Amukotuwa SA ,
      3. Straka M ,
      4. Dehkharghani S , et al
      . Fast automatic detection of large vessel occlusions on CT angiography. Stroke 2019;50:3431–8. doi:10.1161/STROKEAHA.119.027076
      OpenUrlCrossRefPubMed
    16. ↵
      2. Hassan AE ,
      3. Ringheanu VM ,
      4. Rabah RR , et al
      . Early experience utilizing artificial intelligence shows significant reduction in transfer times and length of stay in a hub and spoke model. Interv Neuroradiol 2020;26:615–22. doi:10.1177/1591019920953055
      OpenUrlCrossRefPubMed
    17. ↵
      2. Matsoukas S ,
      3. Stein LK ,
      4. Fifi JT
      . Artificial intelligence-assisted software significantly decreases all workflow metrics for large vessel occlusion transfer patients, within a large spoke and hub system. Cerebrovasc Dis Extra 2023;13:41–6. doi:10.1159/000529077
      OpenUrlCrossRefPubMed
    18. ↵
      2. Elijovich L ,
      3. Dornbos Iii D ,
      4. Nickele C , et al
      . Automated emergent large vessel occlusion detection by artificial intelligence improves stroke workflow in a hub and spoke stroke system of care. J Neurointerv Surg 2022;14:704–8. doi:10.1136/neurintsurg-2021-017714
      OpenUrlAbstract/FREE Full Text
    19. ↵
      2. Woo M ,
      3. Bacon O
      . Alarm fatigue. Agency for Healthcare Research and Quality (US); 2020. Available: https://www.ncbi.nlm.nih.gov/books/NBK555522/
    20. ↵
      2. Murray NM ,
      3. Unberath M ,
      4. Hager GD , et al
      . Artificial intelligence to diagnose ischemic stroke and identify large vessel occlusions: a systematic review. J Neurointerv Surg 2020;12:156–64. doi:10.1136/neurintsurg-2019-015135
      OpenUrlAbstract/FREE Full Text
    21. ↵
      2. Rodrigues G ,
      3. Barreira CM ,
      4. Bouslama M , et al
      . Automated large artery occlusion detection in stroke: a single-center validation study of an artificial intelligence algorithm. Cerebrovasc Dis 2022;51:259–64. doi:10.1159/000519125
      OpenUrlCrossRefPubMed
    22. ↵
      2. Yahav-Dovrat A ,
      3. Saban M ,
      4. Merhav G , et al
      . Evaluation of artificial intelligence – powered identification of large-vessel occlusions in a comprehensive stroke center. AJNR Am J Neuroradiol 2021;42:247–54. doi:10.3174/ajnr.A6923
      OpenUrlAbstract/FREE Full Text
    23. ↵
      2. Sevilis T ,
      3. Figurelle M ,
      4. Avila A , et al
      . Abstract WP81: Validation of artificial intelligence to limit delays in acute stroke treatment and endovascular therapy (VALIDATE). Stroke 2023;54. doi:10.1161/str.54.suppl_1.WP81
    24. ↵
      2. Perry da Câmara C ,
      3. Rodrigues GM ,
      4. Barreira C , et al
      . Abstract WP61: Artificial intelligence for automated detection of large vessel occlusion strokes. Stroke 2020;51. doi:10.1161/str.51.suppl_1.WP61
    25. ↵
      2. Rodrigues GM ,
      3. Barreira C ,
      4. Froehler M , et al
      . Abstract WP71: Multicenter ALADIN: automated large artery occlusion detection in stroke imaging using artificial intelligence. Stroke 2019;50. doi:10.1161/str.50.suppl_1.WP71
    26. ↵
      2. Amukotuwa SA ,
      3. Straka M ,
      4. Smith H , et al
      . Automated detection of intracranial large vessel occlusions on computed tomography angiography. Stroke 2019;50:2790–8. doi:10.1161/STROKEAHA.119.026259
      OpenUrlCrossRefPubMed
    27. ↵
      2. Luijten SPR ,
      3. Wolff L ,
      4. Duvekot MHC , et al
      . Diagnostic performance of an algorithm for automated large vessel occlusion detection on CT angiography. J Neurointerv Surg 2022;14:794–8. doi:10.1136/neurintsurg-2021-017842
      OpenUrlAbstract/FREE Full Text
    28. ↵
      2. Schlossman J ,
      3. Ro D ,
      4. Salehi S , et al
      . Head-to-head comparison of commercial artificial intelligence solutions for detection of large vessel occlusion at a comprehensive stroke center. Front Neurol 2022;13:1026609. doi:10.3389/fneur.2022.1026609
    29. ↵
      2. Dietterich TG
      . Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput 1998;10:1895–923. doi:10.1162/089976698300017197
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      2. Subedi D ,
      3. Zishan US ,
      4. Chappell F , et al
      . Intracranial carotid calcification on cranial computed tomography. Stroke 2015;46:2504–9. doi:10.1161/STROKEAHA.115.009716
      OpenUrlAbstract/FREE Full Text

    Footnotes

    • X @jordantorres_md

    • Correction notice Since this article first published, the following has been added to the COI section. 'HCA Healthcare has a minority equity stake in VIZ.ai and also purchases products from the company for use in its facilities'.

    • Contributors ME, YA, AH, AD and RE all contributed to the design of the study and formation of the data dictionary. ME and YA were involved in data collection. AD performed the statistics. All authors contributed to interpreting these results. AD, CH, EP, and JT were involved in drafting the manuscript. AH, ME and YA performed revisions of this manuscript. All authors were involved in data interpretation and article revision. ME is the primary investigator and coordinated data collection at multiple sites. ME is the guarantor and accepts full responsibility for the finished work and the conduct of the study, had access to the data, and controlled the decision to publish.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Disclaimer This research was supported (in whole or in part) by HCA Healthcare and/or an HCA Healthcare affiliated entity. The views expressed in this publication represent those of the author(s) and do not necessarily represent the official views of HCA Healthcare or any of its affiliated entities.

    • Competing interests HCA Healthcare has a minority equity stake in VIZ.ai and also purchases products from the company for use in its facilities. AH is a consultant and a speaker for Viz.ai, which develops the Viz LVO algorithm that we tested, and he is also involved in a study sponsored by Viz.ai called the LVO Synchronise Study, which looks at the impact of a Viz LVO implementation on patient timing and outcomes. Our other authors do not have any direct conflicts of interest. In the interest of full disclosure, here is a list of conflicts of related companies that are in a related field that we do not believe directly impact this study. YA has stock in Sanavention inc. ME has stock in Galaxy therapeutics. AH is a Consultant/Speaker for Medtronic, Microvention, Stryker, Penumbra, Cerenovus, Genentech, GE Healthcare, Scientia, Balt, Viz.ai, Insera therapeutics, Proximie, NeuroVasc, NovaSignal, Vesalio, Rapid Medical, Imperative Care and Galaxy Therapeutics. AH is also a principal investigator of COMPLETE study – Penumbra, LVO SYNCHRONISE – Viz.ai, Millipede Stroke Trial - Perfuze, RESCUE - ICAD - Medtronic. AH is also on the Steering Committee/Publication Committee of SELECT, DAWN, SELECT 2, EXPEDITE II, EMBOLISE, CLEAR, ENVI, DELPHI, and DISTALS. He is also DSMB of the COMAND trial.

    • Provenance and peer review Not commissioned; externally peer reviewed.