Article Text

Download PDFPDF
Original research
Random expert sampling for deep learning segmentation of acute ischemic stroke on non-contrast CT
  1. Sophie Ostmeier1,
  2. Brian Axelrod2,
  3. Yongkai Liu1,
  4. Yannan Yu3,
  5. Bin Jiang1,
  6. Nicole Yuen4,
  7. Benjamin Pulli1,
  8. Benjamin F J Verhaaren5,
  9. Hussam Kaka1,
  10. Max Wintermark6,
  11. Patrik Michel7,
  12. Abdelkader Mahammedi1,
  13. Christian Federau8,
  14. Maarten G Lansberg4,
  15. Gregory W Albers4,
  16. Michael E Moseley1,
  17. Gregory Zaharchuk1,
  18. Jeremy J Heit9
  1. 1Department of Radiology, Stanford University, Stanford, California, USA
  2. 2Stanford University, Stanford, California, USA
  3. 3Department of Radiology, University of California San Francisco, San Francisco, California, USA
  4. 4Department of Neurology, Stanford University School of Medicine, Stanford, California, USA
  5. 5Department of Radiology, KU Leuven University Hospitals, Leuven, Belgium
  6. 6Department of Radiology, University of Virginia, Charlottesville, Virginia, USA
  7. 7Department of Neurology Service, University of Lausanne, Lausanne, Switzerland
  8. 8Institute for Biomedical Engineering, Zurich, Switzerland
  9. 9Department of Radiology, Neuroadiology and Neurointervention Division, Stanford University School of Medicine, Palo Alto, CA, USA
  1. Correspondence to Dr Jeremy J Heit, Radiology, Neuroadiology and Neurointervention Division, Stanford University School of Medicine, Stanford, California, USA; jheit{at}stanford.edu

Abstract

Background Outlining acutely infarcted tissue on non-contrast CT is a challenging task for which human inter-reader agreement is limited. We explored two different methods for training a supervised deep learning algorithm: one that used a segmentation defined by majority vote among experts and another that trained randomly on separate individual expert segmentations.

Methods The data set consisted of 260 non-contrast CT studies in 233 patients with acute ischemic stroke recruited from the multicenter DEFUSE 3 (Endovascular Therapy Following Imaging Evaluation for Ischemic Stroke 3) trial. Additional external validation was performed using 33 patients with matched stroke onset times from the University Hospital Lausanne. A benchmark U-Net was trained on the reference annotations of three experienced neuroradiologists to segment ischemic brain tissue using majority vote and random expert sampling training schemes. The median of volume, overlap, and distance segmentation metrics were determined for agreement in lesion segmentations between (1) three experts, (2) the majority model and each expert, and (3) the random model and each expert. The two sided Wilcoxon signed rank test was used to compare performances (1) to 2) and (1) to (3). We further compared volumes with the 24 hour follow-up diffusion weighted imaging (DWI, final infarct core) and correlations with clinical outcome (modified Rankin Scale (mRS) at 90 days) with the Spearman method.

Results The random model outperformed the inter-expert agreement ((1) to (2)) and the majority model ((1) to (3)) (dice 0.51±0.04 vs 0.36±0.05 (P<0.0001) vs 0.45±0.05 (P<0.0001)). The random model predicted volume correlated with clinical outcome (0.19, P<0.05), whereas the median expert volume and majority model volume did not. There was no significant difference when comparing the volume correlations between random model, median expert volume, and majority model to 24 hour follow-up DWI volume (P>0.05, n=51).

Conclusion The random model for ischemic injury delineation on non-contrast CT surpassed the inter-expert agreement ((1) to (2)) and the performance of the majority model ((1) to (3)). We showed that the random model volumetric measures of the model were consistent with 24 hour follow-up DWI.

  • thrombectomy
  • stroke
  • CT

Data availability statement

Data are available upon reasonable request.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Data availability statement

Data are available upon reasonable request.

View Full Text

Footnotes

  • Twitter @SophieOstmeier, @stanfordNRAD, @JeremyHeitMDPHD

  • Contributors Guarantors of the integrity of the entire study: JJH. Study concepts/study design or data acquisition or data analysis/interpretation: all authors. Manuscript drafting or manuscript revision for important intellectual content: all authors. Approval of final version of submitted manuscript: all authors. Literature research: SO and JJH. Clinical studies: SO and JJH. Statistical analysis: SO and BA. Manuscript editing: SO, BA, MGL, GZ, and JJH.

  • Funding The German Research Foundation (DFG) supported this project through the Walter-Benjamin fellowship (ID: 517316550).

  • Competing interests CF: founder and CEO, and equity of AI Medical AG. GWA: compensation from iSchemaView for consultant services; compensation from Genentech for consultant services; and stock holdings in iSchemaView. GZ: co-founder, equity of Subtle Medical, funding support GE Healthcare, and consultant for Biogen. JJH: consultant for Medtronic and MicroVention, and member of the medical and scientific advisory board for iSchemaView.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.