Responses

Download PDFPDF
Original research
Using machine learning to optimize selection of elderly patients for endovascular thrombectomy
Compose Response

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Author Information
First or given name, e.g. 'Peter'.
Your last, or family, name, e.g. 'MacMoody'.
Your email address, e.g. higgs-boson@gmail.com
Your role and/or occupation, e.g. 'Orthopedic Surgeon'.
Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.
Statement of Competing Interests

PLEASE NOTE:

  • Responses are moderated before posting and publication is at the absolute discretion of BMJ, however they are not peer-reviewed
  • Once published, you will not have the right to remove or edit your response. Removal or editing of responses is at BMJ's absolute discretion
  • If patients could recognise themselves, or anyone else could recognise a patient from your description, please obtain the patient's written consent to publication and send them to the editorial office before submitting your response [Patient consent forms]
  • By submitting this response you are agreeing to our full [Response terms and requirements]

Vertical Tabs

Other responses

Jump to comment:

  • Published on:
    SPOT as a clinical decision aid
    • Ali Alawieh, Graduate Researcher Department of Neurosurgery, Medical University of South Carolina
    • Other Contributors:
      • Fadi Zaraket, Associate Professor in Electrical and Computer Engineering
      • Mohamed Baker Alawieh, Graduate Assistant
      • Alejandro Spiotta, Neurosurgeon and Neuroendovascular surgeon

    We read with interest the response to our manuscript on using machine learning to optimize elderly patient selection for endovascular thrombectomy (1). We acknowledge here, as the author reports, the limitation of SPOT being based on single center data, and the need for multicenter prospective validation of SPOT as next step in development. The author raises additional technical concerns that we do not necessarily view as applicable to this study.

    First, we would like to stress the general limitations of artificial intelligence based techniques such as the overfitting and the data specific local optima problems. However, the specific comments brought by the author are not applicable in our case. First, studies on the number of events per predictor are applicable for logistic regressions (LRs) which is not used in the SPOT algorithm. In fact, our results show poor LR performance which is consistent with the rule of thumb of 1 to 10 referred to by the author. Hence, while serving as a good guidance for LR, the rule is not binding and more importantly it does not guarantee the generalization of the learned model. To further illustrate, classification models using convolutional neural networks have millions of parameters and are trained with datasets that, in most cases, do not have millions of samples in each group. However, these models have acceptable generalization capabilities and are tested using the data-split method. In SPOT, the model at its core is a regressi...

    Show More
    Conflict of Interest:
    None declared.
  • Published on:
    Response to: "Using machine learning to optimize selection of elderly patients for endovascular thrombectomy"
    • F.C.C. Kremers, Research master student Erasmus University Medical Center Rotterdam
    • Other Contributors:
      • E. Venema, PhD Candidate
      • H.F. Lingsma, Statistician
      • D.W.J. Dippel, Neurologist

    It is with great interest that we read the study of Alawieh et al(1), in which they developed a machine learning algorithm, called ‘SPOT’, to select stroke patients older than 80 years for endovascular therapy (EVT). Prediction modeling to optimize patient selection for EVT is an emerging topic of interest and we agree that predicting individual patient outcomes is increasingly important for decision making in medicine. However, we were surprised by the strong conclusions that were drawn by the authors, considering some serious limitations of the study.

    First, the size of the training set is insufficient to develop a complex model with twelve predictor variables and many correlations. Only 22 patients had a good functional outcome, which means that the number of events per tested predictor variable is less than two. For the development of a reliable model, a sample size of at least ten events per variable is needed to minimize the risk of overfitting(2, 3). It has been suggested that even far more events per variable are needed to achieve stable predictions with machine learning techniques(4). Especially complex models developed on small sample sizes have a high risk of overfitting, resulting in unstable predictions and too optimistic model performance measures. The reported AUC of 0.92 is therefore very likely to be an overestimation.

    Second, the SPOT algorithm provides a treatment advice based on the predicted outcome after treatment, without providing the...

    Show More
    Conflict of Interest:
    Personal disclosures
    FK None reported.
    E. Venema, H.F. Lingsma, and D.W.J. Dippel were involved in the development of a prediction model (MR PREDICTS).

    Venema E, Mulder MJHL, Roozenbeek B, Broderick JP, Yeatts SD, Khatri P, et al. Selection of patients for intra-arterial treatment for acute ischaemic stroke: development and validation of a clinical decision tool in two randomised trials. BMJ. 2017;357.