Responses
Other responses
Jump to comment:
- Published on: 11 July 2019
- Published on: 11 July 2019
- Published on: 11 July 2019SPOT as a clinical decision aid
We read with interest the response to our manuscript on using machine learning to optimize elderly patient selection for endovascular thrombectomy (1). We acknowledge here, as the author reports, the limitation of SPOT being based on single center data, and the need for multicenter prospective validation of SPOT as next step in development. The author raises additional technical concerns that we do not necessarily view as applicable to this study.
First, we would like to stress the general limitations of artificial intelligence based techniques such as the overfitting and the data specific local optima problems. However, the specific comments brought by the author are not applicable in our case. First, studies on the number of events per predictor are applicable for logistic regressions (LRs) which is not used in the SPOT algorithm. In fact, our results show poor LR performance which is consistent with the rule of thumb of 1 to 10 referred to by the author. Hence, while serving as a good guidance for LR, the rule is not binding and more importantly it does not guarantee the generalization of the learned model. To further illustrate, classification models using convolutional neural networks have millions of parameters and are trained with datasets that, in most cases, do not have millions of samples in each group. However, these models have acceptable generalization capabilities and are tested using the data-split method. In SPOT, the model at its core is a regressi...
Show MoreConflict of Interest:
None declared. - Published on: 11 July 2019Response to: "Using machine learning to optimize selection of elderly patients for endovascular thrombectomy"
It is with great interest that we read the study of Alawieh et al(1), in which they developed a machine learning algorithm, called ‘SPOT’, to select stroke patients older than 80 years for endovascular therapy (EVT). Prediction modeling to optimize patient selection for EVT is an emerging topic of interest and we agree that predicting individual patient outcomes is increasingly important for decision making in medicine. However, we were surprised by the strong conclusions that were drawn by the authors, considering some serious limitations of the study.
First, the size of the training set is insufficient to develop a complex model with twelve predictor variables and many correlations. Only 22 patients had a good functional outcome, which means that the number of events per tested predictor variable is less than two. For the development of a reliable model, a sample size of at least ten events per variable is needed to minimize the risk of overfitting(2, 3). It has been suggested that even far more events per variable are needed to achieve stable predictions with machine learning techniques(4). Especially complex models developed on small sample sizes have a high risk of overfitting, resulting in unstable predictions and too optimistic model performance measures. The reported AUC of 0.92 is therefore very likely to be an overestimation.
Second, the SPOT algorithm provides a treatment advice based on the predicted outcome after treatment, without providing the...
Show MoreConflict of Interest:
Personal disclosures
FK None reported.
E. Venema, H.F. Lingsma, and D.W.J. Dippel were involved in the development of a prediction model (MR PREDICTS).
Venema E, Mulder MJHL, Roozenbeek B, Broderick JP, Yeatts SD, Khatri P, et al. Selection of patients for intra-arterial treatment for acute ischaemic stroke: development and validation of a clinical decision tool in two randomised trials. BMJ. 2017;357.