We read with interest the response to our manuscript on using machine learning to optimize elderly patient selection for endovascular thrombectomy (1). We acknowledge here, as the author reports, the limitation of SPOT being based on single center data, and the need for multicenter prospective validation of SPOT as next step in development. The author raises additional technical concerns that we do not necessarily view as applicable to this study.

First, we would like to stress the general limitations of artificial intelligence based techniques such as the overfitting and the data specific local optima problems. However, the specific comments brought by the author are not applicable in our case. First, studies on the number of events per predictor are applicable for logistic regressions (LRs) which is not used in the SPOT algorithm. In fact, our results show poor LR performance which is consistent with the rule of thumb of 1 to 10 referred to by the author. Hence, while serving as a good guidance for LR, the rule is not binding and more importantly it does not guarantee the generalization of the learned model. To further illustrate, classification models using convolutional neural networks have millions of parameters and are trained with datasets that, in most cases, do not have millions of samples in each group. However, these models have acceptable generalization capabilities and are tested using the data-split method. In SPOT, the model at its core is a regression model with continuous output. More importantly, while the overfitting concern is a valid one with the high area under a curve, SPOT was tested in an adequate fashion using data-splitting, a well-accepted validation scheme to test the generalization capability of the model, and thus detect overfitting. In fact, studies discussing events per variable rule for logistic regression use data-splitting as one of the validation method (2). The approach used for testing SPOT using a prospective data not part of the training is the most stringent approach to test a model. In fact, state of the art machine learning model evaluations use the data-splitting technique, and consider the performance of the testing data as the golden metric to judge upon the model’s generalization and over-fitting (3,4).

While we agree with the fact that machine learning are data hungry, the size of the dataset is highly dependent on the problem at hand. For example, in sparse regression models, the number of samples in the training dataset can be orders of magnitude smaller than that of the parameters. However, sparse regression models joined with proper training techniques are able to generalize to unseen data (5). Again, this ability is tested using the data-split method which was used to test SPOT.

Further, we emphasize that although SPOT returns a continuous output of mRS scores, the tool will specifically report grouped outcomes into mRS 0-2 and mRS 3-6. The decision to choose this dichotomy of outcomes was to be consistent with clinical trials that predominantly report functional independence as outcome measure to guide interventions even when elderly patients were included. In response to the concern about returning probability for outcomes, and as stated in the manuscript, when SPOT returns poor outcome prediction, its negative predictive value was 95.2% which represents the probability for a patient that screened negative to have a poor outcome and this probability is reported in the text. In the current form, SPOT does not return a probability for every mRS score.

Finally, while we agree that multicenter data is needed to additionally validate SPOT as a tool as stated in the manuscript, the current version of SPOT does meet technical requirements for a validated tool. We do stress again that “SPOT is designed to aid clinical decision of whether to undergo ET in elderly patients” (1), and not a stand-alone tool.

References
1. Alawieh A, Zaraket F, Alawieh MB, Chatterjee AR, Spiotta A. Using machine learning to optimize selection of elderly patients for endovascular thrombectomy. J Neurointerv Surg. 2019.
2. Harrell FE, Jr., Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361-87.
3. Kohavi, Ron. "A study of cross-validation and bootstrap for accuracy estimation and model selection." Ijcai. Vol. 14. No. 2. 1995.
4. Arlot, Sylvain, and Alain Celisse. "A survey of cross-validation procedures for model selection." Statistics surveys4 (2010): 40-79.
5. Donoho, David Leigh, et al. Sparse solution of underdetermined linear equations by stagewise orthogonal matching pursuit. Department of Statistics, Stanford University, 2006.

We read with interest the article by Soize et al. “Can early neurological improvement after mechanical thrombectomy be used as a surrogate for final stroke outcome?”[1] Based on their results, the authors concluded that early neurological improvement (ENI) 24 hours after thrombectomy is a straightforward surrogate of long-term outcome. However, all patients in this study were treated with conscious sedation (CS), and not general anesthesia (GA). The residual effects of GA may mask ENI and limit its utility as a surrogate for long-term outcome.[2]

We performed a similar analysis of patients enrolled in a prospective single-center registry. The ability of ENI to predict 3-month functional independence was assessed by the area under the receiver operating characteristic curve (AUC) and compared using the independent-samples Hanley test. Multivariable linear regression assessing the relationship between anesthetic technique and ENI was also performed. The analysis received ethics approval.

291 patients were treated with thrombectomy, with 261 (89.7%) procedures performed with GA, and 30 (10.3%) with CS. All patients were de-sedated and extubated more than 12 hours before 24-hour National Institutes of Health Stroke Scale assessment. 174 (59.8%) patients achieved 3-month functional independence. Baseline and procedural characteristics did not differ between GA and CS patients (all P>0.05). ENI demonstrated better prognostic ability in CS (AUC 0.91, 95% confidence interval [CI], 0.80-1.00) than it did with GA treated patients (AUC 0.73, 95% CI, 0.67-0.80; P=0.008). Multivariable regression showed that GA was independently associated with attenuated ENI (P=0.03).

Our findings are in agreement with Soize et al, in that ENI does seem to predict long-term outcome in thrombectomy patients treated with CS, with comparable AUCs (0.91 and 0.93 respectively).[1] However, our results also suggest that ENI is worse at predicting long-term outcome following thrombectomy performed with GA. Furthermore, GA was independently associated with attenuated ENI, suggesting that GA might mask early neurologic recovery. These trends would appear to be in agreement with the SIESTA trial, which reported a greater likelihood of achieving 3-month functional independence in GA than CS patients, in the absence of significant differences in ENI.[3] Post-hoc analysis of SIESTA showed that propofol dose during thrombectomy was independently associated with reduced ENI.[2] The possible mechanisms for GA attenuating ENI may include the residual pharmacological effects of benzodiazepines, opioids, neuromuscular blockers, and intravenous or volatile anesthetic agents, or transient complications of endotracheal intubation such as ventilator-associated complications.[4]

[1] Soize S, Fabre G, Gawlitza M, et al. Can early neurological improvement after mechanical thrombectomy be used as a surrogate for final stroke outcome? J. Neurointerv. Surg. 2019;11(5):450-454.

[2] Schönenberger S, Uhlmann L, Ungerer M, et al. Association of Blood Pressure With Short- and Long-Term Functional Outcome After Stroke Thrombectomy: Post Hoc Analysis of the SIESTA Trial. Stroke. 2018;49:1451–1456.

[3] Schönenberger S, Uhlmann L, Hacke W, et al. Effect of Conscious Sedation vs General Anesthesia on Early Neurological Improvement Among Patients With Ischemic Stroke Undergoing Endovascular Thrombectomy: A Randomized Clinical Trial. JAMA. 2016;316:1986–1996.

[4] Sinclair RCF, Faleiro RJ. Delayed recovery of consciousness after anaesthesia. Continuing Education in Anaesthesia, Critical Care & Pain. 2006;6:114–118.

We had an opportunity to read the article by Lakomkin et al regarding systematic literature review of LVO prevalence. Since one of our studies is part of this review we feel compelled to comment on the paper. We do appreciate the authors’ efforts in conducting this analysis which is important in understanding the burden of disease – but, with respect offer some criticisms. The major limitation of the paper which the authors recognize is the heterogeneity of the included studies. Unfortunately, this limitation is so critical that it yields unreliable information at best and misleading at worst.

The paper intends to study the prevalence of large vessel strokes. However, apart from a couple of population based studies in their review, the rest are a heterogenous mix describing an LVO rate from very selective cohorts of patients from single centers. Several are centered around validation of clinical scales for detecting LVOs. The key features of a population based study include a defined catchment population, access to a large part of that population and a reliable marker of disease. Without these a “prevalence” constitutes a report of a center’s experience of disease rate as it pertains to their patient intake. While still valuable it is not an estimation of the disease burden in the population that the center serves unless an overwhelming majority of that population comes to that center.

The authors determine an average rate of about 30% LVO amongst acute ischemic stroke (AIS) patients based on their review. The critical factor here is the denominator, i.e. the number of AIS patients from which the LVO rate is derived. It can be misrepresentative to extrapolate disease rate to a larger denominator derived from a different methodology. For instance, the oft quoted denominator of about 700,000 ischemic strokes is based on population studies using very specific ICD discharge codes in well-defined populations. Examples include the GCNKSS and the BASIC projects. Unless a study uses the same discharge codes in its methodology for determining the AIS denominator it cannot transplant LVO percentages calculated from its selective cohort to the larger denominator and derive absolute numbers of disease. For instance, a 30% LVO rate in a cohort of patients assessed as having an ischemic stroke by the EMS is not the same as 30% LVO rate amongst all AIS based on specific ICD discharge codes. Using the percentage from one cohort and applying to a different denominator could yield inaccurate absolute numbers.

The study from our center that is included in this analysis was designed to capture the LVO incidence in a unique well defined population of which the vast majority (85%) was served by our hospital system based on each county data reported to the DHHR. Thus, similar to the studies used to derive the total AIS estimates (e.g. GCNKSS, BASIC) we had a defined population, had significant access to the population and used the same ICD discharge codes to determine the denominator as in those studies. We also used a reliable marker of LVO, i.e. CTA performed on every AIS patient as identified by these codes. In our first paper an LVO was defined as ICA-T, MCA and BA to restrict it predominantly to the occlusion sites considered as LVO in the major clinical trials. In our second paper, not included in this analysis, we separately determined an incidence of M2 occlusions and combined with ICA-T, M1, BA this yields a rate of 16% (95CI 14-17). In our population, this gives an incidence of 31 (95CI 26-35)/100,000 people/year. Our second paper also includes a chart showing the location of occluded sites for other vessels not considered as LVOs i.e. ACA, PICA, SCA, PCA etc.

The authors comment on our inclusion of TIA codes in the denominator. We did that because it had been part of previous population studies evaluating AIS incidences. TIA comprised at most 1% of our denominator and if we exclude these patients, the incidence of 16% LVO does not change by more than a percentage point. Another variable to consider in a single center report is that a tertiary level comprehensive center may get transfers of sicker stroke patients from referring hospitals which can further skew the denominator and hence the derived LVO rate.

In summary, it is important to differentiate between population studies and single center experiences – especially when compiling these in a systematic review. It is critical when extrapolating disease incidence from one center’s report to the national disease burden that the methodology of deriving the larger denominator is the same. Nonetheless we do appreciate this review and commend the authors on their efforts. This highlights the need for collaborative efforts to collect population data based on a uniform methodology. These efforts should include state and federal registers that collect health data based on specific disease codes. Such collective ventures can provide more realistic estimates of the LVO burden and help shape systems of care.