Article Text
PDF
Abstract
Background There is limited evidence on the performance of emergent large-vessel occlusion (LVO) stroke screening tools when used by emergency medical services (EMS) and emergency department (ED) providers. We assessed the validity and predictive value of the vision, aphasia, neglect (VAN) assessment when completed by EMS and in the ED among suspected stroke patients.
Methods We conducted a retrospective study of VAN performed by EMS providers and VAN inferred from the National Institutes of Health Stroke Scale performed by ED nurses at a single hospital. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of VAN by EMS and in the ED for LVO and a combined LVO and intracerebral hemorrhage (ICH) outcome.
Results From January 2018 to June 2020, 1,547 eligible patients were identified. Sensitivity and specificity of ED VAN were similar for LVO (72% and 74%, respectively), whereas EMS VAN was more sensitive (84%) than specific (68%). PPVs were low for both EMS VAN (26%) and ED VAN (21%) to detect LVO. Due to several VAN-positive ICHs, PPVs were substantially higher for both EMS VAN (44%) and ED VAN (39%) to detect LVO or ICH. EMS and ED VAN had high NPVs (97% and 96%, respectively).
Conclusions Among suspected stroke patients, we found modest sensitivity and specificity of VAN to detect LVO for both EMS and ED providers. Moreover, the low PPV in our study suggests a significant number of patients with non-LVO ischemic stroke or ICH could be over-triaged with VAN.
- stroke
Data availability statement
Data are available upon reasonable request. The data are deidentified, retrospective from suspected stroke patients at a single center.
Statistics from Altmetric.com
Introduction
Large-vessel occlusions (LVOs) make up roughly one-third of acute ischemic strokes but, given their severity, account for over 60% of stroke-related morbidity and mortality.1 Endovascular thrombectomy (EVT) was recently demonstrated to significantly improve long-term outcomes for patients with emergent LVO stroke.2 Moreover, the benefit of EVT is highly time-dependent: faster reperfusion in even 15-min increments is associated with less disability.3 Currently, many eligible patients experience delays to EVT treatment prior to hospital arrival and during initial triage and evaluation.4 Therefore, early identification and triage of potential LVO is now an essential component of an efficient and effective stroke system of care.5 6
While emergency medical services (EMS) screening for stroke-like symptoms is an established practice in prehospital management, LVOs are less frequently encountered in the field than non-LVO ischemic strokes, and LVO symptoms can be indistinguishable from other types of stroke, making early identification of LVO a significant challenge. There are several stroke severity scales to rapidly assess suspected stroke patients for emergent LVO. However, no single LVO screen has demonstrated both high sensitivity and high specificity, with sensitivity ranging from 47%–73% and specificity from 78%–90%.7 Despite limited accuracy of these tools, their use by EMS to transport potential LVO patients directly to EVT-capable facilities and to activate EVT treatment teams prior to hospital arrival is associated with faster times to EVT.8 9
One quick and simple LVO screen is the vision, aphasia, neglect (VAN) assessment, which was designed to capture the major cortical signs of an LVO stroke.10 In a small study of 62 acute ischemic stroke patients from which it was developed, VAN had 100% sensitivity and 90% specificity to identify LVO when completed by emergency department (ED) nurses.10 However, a recent external validation of VAN used by EMS reported 81% sensitivity and 38% specificity, with the majority of intracerebral hemorrhages (ICH) found to be VAN positive.11 While initial results on the validation of VAN are encouraging, additional investigation is needed on the predictive value of VAN when implemented in a stroke system of care.
The objective of this study was to assess the real-world performance of VAN completed by EMS in the field and in the ED among suspected stroke patients including ischemic and hemorrhagic strokes and stroke mimics. To explore its use in stroke systems, we evaluated the VAN assessment to predict LVO in addition to a combined LVO or ICH outcome.
Methods
Study design and setting
We conducted a retrospective study with existing data on suspected stroke patients presenting to a single community hospital, by either EMS or private vehicle, between January 2018 and June 2020. The study setting was a large metropolitan area of about 1 million residents in central North Carolina and located within the Stroke Belt, a multi-state region in the south-east United States with a disproportionately higher burden of stroke morbidity and mortality than the rest of the country.12 The study hospital has been a Joint Commission-certified stroke center since 2011, was certified as a Thrombectomy-Capable Stroke Center in September 2019, and had EVT capabilities throughout the study period along with two other stroke centers in the region.
The local EMS system provides sole 9-1-1 response and ambulance transport services to an urban/suburban county located in central North Carolina, with an area of 854 square miles and a population of 1,000,000+ residents and receives about 110 000 calls per year. The system operates with centralized medical oversight at an advanced life support level, with at least one paramedic in each ambulance. Since 2009, an EMS triage and destination plan determines the appropriate facility to transport patients screening positive for acute stroke symptoms. In August 2018, the EMS agency added VAN to its prehospital stroke screening protocol. EMS personnel were trained on VAN across July, August, and September 2018 during usual continuing education sessions. The stroke triage and destination plan was updated to route VAN positive patients to the closest EVT-capable stroke center.
The data source for this study was a hospital-based quality improvement database of ‘Code Stroke’ patients, that is, those experiencing acute onset of stroke-like symptoms within 24 hours that led to an activation of the ED acute stroke clinical pathway and the hospital stroke team. Among acute ischemic stroke patients considered for EVT, CT angiography (CTA) imaging is ordered and completed concurrently with the initial neurologic assessment and intravenous alteplase treatment decision and administration. The stroke center team continually updates this database with Code Stroke patient demographics and medical history, initial stroke evaluation and treatment, ED diagnosis, and discharge disposition. Additionally, prehospital data from EMS records are linked directly to the hospital electronic health record system with the ESO (Austin, TX) Health Data Exchange. The University of North Carolina Institutional Review Board approved this study by expedited review.
Study population
We included ‘Code Stroke’ patients that were activated from the field by EMS or in the ED. Eligible patients arrived at the hospital by private vehicle or EMS. To represent the real-world performance of VAN, cancelled activations because the patient was not experiencing a stroke upon arrival were included. Patients were ineligible if less than 18 years' old, in-hospital strokes, or transferred from an outside hospital. Patients were excluded from the analysis if emergent LVO could not be determined. For the analysis of VAN assessed in the ED, we used data on patients starting in January 2018. Since VAN was added to the EMS stroke protocol in August 2018, the analysis of VAN assessed by EMS allowed for a ramp-up period and used data starting in September 2018.
Key measures
The presence of an emergent LVO was determined by initial CTA imaging findings. Specifically, LVO was defined by an acute thromboembolic occlusion of the terminal intracranial carotid, M1 and M2 segments of the middle cerebral, A1 segment of the anterior cerebral, P1 segment of the posterior cerebral, or basilar arteries. Patients with non-ischemic stroke diagnoses were classified as non-LVOs. These diagnoses included transient ischemic attack (TIA), ICH, subarachnoid hemorrhage, subdural hematoma, and stroke mimics (eg, hypoglycemia, seizure, migraine).
EMS VAN results were retrieved directly from linked electronic records of prehospital care. VAN started with an assessment of arm strength. If weakness was present in one or both arms, the assessment proceeded to identify visual disturbance, aphasia, or neglect, in order. A patient was VAN positive if arm weakness and one of the VAN deficits were detected. While ED providers at our stroke center are not required to complete a VAN, the ED nurses complete the National Institutes of Health Stroke Scale (NIHSS) for all Code Stroke patients on arrival. Therefore, ED VAN was inferred from corresponding items on the initial NIHSS (ie, Motor Arm, Vision, Best Language, Best Gaze, and Extinction and Inattention) completed by the ED nurse.
We extracted patient demographics (age, gender, and race), medical history (atrial fibrillation, coronary artery disease, carotid stenosis, diabetes, drugs/alcohol abuse, dyslipidemia, hypertension, obese/overweight, and previous stroke/TIA), time from last known normal to hospital arrival, initial NIHSS score, administration of intravenous alteplase, and final diagnosis. For EMS arrivals, we also categorized patients by EMS provider impression (stroke/TIA, altered mental status, generalized weakness, and other).
Statistical analysis
The study population was characterized with descriptive statistics. Sensitivity and specificity, and positive and negative predictive values (PPV and NPV) were computed for the ED and EMS VAN vs an LVO diagnosis and a combined LVO or ICH outcome. A 95% CI was calculated for each proportion using normal approximation. The agreement between the ED and EMS VAN findings was assessed with percent agreement and kappa.
The main analysis was conducted after excluding patients with missing assessments. In a sensitivity analysis of incomplete documentation of the VAN assessment by EMS, we used multiple imputation by chained equations to create 100 imputed datasets based on observed data.13 The imputation model included covariates that could be predictors of missing EMS VAN (age, gender, race, EMS provider impression, time from last known normal to arrival, and initial NIHSS score). The sensitivity and specificity of EMS VAN to detect LVO were computed for each imputed dataset. Pooled estimates and 95% CIs account for the uncertainty associated with the imputation. All statistical analyses were conducted in SAS 9.4.
Results
We identified 1708 suspected stroke patients, 161 of whom (9%) did not have an initial NIHSS completed in the ED or initial CTA imaging to determine whether there was an LVO, leaving 1547 patients for the ED VAN analysis (figure 1). Patients were on average 70 years' old, and 56% were female (table 1). The study population was predominantly Caucasian (73%) although a substantial proportion were African-American (23%). Most patients (75%) arrived by EMS, and almost half (49%) arrived over 6 hours since time last known normal. The mean initial NIHSS score was 7.1, and 62% of patients had a score less than 6. Although all patients were initially under suspicion of acute stroke, 45% had a non-stroke final diagnosis. Starting in September 2018, 873 patients were eligible for the EMS VAN analysis. Of these, 623 (71%) patients had complete documentation of the VAN assessment by EMS. Two hundred and fifty patients were excluded because of incomplete VAN documentation. Compared with the patients included in the EMS analysis, excluded patients scored on average 3.5 points higher on the initial NIHSS (online supplemental table S1).
Supplemental material
Selection of eligible patients for ED and EMS VAN analyses.
Patient characteristics of overall study population (n=1547)
Table 2 summarizes the accuracy and performance of ED and EMS VAN to detect LVO and combined LVO or ICH diagnoses. ED VAN had similar sensitivity (72%) and specificity (75%) for LVO. EMS VAN was more sensitive (84%) than specific (68%) for LVO. PPVs were low for both ED VAN (21%) and EMS VAN (26%) to detect LVO. Almost half of the false VAN positives were diagnosed as non-LVO ischemic stroke or ICH (table 3). Because of several VAN-positive ICHs, PPVs were substantially higher for both ED VAN (39%) and EMS VAN (44%) to detect LVO or ICH. The NPVs for ED and EMS VAN and both outcomes were greater than 90%.
Validity and predictive values of ED and EMS VAN
Final diagnoses of false positives from ED and EMS VAN
Because 29% of eligible patients did not have a documented EMS VAN assessment, multiple imputation with observed covariate data was used to handle missing data. Derived from imputed data, EMS VAN had a sensitivity of 78% (95% CI 70% to 87%) and specificity of 67% (95% CI 63% to 70%) for LVO (online supplemental table S2), which had overlapping 95% CIs with estimates from the main analysis.
Among 623 patients arriving by EMS, the agreement between the EMS and ED VAN results was 73% with a kappa of 0.41 (95% CI 0.33 to 0.48). Of 171 patients with discordant VAN results, 100 (58%) were VAN positive according to EMS but were VAN negative in the ED, whereas 71 (42%) were found to be negative by EMS but positive in the ED. The majority of LVO diagnoses (69%) were both EMS and ED VAN positive. However, 63% of patients who were both EMS and ED VAN positive were non-LVOs.
Discussion
We found modest sensitivity and specificity of VAN to detect LVO in both the prehospital and ED settings. Moreover, PPVs were relatively low in part due to the underlying low prevalence of LVO among our broad population of suspected stroke patients. The high false-positive rate for prehospital LVO screening has important implications for stroke systems implementing EMS triage and routing algorithms that bypass local hospitals for EVT-capable stroke centers. Our findings suggest a significant number of non-LVO patients would be over-triaged with an EMS bypass protocol, which is consistent with prior studies.14–16 In our study, about one-third of false positives were non-LVO ischemic stroke, suggesting several patients could incur delays to rapid intravenous thrombolysis if unnecessarily transported greater distances.17 Since VAN was designed to be sensitive to the major cortical symptoms, we found a substantial proportion of false positives were due to ICHs and other stroke mimics that often present with similar signs. As noted by others, patients with ICH may also benefit from preferential transport to advanced stroke centers with neurosurgical capabilities.18–20 In our study, VAN in the prehospital setting and in the ED had NPVs of greater than 95% for LVO, indicating early screening with VAN would result in substantially fewer false negatives than false positives. Therefore, our findings suggest the under-triage of LVO patients would be a lesser concern when implementing VAN into EMS or ED triage although strategies will still be needed to ensure under-triaged patients do not miss the opportunity for timely receipt of EVT.
Compared with our findings, a recent study by Birnbaum et al of the EMS use of VAN reported a comparable sensitivity (81%) and lower specificity (38%).11 Incompatibility with our results may be due to differences in the distribution of final diagnoses and the severity of stroke symptoms. Other stroke severity scales like the Rapid Arterial oCclusion Evaluation (RACE)21 and the Cincinnati Stroke Triage Assessment Tool (C-STAT)22 have also shown limited accuracy. Pooled C-STAT studies show a sensitivity of 56% and specificity of 82%, and RACE show a sensitivity of 69% and specificity of 81% among all suspected stroke patients.7 It is expected that scales with more cortical signs such as VAN and RACE would have a greater sensitivity but lower specificity than C-STAT and other scales focused on motor deficits.13 Our findings suggest VAN is at least as sensitive as RACE but less specific. The RACE scale involves a greater number of items and requires the computation of a score following the completion of each item. Further, it was found to have the lowest feasibility compared with other prehospital scales.23 Depending on the goals of the stroke system, the advantage of greater specificity provided by RACE needs to be considered in relation to the simplicity and ease of use of the VAN screening tool.
The low PPV of EMS VAN (26%) found in our study is similar to 29% reported by Birnbaum et al.11 A systematic review of published studies on LVO screens found reported PPVs between 35% and 50% over a wide range of LVO prevalence.7 Navalkele et al recently showed higher PPV for VAN in the ED (53%) although they studied only confirmed acute ischemic strokes with 31% having an LVO.24 Given the relatively low prevalence of LVO in our broad study population (9%), a lower PPV than those previously reported is not surprising. Conversely, the NPV was higher (97%) than other studies although patients who test negative but have an LVO, even if infrequently, need to be planned for so as not to delay EVT in these potentially eligible patients.
We found the agreement between EMS and ED VAN was moderate although the kappa suggests weak agreement. While the lack of agreement in our study could be due to comparing VAN by EMS to one inferred from the NIHSS in the ED, patients’ symptoms may have also changed between the prehospital and ED settings. Our findings suggest suspected stroke patients could be improving as well as declining over time. Initial evaluation of VAN observed perfect agreement between nurses and physicians,10 and recently, prehospital providers were found to agree over 90% on the Field Assessment Stroke Triage for Emergency Destination stroke severity scale.25 While the inter-rater reliability of VAN and other LVO screening tools may be high within a certain setting, our findings support additional investigation into how evolving symptoms from the field to hospital arrival could influence early LVO screening and triage.
Recent research, in addition to our study, has demonstrated the feasibility of implementing LVO screening tools, such as RACE and VAN, for EMS stroke triage.11 26 Still, it is critical for a stroke system to understand the magnitude of false positives and false negatives and their potential adverse impact on the use of medical resources and patient outcomes within the system’s local and regional context. For well-resourced regions with multiple EVT-capable centers such as our system, over-triage may be less of a concern, and strategies to minimize under-triage would be a higher priority. As additional clinical evidence and experience emerges on EVT for acute ischemic stroke due to more distal occlusion sites,27 early triage of EVT candidates based exclusively on symptom severity will likely lead to new challenges. To address the inherent limitations of current symptom-based screening methods, non-invasive brain monitoring technologies, such as ultrasound or electroencephalography devices, may offer novel solutions.28 29
Our study limitations include data only from a single stroke center and EMS system though a large sample (n=1547) from a demographically diverse region within the Stroke Belt of the United States. With data on 623 patients arriving by EMS, our study is the largest to date on VAN used in the prehospital setting. Further, our assessment of prehospital VAN was based on a single EMS system with uniform protocols and training. Because we inferred ED VAN from the NIHSS completed by the ED nurse, our results may not be generalizable to EDs that implement the VAN screening tool as it was designed. Also, by inferring VAN from the NIHSS, our evaluation of agreement between EMS and ED providers was not a direct comparison, and the results should be interpreted with caution. A substantial proportion of EMS arrivals did not have complete documentation to determine the VAN assessment, possibly because EMS providers may not have felt VAN was needed to appropriately triage and transport more mild or severe strokes. We observed patients excluded because of missing EMS VAN had on average higher NIHSS scores, suggesting more severe symptoms. Using the NIHSS score and other covariate data, multiple imputation analysis showed similar estimates of sensitivity and specificity. However, this method does not account for non-random missing data.
Conclusions
In this large study of a broad population of suspected stroke patients, VAN had comparable sensitivity and specificity to other stroke severity scales and screening tools for rapid detection of LVO stroke. For a population with a low prevalence of LVO, the modest specificity of VAN may result in a considerable number of false positives, which can have important clinical and economic implications for a stroke system of care. Given the limited accuracy of existing LVO screens, future research is required on the optimal use of these tools within stroke systems accounting for local resources and needs.
Data availability statement
Data are available upon reasonable request. The data are deidentified, retrospective from suspected stroke patients at a single center.
Ethics statements
Patient consent for publication
Ethics approval
The University of North Carolina Institutional Review Board approved this study by expedited review.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @epid_mehul
Contributors MDP, JT, OK-H: conception and design of the study. JT, JGG, JGW, EL, MB, MAM, CL, LM, TB, OK-H: acquisition of data. MDP, JT: analysis of data. MDP: drafting of the manuscript. All authors: interpretation of data, critical review, and final approval of the manuscript.
Funding MDP is supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant KL2TR002490.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.