Article Text

Download PDFPDF
Original research
Feasibility study for use of angiographic parametric imaging and deep neural networks for intracranial aneurysm occlusion prediction
  1. Mohammad Mahdi Shiraz Bhurwani1,2,
  2. Muhammad Waqas1,3,
  3. Alexander R Podgorsak1,2,
  4. Kyle A Williams1,2,
  5. Jason M Davies1,3,4,
  6. Kenneth Snyder1,3,
  7. Elad Levy1,3,
  8. Adnan Siddiqui1,3,
  9. Ciprian N Ionita1,2,3
  1. 1 Canon Stroke and Vascular Research Center, Buffalo, New York, USA
  2. 2 Biomedical Engineering, University at Buffalo - The State University of New York, Buffalo, New York, USA
  3. 3 Neurosurgery, University at Buffalo School of Medicine and Biomedical Sciences, Buffalo, New York, USA
  4. 4 Biomedical Informatics, University at Buffalo,The State University of New York, Buffalo, New York, United States
  1. Correspondence to Dr Ciprian N Ionita, Biomedical Engineering, University at Buffalo, State University of New York, Buffalo, NY 14203, USA; cnionita{at}


Background Angiographic parametric imaging (API), based on digital subtraction angiography (DSA), is a quantitative imaging tool that may be used to extract contrast flow parameters related to hemodynamic conditions in abnormal pathologies such as intracranial aneurysms (IAs).

Objective To investigate the feasibility of using deep neural networks (DNNs) and API to predict IA occlusion using pre- and post-intervention DSAs.

Methods We analyzed DSA images of IAs pre- and post-treatment to extract API parameters in the IA dome and the corresponding main artery (un-normalized data). We implemented a two-step correction to account for injection variability (normalized data) and projection foreshortening (relative data). A DNN was trained to predict a binary IA occlusion outcome: occluded/unoccluded. Network performance was assessed with area under the receiver operating characteristic curve (AUROC) and classification accuracy. To evaluate the effect of the proposed corrections, prediction accuracy analysis was performed after each normalization step.

Results The study included 190 IAs. The mean and median duration between treatment and follow-up was 9.8 and 8.0 months, respectively. For the un-normalized, normalized, and relative subgroups, the DNN average prediction accuracies for IA occlusion were 62.5% (95% CI 60.5% to 64.4%), 70.8% (95% CI 68.2% to 73.4%), and 77.9% (95% CI 76.2% to 79.6%). The average AUROCs for the same subgroups were 0.48 (0.44–0.52), 0.67 (0.61–0.73), and 0.77 (0.74–0.80).

Conclusions The study demonstrated the feasibility of using API and DNNs to predict IA occlusion using only pre- and post-intervention angiographic information.

  • aneurysm
  • flow diverter
  • angiography
  • intervention
  • blood flow

Statistics from


Hemodynamic factors such as flow patterns, wall shear stress, and velocity play important roles in the initiation, growth, and rupture of intracranial aneurysms (IAs).1–4 The aim of endovascular treatments such as coil embolization5 and flow diversion6–11 are to occlude the IAs and reduce the risk of rupture by modifying the hemodynamics in the aneurysmal dome. These therapies reduce the mechanical stress exerted on the aneurysm wall, while catalyzing the endovascular clotting cascade.12 13 For IAs treated with a Pipeline Embolization Device (PED), occlusion may occur after several months, and minimum 6 months post-procedure follow-ups are required to assess the occlusion status of the IA. For IAs treated with a PED, 21.8% remain partially to completely unoccluded,14 and there is currently no accurate imaging-based diagnosis tool which may predict IAs at risk of poor outcome.

Digital subtraction angiography (DSA)15 16 is the standard diagnostic imaging technique for the evaluation of IAs. DSAs are used to assess morphology and semiquantitative intra-aneurysmal flow. There have been attempts to use this temporal information of contrast propagation for better diagnosis.17 The temporal and spatial contrast distribution may be used to perform angiographic parametric imaging (API) by recording each pixel intensity in an image sequence. This method uses time–density curves to calculate parameters such as mean transit time (MTT), time to peak (TTP), time to arrival (TTA), peak height (PH), and area under the time–density curve (AUC). These parameters may be used to analyze IA flow aspects,18 19 or to evaluate flow changes due to treatment.18 20–23

The challenge in the interpretation of API maps arises from incomplete knowledge of the correlation between the imaging biomarkers and the aneurysm’s blood flow conditions. Combining multiple pre- and post-treatment parameters creates complicated analyses with poorly understood relationships, making prediction of IA occlusion an ideal application for machine learning in general and deep neural networks (DNNs) in particular.

The main goals for this project were twofold. First, we aimed to determine whether we could predict the minimum 6-month follow-up IA occlusion using API on pre- and post-intervention angiographic scans in conjunction with DNNs. Second, we aimed to investigate whether various normalization approaches applied to the API data can reduce errors due to injection variability and x-ray view foreshortening.


Data collection

Patient data collection and analysis was approved by our institution’s Institutional Review Board. The study schema is shown in figure 1. DSA scans from 180 patients with IAs treated only with a PED were collected for this study. All cases were collected from one center and the procedures were performed by four neurosurgeons. Only cases with a follow-up scan at least 6 months from the date of treatment were considered.

Figure 1

Study inclusion schema. P refers to number of patients and N refers to number of intracranial aneurysms.

Due to image artifacts which prevented API analysis, 17 cases were excluded. In summary, 84.0% of cases had one aneurysm treated per intervention, 15.3% had 2%, and 0.6% had 3. In total, we used 190 IAs for our API analysis. The mean and median duration between treatment and follow-up were 9.8 and 8.0 months, respectively. For each IA, we recorded DSA sequences at three time points: pre-treatment, immediately post-treatment, and follow-up. Of the 190 IAs considered, 171 were on the internal carotid artery (ICA); of these, 3 were anterior choroidal, 53 cavernous, 8 cervical, 40 paraophthalmic, 31 supraclinoid (non-branch variants), 3 petro-cavernous, 1 terminus, 15 were on the posterior communicating artery (PComA), 17 were on the superior hypophyseal artery (SHA). Of the 19 IAs not on the ICA, 2 were on the anterior cerebral artery, 2 on the anterior communicating artery, 3 on the middle cerebral artery, 1 on the posterior cerebral artery, 5 on the posterior communicating artery, and 6 were vertebra-basilar.

All angiograms in each patient study were observed retrospectively and pre- and post-treatment DSAs with best visualization of IAs were used to generate an API database. Occlusion outcome binary labels (occluded/unoccluded) were attained by neurosurgery attendings from the follow-up DSAs. API maps were generated from DSA sequences in an Angiographic Parametric Imaging software (Canon Medical Systems, Tustin, California, USA). For each case, expert users drew regions of interest over the aneurysm sac and the main artery to compute average MTT, TTP, PH, and AUC values. Interuser variability was assessed by a single-tailed heteroscedastic t-test.

Injection variability and projection view variability reduction

Since the contrast injection rates during DSA acquisitions are dependent on the neurosurgeon, we needed to normalize the API parameters to reduce injection variability. Additionally, projection view may play an important role due to vessel foreshortening, where the contrast intensity in the angiogram correlates with the length of the x-ray pathway through structures. In order to reduce foreshortening error, we required a second normalization, which assumes pre- and post-treatment views are identical. Thus, we implemented a two-step normalization method proposed by Ionita et al.20 21 First, we divided pre- and post-treatment IA API values by the corresponding main artery API values measured at 2–3 cm proximal to the IA ostium. Next, we divided the post-treatment normalized API values by the corresponding pre-treatment normalized ones. To determine the normalization effect, subgroup analysis was carried out on all data at all instances: no normalization (un-normalized set), arterial normalization (normalized set), and post-/pre-treatment normalization (relative set).

Occlusion outcome predictor development

Individual API parameter use for IA occlusion prediction, without machine learning, limits the ability to use all parameters and their correlations. To address this multidimensional problem, we developed a DNN using Keras24 to predict IA occlusion outcomes as a binary output: occluded or unoccluded. The DNN architecture design was an iterative process based on various network parameters optimization such as convergence, predictive accuracy, and ROC metrics.

The final DNN architecture used contained three dense layers with 20, 60, and 20 output nodes in succession followed by a 20% dropout layer and a final dense layer with two nodes that made the prediction. A robust optimizer, Adadelta, that adapts learning rates based on a moving window of gradient updates was used as it automatically adjusts the learning rate as training progresses. DNNs were trained and tested on a single NVIDIA (Nvidia Corporation, Santa Clara, California, USA) P2000 Graphics Processor Unit (GPU).

The DNN was evaluated using two quantitative metrics: classification accuracy and receiver operating characteristic (ROC) curves. To test the network robustness and ensure that the results were not obtained due to a specific training-testing split, a Monte Carlo Cross Validation (MCCV)25 was conducted. Using this approach, the total dataset was randomly split 20 times into 70% (n=135) training and 30% (n=55) testing cohorts. Next, to prevent the DNN from overfitting, we used a method previously described which implements Gaussian noise to augment the training and testing datasets separately, thus tripling the initial dataset.26 The process of augmentation was performed separately on the training and testing cohorts to avoid contribution of training cases in the testing cohort. Also, in order to set a benchmark and compare the performance of the DNN, we trained and tested a logistic regression (LR) model on the relative augmented training and testing datasets.

Statistical analysis

Subgroup analysis was conducted on the individual API parameters (TTP, MTT, PH and AUC), normalization method, and treatment status as a function of occlusion outcome (occluded and unoccluded). Box and whisker plots were generated for each parameter and normalization method displaying the difference between occluded and unoccluded IAs. In addition, area under the ROC curve (AUROC) was computed for each API parameter as an individual predictor of IA occlusion outcome. P values were calculated using a single-tailed heteroscedastic t-test to check if there was a statistically significant difference in the DNN performance between the three subgroups (un-normalized, normalized, and relative).


API data collection and inter-reader variability

Raw API value analyses are displayed in the form of box and whisker plots for each API parameter and normalization method in figure 2. AUROCs for each API parameter as an individual IA occlusion predictor are shown in table 1. The p values for the inter-reader variability for API parameter extraction from the aneurysm sac and arterial inlet among the three expert users were statistically not significant: 0.12, 0.16, and 0.46.

Figure 2

Box and whisker plots for each API parameter and normalization method. For each API parameter, within each normalization method, there is overlap and not much separation of values between occluded and unoccluded. This indicates the prediction of occlusion outcome cannot be achieved by assessing each API parameter individually, no matter the normalization procedure employed. API, angiographic parametric imaging.

Table 1

Performance of the DNN in predicting occlusion outcome of intracranial aneurysms in the form of average accuracies and average AUROCs along with their SD and 95% CI

On observing the higher performance of PH individually as a predictor of IA occlusion outcome, the Youden index, which calculates the optimal classification threshold,27 was generated. For PH as a single occlusion outcome predictor, the optimal relative threshold was 0.87. At this value the diagnostic sensitivity and specificity for occlusion outcome were 0.92 and 0.57, respectively.

DNN performance

The DNN required approximately 29 s to train and, once trained, the DNN required 2 ms to make a prediction on each case. DNN prediction examples on an occluded and unoccluded IA are shown in figure 3. The DNN predicted with a probability of 81% that the aneurysm in case (A) of figure 3 will occlude; this prediction matched the label as that aneurysm did occlude in the follow-up. For the aneurysm in case (B) of figure 3, the DNN predicted with a probability of 99.8% that the aneurysm would not occlude, which matched the label as that aneurysm did not occlude in the follow-up.

Figure 3

The workflow to testing two specific cases is shown in (A) and (B). In both cases the DSA sequence is used to obtain API maps from which API parameter values are obtained. These values are run through the DNN which gives a probability of the IA occluding or not occluding. In case (A), the DNN predicted with 81% probability that the IA would occlude, and that was confirmed by the follow-up DSA sequence shown on the far right. In case (B), the DNN predicted with a 99.8% probability that the IA would not fully occlude, and this was confirmed by the follow-up DSA shown on the far right. Red arrows depict locations of aneurysms (or location where the aneurysms were before occlusion) on the DSA images and API maps. DSA, digital subtraction angiography; API, angiographic parametric imaging;, DNN, deep neural network; IA, intracranial aneurysm.

Average AUROCs and average accuracies along with the SD and 95% CI displaying DNN performance on the different normalization subgroups and augmentation datasets are displayed in table 1. A figure showing ROC curves indicating performance of the DNN on the three sub-groups (un-normalized, normalized and relative) is available as a supplementary file. P values were calculated, using a single-tailed heteroscedastic t-test, between DNN performances on the three subgroups. All p values were in the range of 2.8E-3 to 3.1E-14, and thus below the statistically significant threshold of 0.05.


In this technical efficacy study we established three main findings. First, we demonstrated that DNNs can use quantitative imaging information from API to predict IA occlusion immediately post-device deployment instead of waiting a period of minimum 6 months post-treatment. Second, we established that normalization of API data to the vessel inlet as well as to the pre-treatment API values is needed to reduce injection variability and foreshortening. Third, we showed that each API parameter cannot be used individually to make an accurate IA occlusion prediction.

In order to assess the ability of DNNs to predict occlusion of IAs, a DNN was trained and tested using API data derived from DSAs pre- and post-treatment with occlusion outcomes that were assigned by neurosurgeons from the follow-up scans. A 20-fold MCCV was conducted that allowed us to test the network robustness by providing average, SD, and 95% CI values for each evaluation metric. Results from the DNN on the un-normalized, normalized, and relative subgroups (table 1) were compared to test the technical feasibility of the normalization process employed. The DNN performed better on the relative subgroup than the normalized and un-normalized subgroups by 7.1% and 15.5% in terms of average accuracy and 0.1 and 0.29 in terms of average AUROC, respectively. The p values between the three subgroups were below the statistically significant threshold of 0.05, indicating a significant difference between performance on the un-normalized, normalized, and relative data. This indicates a significant advantage of using a two-step normalization process to predict the occlusion outcome of IAs using DNNs.

Using the analysis in figure 2, we confirmed that there was virtually no difference in the mean values between the occlusion outcomes for the un-normalized post-treatment API parameters. After the arterial normalization method, some separation was observed for PH and AUC; however, this separation is not enough, as seen by the performance of these parameters as individual predictors in table 1. For the relative set following foreshortening correction, further mean value separation was observed between occluded and unoccluded cohorts. In general, the AUROCs of each individual API parameter is less than the DNN AUROC (also shown in table 1) for each respective subgroup. The AUROC of PH as an individual predictor is higher than the other parameters, thus the Youden index indicating the optimal classification threshold and the sensitivity and specificity at that point were calculated. While the sensitivity had a high value of 0.92 at the optimal point, the specificity was only 0.57. Following this analysis, we concluded that, for occlusion outcome prediction in IAs treated with PED, we should avoid using API parameters as single diagnostic indicators. Instead, we should implement DNNs which may use the subtle correlations between these parameters to perform a more accurate prediction.

Both machine learning approaches, DNN and LR models, achieved similar performances when using the augmented training and testing relative dataset, which follows the trends previously reported in other medical applications.28 There was no statistically significant advantage to using either of the two methods (p value >0.05). However, simple LR models have lower flexibility and require linearly separable data, as shown by Dreiseitl et al,29 increasing the number of input features would require greater model flexibility and less reliance on the presence of linearly separable data which is ideal for DNNs. Thus, while the performances are close, use of the DNN may prove to be more advantageous than an LR model if additional quantitative angiographic parameters are included.

One limitation of this study is that it was performed at only one institution, and despite including all cases with known outcome from follow-up angiographies, only 190 IAs were eligible for analysis. Thus, we are limited to demonstrating only a technical efficacy of using machine learning to predict occlusion of IAs treated endovascularly. To compensate for the smaller dataset and to maximize diagnostic accuracy, augmentation was performed to increase both training and testing cohort sizes by a factor of 3. The reasoning behind augmenting the training and testing set, instead of only the training set, is that this process of augmentation does not create cases that will not be seen in the real world. This process of augmentation uses a Gaussian distribution to create API parameters which may be seen in cases from the angiographic suites. In order to ensure no contribution of cases from the training set on the testing set, we conducted the augmentation after splitting the data into the training and testing sets. The difference in performance with and without augmentation on the training set is reported in table 1. Better performance was observed when augmentation was done on both training and testing cohorts. Performances were lower in the other two augmentation scenarios, which can be attributed to the number of cases in the testing set for those scenarios being one-third that of the first scenario. Once the dataset has been expanded, we may move away from augmenting the testing cohort and even the training cohort without negatively impacting the model predictive accuracy. The augmentation process in general may introduce DNN overfitting. To demonstrate absence of this in our study, we conducted a 20-fold MCCV which demonstrated small variability in the predictive capability of our network (table 1).

Another limitation was the inclusion of only saccular aneurysms treated with PEDs. The DNN trained with this dataset will have lower performance in predicting occlusion outcome of different types of aneurysms such as fusiform aneurysms, or using other methods of treatment such as coiling, stent-coiling, and other methodologies that employ flow diversion. Future work will involve including API values computed using image data from a larger more diverse dataset. This will be used to expand the scope of this analysis to a wider range of patients.

One of the most impactful findings of this study was that a DNN might have the capability to be used in angiographic suites in real time to predict the occlusion outcome immediately after device deployment. Based on the network prediction, the neurosurgeon may choose to adjust the treatment by deploying a second device or to perform appropriate follow-ups with the patient. There are three technical features which might improve the algorithm accuracy. First, the use of an auto-injector to standardize the rate of injections between neurosurgeons across different institutions would allow for a consistency in the database and would remove the need for arterial normalization. Second, the use of a higher frame rate than the current 3 frames/s during acquisition of the pre- and post-treatment DSAs would allow better-sampled time–density curves to be obtained and thus increase the API accuracy. Third, maintaining the same C-arm angle during acquisition of the pre- and post-treatment sequences would prevent variability in the projection view, allowing for a more consistent relative API correction. These technical adjustments combined with an automatic aneurysm detection and radiomic feature extraction30 could provide a precise technology for decision support in the angiographic suites for the neurosurgeons.


In this technical efficacy study, we used a DNN to predict the IAs occlusion outcome using only intraoperative information. In addition, we implemented a normalization method to eliminate intra-operator and vessel architecture variability. The DNNs performed best when using the two-step normalization process showing a prediction accuracy of 77.9% and AUROC of 0.77. This is a novel attempt at analyzing API data using DNNs, and indicates the possibility to analyze correlation of API parameters with the blood flow and make predictions based on the analysis. In the future, such models may be able to be used in the operating room to predict whether the intervention performed was successful or not based on the API data measured pre- and post-treatment.



  • Contributors MMSB, ARP, and CNI conceived and designed the research. JMD, EL, KS, and AS performed all the clinical procedures.MMSB, MW and CNI collected the data. MMSB, CNI, and KAW analyzed the data. MMSB performed the statistical analysis. CNI handled funding and supervision. MMSB drafted the manuscript. All authors made critical revisions to the manuscript and reviewed the final version.

  • Funding This project was partially supported by Canon Medical Systems and the James H. Cummings Foundation.

  • Competing interests CNI: Equipment grant from Canon Medical Systems, support from the Cummings Foundation, NIH R21 grant. JMD: Research grant: National Center for Advancing Translational Sciences of the National Institutes of Health under award number KL2TR001413 to the University at Buffalo. Speakers’ bureau: Penumbra; Honoraria: Neurotrauma Science. Shareholder/ownership interests: RIST Neurovascular. KS: Consulting and teaching for Canon Medical Systems Corporation, Penumbra, Medtronic, and Jacobs Institute. Co-Founder: Neurovascular Diagnostics. EL: Shareholder/Ownership interests: NeXtGen Biologics, RAPID Medical, Claret Medical, Cognition Medical, Imperative Care (formerly the Stroke Project), Rebound Therapeutics, StimMed, Three Rivers Medical. National Principal Investigator/Steering Committees: Medtronic (merged with Covidien Neurovascular) SWIFT Prime and SWIFT Direct Trials. Honoraria: Medtronic (training and lectures). Consultant: Claret Medical, GLG Consulting, Guidepoint Global, Imperative Care, Medtronic, Rebound, StimMed. Advisory Board: Stryker (AIS Clinical Advisory Board), NeXtGen Biologics, MEDX, Cognition Medical, Endostream Medical. Site Principal Investigator: CONFIDENCE study (MicroVention), STRATIS Study—Sub I (Medtronic). AS: Research grant: NIH/NINDS 1R01NS091075 as a co-investigator for “Virtual Intervention of Intracranial Aneurysms”. Financial interest/investor/stock options/ownership: Amnis Therapeutics, Apama Medical, Blink TBI, Buffalo Technology Partners, Cardinal Consultants, Cerebrotech Medical Systems, Cognition Medical, Endostream Medical, Imperative Care, International Medical Distribution Partners, Neurovascular Diagnostics, Q’Apel Medical, Rebound Therapeutics Corp, Rist Neurovascular, Serenity Medical, Silk Road Medical, StimMed, Synchron, Three Rivers Medical, Viseon Spine. Consultant/advisory board: Amnis Therapeutics, Boston Scientific, Canon Medical Systems USA, Cerebrotech Medical Systems, Cerenovus, Corindus, Endostream Medical, Guidepoint Global Consulting, Imperative Care, Integra LifeSciences Corp, Medtronic, MicroVention, Northwest University–DSMB Chair for HEAT Trial, Penumbra, Q’Apel Medical, Rapid Medical, Rebound Therapeutics Corp., Serenity Medical, Silk Road Medical, StimMed, Stryker, Three Rivers Medical, VasSol, W.L. Gore & Associates. Principal investigator/steering comment of the following trials: Cerenovus LARGE and ARISE II; Medtronic SWIFT PRIME and SWIFT DIRECT; MicroVention FRED & CONFIDENCE; MUSC POSITIVE; and Penumbra 3D Separator, COMPASS, and INVEST.

  • Patient consent for publication Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement Data are available upon reasonable request.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.