Background and purpose To evaluate factors influencing the confidence of management recommendation for unruptured intracranial aneurysms (UIAs) and to assess the ability of neurointerventionalists to predict procedure-related neurological complications compared with a 3-point risk score.
Materials and methods Twenty-eight neurointerventionalists were asked to evaluate digital subtraction angiographies examinations of patients with UIAs by determining the best management approach, their level of confidence in their management recommendation, and estimating the risk of procedure-related neurological complications. Knowledge and experience in interventional neuroradiology (INR) of each participant were assessed.
Results Reliability was moderate regarding any treatment recommendation (ICC=0.49) and low regarding the estimation of risk of complications (ICC=0.38). The recommendation of clipping was less likely with more experience in INR (OR=0.6) and more likely with increasing knowledge (OR=1.7). Odds of recommending WEB device were lower with more experience in INR (OR=0.6), higher in patients with multiple aneurysms (OR=3.6) and increasing neck width (OR=2.7). The recommendation of stent-assisted coiling was more likely with increasing neck width (OR=2.4) and when cerebral ischemic comorbidities were present (OR=2.9). The participants were significantly worse than the risk score (mean area under the curve of 0.53) and not better than random guess in predicting complications. Neither knowledge nor experience in INR was significantly associated with the participants’ ability to predict neurological complications.
Conclusions Our study shows a moderate interrater reliability of treatment recommendations of UIAs. Confidence in treatment recommendation varied significantly according to recommended treatments. Overall performance in predicting neurological complications was worse than the risk score and not better than random guess.
- flow diverter
Statistics from Altmetric.com
Unruptured intracranial aneurysms (UIAs) are seen in nearly 3% of the adult population, with an increasing prevalence as intracranial vascular imaging is used more frequently.1 The endovascular treatment options for UIAs have increased substantially in recent decades. However, procedure-related neurological complications are common after endovascular treatment of UIAs with 5%.2 3 To this date, in the absence of high-level evidence from clinical trials, management recommendations are mainly based on expert opinion. Little is known about the agreement and confidence of neurointerventionalists in their management recommendation. Recent studies reported substantial variation in the management of UIAs among clinicians.4 5 To guide management and decision making in patients with UIAs, the unruptured intracranial aneurysm treatment score (UIATS) was developed by 69 multidisciplinary specialists.6 However, this scoring system has not been broadly adopted and a recent study comparing the UIATS' recommendations with the real-world cerebrovascular practice showed several differences.7 Medical decision making plays an important role in neurovascular interventions, and studies analyzing this multifactorial and complex process are limited. The decision process is affected by a variety of factors starting with personal experiences and prior training ranging to cognitive biases and medicolegal considerations.8
We aimed to assess the interrater reliability of management recommendations and to evaluate factors influencing the management recommendation as well as the confidence of neurointerventionalist in their treatment recommendation. As the risks of aneurysmal rupture have to be compared with the risk associated with the intervention, we aimed to assess the ability of neurointerventionalists to predict procedure-related neurological complications compared with a recently developed score to quantify individual patient risk.2 Moreover, we analyzed factors influencing the ability of risk prediction.
Materials and methods
Twenty-eight participants of the Advanced Course in Endovascular Interventional Neuroradiology of the European Society of Neuroradiology, held in Hamburg from February 27 to March 1 2017 agreed to participate in this study. The study was approved by the local ethics committee (Ethik-Kommission Ärztekammer Hamburg, WF-030/14), and the requirement for written informed consent was waived. Participants’ information and test scores were anonymized and de-identified before analysis.
The study was performed in the format of a teaching seminar featuring case discussions as part of the curriculum of the Advanced Course. Therefore the time frame for the scientific content needed to be limited.
Assessment of the participants’ experience and knowledge
To assess the experience in interventional neuroradiology (INR), the participants were invited to answer an online questionnaire prior to the course (see Supplementary Figure 2). This online survey was a multiple-choice questionnaire with the opportunity to provide a commentary for each question. It consisted of 10 questions regarding their qualifications as well as the number of procedures they assisted or performed as a primary operator in aneurysm embolization, endovascular treatment of arteriovenous malformations or dural arteriovenous fistulas, and mechanical thrombectomy. As reported elsewhere in detail,9 work experience and aneurysm treatment experience were calculated from corresponding items and expressed as standardized scores (z scores with a mean of zero and an SD of !). To assess knowledge, the participants had to answer 15 multiple-choice questions concerning neuroembryology and neuroanatomy, pathophysiology, materials, and techniques as well as studies in interventional neuroradiology. Knowledge was also expressed as z scores.
Cases and measures
Digital subtraction angiographies (DSA) of 24 patients who had undergone endovascular treatment of UIAs at our institution between 2010 and 2017 were evaluated. Eligible were only cases that included at least two angiographic series in standard antero-posterior and lateral projection, and a 3D-rotational angiography pretreatment and two angiographic series in standard antero-posterior and lateral projection posttreatment along with relevant clinical data. Clinical data that were retrieved included patient’s age and sex, cerebral ischemic comorbidities, and existence of multiple aneurysms, as well as the aneurysm location, size, and neck width, and occurrence of procedure-related neurological complications. The ratio of cases with and without complications was held to be 1:1. Case characteristics are shown in figure 1.
For each case, the angiographic series were presented as movies to the participants. Participants were informed about the patient’s age and sex, cerebral ischemic comorbidities, and existence of multiple aneurysms, as well as the aneurysm location, size, and neck width. For each case, the participants answered four questions. First, participants were asked to determine the best therapeutic approach. Second, they were asked to indicate how confident they were in their treatment recommendation on a 6-step scale (complete guess, very uncertain, rather uncertain, rather certain, very certain, or completely certain). Third, if the presented patient was younger than 50 years of age, the participants were asked to recommend the best therapeutic approach if the patient was 70 years of age: in cases in which the presented patient was older than 50 years of age, they were asked to recommend the best therapeutic approach if the patient was 30 years of age. Fourth, they were told how the aneurysm was treated in reality and they were asked to estimate the risk for neurological complications after treatment (low, moderate, or high). A neurological complication was defined as any transient or permanent increase in the modified Rankin Scale score after aneurysm embolization. Finally, the posttreatment angiographic series were shown and the occurrence of procedure-related neurological complications was revealed.
Assessment of S-C-C risk score
Based on cerebral ischemic comorbidities, size, and core areas the recently developed 3-point S-C-C risk score was determined for each case. The S-C-C risk score is a simple scoring system to identify patients at high risk of neurological complications, including transient or permanent neurological deficits or death, after endovascular treatment of UIAs.2
Generalized linear mixed models (GLMM) and linear mixed models (LMM) were used to investigate the association of various variables with recommending any treatment (GLMM with binomial distribution and logit link), recommending specific treatments (GLMM with multinomial distribution and logit link), confidence of treatment recommendation (LMM), and estimated risk of complications (GLMM with binomial distribution and logit link). For all analyses, a cross-nested model was fitted accounting for variance both across raters (neuroradiologists) and across rated images (cases). Although a priori defined as an independent variable, ‘experience in years’ was removed from all analyses due to collinearity with ‘aneurysm treatment experience’ (r>0.7).
For quantifying interrater reliability for the treatment recommendation and the estimation of risk of complications, we calculated intraclass correlation coefficients (ICCs) from models with intercept only. ICCs were specified as the ratio between the variance across cases and the total variance.10 We used the following categories for interpreting ICCs: poor to fair (below 0.4); moderate (0.41–0.60); substantial (0.61–0.80); and almost perfect (0.81–1.00).11
Receiver operating characteristic curve analysis was performed to evaluate the discrimination of risk prediction (based on objective scores or neuroradiologist rating). We used the following categories for interpretation: 0.50 means chance (random concordance); below 0.50 worse than chance; and above 0.50 better than chance.12 13
For analyses of associations at the participant and the case level, we used bivariate correlation techniques (Pearson correlation coefficient or point-biserial correlation coefficient) and multivariable regression (linear or logistic, depending on the outcome).
Results with P<0.05 were considered as statistically significant, but in line with the explorative character of the study we also marked findings with P<0.10 as potentially relevant. All analyses were performed with SPSS 21 (IBM Corp., Armonk, NY).
Participant characteristics, experience in interventional neuroradiology, and knowledge
All 28 participants answered all questions in the online survey, resulting in a response rate of 100%. Eight of the 28 participants were female (28.6%). One participant (3.6%) was in the first year of radiology training, three (10.7%) were in their fifth year, and 20 participants (71.4%) had completed their radiology residency. Each participant had at least 1 year of experience working in interventional neuroradiology, with nine (32.1%) reporting more than 4 years. Thirteen of the 28 participants (46.4%) were certified neuroradiologists in their country. The high number of certified radiologists is due to the fact that in most European countries radiological certification is a prerequisite for specialization in INR. Experience in terms of number of performed or assisted interventional procedures are shown in Online supplementary figure 1. One participant did not attend the final knowledge test. Twelve participants (44.4%) answered at least 80% of the 15 multiple-choice questions correctly (median, 11/73% correct answers).
Supplementary file 1
Interrater reliability was fair to moderate with regard to any treatment recommendation (vs. no treatment; ICC=0.49, CI 0.36 to 0.65); substantial with regard to the recommendation of clipping (ICC=0.72, CI 0.61 to 0.83); moderate to substantial with regard to coiling (ICC=0.61, CI 0.48 to 0.75) and WEB device (ICC=0.59, CI 0.47 to 0.74); and poor to moderate with regard to flow diverter (ICC=0.33, CI 0.22 to 0.49) and stent-assisted coiling (ICC=0.34, CI 0.23 to 0.50). Factors associated with general treatment recommendation and specific retreatment recommendations are shown in table 1. The recommendation of any treatment tended to be less likely with increasing estimated risk of complications (OR=0.24, P<0.05). The recommendation of clipping tended to be less likely with more experience in endovascular treatment of aneurysms (OR=0.6, P<0.10) and more likely with increasing knowledge in INR (OR=1.7, P<0.10). Odds of recommending WEB device tended to be lower with more experience in endovascular aneurysm treatment (OR=0.6, P<0.10), higher in patients with multiple aneurysms (OR=3.35, P<0.10) and increasing neck width (OR=2.7, P<0.05). The recommendation of stent-assisted coiling tended to be more likely with increasing neck width (OR=2.4, P<0.01), when cerebral ischemic comorbidities were present (OR=2.9, P<0.10), and when the aneurysm was located in the posterior circulation (OR=2.5, P<0.10). Odds of recommending flow diverter tended to increase with increasing neck width (OR=2.4, P<0.10).
Confidence in treatment recommendation
Overall confidence in the treatment recommendation was low (median, ‘rather certain’; mean 4.18). Factors associated with confidence of treatment recommendation are shown in table 2. Confidence in treatment recommendation varied statistically significantly according to recommended treatments with highest confidence for flow diverter and coiling, followed by clipping, WEB device, and stent-assisted coiling.
Estimation of procedure-related neurological complications
Interrater reliability was fair to moderate with regard to the estimation of the risk for neurological complications after treatment (ICC=0.38, CI 0.26 to 0.55). Factors associated with estimated risk of complications are shown in table 3. The risk of neurological complications tended to be estimated lower with increasing aneurysm size and higher with increasing neck width.
Ability to predict procedure-related neurological complications
The S-S-C risk score had a moderate predictive quality (area under the curve [AUC ]of 0.69, 95% CI 0.45 to 0.92). At average, the participants were significantly worse than the S-C-C risk score (mean AUC of 0.53, 95% CI of the mean 0.48 to 0.58) and statistically not better than random guess in predicting procedure-related neurological complications. However, some of them were good (best participant AUC: 0.81), but some were even worse than random guess (worst participant AUC: 0.32).
Factors influencing the ability to predict procedure-related neurological complications
Interrater reliability was low with regard to the estimation of risk of complications (ICC=0.38). In correlation and linear regression analysis, neither knowledge nor experience in endovascular aneurysm treatment was statistically significantly associated with the participant’s ability to predict procedure-related neurological complications (as measured by the AUC). The risk of complications was not estimated significantly higher if procedure-related neurological complications occurred in the previous case.
Factors predicting neurological complications
The presence of cerebral ischemic comorbidities was statistically significantly associated with procedure-related neurological complications (r=0.61; P=0.002).
In accordance with previous studies, our findings confirm the overall suboptimal interrater reliability regarding optimal management of UIAs.7 We evaluated numerous patient and rater variables to analyze possible reasons behind the wide variation among raters. Our study shows that the recommendation of treatment is influenced by not only case-specific characteristics such as neck width, location, and cerebral ischemic comorbidities, but also raterspecific characteristics such as work experience and knowledge in the field of INR.
The observation that the recommendation of clipping and WEB device tended to be less likely with more experience in endovascular treatment of aneurysms might be explained by the concept of grounded rationality, that is, that physicians typically recommend treatment with which they are familiar.14 15 Moreover, we found an association between the participant’s confidence in his treatment recommendation and the treatment he recommended. The underlying reason might be a different degree of experience with the different treatment modalities. Confidence in treatment recommendations reflects the uncertainty that accompanies each healthcare management decision. Low confidence can lead to anxiety and fear of reprimand, leading to overreliance on standardized protocols and avoidance or passing the responsibility onto the patient, thereby missing the true and individual needs of the patient.16 Thus, interventions performed with lower confidence are expected to be performed more often erroneously or with a lower quality than treatments accompanied by higher confidence.
Though the overall interrater reliability was suboptimal, the agreement was substantial with regard to the recommendation of clipping. Interestingly, most participants rarely recommended ‘no treatment’. This observation might be explained by the common practice of positive defensive medicine. Thus, the therapist might fear being blamed for undertreatment in case an aneurysm ruptures. These findings underline that guidelines based on expert opinions, the lowest level of acceptable evidence, should be followed with caution.
With regard to the prediction of procedure-related neurological complications, the overall performance was worse than the S-C-C risk score and not better than random guess. These results support the use of heuristics, that is, strategies that ignore part of the information, in order to make decisions more accurately and faster than complex statistical models.17 These so-called ‘rules of thumb’ are recommended in the process of rational decision making in the presence of uncertainty and sometimes outperform complex methods, a phenomenon called ‘less is more’.15 18
Our research suggests that the S-S-C score is more useful for risk stratification than the raters’ intuition alone. Further studies are needed to analyze if the S-S-C score is better in estimating the risk of neurological complication than complex prediction models.
As a strength of this study, all angiograms were presented as movies in two projections and 3D-rotational angiograms to recreate a more realistic clinical scenario. Moreover, we provided aneurysm-specific information to the raters, including location, size, neck width, and cerebral ischemic comorbidities.
Interestingly, we found no significant association between patient age and treatment recommendation. This is in contrast to previous studies analyzing factors influencing retreatment recommendation in patients with coiled aneurysms.19 20 They found that the recommendation for retreatment was less likely in elderly patients.
Although in real-world cerebrovascular practice the final treatment recommendation is influenced by further factors such as patient’s anxiety and preferences, our experimental setting allowed us to control for these factors and to concentrate on the influence of specific rater and aneurysm characteristics.
The odds of complications was intentionally held 1:1 in the case pool in order to achieve sufficient precision in estimating diagnostic accuracy parameters and to prevent participants with invariant ratings from being erroneously favored.
In correlation and logistic regression analysis, only the presence of comorbidities was statistically significantly associated with an increased likelihood of complications. For other factors such as patient age, cerebral ischemic comorbidities, core area, aneurysm size, aneurysm neck width, multiple aneurysms, and type of received treatment no significant association was observed. Similarly, we did not find a significant association between treatment and estimated risk of neurological complication. This might be explained by the limited number of cases and limited power of analyses. As we aimed to guarantee that the participants gave reliable and authentic responses, and given that concentration diminishes with time, we had to limit the number to 24 cases. With 24 cases, the participants voluntarily answered 98 questions. Given the limited time and number of cases, we decided to show only cases that were treated. Thus, we were able to assess the estimation of the risk for neurological complications after treatment at the same time. Participants were not informed that only cases that were treated in reality were shown. Thus, ‘no treatment’ was an equal option to be considered for them.
Neither aneurysm treatment experience nor knowledge was statistically significantly associated with the participant’s ability to predict complications. However, this might be due to the low number of participants. Future studies with more participants and cases, including patients that were not treated, are desirable. Moreover, this would allow the analyses of more factors traditionally considered to favor conservative management such as anticoagulation, no smoking, low blood pressure, negative family history, and comorbidities influencing perioperative morbidity and mortality related to anesthesia.
Only 13 of the 28 participants were certified neuroradiologists;: thus, our study does not necessarily represent the community of INRs. However, the group is well described, with the detailed assessment of knowledge and working experience.
A previous study analyzing the degree of interobserver variability regarding retreatment decisions of recurrent and residual aneurysms observed that neurosurgeons were significantly more likely to retreat and recommend different types of treatment compared with neuroradiologists.20 Future studies with more participants should analyze treatment patterns for unruptured intracranial aneurysms across different specialties.
Our study shows a moderate interrater reliability of treatment recommendations of unruptured intracranial aneurysms. Confidence in treatment recommendation varied statistically significantly according to recommended treatments with highest confidence for flow diverter and coiling. Overall performance in predicting procedure-related neurological complications was worse than the S-C-C risk score and not better than random guess.
Supplementary file 2
Contributors ME designed data collection tools, monitored data collection for the whole trial, wrote the statistical analysis plan, cleaned and analyzed the data, and drafted and revised the paper. She is guarantor. LK helped to develop the research questions and to design the study: he contributed to the development of the survey tool, performed the statistical analyses, and contributed to the interpretation of the findings and to the preparation of the manuscript. JB and JF initiated the collaborative project, designed data collection tools, monitored data collection for the whole trial, analyzed the data, and drafted and revised the paper. UH and AF analyzed the data, and drafted and revised the paper.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial, or not-for-profit sectors.
Competing interests None declared.
Patient consent Not rquired.
Ethics approval The study was approved by Ethik-Kommission Ärztekammer Hamburg WF-030/14.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.