Article Text
PDF
Abstract
Background Accurate vessel size measurement is important for neurointervention. Modern angiographic equipment offers various two-dimensional (2D) and 3D measurement methods that have not been systematically evaluated for accuracy and reliability.
Objective To evaluate these methods using anthropomorphic vessel phantoms.
Materials and methods Tubing of known sizes (2–5 mm, 1 mm increments) was embedded in 3D-printed skulls to simulate the middle cerebral artery, internal carotid artery, and basilar artery. Each phantom was imaged to gain 3D DSA, 2D DSA, and DynaCT images. Three identical measurement locations were identified on each simulated vessel. Eight measurement methods (four 2D, three 3D, and one DynaCT) were evaluated. Measurements were performed by three independent experienced users on three separate occasions. Intraclass correlation and independent non-parametric analysis were carried out to evaluate the reliability and accuracy of these measurement methods.
Results Better reliability was noted for the automatic measurement methods than for the corresponding manual measurement methods. The mean differences with the ground truth for all methods ranged from −0.12 to 0.03 with small SEs (0.02–0.03) and SDs (0.10–0.18). The smallest absolute mean differences were achieved in two automatic measurement methods based on 2D manual calibration and 3D images. In comparison with these two methods, results of measurements based on 2D autocalibration were statistically different.
Conclusions In our study, automatic analysis using 3D or 2D was the preferred measurement method. Manual calibration on 2D angiograms is necessary to improve the measurement accuracy. It is not known how our results may pertain to other angiographic systems.
- Stent
- Flow Diverter
- Technique
- Artery
Statistics from Altmetric.com
Introduction
With the rapid progress of neurointervention, intracranial stents and flow diverters (FDs) are increasingly used for treatment of different types of cerebrovascular diseases.1–3 Complete opening and apposition to the vessel wall are essential for safe and effective use of these devices.4–6 Neither of these is possible unless correct sizing is possible. Selection of a stent or FD that is too small relative to the parent artery may result in poor apposition and can result in migration, suboptimal coverage, or thrombus formation. Selection of a stent or FD that is too large relative to the parent artery may result in incomplete opening and/or injury to the vessel wall that can then result in thrombus formation and/or endothelial hyperplasia and in-stent stenosis. For these reasons appropriate size selection of these devices is crucial. This is only possible when accurate measurement of the parent vessel is achieved.
As the importance of accurate sizing has become apparent, there has been a corresponding increase in the sizes of stents and FDs that are available. For example, the Pipeline embolization device is available in increments of 0.25 cm from 2.5 to 5.0 cm. Correct selection depends upon an understanding of both the optimal tool to use for measurement and the sources of errors that are inherent in the various measurement tools. Measurement errors result in the wrong selection of a device size which, in turn, can result in a decrease in long-term effectiveness.1 ,2
In spite of the availability of multiple measurement tools on angiographic systems, there has been, to our knowledge, no systematic study of the errors inherent in each tool or which tool provides the most accurate and reproducible results. The purpose of our study was thus to evaluate these measurement methods using anthropomorphic vessel phantoms.
Materials and methods
Phantom setup
Four plastic tubes (McMaster-Carr, Elmhurst, Illinois, USA) with a nominal inner diameter of 2, 3, 4, and 5 mm were used to mimic intracranial arteries. The outer and inner diameters of each tube were measured 10 times at 10 mm intervals using a coolant-proof LCD micrometer (No 293–340, Mitutoyo, Aurora, Illinois, USA). These measurements were used both to test the uniformity of the tubes’ dimensions, as ground truth for the manual calibrations of the angiographic measurement tools, and for statistical analysis. After confirming the tubes’ uniformity, two segments of each of the four different diameter tubes were embedded into a three-dimensional (3D) printed skull to simulate the anatomical course of the middle cerebral artery (MCA), internal carotid artery (ICA), and basilar artery (BA) (figure 1A). The inlets of both segments were connected with a three-way stopcock to facilitate filling of the tubes with contrast medium.
Two segments of tube were embedded into a three-dimensional (3D)-printed skull (A) to simulate the internal carotid artery, middle cerebral artery, and basilar artery (BA). Three measurement locations were indicated using arrows (B) on 2D working projection of the BA. The indicated 2D image was stored and opened in front of the 3D angiogram on the same projection (C). We used open-source software to make the front window transparent. After adjusting the magnification and position of the 3D angiogram, the vessels on the back and front windows aligned with each other perfectly (D). Three yellow arrows were inserted to indicate identical measurement locations (E). A 3D angiogram was displayed using default settings and overlaid with measure grid (F). For DynaCT images, window adjustment is necessary but difficult in order to identify real vessel lumen from vessel wall (G–I).
Data acquisition
Each phantom was placed in a water bath and image obtained using a clinical biplane C-arm system (Artis Z Siemens Healthcare, Forchheim, Germany). The 3D DSA data for ICA, MCA, and BA were acquired simultaneously using a conventional mask and fill run acquisition (70 kVp, 0.36 µGy/Fr, 200°, 133 images). The fill run was triggered manually in order to maintain constant contrast filling inside the tube during the acquisition (ie, to be sure there was consistency between the projections). According to the 3D images, the working projections were determined to obtain images of the BA, ICA, and MCA respectively. Then, the enhanced C-arm DynaCT data (70 kVp, 1.2 µGy/Fr, 200°, 496 images) were also acquired using a hand injection of 10% contrast agent (Isovue 300, 300 mg/L, Monroe, New Jersey, USA). All the 3D DSA and 2D DSA images were acquired with 50% contrast strength.
Image postprocessing
All 2D DSA projection data were processed in the vendor’s image processing pipeline using standard vendor-supplied settings and reviewed and evaluated on the supplied system control workstation (Artis Workplace, Siemens Healthcare). Acquisition used a 2×2 binned detector, resulting in a homogeneous pixel size of 0.3 mm. All rotational projection images were transferred to a separate workstation (X-Workplace, Siemens Healthcare) for reconstruction. For 3D DSA datasets, a ‘normal’ edge enhancing kernel with system spatial resolution preserving characteristics was chosen for image preprocessing (recommended by the manufacturer for 3D DSA). 3D reconstruction was performed using conventional filtered backprojection, resulting in a 512×512×384 voxel volume with a homogeneous voxel size of 0.5 mm. DynaCT datasets were reconstructed using a ‘sharp’ edge enhancing kernel aimed at providing an image with native system spatial resolution. While this approach enhances the image noise, it minimizes spatial blur of highly enhanced objects, such as contrasted vessels for reconstruction from continuous projection image data. 3D reconstruction was performed identically to 3D DSA, using conventional filtered backprojection, resulting in a 512×512×384 voxel volume with a homogeneous voxel size of 0.5 mm. Data were presented to the reviewers on the dedicated workstation in volume rendered and slice view modes.
Vessel measurement
For the 2D process, measurement projections, such as those that would be used for vessel measurement when choosing a stent or PED, were selected for each of the three simulated arteries (MCA, BA, and ICA). On each artery, three measurement locations were selected. Using the annotation tool, these were marked with arrows on the 2D images. These three locations were then identified on the same projection of the 3D images (including DynaCT images) by an overlaying method facilitated by use of open-source software (See Through Windows V.1.0.6 (Mobzystems, Amsterdam, the Netherlands)), which makes the front window an overlay transparency (figure 1B–E). These 3D images were then saved as a preset (bookmark) so that the projection used by each evaluator was identical.
The 2D measurements were made using both automatic and manual calibration methods. For automatic calibration (AC), the system’s known exposure geometry is used. The system is calibrated for an object situated at the center of rotation of the system (isocenter) at the time of installation. Every centimeter distance of the object from the isocenter imparts a 1.5% measurement uncertainty. For manual calibration (MC) a catheter-based method is used. Vessel measurements on these 2D images were made both by manual measurement (MM) based on line drawing, and automated measurement (AM). For 3D images, the vessel sizes were measured using three methods: (a) MM-line drawing (b) MM-overlay grid, and (c) AM. For the 3D measurements the mean value of the maximum and minimum diameter was used as the measurement. For DynaCT images, the measurements were obtained using an overlay grid method. Therefore, in summary, a total of eight measurement methods were evaluated including four 2D methods (MM-AC, AM-AC, MM-MC, and AM-MC), three 3D methods (MM-overlay grid, MM-line drawing, and AM), and one DynaCT method (MM-overlay grid).
Three people with long experience of using the angiographic equipment (one interventional neuroradiologist with more than 30 years’ experience, one endovascular neurosurgeon with 8 years’ experience, and one biomedical engineer with 5 years’ experience) independently performed the vessel size measurements using the above-mentioned eight different methods on three separate occasions.
Statistics
The statistical analysis was performed using SPSS V.20.0. The difference between our measurements and the ground truth was calculated for each measurement and was used for further statistical analysis. Intraclass correlation analysis was performed and Cronbach's α coefficients were calculated to evaluate the intraobserver reproducibility and interobserver consistency for each measurement method. The reproducibility and consistency were described as unacceptable (α<0.5), poor (0.5≤α<0.6), acceptable (0.6≤α<0.7), good (0.7≤α<0.9), or excellent (α≥0.9). Independent non-parametric tests were used to detect any difference among different measurement methods. Only those measurement methods with good intraobserver reproducibility and interobserver consistency were included for the independent non-parametric tests (Kruskal–Wallis test). The level of statistical significance was set at the 0.05 level.
Results
Intra-rater reproducibility and inter-rater consistency
The detailed results showing intra-rater reproducibility and inter-rater consistency are given in table 1. For intra-rater reproducibility, most of the measurement methods showed good to excellent reproducibility for all three raters. However, when using manual measurement on DynaCT, the reproducibility was poor for raters 1 (α=0.514) and 2 (α=0.525), and acceptable for rater 3 (α=0.677). The intra-rater reproducibility was also poor for rater 3 when using manual measurement methods on 2D images (α=0.520 and 0.591). Inter-rater consistency was extremely unacceptable for the measurements made on DynaCT images (α=0.103), and was only acceptable (α=0.679) for manual measurement with International Organization for Standardization autocalibration on 2D images. For the other measurement methods, good to excellent inter-rater consistency was achieved. Better intra-rater reproducibility and inter-rater consistency were noted for automatic measurement methods than for the corresponding manual measurement methods.
Results of intraclass correlation analysis to evaluate intra-rater reproducibility and inter-rater consistency
Evaluation of measurement accuracy
Owing to unacceptable intra-rater reproducibility and inter-rater consistency, the measurement results based on DynaCT images were excluded from further analysis. The calculated mean differences of the other seven methods ranged from −0.1154 to 0.0257 as compared with the ground truth (table 2). For each method, 324 measurements were made (three reviewers, three sessions, three vessels, three sites for each vessel, and four vessel diameters). Therefore, the SE for the mean difference of each method was very small (0.01666–0.03000). The absolute values of the mean differences were smallest for the AM-MC-2D method (0.0051) and the AM-3D method (0.0085), while the SD for the AM-MC-2D method (0.18002) was higher than that for the AM-3D method (0.10848). An independent-samples Kruskal–Wallis test showed that the differences among different measurement methods were statistically significant (p=0.000). Pairwise comparisons further demonstrated that the mean differences of the MM-AC-2D and the AM-AC-2D methods were statistically different from those of the AM-MC-2D and the AM-3D methods, whose measurements were the closest to the ground truth.
Mean differences of the measurement results with the ground truth
Discussion
In our study, we have found that most of the measurement methods can achieve clinically acceptable results, except that used for DynaCT (MM-overlay grid). This was unacceptable owing to the low intra-rater reproducibility and inter-rater consistency. Measurements using the AM-MC-2D and the AM-3D were most accurate, with only the measurements based on autocalibration for the 2D images being statistically different (more errors) from them. These findings suggest that the use of automated tools on both 2D and 3D images is preferred to gain reproducible and accurate vessel measurement. Manual calibration was more reliable than autocalibration for making measurements on 2D images.
Two characteristics of an ideal measurement method are (1) a good intra-rater reproducibility and (2) good inter-rater consistency. From our study, DynaCT measurements had the worst intra-rater reproducibility and inter-rater consistency. This is because the tubes had a relatively thick wall and it was difficult to adjust the window level to identify the real vessel lumen from the tube wall (figure 1G–I). However, the real cerebral arterial wall in a human subject is thinner and the enhanced vessel lumen can be more easily identified than with the tested tubes. Thus, their impact on vessel size measurement may not be as dramatic. Some previous reports also showed that contrast-enhanced cone beam CT can be reliably used to measure vessel size and calculate the rate of stenosis.7 ,8
Very good intra-rater reproducibility and excellent inter-rater consistency were achieved for all 3D measurements. Two reasons may account for this. First, we ensured constant filling of the tested tube by triggering the fill acquisition manually after confirming good vessel delineation on real-time fluoro monitoring. Second, we used the default reconstruction settings to display the 3D angiogram; this avoided the variances from window adjustment by different raters.
Another finding of this study was that automatic measurements were more reliable than manual measurements. This reflects the inherent variance in manual measurement methods due to the inability to precisely mark identical vessel edges by different users in different sessions. Despite these differences the mean differences between automatic measurement methods and their corresponding manual measurement methods were not statistically significant. In our opinion this indicated comparable measurement accuracy of different measurement tools. Thus, based on these results, automatic measurement should be preferred for clinical practice. This suggestion was further supported by the finding that the smallest mean difference was achieved in two automatic measurement methods: the AM-MC-2D and the AM-3D methods. However, the SD for the AM-MC-2D method was almost twice as high as that of the AM-3D method, which may come from the process of manual calibration. Therefore, the AM-3D method seems better than the AM-MC-2D method according to this study. Furthermore, compared with the above-mentioned two most accurate measurement methods, measurements based on autocalibration on 2D images were statistically different, which suggests that autocalibration is not reliable on 2D measurement because there is unavoidable error due to system magnification.
This study has several limitations. First, the reliability of the measurement method based on DynaCT images may be underestimated owing to the difficulty in finding the real vessel lumen, which is created by the tested tubes and may not exist in human vessels. Second, we used a Siemens biplane C-arm system to perform this study, which limits generalization of our results to other flat detector angiographic systems. Despite this, however, the inaccuracies due to operator variances—for example, manual measurements and windowing, should, to some degree be independent of the angiographic system used for measurement. Finally, some other factors may affect the accuracy of vessel size measurement—for example, source to image distance, detector binning, and reconstruction kernel, etc. In this study, we did not evaluate the effect of these factors.
Conclusion
Our study showed that automatic measurement using 3D or 2D is the preferred method of measuring vessel size. Manual calibration on 2D images is necessary to achieve accurate measurement. Further study is required to evaluate these different measurement methods in humans and the possible effect of some other factors (source to image distance, detector binning, and reconstruction kernel, etc) on the measurement accuracy.
Footnotes
Contributors All authors participated in designing and performing this study, and drafting the manuscript.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.