Background and purpose Optical see-through head mounted displays (OST-HMDs) offer a mixed reality (MixR) experience with unhindered procedural site visualization during procedures using high resolution radiographic imaging. This technical note describes our preliminary experience with percutaneous spine procedures utilizing OST-HMD as an alternative to traditional angiography suite monitors.
Methods MixR visualization was achieved using the Microsoft HoloLens system. Various spine procedures (vertebroplasty, kyphoplasty, and percutaneous discectomy) were performed on a lumbar spine phantom with commercially available devices. The HMD created a real time MixR environment by superimposing virtual posteroanterior and lateral views onto the interventionalist’s field of view. The procedures were filmed from the operator’s perspective. Videos were reviewed to assess whether key anatomic landmarks and materials were reliably visualized. Dosimetry and procedural times were recorded. The operator completed a questionnaire following each procedure, detailing benefits, limitations, and visualization mode preferences.
Results Percutaneous vertebroplasty, kyphoplasty, and discectomy procedures were successfully performed using OST-HMD image guidance on a lumbar spine phantom. Dosimetry and procedural time compared favorably with typical procedural times. Conventional and MixR visualization modes were equally effective in providing image guidance, with key anatomic landmarks and materials reliably visualized.
Conclusion This preliminary study demonstrates the feasibility of utilizing OST-HMDs for image guidance in interventional spine procedures. This novel visualization approach may serve as a valuable adjunct tool during minimally invasive percutaneous spine treatment.
Statistics from Altmetric.com
Adequate image guidance is necessary to reliably visualize anatomic landmarks and successfully deliver medical devices during percutaneous spine procedures. Radiography monitors displaying the fluoroscopic images used for guidance purposes are typically not aligned with the procedural axis,1 rendering an indirect visualization shown to hinder hand–eye coordination.2 3 Optical see-through head mounted displays (OST-HMDs) with high resolution and high contrast capabilities offer real time mixed reality (MixR) visualization of radiographic images that can be projected over the interventional site without hampering direct visual control of procedural manipulations.
MixR/hybrid reality is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time.4 MixR is a mix of reality and virtual reality (depicts objects and events from both the physical and virtual world, therefore encompassing both augmented reality and augmented virtuality via immersive technology).4 5
Three routinely performed percutaneous procedures, vertebroplasty, kyphoplasty, and discectomy, were selected in order to demonstrate the feasibility of this novel visualization approach. Percutaneous vertebroplasty (PV) consists of injection of polymethylmethacrylate (PMMA) into fractured vertebral bodies; it is frequently used for the treatment of osteoporotic or metastatic lesions.2 6–10 Kyphoplasty differs from PV by first creating an intravertebral cavity to attempt more controlled PMMA delivery.10 11 Various percutaneous disc decompression (PDD) techniques have been proposed as alternatives to open surgical disc decompression, based on the premise that a reduction in central nucleus pulposus volume decreases the intradiscal pressure and results in retraction of the herniated fragment.12–18
The study was performed in a biplane angiography suite (Artis Zee, Siemens Healthcare GmbH, Forchheim, Germany). The software and hardware system for real time visualization of medical images in a MixR surgical environment using an OST-HMD has been previously detailed.1 It includes a frame grabber, an image processing unit (in this case a personal computer), a data transfer network, and a head mounted display (figure 1). The system is compatible with all conventional imaging formats and allows for real time MixR visualization of medical images in head, world, or body anchored display, with an average delay of 214 (±30) ms and a refresh rate of 41.4 (±32.0) Hz for typical image sizes.1
The operator was a neurointerventional fellow with 1 year’s prior experience of vertebroplasty and kyphoplasty (approximately 25 procedures as the primary operator). The HMD created a real time MixR environment by superimposing virtual posteroanterior and lateral projections onto the interventionalist’s field of view (figures 2a and b). Unipedicular PV and kyphoplasty were performed at L2 and L3, respectively, and PDD on the L3–L4 disc. The angiography suite monitors were blinded when the procedures were conducted under OST-HMD guidance. The procedures were performed by a single operator on a lumbar spine phantom (figure 3a) using commercially available PV, kyphoplasty , and PDD kits (results demonstrated in figure 3b). MixR visualization was achieved using the HoloLens OST-HMD system (Microsoft Corporation, Redmond, W ashington, USA) (figures 4a and b).
Procedural steps for PV
Posteroanterior, lateral, and oblique working projections were stored.
A 13 g needle was advanced into the anterior third of the vertebral body through the left pedicle using oblique and lateral projections to monitor the progression of the needle and ensure adequate placement without encroaching the medial or inferior aspect of the pedicle.
PMMA was prepared (Autoplex Cement Delivery System and half-dose Verteplex cement; Stryker Corporation, Kalamazoo, Michigan, USA).
The needle stylet was removed and PMMA injected under posteroanterior and lateral visualization.
After PMMA administration, the stylet was replaced under lateral visualization and the needle withdrawn.
Procedural steps for kyphoplasty
Same as for PV (see above).
Same as for PV (see above).
Once the appropriate needle tip position was reached, the stylet was removed and the kyphoplasty balloon advanced into the anterior third of the vertebral body.
The balloon was inflated (with iodinated contrast) under lateral plane visualization. After complete inflation, the balloon was deflated and withdrawn.
PMMA was prepared (Autoplex Cement Delivery System and half-dose Verteplex cement; Stryker Corporation).
PMMA was injected under posteroanterior and lateral visualization.
After PMMA administration, the stylet was replaced under lateral visualization and the needle withdrawn.
Procedural steps for PDD
Posteroanterior, lateral, and oblique working projections were stored. The craniocaudal and lateral angulations on the A plane were adapted to optimize disc space visualization.
A 13 g needle was advanced into the center of the intervertebral disc.
The stylet was removed and the Dekompressor device (Stryker Corporation) advanced into the disc. Once the tip of the device was adequately located, the Dekompressor was turned on to simulate the removal of nucleus pulposus.
In order to facilitate a comparison between traditional and HMD visualization modes, the key portions of each procedure were repeated four times, once using the standard angiographic display and once with each HMD visualization mode. The order in which visualization modes were used was changed from procedure to procedure to ensure a comparison that allowed for a degree of learning effect. PMMA preparation and injection was performed only once each for PV and kyphoplasty, using a combination of head anchored and world anchored modes.
The following time points were recorded:
Time from the start of the procedure to establishing and recording all projections, including the oblique working projection (step 1).
Time to place and stabilize the needle in the appropriate access axis for ‘end on’ transpedicular advancement on the working projection (steps 2 and 3).
Time to advance the tip of the needle into the anterior third of the vertebral body with the tip crossing the midline (steps 4 and 5) or into the center of the disc.
Time to inflate the kyphoplasty balloon or complete utilization of the Dekompressor device.
When using the HMD, the procedure was filmed through the operator’s point of view using each of the three HMD modes:
Head anchored. The MixR display remains at a constant position within the operator’s field of view. The display does not change depth position when objects are encountered in the environment.
World anchored. The display is locked to a specific position in the environment by the operator (with the use of a hand gesture) and may therefore move in and out of the operator’s field of view.
Body anchored. The display consistently remains in a specific position within the operator’s field of view, only moving when the operator’s head movement would result in the display completely exiting his field of view.
Video recordings were reviewed to assess whether key anatomic landmarks could be consistently and reliably visualized (see online supplementary videos 1 and 2). Procedural dosimetry and duration were recorded. The operator completed a descriptive, qualitative questionnaire following the procedure, detailing the benefits and limitations of each visualization mode. The questionnaire included questions regarding which HMD visualization mode(s) was/were optimal for each portion of the procedure, their ease of use, and comfort of the operator.
All simulated procedures were performed successfully. PV and kyphoplasty were technically adequate, with PMMA filling 60–70% of the L2 and L3 vertebrae, endplate to endplate deposition, an equal amount of cement on either side of the midline, but no unwanted extension towards the posterior quarter of the vertebral body, posterior elements, or extra-vertebral structures. The Dekompressor device was successfully placed into the mid portion of the L3–4 intervertebral disc.
Head anchored, world anchored, and body anchored visualization modes provided equally effective image guidance. Each mode offered specific advantages depending on the portion of the procedure.
Head anchored mode
This mode was especially useful when utilizing different visual spaces—for example, at the beginning of the procedures, when the operator must look at the patient (or phantom), the angiography C arm, and the angiography table control panel.
World anchored mode
This mode proved particularly helpful to work in a single visual space, notably when advancing a needle or device under fluoroscopic guidance.
Body anchored mode
This display mode combines the convenient features of both the world and head anchored modes. However, the constant repositioning of the virtual monitor within the operator’s field of view requires a period of acclimation to the MixR environment.
Key anatomic landmarks, devices, and material components were reliably visualized using both the conventional and virtual environment (in all three visualization modes). During the initial set-up and planning, the operator was able to visualize the superior and inferior endplates of the targeted levels and obtain a working view that clearly outlined the margins of the pedicle and the articulating processes of the facet joints. The needle tip (including the bevel orientation) and the pedicle margins were consistently visualized in both selected working projections; in addition, the lateral view delineated the posterior elements (including the pedicle trajectory), while the posteroanterior view identified the needle tip position in regard to the superior and inferior endplates. During PMMA injection, the cement was adequately visualized both within the needle and the vertebral body. The distribution of PMMA was adequately controlled and extravertebral deposition avoided.
Procedural duration and dosimetry
The key procedural steps (as listed above) were repeated four times for each procedure, first using the standard angiographic display and then three times with the HMD, using each of the visualization modes. The results are summarized in the online supplementary tables 1.1–3.2. Total procedural times, key intra-procedural times, beam time, and dose area product measurements were similar when comparing HMD visualization modes to the traditional monitor for vertebroplasty (see online supplementary tables 1.1, 1.2), kyphoplasty (see online supplementary tables 2.1, 2.2), and disc decompression (see online supplementary tables 3.1, 3.2).
Supplementary file 1
Operator preferences and observations
While all procedural steps and salient structures were adequately visualized using all three visualization modes, there was a significant learning curve in becoming familiar with HMD’s controls and getting used to the presence of the virtual monitor. Initially, the world locked mode was preferred, being the closest to a standard monitor and therefore more intuitive. However, the benefits of the body locked mode became apparent as the operator grew familiar with the MixR environment and functionalities.
After an adjustment period, the operator found the HMD unobtrusive and was able to wear it throughout the procedure without discomfort and without significant field of view impairments during any procedural step using any of the visualization modes. The motion of the display in the body anchored mode was occasionally found to be disorienting, particularly during placement and advancement of the needle.
Indirect or off axis visualization of radiographic images used for procedural guidance interferes with visual and motor coordination. This problem may be solved by techniques allowing for simultaneous or in-line visualization of the procedural site and guiding images.19–22 Recently introduced OST-HMD headsets can display high resolution images directly in the operator’s ﬁeld of view without obstructing the rest of the procedural scene.23 24 OST-HMD headsets offer a synchronized assessment of body, image, and action that can confer a significant advantage over traditional display modes.25
MixR interventions using HMDs are categorized by the type of HMD used, the co-registration mode between the virtual and real environments, and the type of reality augmentation. Visualization modes include head anchored, body anchored, world anchored, and object anchored. Object anchored display keeps the operator’s perception location close to the procedural site, without impairing direct visualization. It offers a virtual monitor that resembles physical monitors that are widely used and perceptually accepted by interventionalists. Therefore, the transition from a classic clinical set-up to an object anchored display requires less adaptation than other visualization modes.26 27
We demonstrated the feasibility of performing percutaneous spine interventions using OST-HMD visualization (without the use of traditional monitors). In this single user preliminary study, OST-HMD visualization was not found to be inferior to traditional monitors in terms of procedural duration, dosimetry, or ability to visualize key anatomic structures, devices, or material components. There was no perceived impairment in the operator’s ability to visualize real world structures while wearing the HMD and using the virtual monitor display, as demonstrated by the completion of all procedural steps without removing the HMD, turning off the virtual monitor, or resorting to a traditional monitor.
HMD monitoring of a percutaneous procedure performed in the angiography suite carries several potential advantages. First, the key imaging plane(s) and the procedural site are constantly present in the operator’s field of view. This may be in a single fixed location (as in world anchored mode) or following the operator’s field of view in other modes. Operators do not need to modify their position, significantly rotate their head, or move their field of view away from the procedural site in order to obtain image guidance information. In addition, the operator can independently adjust display characteristics (eg, location, size, contrast, luminosity) using hand gestures or voice control, without relying on other team members to adjust the display’s position or characteristics.
HMDs may also prove advantageous in percutaneous procedures taking place in other environments—for example, intraoperative angiography performed in operating rooms using mobile radiographic equipment and displays, which are often cumbersome and placed in ergonomically challenging positions for the operator due to space limitations. The potential role of HMDs might even be expanded further if the possibility of conducting procedures outside angiography or operating rooms is considered. Having a compact, lightweight, wearable, and easily transportable display opens the door for image guided percutaneous procedures to be performed in an outpatient clinic setting or at the bedside, for example, taking advantage of constantly smaller, more mobile image acquisition equipment, including in-office fluoroscopy, ultrasound, or even CT/MRI scanners.
In this feasibility study, a limited number of procedures were performed by a single operator. In order to minimize a potential learning effect from procedural repetition, the first attempt of each procedure was discarded. This measure appeared to be effective at limiting a learning effect, as the total procedural and fluoroscopy times for the four subsequent procedural attempts were similar.
Currently available HMD systems only offer visualization of a single plane (A or B) at any given moment. Real time biplane fluoroscopic visualization would represent a significant technical improvement. In addition, this proof of concept study only used two-dimensional images and a two-dimensional display, closely reflecting a typical spine intervention set-up in current clinical practice. However, future HMD guidance will certainly take advantage of the advanced three-dimensional imaging capabilities of modern angiography suites. Three-dimensional datasets might be rendered into two-dimensional multiplanar reconstructions projected on the virtual display or even into three-dimensional reconstructions that can be rotated in the virtual working space.
This preliminary study demonstrates the feasibility of using OST-HMD as an alternative to conventional physical displays for image guided interventional spine procedures. This novel visualization approach may represent a valuable adjunct tool for minimally invasive percutaneous procedures in general, notably when performed in spatially limited environments.
Supplementary file 2
Supplementary file 3
Contributors GD: conception and design, acquisition of the data, analysis and interpretation of the data,drafting article and critical revisions of the article, final approval of the article, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. AJ and MU: conception and design, acquisition of the data, analysis and interpretation of the data,critical revisions of the article, final approval of the article, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. KY: acquisition of the data, analysis and interpretation of the data, critical revisions of the article, final approval of the article, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. SA and LQ: acquisition of the data, critical revisions of the article, final approval of the article, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. GO, NN, FH, and PG: conception and design, critical revisions of the article, final approval of the article, and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Competing interests None declared.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.