Imagine this: you put on a pair of glasses, and in an instant, the world before you transforms into a digital playground teeming with fantastical creatures and marvels. The physical reality seamlessly melds with virtual elements projected onto the glasses' visual frame, presenting an augmented view of reality reminiscent of a video game. Astonishingly, as this blend of technology and the tangible world rapidly reshapes the entertainment and marketing industries, augmented reality (AR) has also made its mark in the medical field, notably within the confines of the operating room.
AR works by overlaying digital content, such as images, text, or 3D models, onto the user's view of the real world in real time. AR devices, like smartphones or smart glasses, use sensors to capture the user's surroundings and recognize markers or features to understand the environment's context (Dennler et al., 2020). The system then seamlessly renders and displays the virtual content on the device's screen through which the users are looking, allowing them to interact with both the real and virtual elements simultaneously (Moawad et al., 2020).
Imagine this in a medical setting: how much easier would it be if surgeons could visualize their patient’s X-ray scans right in front of them in the operating room while performing the operation itself? Instead of needing to divert their gaze from their work to consult the scan, physicians could view a direct projection of it onto their patient’s body. The extra aid provided through screens on AR devices could help reduce surgical time and improve the physicians’ overall performance. As Timothy Witham, professor of neurosurgery at Johns Hopkins, puts it, “it’s like having a GPS navigator in front of your eyes in a natural way so you don’t have to look at a separate screen to see your patient’s CT scan” (Johns Hopkins Medicine, 2021).
In 2020, neurosurgeons at Rush University Medical Centre successfully used the first AR headset approved by the FDA to perform a minimally invasive spinal surgery on a 78-year-old patient suffering from degenerative spine disease (Business Wire, 2020). With the aid of a head-mounted display that provided a see-through view of the patient’s internal anatomy via CT scans (an imaging procedure that combines X-rays and computers to create images of the body), the surgeons were able to insert implants into the bone and treat compressed nerves in the lower spine with a clinical accuracy of 98.9% (Molina et al., 2021).
The headset consists of a transparent display that uses artificial intelligence (AI) to accurately locate surgical tools during the surgery and superimpose CT scan data onto the surgeon’s retina (a crucial part of the eye that captures light), essentially giving them X-ray vision superpowers (Business Wire, 2020). Other AR-guided surgical headsets have made further use of AI to guide surgeons, using images taken during the operation to autonomously classify key anatomical structures and suggest patient-specific screw trajectories that help surgeons plan their procedures (Moawad et al., 2020; Park et al., 2020; Hospimedica, 2022). Known as Targeted Guided Surgery, this AR/AI approach can likely facilitate surgeons’ decision-making and efficiently reduce operational time (Moawad et al., 2020).
“Being able to place minimally invasive spinal instrumentation extremely accurately and efficiently, reducing surgical time and complication risk, is critical to improving outcomes for spinal surgery,” said Dr. Phillips, the head surgeon of the spinal operation that used the first FDA-approved AR headset (Business Wire, 2020). With future developments, such methods in the operating room could drastically improve surgical precision and safety, reducing the risk of complications and expediting surgical durations. In the future, this advancement could lead to improved patient outcomes and, once any stigmas related to the use of technology in surgery are overcome, foster a higher level of confidence in surgical interventions.
However, ethics related to the use of AR/AI must also be carefully considered. As AI systems handle sensitive medical information, the protection of patient data becomes of utmost importance. While AI may be ethically neutral in theory, it is also vulnerable to algorithmic biases that could result in discriminatory outcomes and significantly impact patient care. Ensuring patient consent and their comprehension of these emerging technologies is equally critical to establish a trustworthy role for AR/AI within the medical field (Farhud & Zokaei, 2021).
Regardless, the integration of AR/AI in surgery presents a remarkable leap forward into the intersection of technology and medicine. With further research and regulations, the potential of AR/AI tools projects a vision of a safer healthcare experience in the operating room for all — literally and figuratively.
The image above displays a simplified overview of the process by which an AR headset uses AI to identify points on a patient’s body that are important for a surgical operation. It then creates a projection onto the patient’s real body through the headset display, facilitating the surgeon’s ability to conduct surgery. Created with BioRender.com.
References
Dennler, C., Bauer, D. E., Scheibler, A.-G., Spirig, J., Gotschi, T., Furnstahl, P., & Farshad, M. (2020, November 25). Augmented Realities, Artificial Intelligence, and Machine Learning: Clinical Implications and How Technology Is Shaping the Future of Medicine. NCBI. Retrieved August 1, 2023, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7761251/
Dr. Frank Phillips is First in the World to Use Augmented Reality Surgical Guidance in Minimally Invasive Spine Surgery. (2020, June 17). Business Wire. Retrieved August 1, 2023, from https://www.businesswire.com/news/home/20200617005352/en/Dr.-Frank-Phillips-World-Augmented-Reality-Surgical
Farhud, D. D., & Zokaei, S. (2021). Ethical Issues of Artificial Intelligence in Medicine and Healthcare. Iranian journal of public health, 50(11), i–v. https://doi.org/10.18502/ijph.v50i11.7600
Johns Hopkins Performs Its First Augmented Reality Surgeries in Patients. (2021, February 16). Johns Hopkins Medicine. Retrieved August 1, 2023, from https://www.hopkinsmedicine.org/news/articles/johns-hopkins-performs-its-first-augmented-reality-surgeries-in-patients
Moawad, G. N., Elkhail, J., Klebanoff, J. S., Rahman, S., Habib, N., & Alkatout, I. (2020, November 25). Augmented Realities, Artificial Intelligence, and Machine Learning: Clinical Implications and How Technology Is Shaping the Future of Medicine. NCBI. Retrieved August 1, 2023, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7761251/
Molina, C. A., Sciubba, D. M., Greenberg, J. K., Khan, M., & Witham, T. (2021, February 16). Clinical Accuracy, Technical Precision, and Workflow of the First in Human Use of an Augmented-Reality Head-Mounted Display Stereotactic Navigation System for Spine Surgery. PubMed. Retrieved August 1, 2023, from https://pubmed.ncbi.nlm.nih.gov/33377137/
Park, B. J., Hunt, S. J., Nadolski, G. J., & Gade, T. P. (2020, October 29). Augmented reality improves procedural efficiency and reduces radiation dose for CT-guided lesion targeting: a phantom study using HoloLens 2. Nature. Retrieved August 1, 2023, from https://www.nature.com/articles/s41598-020-75676-4
World's First Surgical Guidance System Incorporates AI and AR. (2022, September 16). Hospimedica.com. Retrieved August 1, 2023, from https://www.hospimedica.com/surgical-techniques/articles/294794612/worlds-first-surgical-guidance-system-incorporates-ai-and-ar.html