How Augmented Reality is Rewriting the Rules of High-Stakes Surgery

Every surgeon is aware that there is a time during surgery when the hands must move with a certainty that the eyes are unable to fully provide. Beneath the skin, the anatomy remains hidden. Millimeters, sometimes even less, separate a tumor from a blood vessel. Surgeons used training, experience, and a sort of educated intuition to navigate this uncertainty for decades. Something is starting to alter that equation now.

When you walk into some operating rooms these days, you might notice something strange: a surgeon wearing what appear to be futuristic ski goggles. That would be the Microsoft HoloLens, a mixed reality headset that has been subtly making its way into operating rooms in Seattle, London, Bologna, and other places since 2018. A surgeon can see more than just the patient on the table through its lenses.

Similar to an X-ray that tracks your gaze, they see a three-dimensional digital model of the patient’s interior, including blood vessels, bones, and tissue, superimposed directly onto the body in front of them.

The experience is difficult to forget, according to Dr. Philip Pratt, a Research Fellow in Robotic Surgery at Imperial College London. He said that surgeons could practically see inside a patient’s leg. Not in a symbolic sense. In actuality. The surgeon doesn’t need to take their eyes off the patient to see the three-dimensional CT scan data. It sounds like science fiction until you realize that it was already taking place in St. Mary’s Hospital a few years prior, and Imperial College surgeons were praising it for increasing accuracy and saving time.

It’s possible that no single instrument in the history of modern surgery has raised as many questions and inspired as much real excitement. According to a 2019 University of Pisa study, wearing the HoloLens actually made participants make more mistakes on precision tasks than they would have with the naked eye.

This finding was explained by focal rivalry, which is the eye’s limited capacity to focus on both a real and a projected image at the same time. That study was criticized. Defenders pointed back, pointing out that the tasks were artificial, the participants weren’t trained surgeons, and the technology has continued to advance ever since. The argument hasn’t been resolved because both sides have a point.

Then February 2020 arrived. The world’s first augmented reality-guided surgery was performed at Sant’Orsola Hospital in Bologna, according to the University of Pisa, the same organization that had questioned the accuracy of AR headsets only months earlier. Throughout the intricate jaw-repositioning procedure, the lead surgeon, Dr. Giovanni Badiali, wore an AR visor.

He could view pre-operative imaging from CT, MRI, and 3D ultrasound scans as well as the patient’s heart rate, blood pressure, and breathing rate via his headset. He did all of this in real time without looking at a monitor. It is on the verge of a medical revolution, according to Badiali. Although that phrase might typically elicit skepticism, it’s difficult not to take him seriously after seeing the timeline of what his team accomplished.

The University of Pisa and twelve European partners collaborated on the VOSTARS initiative, or Video and Optical See-Through Augmented Reality Surgical Systems, which provided the technology used in that procedure. The next development is a live augmented reality video feed of camera-equipped instruments, which will enable surgeons to track real-time movements inside the body without ever looking away. The headset is anticipated to become widely available within a few years. The aspirations are astounding. The technical intricacy is just as important.

In the meantime, a different type of augmented reality application was subtly resolving a different issue: geography. Dr. Nadine Hachach-Haram, a reconstructive plastic surgeon, founded Proximie, a London-based platform that uses AR, AI, and live video to allow surgeons to participate in procedures from thousands of miles away. In 2016, a bomb blast victim in Gaza underwent the first use of the technology. By September 2020, the platform had facilitated over 5,000 AR-assisted procedures in 35 countries on five continents.

One remarkable instance was when Dr. Jim Porter, Director of Robotic Surgery at Providence St. Joseph Health in Seattle, led a surgical team from London through a challenging cancer procedure from 4,700 miles away. He observed that the systems he had previously used had no lag, latency, or anything similar. Proximie is currently used by one-third of NHS hospitals. That program is no longer a pilot. Infrastructure is that.

At the beginning of 2020, GlobalData predicted that the AR market in healthcare would surpass virtual reality and reach $76 billion by 2030. Although it’s still unclear if that timeline will hold—markets have a way of undermining forecasters—the direction appears to be getting harder to argue against. AR is influencing neurosurgery, spine surgery, oncological tumor resection, and robotic procedures by providing real-time overlays that assist surgeons in avoiding nerves and vessels, placing screws more precisely, and removing tumors more thoroughly.

The difficulties are genuine and shouldn’t be disregarded. Each unit may cost hundreds of thousands of dollars to implement. It takes time to teach surgeons how to use AR naturally. Regulators are still figuring out where the boundaries should be, especially when it comes to liability when a high-stakes procedure and a technical failure collide. These are serious issues.

However, looking back over the last few years, it seems as though the technology has already passed some sort of threshold, going from experiment to early adoption, from curiosity to clinical utility. Under extreme pressure, nerve and knowledge have always met in the operating room. Neither of those things is replaced by augmented reality. It simply gives the surgeon a little more to work with in a subtle and gradual manner.