Mixed reality in the operating room

IT in the care sector

Tim Tollens is the head of surgery at the Imelda Hospital. He is himself a specialist in abdominal surgery and, more specifically, complex incisional hernia surgery. “It is true that the development of IT is visible in our sector but, in my opinion, it is going too slowly,” he notes. “Corona may provide an acceleration in that area. The patient is now much more familiar with video consultations and remote advice.” The doctor himself is particularly interested in the use of IT in his profession and is always looking for new applications. That led to a Flemish first.

Livestream with smartglasses

“Through the medical technology company Becton Dickinson, I came into contact with the HoloLens, Microsoft's mixed reality headset,” he says. “In that context, I also became acquainted with Mr. WattsNew window, a Belgian company that develops applications for the HoloLens.” Dr. Tollens already had experience filming surgical procedures for training and research.

With the HoloLens, a camera was added, which allows the others to follow the procedure through the surgeon's eyes, as it were. Dr. Tollens performed a complex incisional hernia operation with the smartglasses on. Doctors followed the live stream from 280 locations in 49 countries.

Via the HoloLens, the surgeon at the operating table has access to all kinds of information and medical images.

Dr. Tim Tollens

Projection on the patient

“The real added value is in the application Mr. Watts developed,” continues Dr. Tollens. “It creates a virtual space that you can access via the HoloLens. This way, as a surgeon, you can open the patient's medical file in that virtual space.” The app is controlled by voice, gestures or eye movements. “I don't have to leave the operating table for that and I don't have to take off my gloves to check something on the PC. Via the HoloLens, all information is immediately available in a safe manner at the operating table.”

3D visualization

The biggest advantage, however, lies in the use of mixed reality (MR). The application displays medical images – for example, from a CT scan – via the HoloLens and projects them onto the patient. In this way, the surgeon can see in the MR image where the exact location of the problem is in the patient. It allows the surgeon to determine the location of the procedure in a very specific way, without having to be invasive. “That technique – 3D visualization and superimposition – is really the future as far as I'm concerned,” says Dr. Tollens.

Advantage in the operating room

The surgery that the doctor performed, supported by the HoloLens and Mr. Watts, seemed to have some immediate and important practical advantages. “In the operating theater there is no longer a need for expensive video screens with all kinds of complex cabling,” he says.

“The images the surgeon needs appear on his smartglasses. That not only makes it more convenient for the entire team's work, but also saves a lot of money when setting up an operating room.”

Created for 5G

Dr. Tollens looks back enthusiastically on his first experience with the headset and MR application, although there is still room for improvement. “Rendering 3D structures and positioning them in a correctly calibrated way remains a challenge. But still, it’s only a matter of time before that is further refined.” When using the HoloLens, large volumes of data move between the smartglasses and the application running in the cloud. “It’s the kind of application that 5G is made for.” Every time the doctor moves his head, the application adjusts the image in real time, without delay.

It is not the only application where 5G will make a difference in the future. “I am looking with great interest at Microsoft Mesh, which will enable remote holographic collaboration,” says Dr. Tollens. “In the hospital, we want to focus on a holographic training and consultation platform, among other things.” There is every indication that the use of holographic applications – such as VR and MR – will grow rapidly, also in the medical sector, thanks to 5G.