In Erlangen, pediatric cardiologist Muhannad Alkassar, MD, and his team use an augmented reality prototype to prepare quickly and with maximum accuracy the most complex heart surgeries. The mixed-reality HoloLens device visualizes CT scans in photorealistic 3D to represent even the tiniest detail. The physicians are currently comparing the technology with 3D printing.
Photos: Stefan Hobmaier
Download your PDF here.
Doctors in the maternity ward of the Erlangen University Hospital were worried by the low oxygen saturation of only 85 percent in the newborn’s blood. A prenatal examination had not indicated any heart defect. But because the baby’s skin had turned bluish due to cyanosis, the physicians alerted pediatric cardiologist Muhannad Alkassar less than an hour after the birth. “Within 30 minutes we had done a CT scan that confirmed our suspicion of a pulmonary vein malformation,” recalls the 40-year-old. For such emergency surgeries – Alkassar’s specialty – success depends on the best possible surgical planning. The latest technology meant the surgeon could examine the newborn as if he were lying on the operating table with his chest open. And as if his walnut-sized heart were made of glass.
“When we look at an ultrasound image of a heart that is affected like this, we have to put the different layers together in our head to create a three-dimensional image,” explains Dr. Alkassar. “This is particularly difficult for a surgeon who doesn’t do this quite as often as we cardiologists do.” Every heart defect is unique. In the case of the newborn, the oxygen-enriched blood did not flow from the lungs into the left heart chamber, but instead through an abdominal vessel into the inferior vena cava. The surgeon and Alkassar were able to study the vessels and the heart in color and 3D – using the HoloLens 2 developed by Microsoft. This presented them with a hologram of the patient’s heart, based on the CT scan, allowing them to view the organ from all angles and down to the smallest detail. “Once you’ve tried HoloLens technology, you’ll never want to do without it,” smiles Alkassar.
Help from Hollywood
When Dr. Alkassar first put on the HoloLens headset more than a year ago, he admits he was skeptical. “I never imagined that the image of the object would be in such high resolution. I had seen the photorealistic image on the computer screen, but seeing a spatial representation of the heart in 3D in this quality is something else altogether.” The CT images are processed with an app using the Cinematic Rendering process, a technology that draws inspiration from Hollywood’s computer-generated images to incorporate the shadows cast by every single pixel in the image rendered. “At first it sounds like a gimmick, but our eyes are so used to seeing the effects of light. And Cinematic Rendering means – considering the delicate vessels and structures we’re dealing with – that we can look and work the way we’re used to. That makes such a huge difference.” And it’s a difference that can be measured:
In the case of 26 patients, Alkassar and his team compared the procedure and quality of surgical planning using HoloLens augmented reality with using a 3D-printed model – the method of choice for preparing surgery. In the subjective assessment of the physicians, the HoloLens performed significantly better – probably because in addition to the heart it also visualized the surrounding muscles, bones and vessels. This explains why surgeons need less time to prepare for a heart surgery with HoloLens than they would without it. And what is more: it only takes a few minutes to process the data. “I have to wait about 24 hours for 3D printing,” says Alkassar.
”I never imagined that the image of the object would be in such high resolution. I had seen the photorealistic image on the computer screen, but seeing a spatial representation of the heart in 3D in this quality is something else altogether.”
Science fiction becomes reality
What would Wilhelm Conrad Röntgen, born 175 years ago, have made of HoloLens and cinematic 3D images from inside the body? “I think that would have been considered magic at the time,” says Dr. Alkassar, smiling whimsically. “But you don’t even have to go back as far as that: even 30, 40 years ago you couldn’t have imagined seeing these kinds of images on a monitor. Now, what used to happen only in Star Trek, in the holodeck device, will become reality.” And that is not all. “It definitely makes sense to use this technology in the future to provide support in the operating room, but the technology still needs to be developed further, for example in the area of artificial intelligence.” Muhannad Alkassar is in no doubt that this can be achieved. Everything seems possible for a doctor who sees into children’s hearts on a daily basis.
Muhannad Alkassar, MD
Muhannad Alkassar, MD, (40) holds a doctorate in biology and medicine. The son of an engineer and a physicist and chemist, who grew up in the Saarland town of Homburg, has been working and researching in the Department of Pediatric Cardiology at the Erlangen University Hospital for five years.
About the Author
As a freelance journalist and author, Marc Engelhardt reports worldwide on the latest developments in business, science and politics. He has worked as a correspondent for a number of German-language media including Neue Zürcher Zeitung, ARD, and Die Zeit.