How to mend a broken heart

More and more cardiac interventions are now performed by inserting a small tube called a catheter into a blood vessel in the arm or groin, and from there into the heart chambers. This is much less traumatic than open heart surgery, but training young doctors to do this takes a long time.

Enter the HEARTS (Heart Augmented Reality Training System) project, in which researchers from EPFL’s Computer Vision Laboratory (CV Lab) in the School of Computer and Communication Sciences (IC), physicians from the interventional MRI center of the Cardiology Division of Lausanne’s University Hospital (CHUV), and the Swiss company ADIS are jointly developing an Augmented Reality training system to enable doctors to practice on virtual 3D models of real patients’ hearts.

ADIS is in charge of developing the cardiac intervention simulator that allows a trainee to insert a real catheter into the "patient", which is nothing more than an empty box monitored by cameras observing the moving catheter. Its motion is transcribed into a 3D model of the catheter that is then inserted into a 3D heart model. In this way, the trainee can practice without any risk to a real patient.

At EPFL the focus is on building the 3D heart models. “We have developed ImHeart, a generative 3D heart model that is designed to properly model the composite nature of the heart with its four separate chambers. It incorporates technological topological constraints that ensure that they properly fit together and is parameterized by a latent vector. We are now working on extracting and refining this latent vector from Magnetic Resonance Imaging, or MRI, images. The goal is to fully automate the production of complete heart models, including scars and conduction channels,” explained Professor Pascal Fua, Head of the CV Lab.

The interventional MRI Center of the CHUV first provided a large MRI dataset containing data from a significant number of patients consisting of annotations of the epicardium, endocardium and the information necessary to compute scar tissue regions in the left main chamber of the heart. The quality of the annotations allowed CV Lab researchers to obtain good results on test data.

“A second large MRI dataset from the interventional MRI Center of the CHUV has whole heart information and we are currently working on extracting accurate 3D heart models from this data with their help. Meanwhile, ADIS has built an integrated system that takes the information from the catheter tracking module and visualizes it with the 3D heart,” Fua continued.

Dual-Purpose Platform

The project partners designed the HEARTS platform with a dual purpose in mind.

First, it will enable doctors in training to practice on virtual 3D models of real patients' hearts. As with flight simulators for pilots, the heart augmented reality simulator will lower risks, promote more thorough intervention planning and reduce interventional training costs, preparing medical personnel to then operate in the far more expensive settings required for real interventions.

Second, in a more operational context, it will allow interventionalists to simulate and plan operations before actually performing them. This is particularly important for patients who survive acute myocardial infarctions. With current state-of-the-art medical management, more of these patients survive but can suffer post-infarction inhomogeneous scar formation that can result in malignant arrhythmias and sudden cardiac death. To treat this, doctors use catheter ablation treatment to destroy dangerous “conducting channels” in the scar tissue, which requires highly trained interventionalists and meticulous pre-operation planning.

“To this end, the patient's actual scars will be incorporated into the simulation environment in which the damaged heart tissue will have been modelled from MRI images, established as one of the key modalities to detect abnormalities in the structure and function of the postinfarct heart,” said Fua.

The team has already built five platforms. One is installed at CHUV and the other four are intended for early customers. They comprise the catheter, three stereo cameras, a computing device, and the supporting infrastructure. The three stereo cameras generate a point cloud of the catheter that is then fed to a neural network, which returns estimates of the 3D catheter shape in real-time. These estimates are then used to predict what a real catheter would do when inserted in a real heart and in contact with the heart walls.

In short, the HEARTS platform makes it possible to train doctors to perform endoscopic treatment of malignant arrhythmias by exposing the trainees to many different hearts. It also allows for procedure planning and execution of simulated interventions on the anatomy of actual patients.