Creating an avatar from a 3D selfie

“We wanted the process to be fast and easy: all you have to do is take a video of yourself and then snap a few more shots to get facial expressions, and our algorithm does the rest,” said researcher Alexandru Ichim from the Computer Graphics and Geometry Laboratory at EPFL. The user’s digital double can be displayed on a screen and animated in real time if so desired with a video camera that follows the user’s movements. “The goal was to make the process accessible to anyone with a smartphone, even an old model, as long as it can take video,” said the researcher.

The program’s designers see a number of possible uses in the near future: gaming, virtual reality, online discussions with other avatars, embedding in films, video conferences, and even avatar therapy for people suffering from schizophrenia – something that’s already done at University College London.

Smiles are unique

Creating an avatar is simple for the user, but the underlying algorithms are not. Using a smartphone to replace studio conditions – which include proper lighting and numerous cameras – was a real challenge. “We begin by assuming that people will take pictures of themselves in conditions that are impossible to control,” said Alexandru. The main difficulties: changes in the light, blurry shots without a tripod, and limited picture quality depending on the smartphone's camera.

All these factors had to be juggled to achieve a good final result. Because to be credible, an avatar has to be nearly perfect. “A small detail will turn people off immediately,” said Alexandru. “The avatar has to have the right facial geometry and reproduce the texture, color and details like face wrinkles.” Then there’s the animation: everyone has their own way of smiling, yawning and frowning. Faithfully recreating such expressions required new algorithms for real time face animation to bring the reconstructed 3D avatars alive, even on mobile devices.

In the future, the EPFL researchers hope to further refine their process, including getting past the limits they currently face. Precisely duplicating hair, for example, is very complicated. And they have not yet customized the inside of the mouth – with the teeth and tongue – or the ears. Modeling the entire body is also in the works, although the researchers wanted to focus on the face first, since it is the most expressive part of human communication.

Source: Dynamic 3D avatar creation from hand-held video input, Alexandru-Eugen Ichim, Sofien Bouaziz, Mark Pauly, ACM Transactions on Graphics (Proceedings of SIGGRAPH), 2015

Pictures: © Murielle Gerber, EPFL 2015.