Unravelling light to understand the visible world

Fully computer-generated images that appear indistinguishable from reality, have become a routine occurrence in our daily lives: we see them in product catalogs, magazines, and films, often without being aware that they aren’t real. Assistant Professor Wenzel Jakob, head of EPFL’s Realistic Graphics Lab (RGL), has worked on improving the realism of such images for more than a decade, simulating subtle aspects of how light interacts with objects in a virtual world.

Now, Jakob and his team are turning this idea on its head by running the simulation in reverse, unravelling the physics of light and instead of creating life like images of a virtual world, they are developing algorithms that start with one or more real images and produce a three-dimensional virtual world as output.

One exciting potential application of UNRAVEL’s work, that has just received a 1.6 million CHF starting grant from the European Research Council (ERC), is to develop inversion algorithms tailored to specific scientific disciplines, including the observation of the Earth’s atmosphere. 

“Think of today’s climate research: everything hinges on satellite-based measurements of atmospheric gases like CO₂. Of course, these satellites are in space, so they cannot measure the atmosphere directly! What they do instead is to capture photographs that must be processed to estimate the gas concentrations. Our objective is to contribute algorithms that will greatly improve the accuracy of exactly those kinds of calculations”, says Jakob. 

This work expands the traditional boundaries of computer graphics and Jakob is looking forward to the broader impact that it might have on scientic disciplines beyond earth observation, including fields as diverse as architecture and microscopy. 

“Suppose we’re looking at an object that has a certain color in a photograph. Can we really be sure of this? Perhaps the object is actually white, and the color arises from a reflection of the surrounding environment”, explains Jakob. “We can now begin to consider these questions thanks to the dramatic progress in an area known as differentiable rendering that lies at the core of UNRAVEL. My lab has been at the forefront of this new development and now the ERC funding gives us the resources to dramatically improve it, learn how to wield it, and pursue several different applications.”

ERC Starting Grants are awarded to researchers of any nationality with two to seven years of experience, and “a scientific track record showing great promise”. Awardees receive up to €2.5 million per grant for work to be carried out over a maximum of five years.