Call for imaging projects: interdisciplinarity in the spotlight!

The EPFL Center for imaging is pleased to see the increasing number of collaborations and cross-fertilizations being set up between EPFL imaging scientists with very different backgrounds. The four projects that were selected for this second call launched by the Center for Imaging are genuinely interdisciplinary, as were the other submissions. From astronomy to the imaging of historical buildings, from smart microscopy to behavior analysis, they promise to tackle ambitious challenges and bring innovative solutions to the latter.

Interdisciplinarity and cross-fertilization between various disciplines are key for the future progress in imaging. All five Schools and the CDH, as well as an EPFL platform, are represented in the projects that will start in 2023. “We are delighted to see that collaborations in imaging at EPFL are truly involving actors from all across the campus”, says Laurène Donati, executive head of the Center. The scientific committee selected four projects that, beyond their ambitious targets, are of very high scientific quality and have the potential to lead to new pioneering developments in imaging technology that could benefit the community at large.

Summaries of the four projects

Learned Scalable High Dynamic Range Imaging in Radio Astronomy

Jean-Philippe Thiran (LTS5- STI), Jean-Paul Kneib (LASTRO-SB), Emma Tolley (Scitas)

Next-generation radio telescopes such as the Square Kilometer Array (SKA) will observe the sky with unprecedented resolution, sensitivity, and survey speed. However, this precise instrument will demand reliable, precise, and high dynamic range deconvolution techniques to form images. The popular CLEAN algorithm, while efficient, often produces images of suboptimal quality. In recent years convex and nonconvex optimization algorithms have been demonstrated to produce images with superior quality, but at the cost of efficiency and scalability. Deep learning solutions offer a compromise between speed and quality, but at the cost of reliability and generalizability. In this project the researchers will leverage the expertise between the astronomy and signal processing research groups at EPFL to develop an end-to-end imaging solution that is precise, robust, and scalable.

Video-Based Action Segmentation by Learning World Models From Language

Alexander Mathis (Mathis Group - SV), Antoine Bosselut (NLP Lab - IC)

Many questions in biology, from development to neuroscience and medicine require the identification of fine-grained behaviors. With this project, researchers will develop novel computer vision and natural language processing technology to improve behavioral analysis in biology and medicine. Specifically, they will build deep learning models that can efficiently learn joint representations from video and heterogeneous data sources (e.g., textual descriptions, knowledge graphs). To do so, they intend to mine the written literature as well as video sharing platforms to extract a knowledge graph of behavior and then learn tri-modal models based on vision, language and this knowledge graph. The goal is to make these models able to more robustly and efficiently generalize to various applications in biology.

Spatiotemporal Adaptive Microscope Control, Driven by Biological Events

Suliana Manley (LEB- SB), Andrew Oates (Oates Lab – SV)

A key tool for studying the dynamics of living systems is the light microscope. Microscopes allow real-time recording of spontaneous or evoked spatio-temporal dynamics, data that can be used to develop models for how complex systems function. Today, cutting-edge microscopes can image below the diffraction limit of light (super-resolution microscopy), or over days, gently enough to allow an organism to develop and walk away (light-sheet microscopy). Yet, microscopy studies of biological systems largely rely on human control or pre-defined acquisition parameters, to identify features of interest, perturb the system, and collect data in a given location and at a given timescale. This is because subtle changes in protein dynamics and assembly patterns often herald events of interest, but they are too subtle and unreliable to act as inputs to existing microscope automation.

Advances in intelligent systems and adaptive control have the potential to revolutionize how microscopy data is collected, and to then enable breakthroughs in our understanding of biological systems. The researchers propose to develop a neural network-based microscope controller that is capable of detecting image signatures related to biological activity, and in response, adapting illumination patterns at multiple locations across an imaging field of view. The proposed project aims to build upon a neural-network microscope control framework previously developed in the Manley group, to make it suitable for spatially and temporally adapted control. They will apply this to push beyond the state-of-the art, using as proof-of-concept organismal studies performed in the Oates group, and biofilm studies in the Manley group.

3D imaging of Historical Buildings

Katrin Beyer (EESD- ENAC), Frederic Kaplan (DHLAB- CDH)

When it comes to generating 3D digital geometric models of historical buildings, the automation of methods is still limited. Existing research focused on sacral structures, on 3D model generation of the exterior envelope of buildings and on segmentation of interior spaces. The goal of this project is to develop a data acquisition and post-processing pipeline for deriving the exterior and interior geometry of historical buildings in terms of 3D point clouds derived from spherical RGB images, to augment this data with information extracted from historical architectural drawings, and to approximate the 3D point clouds by geometrical primitives describing the architectural and structural elements in historical buildings. Because interior spaces are often very privacy-sensitive spaces, privacy-issues will be considered from the start, limiting the stored data to data containing architectural and structural elements. The derived models can be used for a wide range of research applications, such as 4D modelling showing the evolution of a historical building over time and structural modelling of historical buildings, which requires as input a geometric model of the building.