IC researchers recognized with ICML Outstanding Paper Award
Neural network for artificial intelligence © iStock / EPFL 2023
The paper Generalization on the Unseen, Logic Reasoning and Degree Curriculum tackles the problem of better understanding the reasoning capabilities of neural networks like Transformers, and how such models can be improved to afford better “extrapolation” properties.
The International Conference on Machine Learning (ICML) is one of the most important and fastest growing conferences on machine learning with more than 6500 paper submissions in 2023.
This year six papers were chosen as recipients of the Outstanding Paper Award including the paper submitted by EPFL researchers from the School of Computer and Communications Sciences (IC) and colleagues at Apple. Generalization on the Unseen, Logic Reasoning and Degree Curriculumconsiders the learning of logical (Boolean) functions with focus on the generalization on the unseen (GOTU) setting, a strong case of out-of-distribution generalization.
This is motivated by the fact that the rich combinatorial nature of data in certain reasoning tasks (e.g., arithmetic/logic) makes representative data sampling challenging, and learning successfully under GOTU gives a first vignette of an 'extrapolating' or 'reasoning' learner. In particular, the paper shows that Transformers tend to have a ‘minimal-degree-bias’ that prevents extrapolation in certain Boolean settings.
“Our findings lead to two implications: they have allowed us to provide an explanation to the length generalization problem and secondly, they lead to the development of a curriculum learning algorithm called Degree-Curriculum that learns certain hard functions more efficiently by feeding the the data to the model in an incremental complexity order,” explained Professor Emmanuel Abbé, Chair of Mathematical Data Science at EPFL.
Abbé hopes the research will lead to further developments on how to improve current architectures in reasoning tasks and in particular mathematical tasks.
This year’s six Outstanding Papers papers marked the 40th anniversary of the ICML and the Award Paper Committee said it believes the community will appreciate the extremely strong contributions of these papers.
Generalization on the Unseen, Logic Reasoning and Degree Curriculum was authored by Emmanuel Abbé (EPFL, Apple) , Samy Bengio (Apple), Aryo Lotfi (EPFL) and Kevin Rizk (EPFL).