Generative Flow Networks as Entropy-Regularized Reinforcement learning

• 主讲人：Alexey Naumov (National Research University Higher School of Economics)
• 举办方： Beijing-Saint Petersburg Mathematics Colloquium
• 时间： 2024-01-04 21:00 - 2024-01-04 22:00
• 地点： online

Recording: https://disk.pku.edu.cn:443/link/6635F43357C55A250F7FD461806C16DC
Valid Until: 2025-02-01 23:59

 

Abstract: The recently proposed generative flow networks (GFlowNets) are a method of training a policy to sample compositional discrete objects with probabilities proportional to a given reward via a sequence of actions. GFlowNets exploit the sequential nature of the problem, drawing parallels with reinforcement learning (RL). Our work extends the connection between RL and GFlowNets to a general case. We demonstrate how the task of learning a generative flow network can be efficiently redefined as an entropy-regularized RL problem with a specific reward and regularizer structure. Furthermore, we illustrate the practical efficiency of this reformulation by applying standard soft RL algorithms to GFlowNet training across several probabilistic modeling tasks. Contrary to previously reported results, we show that entropic RL approaches can be competitive against established GFlowNet training methods. This perspective opens a direct path for integrating reinforcement learning principles into the realm of generative flow networks. The talk is based on the joint work with Nikita Morozov, Daniil Tiapkin and Dmitry Vetrov.

 

Bio: Alexey Naumov is a professor and a head of the International laboratory of stochastic algorithms and high-dimensional inference of the National Research University Higher School of Economics.  He graduated from the Lomonosov Moscow State University (MSU) in 2010, received a PhD degree in mathematics from Bielefeld University in 2013, candidate of science in mathematics and physics from MSU in 2013 and doctor of computer science degree in 2022. Alexey is the world-famous specialist  in the areas of high-dimensional probability and mathematics of data science. His research interests span a broad range of problems extending from fundamental, theoretical questions in probability to applications in various branches of data science. Alexey Naumov is an author of more than 30 papers in leading journals and conferences of the field including Probability Theory and Related Fields, Bernoulli, Statistics and Computing, COLT, ICML and NeurIPS.