Session: Optimization on Riemannian manifolds and stratified sets (2/2)
Chair: Guillaume Olikier
Cluster: Optimization on Manifolds
Talk 1: A space-decoupling framework for optimization on bounded-rank matrices with orthogonally invariant constraints
Speaker: Bin Gao
Abstract: Imposing additional constraints on low-rank optimization has garnered growing interest recently. However, the geometry of coupled constraints restricts the well-developed low-rank structure and makes the problem nonsmooth. In this paper, we propose a space-decoupling framework for optimization problems on bounded-rank matrices with orthogonally invariant constraints. The "space-decoupling" is reflected in several ways. Firstly, we show that the tangent cone of coupled constraints is the intersection of the tangent cones of each constraint. Secondly, we decouple the intertwined bounded-rank and orthogonally invariant constraints into two spaces, resulting in optimization on a smooth manifold. Thirdly, we claim that implementing Riemannian algorithms is painless as long as the geometry of additional constraint is known a prior. In the end, we unveil the equivalence between the original problem and the reformulated problem. The numerical experiments validate the effectiveness and efficiency of the proposed framework.
Talk 2: A Riemannian rank-adaptive method for tensor completion in the tensor-train format
Speaker: Charlotte Vermeylen
Abstract: Riemannian rank-adaptive methods (RRAMs) are state-of-the-art methods that aim to optimize a continuously differentiable function on a low-rank variety, a problem appearing notably in low-rank matrix or tensor completion. The RRAMs that are developed for the set of bounded rank matrices iteratively optimize over smooth fixed-rank manifolds, starting from a low initial rank. They increase the rank by performing a line search along a descent direction selected in the tangent cone to the variety. This direction can be the projection of the negative gradient onto the tangent cone but does not need to; for instance, the RRAM developed by Gao and Absil uses the projection of the negative gradient onto the part of the tangent cone that is normal to the tangent space. Additionally, they decrease the rank based on a truncated SVD when the algorithm converges to an element of a lower-rank set. This is possible because the manifold is not closed. We aim to generalize these RRAMs to the tensor-train (TT) format. In this format, only the RRAM by Steinlechner is known to us from the literature. This method is developed for high-dimensional tensor completion and has a random rank update mechanism in the sense that each TT-rank is increased subsequently by one by adding a small random term in the TTD of the current best approximation such that the cost function does not increase much. No rank reduction step is included which makes the algorithm prone to overfitting. We improve this RRAM by including a method to increase the TT-rank based on an approximate projection of the negative gradient onto the tangent cone to the variety. The tangent cone is the set of all tangent vectors to the variety. The approximate projection ensures that the search direction is sufficiently gradient related. Furthermore, a method is proposed to determine how much the rank should be increased for the low-rank tensor completion problem (LRTCP). Lastly, a method to decrease the rank is proposed, which is necessary when the iterate comes close to a lower-rank set. This is possible because, as for the manifold of fixed-rank matrices, the manifold of fixed-rank TTDs is not closed.
Talk 3: Low-rank optimization on Tucker tensor varieties
Speaker: Renfeng Peng
Abstract: In the realm of tensor optimization, the low-rank Tucker decomposition is crucial for reducing the number of parameters and saving storage. In this talk, we delve into the geometry and optimization methods for Tucker tensor varieties---the set of tensors with bounded Tucker rank---which is notably more intricate than the well-explored matrix varieties. We give an explicit parametrization of the tangent cone of Tucker tensor varieties and leverage its geometry to develop provable gradient-related line-search methods for optimization on Tucker tensor varieties. In practice, low-rank tensor optimization suffers from the difficulty of choosing a reliable rank parameter. To this end, we incorporate the established geometry and propose a Tucker rank-adaptive method that aims to identify an appropriate rank. Numerical experiments on tensor completion reveal that the proposed methods are in favor of recovering performance over other state-of-the-art methods. This is joint work with Bin Gao (AMSS) and Ya-xiang Yuan (AMSS).
