Session: Regularization Methods for stochastic operator splitting dynamics
Chair: Mathias Staudigl
Cluster: Nonsmooth Optimization
Talk 1: Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces
Speaker: Rodrigo Maulen
Abstract: To solve convex optimization problems with a noisy gradient input, we analyze the global behavior of subgradient-like flows under stochastic errors. The objective function is composite, being equal to the sum of two convex functions, one being differentiable and the other potentially non-smooth. We then use stochastic differential inclusions where the drift term is minus the subgradient of the objective function, and the diffusion term is either bounded or square-integrable. In this context, under Lipschitz's continuity of the differentiable term and a growth condition of the non-smooth term, our first main result shows almost sure weak convergence of the trajectory process towards a minimizer of the objective function. Then, using Tikhonov regularization with a properly tuned vanishing parameter, we can obtain almost sure strong convergence of the trajectory towards the minimum norm solution. We find an explicit tuning of this parameter when our objective function satisfies a local error-bound inequality. We also provide a comprehensive complexity analysis by establishing several new pointwise and ergodic convergence rates in expectation for the convex and strongly convex case.
Talk 2: Accelerated Variance-Reduced Forward-Reflected Methods for Generalized Equations
Speaker: Quoc Tran-Dinh
Abstract: In this talk, we propose a novel class of Nesterov’s stochastic accelerated forward- reflected-based methods with variance reduction to solve [non]linear equations under a co-coerciveness assumption. Our algorithm is single-loop and leverages a new family of unbiased variance-reduced estimators specifically designed for a wider class of root-finding algorithms. It achieves both O(L^{2}/k^{2}) and o(1/k^2)-last-iterate convergence rates in terms of expected operator squared norm, where k denotes the iteration counter. We instantiate our framework for two prominent estimators: SVRG and SAGA. By an appropriate choice of parameters, both variants attain an oracle complexity of O(n+ Ln^{2/3}ϵ−1) to reach an ϵ-solution, where n represents the number of summands in the finite-sum operator. Furthermore, under µ-strong quasi-monotonicity, our method achieves a linear convergence rate and an oracle complexity of O(n+ max{n,n^{2/3}κ} log(ϵ−1)), where κ := L/µ. We extend our approach to solve a class of finite-sum monotone inclusions, demonstrating that our schemes retain the same theoretical guarantees as in the equation setting. Finally, we present some numerical experiments to validate our algorithms and demonstrate their promising performance compared to state-of-the-art methods.
Talk 3: Tikhonov regularized exterior penalty dynamics for constrained variational inequalities
Speaker: Mathias Staudigl
Abstract: A recent trend in optimization and variational analysis is the investigation of dynamical systems, either in discrete or continuous time, with explicit regularization of the trajectory using either Tikhonov of the Halpern method. These schemes exhibit a higher degree of stability and in some cases also exhibit interesting acceleration phenomena. Within an infinite-dimensional Hilbert space context, this paper is concerned with the study of a class of constrained variational inequalities in which the constrained domain can be represented as the zero set of a maximally monotone operator. The Tikhonov regularization parameter is assumed to tend to zero as time tends to infinity, which preserves equilibria and serves to enforce strong convergence of the trajectory. To steer the dynamical system towards the feasible set over which the original variational problem is to be solved, we append to the trajectory and exterior penalization term. The combination of the Tikhonov regularization and the penalization lead to a multiscale dynamical system, pioneered in the work of Attouch, Czarnecki, Peypouqet, and many others. In this paper we study the case of a cocoercive and non-cocoercive monotone operator separately. Full Splitting based dynamical systems, based on the Forward-Backward and Forward-Backward-Forward Splitting method are constructed. Proofs of existence and uniqueness of the solution trajectories of these new time-varying dynamical systems, as well as the strong convergence towards the least norm solution of the underlying variational inequality problem are derived. Numerical experiments conclude this work, showing the reconstruction power of the approach.
