Session: Frontiers of Optimization for Machine Learning - Part III
Chair: Fred Roosta
Cluster: Nonlinear Optimization
Talk 1: Randomized Techniques for Fast and Scalable Operator
Speaker: Michael Mahoney
Abstract: The approximation of linear operators and their inverses lies at the heart of many optimization and machine learning algorithms. This work improves the efficiency and scalability of these tasks by integrating two central computational tools of recent decades-- randomization and preconditioning. A particular focus is placed on addressing large-scale applications on modern hardware architectures with stringent communication and memory constraints. Notably, the proposed methods are designed to enable effective recycling of computations when dealing with sequences of operators with similar intrinsic properties, a scenario frequently arising in iterative optimization algorithms.
Talk 2: Approximation and Preconditioning
Speaker: Matus Telgarsky
Abstract: This talk will demonstrate a duality-based proof technique for establishing that coordinate and gradient descent follow specific paths (and not just limit points) for linear and deep network classifiers.
Talk 3: Example Selection for Distributed Learning
Speaker: Christopher de Sa
Abstract: Training example order in SGD has long been known to affect convergence rate. Recent results show that accelerated rates are possible in a variety of cases for permutation-based sample orders, in which each example from the training set is used once before any example is reused. This talk will cover a line of work in my lab on decentralized learning and sample-ordering schemes. We will discuss the limits of the classic gossip algorithm and random-reshuffling schemes and explore how both can be improved to make SGD converge faster both in theory and in practice with little overhead.
