CoSy zoom seminar

Abstract: Deep networks are typically trained with many more parameters than the size of the training dataset. Recent empirical evidence indicates that the practice of overparameterization not only benefits training large models, but also assists – perhaps counterintuitively – building lightweight models. Specifically, it suggests that overparameterization benefits model pruning / sparsification. In this talk we shed light on these empirical findings by discussing our theory on the high-dimensional asymptotics of model pruning in the overparameterized regime. The theory presented addresses the following core question: ``should one train a small model from the beginning, or first train a large model and then prune?''. We analytically identify regimes in which, even if the location of the most informative features is known, we are better off fitting a large model and then pruning rather than simply training with the known informative features. This leads to a new double descent in the training of sparse models: growing the original model, while preserving the target sparsity, improves the test accuracy as one moves beyond the overparameterization threshold. Our analysis further reveals the benefit of retraining by relating it to feature correlations. We find that the above phenomena are already present in linear and random-features models. The intuition gained by analytically studying simpler models exhibit a remarkable match with practical neural networks.

Bio:Dr. Oymak is an Assistant Professor at the Department of Electrical and Computer Engineering at the University of California, Riverside. Prior to UCR, he spent three years as a machine learning researcher working in software industry and algorithmic finance. During his postdoc, he was a recipient of the Simons Fellowship from UC Berkeley. In 2015, he received the Charles Wilts Prize for the best departmental PhD thesis from the California Institute of Technology for data-efficient machine learning approaches. He obtained his BS degree from Bilkent University and MS degree from Caltech in 2009 and 2011 respectively. He is also a recipient of the UCR Regents' Faculty Fellowship and the NSF CAREER award.