ABSTRACT: Classical theory that guides the design of nonparametric prediction methods like deep neural networks involves a tradeoff between the fit to the training data and the complexity of the prediction rule. Deep learning seems to operate outside the regime where these results are informative, since deep networks can perform well even with a perfect fit to noisy training data. We investigate this phenomenon of ‘benign overfitting’ in the simplest setting, that of linear prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of effective rank of the data covariance. It shows that overparameterization is essential: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. It also shows an important role for finite-dimensional data: benign overfitting occurs for a much narrower range of properties of the data distribution when the data lies in an infinite dimensional space versus when it lies in a finite dimensional space whose dimension grows faster than the sample size. We discuss implications for deep networks, for robustness to adversarial examples, and for the rich variety of possible behaviors of excess risk as a function of dimension. This is joint work with Phil Long, Gábor Lugosi, and Alex Tsigler.
SPEAKER: Peter Bartlett is professor of Computer Science and Statistics at the University of California, Berkeley, Associate Director of the Simons Institute for the Theory of Computing, Director of the Foundations of Data Science Institute, and Director of the Collaboration on the Theoretical Foundations of Deep Learning. His research interests include machine learning and statistical learning theory and he is the co-author of the book Neural Network Learning: Theoretical Foundations. He has been an Institute of Mathematical Statistics Medallion Lecturer, a winner of the Malcolm McIntosh Prize for Physical Scientist of the Year, and an Australian Laureate Fellow, and he is a Fellow of the IMS, ACM, and the Australian Academy of Science.
Ещё видео!