Empirical Study of Extreme Overfitting Points of Neural Networks

Posted in Papers

In this paper we propose a method of obtaining points of extreme overfitting—parameters of modern neural networks, at which they demonstrate close to 100% training accuracy, simultaneously with almost zero accuracy on the test sample. Despite the widespread opinion that the overwhelming majority of critical points of the loss function of a neural network have equally good generalizing ability, such points have a huge generalization error. The paper studies the properties of such points and their location on the surface of the loss function of modern neural networks

I am PhD student at Skolkovo Institute of Science and Technology and Senior Lecturer at Moscow Institute of Physics and Technology. I love math, teaching, history, travelling and the ambiguity of being.

Read Next

Сказ 5. The Rise of Colab. Матрешка. Невозможность