High-capacity hypothesis spaces in modern statistical learning

This work attempts to give an overview over the modern questions in statistical learning sparked by the success of deep learning. In particular, benign overfitting and overparameterization are discussed by means of kernel ridge(less) regression. Due to recent interest in kernel methods in this field, the underlying theory is developed in-depth. Finally, a simple model is proposed that seems to reveal many of the interesting aspects of deep learning, and this is demonstrated experimentally by a brief discussion of the "double descent" phenomenon.