Statistical learning and regularisation for regression

Abstract or Introduction

This thesis deals with the problem of statistical learning. By "learning"', we mean a process by which we obtain a model of a phenomenon using data measured on it. We focus on system identification and time series modelling, and our work is naturally slightly influenced by this concern. We will for example evoke only regression estimation, and no classification problem. However, most theoretical and practical aspects presented herein can easily be adapted.

We study neural networks models. This research has traditionally been linked to computer science and artificial intelligence. In the last few years however, two distinct lines of thoughts seem to diverge: the first one stays close to the biological origins of the term - we will call it "neuro-biological"; the second considers neural networks as a model, and studies them from a statistical point of view.

Our work concerns the second of these lines. Furthermore, we try to stay independent of any specific application, and keep a general approach to neural networks. We attempt to exhibit the links between neural networks and statistics, and show that neural computation can be naturally placed in the realm of traditional statistics. We thus compare neural networks to other models: linear regression as well as non-parametric estimators. We also consider some of the numerous
learning techniques developed for neural models, and try to compare them, from both a theoretical and applied point of view.