Dynamics-Aware Theory of Deep Learning

Introduction

The recent advances in deep learning (DL) have transformed many scientific domains and have had major impacts on industry and society. Despite their success, DL methods do not obey most of the wisdoms of statistical learning theory, and the vast majority of the current DL techniques mainly stand as poorly understood black-box algorithms.

Theoretical Gaps

Even though DL theory has been a very active research field in the past few years, there is a significant gap between the current theory and practice:

1. The current theory often becomes vacuous for models with a large number of parameters (which is typical in DL).
2. It cannot capture the interaction between data, architecture, training algorithm, and its hyper-parameters, which can have drastic effects on the overall performance.

Due to this lack of theoretical understanding, designing new DL systems has been dominantly performed by ad-hoc, 'trial-and-error' approaches.

Objectives

The main objective of this proposal is to develop a mathematically sound and practically relevant theory for DL, which will ultimately serve as the basis of a software library that provides practical tools for DL practitioners. In particular:

1. We will develop error bounds that closely reflect the true empirical performance by explicitly incorporating the dynamics aspect of training.
2. We will develop new model selection, training, and compression algorithms with reduced time/memory/storage complexity by exploiting the developed theory.

Theoretical Framework

To achieve the expected breakthroughs, we will develop a novel theoretical framework, which will enable tight analysis of learning algorithms in the lens of dynamical systems theory. The outcomes will help relieve DL from being a black-box system and avoid the heuristic design process.

Software Development

We will produce comprehensive open-source software tools adapted to all popular DL libraries and test the developed algorithms on a wide range of real applications arising in computer vision, audio/music/natural language processing.