Modern Challenges in Learning Theory

Introduction

Recent years have witnessed tremendous progress in the field of Machine Learning (ML). Learning algorithms are applied in an ever-increasing variety of contexts, ranging from engineering challenges such as self-driving cars all the way to societal contexts involving private data.

Challenges in Machine Learning

These developments pose important challenges:

1. Many of the recent breakthroughs demonstrate phenomena that lack explanations and sometimes even contradict conventional wisdom. One main reason for this is because classical ML theory adopts a worst-case perspective, which is too pessimistic to explain practical ML. In reality, data is rarely worst-case, and experiments indicate that often much less data is needed than predicted by traditional theory.

2. The increase in ML applications that involve private and sensitive data highlights the need for algorithms that handle the data responsibly. While this need has been addressed by the field of Differential Privacy (DP), the cost of privacy remains poorly understood. Specifically, how much more data does private learning require compared to learning without privacy constraints?

Research Objective

Inspired by these challenges, our guiding question is: How much data is needed for learning?

Proposed Approach

Towards answering this question, we aim to develop a theory of generalization which complements the traditional theory and is better fit to model real-world learning tasks. We will base it on:

• Distribution-dependent perspectives
• Data-dependent perspectives
• Algorithm-dependent perspectives

These approaches complement the distribution-free worst-case perspective of the classical theory and are suitable for exploiting specific properties of a given learning task.

Study Settings

We will use this theory to study various settings, including:

• Supervised learning
• Semisupervised learning
• Interactive learning
• Private learning

Expected Impact

We believe that this research will advance the field in terms of efficiency, reliability, and applicability. Furthermore, our work combines ideas from various areas in computer science and mathematics; we thus expect further impact outside our field.