Algorithmic Bias Control in Deep learning

Introduction

Deep Learning (DL) has reached unparalleled performance in many domains. However, this impressive performance typically comes at the cost of gathering large datasets and training massive models, requiring extended time and prohibitive costs.

Research Efforts

Significant research efforts are being invested in improving DL training efficiency, i.e., the amount of time, data, and resources required to train these models. This can be achieved by:

• Changing the model (e.g., architecture, numerical precision)
• Modifying the training algorithm (e.g., parallelization)

Other modifications aim to address critical issues, such as credibility and over-confidence, which hinder the implementation of DL in the real world. However, such modifications often cause an unexplained degradation in the generalization performance of DL to unseen data.

Algorithmic Bias

Recent findings suggest that this degradation is caused by changes to the hidden algorithmic bias of the training algorithm and model. This bias selects a specific solution from all solutions which fit the data. After years of trial-and-error, this bias in DL is often at a "sweet spot" which implicitly allows ANNs to learn well, due to unknown key design choices.

However, performance typically degrades when these choices change. Therefore, understanding and controlling algorithmic bias is the key to unlocking the true potential of deep learning.

Project Goal

Our goal is to develop a rigorous theory of algorithmic bias in DL and to apply it to alleviate critical practical bottlenecks that prevent such models from scaling up or being implemented in real-world applications.

Approach Objectives

Our approach has three objectives:

1. Identify the algorithmic biases affecting DL
2. Understand how these biases affect the functional capabilities and generalization performance
3. Control these biases to alleviate critical practical bottlenecks

Feasibility

To demonstrate the feasibility of this challenging project, we describe how recent advances and concrete preliminary results enable us to effectively approach all these objectives.