DEep COgnition Learning for LAnguage GEneration

Introduction

In recent years, transformer-based deep learning models such as BERT or GPT-3 have led to impressive results in many natural language processing (NLP) tasks, exhibiting transfer and few-shot learning capabilities.

Limitations of Current Models

However, despite faring well in benchmarks, current deep learning models for NLP often fail badly in the wild. The main issues include:

• Poor out-of-domain generalization
• Inability to exploit contextual information
• Poor calibration
• Non-traceable memory

These limitations stem from their monolithic architectures, which are good for perception but unsuitable for tasks requiring higher-level cognition.

Project Goals

In this project, I attack these fundamental problems by bringing together tools and ideas from various fields, including:

1. Machine learning
2. Sparse modeling
3. Information theory
4. Cognitive science

This interdisciplinary approach aims to address the limitations of current models.

Methodology

Utility-Guided Controlled Generation

First, I will use uncertainty and quality estimates for utility-guided controlled generation. This will involve:

• Combining the control mechanism with efficient encoding of contextual information
• Integrating multiple modalities

Sparse and Structured Memory Models

Second, I will develop sparse and structured memory models, along with attention descriptive representations aimed at conscious processing.

Mathematical Models for Sparse Communication

Third, I will build mathematical models for sparse communication, which will involve:

• Reconciling discrete and continuous domains
• Supporting end-to-end differentiability
• Enabling a shared workspace for communication among multiple modules and agents

Application of Innovations

I will apply the innovations above to highly challenging language generation tasks, including:

• Machine translation
• Open dialogue
• Story generation

Collaborations

To reinforce interdisciplinarity and maximize technological impact, collaborations are planned with:

• Cognitive scientists
• A scale-up company in the crowd-sourcing translation industry