Graphs without Labels: Multimodal Structure Learning without Human Supervision

Introduction

Multimodal learning focuses on training models with data in more than one modality, such as videos capturing visual and audio information or documents containing image and text. Current approaches use such data to train large-scale deep learning models without human supervision by sampling pair-wise data, e.g., an image-text pair from a website, and train the network to identify matching vs. not matching pairs to learn better representations.

Argument for Multimodal Learning

We argue that multimodal learning can do more: by combining information from different sources, multimodal models capture cross-modal semantic entities. As most multimodal documents are a collection of connected modalities and topics, multimodal models should allow us to capture the inherent high-level topology of such data.

Project Goals

The goal of the following project is to learn semantic structures from multimodal data to capture long-range concepts and relations in multimodal data via multimodal and self-supervision learning without human annotation.

1. We will represent this information in the form of a graph, considering latent semantic concepts as nodes and their connectivity as edges.
2. Based on this structure, we will extend current unimodal approaches to capture and process data from different modalities in a single structure.

Challenges and Opportunities

Finally, we will explore the challenges and opportunities of the proposed idea with respect to their impact on two main challenges in machine learning:

• Data-efficient learning
• Fairness in label-free learning.

Bridging Trends

By bridging the gap between those two parallel trends, multimodal supervision and graph-based representations, we combine their strengths of generating and processing topological data. This will not only allow us to build new applications and tools but also open new ways of processing and understanding large-scale data that are currently out of reach.