Spatial 3D Semantic Understanding for Perception in the Wild

Introduction

Understanding the 3D spatial semantics of the world around us is core to visual perception and digitization. Real-world environments are spatially three-dimensional and must be understood in their 3D context, even from 2D image observations.

Importance of 3D Perception

This understanding will lead to spatially-grounded reasoning and a higher-level perception of the world around us. Such 3D perception will provide the foundation for transformative, next-generation technology across various fields, including:

• Machine perception
• Immersive communications
• Mixed reality
• Architectural or industrial modeling
• And more

New Paradigm in Semantic Understanding

This will enable a new paradigm in semantic understanding that derives primarily from a spatially-consistent, 3D representation rather than relying on image-based reasoning that captures only projections of the world.

Challenges in 3D Semantic Reasoning

However, 3D semantic reasoning from visual data such as RGB or RGB-D observations remains in its infancy. This is due to challenges in:

1. Learning from limited amounts of real-world 3D data
2. The complex, high-dimensional nature of the problem

Proposed Solutions

In this proposal, we will develop new algorithmic approaches to effectively learn robust visual 3D perception. This will include new learning paradigms for features, representations, and operators to encompass 3D semantic understanding.