Seeing Stuff: Perceiving Materials and their Properties

Introduction

Different materials, such as silk, soil, steel, and soap, exhibit an astonishing variety of physical properties, appearances, and behaviors. The material properties of objects and substances are central to practically every task we perform, from selecting and preparing food to detecting slippery ground and using tools effectively. Without touching a surface, we usually enjoy a vivid impression of what it would feel like through the sense of sight. Yet, how we do so remains mysterious.

Research Focus

Decades of research have focused on the visual recognition of objects, faces, and scenes. By comparison, how we see, think about, and interact with ‘stuff’ has been relatively neglected. STUFF addresses this major gap in our understanding.

Challenges in Material Perception

Material perception poses unique and fascinating challenges. The image of a surface is a complex and ambiguous combination of lighting, shape, and material properties.

Key Questions

1. How does the visual system disentangle these intermingled physical factors?
2. Deformable materials like liquids and textiles move and change shape in complex yet lawful ways. How do we infer intrinsic properties like viscosity, compliance, and elasticity from such ever-changing stimuli?
3. How do we reason about and predict their future behaviors as they interact with their surroundings?
4. How do we adapt our own interactions with objects to take into consideration their hardness, density, friction, and other physical characteristics, allowing us to pluck a raspberry without crushing it or pick up wet soap without it slipping through our fingers?

Interdisciplinary Approach

STUFF takes a radically interdisciplinary approach to these questions in five tightly interconnected work packages. This initiative brings together state-of-the-art methods from:

• Experimental psychology and behavioral neuroscience
• Computer graphics and computational image analysis
• Machine learning
• Art

We draw on real and simulated materials to uncover how we perceive, reason about, predict, and interact with materials and their properties.