Towards a computational account of natural sequential behavior

Introduction

Imagine navigating through your kitchen, walking around your table and chairs, reaching the counter, sliding a cup on the countertop, pouring yourself some coffee, and preparing a sandwich. Such an everyday situation requires a sequence of behaviors involving the sophisticated interplay of perception, attention, cognition, planning, and action, all unfolding in an ambiguous and uncertain world.

Background

Yet, although past research in the behavioral sciences has revealed many properties of these constituent faculties, the understanding of how humans coordinate these cognitive abilities when carrying out everyday tasks is still rudimentary.

Project Goals

The goal of this project is to carry out behavioral experiments and develop computational models to identify and characterize perception, cognition, and action in the wild. Our hypothesis is that sequential actions involving:

1. Perceptual uncertainty
2. Action variability
3. Internal and external costs and benefits

can be understood and modeled in the unified framework of optimal sequential decision making under uncertainty.

Methodology

To achieve this goal, we integrate ideas and methods from:

• Psychophysics
• Statistical decision theory
• Active vision
• Intuitive physics
• Optimal control under uncertainty
• Navigation
• Inverse reinforcement learning

The proposal is structured in such a way that subsequent work packages consider tasks with increasing naturalness, from classical psychophysical production tasks, visuomotor control, to navigation and sandwich making.

Expected Outcomes

We will develop methods that allow recovering subjects’:

• Perceptual uncertainties
• Beliefs about tasks
• Subjective costs and benefits, including effort
• Action variability

Based on our previous experimental work on visuomotor behavior and theoretical work in modeling such behavior on an individual-by-individual and trial-by-trial basis, we seek to elucidate how perception, cognition, and action are inseparably intertwined in extended, sequential behavior.