Artificial User

PROBLEM

Despite decades of research on human-computer interaction (HCI), the problem of how to predict human performance with a given user interface (UI) is unsolved. Existing computational models are limited in scope or require large datasets or expert input. Instead, empirical methods are used that are costly and error-prone.

OBJECTIVE

This project establishes the foundations of simulation intelligence in HCI through computational rationality. Given a design and a task environment, a simulator generates human-like moment-to-moment behavior autonomously, from which key metrics can be computed, i.e. for learning, performance, and ergonomics. “Artificial users” can be commanded using natural language without modeling expertise.

APPROACH

We seek a breakthrough through the theory of computational rationality that would dramatically expand the models’ scope and actionability. The theory posits that interactive behavior is an emergent consequence of a control policy adapted to internal bounds (cognition) and rewards.

While previous work has demonstrated progress, scope has been limited to simple sensorimotor tasks and required reward engineering. This project will study the principles of computationally rational agents that:

1. Learn skills like humans
2. Operate autonomously
3. Can be commanded via natural language

We design a workflow for building dramatically larger models via self-supervised pretraining.

IMPACT

We develop a strong complement to existing evaluation methods in HCI. This pushes forward generative modeling in HCI by combining theory-driven causal assumptions about people and modern ML, while being deployed directly in high-fidelity simulators.

This allows dealing with novel task environments with higher accuracy. This will be a leap forward in applications of design and engineering (rapid evaluation) and ML (realistic synthetic data) in HCI.