A Foundation for Empirical Multimodality Research

Introduction

This project lays a foundation for conducting data-driven empirical research on multimodality, that is, how human communication and interaction rely on combinations of 'modes' of expression. Theories of multimodality are rapidly gaining currency in diverse fields concerned with human communication, interaction, and cultural production. However, most theories of multimodality are based on conjecture and remain without an empirical foundation due to the lack of large, richly-annotated multimodal corpora and methods for their analysis.

Methodology Development

FOUNDATIONS solves this problem by developing a novel methodology for conducting empirical research on multimodality in everyday cultural artefacts, such as newspapers, textbooks, magazines, and social media videos. This methodology allows for critically examining key theoretical concepts in the field – medium, semiotic mode, and genre – and their joint contribution to meaning-making. This renewal enhances our understanding of key theoretical concepts in multimodality research and places their definitions on a solid empirical foundation.

Crowdsourcing and Data Collection

To do so, FOUNDATIONS creates large and reproducible multimodal corpora using microtask crowdsourcing, which breaks complex tasks into piecemeal work and distributes this effort to non-expert workers on online platforms. These crowdsourced descriptions are combined with computational representations into graphs that represent the structure of multimodal discourse.

Analysis Techniques

For analysing these corpora, the project develops novel methods based on neuro-symbolic artificial intelligence, which enables combining crowdsourced human insights with the pattern recognition capability of neural networks.

Impact and Future Directions

The groundbreaking theoretical and methodological advances in FOUNDATIONS go far beyond the state of the art by enabling large-scale empirical research while preserving the analytical depth needed for multimodality research. This opens up new domains of inquiry for studying multimodality across cultures, situations, artefacts, and timescales.