Scaling and Concentration Laws in Information Theory

Introduction

Shannon’s 1948 paper established the mathematical foundations of digital compression and transmission and paved the way for the information age. A notion embedded in Shannon’s and most work in Information Theory is the concept of rate, defined as the exponential growth rate of the number of messages.

Problem Statement

The probabilistic law governing general information processing systems may be such that the optimal number of messages does not scale exponentially with the length of the sequences. The vast majority of the Information Theory literature assumes an exponential number of messages and thus, ignores the rich amount of possible scaling functions in important settings.

Need for New Techniques

When the system probability law is such that the optimal rate scaling is not exponential, standard techniques cannot be applied and new techniques must be sought. A fundamental understanding of the impact of general scaling laws is crucial to harvesting the potential gains in practical system design.

Project Goals

This project is aimed at contributing towards the ambitious goal of providing a unified framework for the study of Information Theory with arbitrary scaling laws. The unified framework will be based on information spectrum and concentration techniques and will also consider the penalties incurred by sub-optimal decoding for systems described by general probabilistic laws.

Expected Outcomes

The project will also provide coding solutions that approach the theoretical limits. This unconventional and challenging treatment of Information Theory will advance the area and will contribute to Information Sciences and Systems disciplines where Information Theory is relevant.

Significance

A comprehensive study of the fundamental limits and optimal code design with general scaling laws will represent a major step forward in the field, with the potential to provide new tools and techniques to solve open problems in close disciplines.