Information theory and measure theory

This article discusses how information theory (a branch of mathematics studying the transmission, processing and storage of information) is related to measure theory (a branch of mathematics related to integration and probability).

Many of the concepts in information theory have separate definitions and formulas for continuous and discrete cases. For example, entropy ${\displaystyle H(X)}$ is usually defined for discrete random variables, whereas for continuous random variables the related concept of differential entropy, written ${\displaystyle h(X)}$, is used (see Cover and Thomas, 2006, chapter 8). Both these concepts are mathematical expectations, but the expectation is defined with an integral for the continuous case, and a sum for the discrete case.

...
Wikipedia