State source coding theorem. State channel capacity theorem. State channel coding theorem for a discrete memory less channel. What is prefix coding?
|Published (Last):||11 June 2019|
|PDF File Size:||19.43 Mb|
|ePub File Size:||15.89 Mb|
|Price:||Free* [*Free Regsitration Required]|
Overview[ edit ] Information theory studies the transmission, processing, extraction, and utilization of information. Abstractly, information can be thought of as the resolution of uncertainty. In the case of communication of information over a noisy channel, this abstract concept was made concrete in by Claude Shannon in his paper "A Mathematical Theory of Communication", in which "information" is thought of as a set of possible messages, where the goal is to send these messages over a noisy channel, and then to have the receiver reconstruct the message with low probability of error, in spite of the channel noise.
Information theory is a broad and deep mathematical theory, with equally broad and deep applications, amongst which is the vital field of coding theory. Coding theory is concerned with finding explicit methods, called codes, for increasing the efficiency and reducing the error rate of data communication over noisy channels to near the channel capacity.
These codes can be roughly subdivided into data compression source coding and error-correction channel coding techniques. A third class of information theory codes are cryptographic algorithms both codes and ciphers.
Concepts, methods and results from coding theory and information theory are widely used in cryptography and cryptanalysis. See the article ban unit for a historical application.
Main article: History of information theory The landmark event that established the discipline of information theory and brought it to immediate worldwide attention was the publication of Claude E.
Prior to this paper, limited information-theoretic ideas had been developed at Bell Labs , all implicitly assuming events of equal probability. The unit of information was therefore the decimal digit , which has since sometimes been called the hartley in his honor as a unit or scale or measure of information.
Alan Turing in used similar ideas as part of the statistical analysis of the breaking of the German second world war Enigma ciphers. Much of the mathematics behind information theory with events of different probabilities were developed for the field of thermodynamics by Ludwig Boltzmann and J. Willard Gibbs. Connections between information-theoretic entropy and thermodynamic entropy, including the important contributions by Rolf Landauer in the s, are explored in Entropy in thermodynamics and information theory.
Quantities of information[ edit ] Main article: Quantities of information Information theory is based on probability theory and statistics. Information theory often concerns itself with measures of information of the distributions associated with random variables. Important quantities of information are entropy, a measure of information in a single random variable, and mutual information, a measure of information in common between two random variables.
The former quantity is a property of the probability distribution of a random variable and gives a limit on the rate at which data generated by independent samples with the given distribution can be reliably compressed. The latter is a property of the joint distribution of two random variables, and is the maximum rate of reliable communication across a noisy channel in the limit of long block lengths, when the channel statistics are determined by the joint distribution. The choice of logarithmic base in the following formulae determines the unit of information entropy that is used.
A common unit of information is the bit, based on the binary logarithm. Other units include the nat , which is based on the natural logarithm , and the decimal digit , which is based on the common logarithm. This is justified because lim.
IT2302-Information Theory and Coding
IT2302- INFORMATION THEORY AND CODING