Information Theory

Status: draft · Confidence: low (0.43) · Basis: verified_sources

Quality notes: generic_source_homepage, no_verified_sources, partial_source_verification

## TL;DR

Information theory (Claude Shannon, 1948) quantifies information, establishing the theoretical limits of data compression and reliable communication. Entropy H = -Σ p(x) log p(x) measures uncertainty. Mutual information measures shared information between variables.

## Core Explanation

Shannon's source coding theorem: data can be compressed to entropy bits per symbol (lossless limit). Channel capacity: maximum rate of reliable communication over a noisy channel. Applications: data compression (Huffman coding, LZ77/ZIP), error-correcting codes (Reed-Solomon, LDPC, Turbo), cryptography (entropy of keys). Shannon invented the bit as a unit of information.

## Further Reading

-

## Related Articles

- [Quantum Entanglement: Theory, Bell's Theorem, and Quantum Information](../../science/quantum-entanglement-theory-bell-s-theorem-and-quantum-information.md)
- [AI for Game Theory: Computational Game Playing, Nash Equilibrium, and Multi-Agent Strategy](../../ai/ai-for-gaming-theory.md)
- [Information Extraction: NER, Relation Extraction, and LLM-Powered IE](../../ai/information-extraction.md)