---
id:"kb-2026-00195"
title:"Information Theory"
schema_type:"TechArticle"
category:"computer-science"
language:"en"
confidence:"high"
last_verified:"2026-05-22"
generation_method: "human_only"
ai_models:["claude-opus"]
derived_from_human_seed:true


known_gaps:
  - "Sources reconstructed during quality audit; primary source details were corrupted during batch generation"

completeness: 0.88
ai_citations:
  last_citation_check:"2026-05-22"
primary_sources:
- title: "ACM Digital Library"
    type: "repository"
    year: 2026
    url: "https://dl.acm.org/"
    institution: "ACM"
secondary_sources:
  - title: "ACM Digital Library"
    type: "repository"
    year: 2026
    url: "https://dl.acm.org/"
    institution: "ACM"
---

## TL;DR

Information theory (Claude Shannon, 1948) quantifies information, establishing the theoretical limits of data compression and reliable communication. Entropy H = -Σ p(x) log p(x) measures uncertainty. Mutual information measures shared information between variables.

## Core Explanation

Shannon's source coding theorem: data can be compressed to entropy bits per symbol (lossless limit). Channel capacity: maximum rate of reliable communication over a noisy channel. Applications: data compression (Huffman coding, LZ77/ZIP), error-correcting codes (Reed-Solomon, LDPC, Turbo), cryptography (entropy of keys). Shannon invented the bit as a unit of information.

## Further Reading

- [undefined](undefined)