📡 Information Theory — Intermediate#

Scope — Quantitative measures of information, coding efficiency, and limits on reliable communication.

Key concepts#

• Shannon entropy — average information per symbol in a source.
• Mutual information — shared information between variables; reduction in uncertainty.
• Source and channel coding — compression and error correction strategies.

Seed Q&A triads#

• Q: How does Shannon entropy quantify information?
  A: It computes the expected value of information content based on symbol probabilities.

• Q: What does mutual information tell us about two variables?
  A: How much knowing one variable reduces uncertainty about the other.

• Q: Why is data compression possible without losing information?
  A: Redundancy in sources allows efficient encoding that preserves essential information.

Short exercises#

• Calculate entropy for a simple discrete source with given symbol probabilities.
• Interpret mutual information between two correlated signals.