lecture06: Entropy and Mutual Information

Topics covered

• Simple Properties
• Entropy Chain Rule
• Entropy Chain Rule: Corollaries
• Entropy Chain Rule: General form
• Relative Entropy Chain Rule
• Relative Entropy Properties
• Corollary
• Convexity of relative entropy
• Corollary: concavity of $H$
• Conditioning reduces entropy
• Motivation
• Mutual Information
• Relationship between entropy and mutual information
• Mutual Information Properties
• Assignment