Information theory studies the fundamental limits of the representation and transmission of information. This course provides an introduction to information theory, studying fundamental concepts such as probability, information, and entropy and examining their applications in the areas of data compression, coding, communications, pattern recognition and probabilistic inference.

Upon successful completion of the course, the student will have background knowledge necessary to understand problems in data compression, storing and communication and undertake advanced courses on statistical inference, machine learning and information engineering. In particular, the student will be able to:

• Understand and apply fundamental concepts in information theory such as probability, entropy, information content and their inter-relationships.
• Understand the principles of data compression.
• Compute entropy and mutual information of random variables.
• Implement and analyse basic coding and compression algorithms.
• Understand the relationship of information theoretical principles and Bayesian inference in data modelling and pattern recognition.
• Understand some key theorems and inequalities that quantify essential limitations on compression, communication and inference.
• Know the basic concepts regarding communications over noisy channels.

Assignment 1 (10%) Assignment 2 (20%) Assignment 3 (20%) Final Exam (50%)

Twenty-six one-hour lectures and five two-hour tutorial sessions.

Some background in elementary statistics and probability.

See Assumed Knowledge

Some background in elementary statistics and probabilities and programming experience.

Information Theory, Inference, and Learning Algorithms by David MacKay, Cambridge University Press, 2003.
Additional reading: Elements of Information Theory by Cover and Thomas, 2nd Edition, New York, Wiley, 2006.