An intuitive introduction to information theory

Ivo Grosse Leibniz Institute of Plant Genetics and Crop Plant Research Gatersleben Bioinformatics Centre Gatersleben-Halle

Outline

 Why information theory?

 An intuitive introduction

2

History of biology

St. Thomas Monastry, Brno

3

Genetics 4

Gregor Mendel 1822 – 1884 1866 Mendel‘s laws Foundation of Genetics Ca. 1900: Biology becomes a quantitative science

50 years later … 1953

James Watson & Francis Crick

5

50 years later … 1953 6

7

DNA 8

Watson & Crick 1953 Double helix structure of DNA 1953: Biology becomes a molecular science

1953 – 2003 … 50 years of revolutionary discoveries 9

1989 10

1989

Goals:  Identify all of the ca. 30.000 genes  Identify all of the ca. 3.000.000.000 base pairs  Store all information in databases  Develop new software for data analysis

11

2003 Human Genome Project officially finished 12

2003: Biology becomes an information science

2003 – 2053 … biology = information science 13

2003 – 2053 … biology = information science

Systems Biology

14

What is information?

 Many intuitive definitions  Most of them wrong  One clean definition since 1948  Requires 3 steps - Entropy - Conditional entropy - Mutual information

15

Before starting with entropy …

Who is the father of information theory?

Who is this?

Claude Shannon 1916 – 2001 A Mathematical Theory of Communication.

Bell System Technical Journal, 27, 379–423 & 623–656, 1948

16

Before starting with entropy …

Who is the grandfather of information theory?

Simon bar Kochba Ca. 100 – 135 Jewish guerilla fighter against Roman Empire (132 – 135)

17

Entropy

    Given a text composed from an alphabet of 32 letters (each letter equally probable) Person A chooses a letter X (randomly) Person B wants to know this letter B may ask only binary questions  Question: how many binary questions must B ask in order to learn which letter X was chosen by A  Answer: entropy H(X)  Here: H(X) = 5 bit

18

Conditional entropy (1)

 The sky is blu_   How many binary questions?

5?

   No!

Why?

What’s wrong?

 The context tells us “something” about the missing letter X

19

Conditional entropy (2)

     Given a text composed from an alphabet of 32 letters (each letter equally probable) Person A chooses a letter X (randomly) Person B wants to know this letter B may ask only binary questions A may tell B the letter Y preceding X    E.g.

L_ Q_  Question: how many binary questions must B ask in order to learn which letter X was chosen by A  Answer: conditional entropy H(X|Y)

20

Conditional entropy (3)

 H(X|Y) <= H(X)  Clear!

 In worst case – namely if B ignores all “information” in Y about X – B needs H(X) binary questions  Under no circumstances should B need more than H(X) binary questions  Knowledge of Y cannot increase the number of binary questions  Knowledge can never harm! (mathematical statement, perhaps not true in real life  )

21

Mutual information (1)

 Compare two situations:   I: learn X without knowing Y II: learn X with knowing Y   How many binary questions in case of I? How many binary questions in case of II?

  H(X) H(X|Y)   Question: Question: How many binary questions could B save in case of II?

How many binary questions could B save by knowing Y?

 Answer: I(X;Y) = H(X) – H(X|Y)  I(X;Y) = information in Y about X

22

Mutual information (2)

 H(X|Y) <= H(X)  I(X;Y) >= 0  In worst case – namely if B ignores all information is no information in Y about X – then I(X;Y) = 0 in Y about X or if there  Information in Y about X can never be negative  Knowledge can never harm! (mathematical statement, perhaps not true in real life  )

23

Mutual information (3)

  Example 1: random sequence composed of A, C, G, T (equally probable) I(X;Y) = ?

   H(X) H(X|Y) I(X;Y) = = = 2 bit 2 bit H(Y) – H(X|Y) = 0 bit      Example 2: deterministic sequence … ACGT ACGT ACGT ACGT … I(X;Y) H(X) H(X|Y) I(X;Y) = = = = ?

2 bit 0 bit H(Y) – H(X|Y) = 2 bit

24

Mutual information (4)

   I(X;Y) = I(Y;X) Always! For any X and any Y!

Information in Y about X = information in X about Y  Examples:   How much information is there in the amino acid sequence about the secondary structure? How much information is there in the secondary structure about the amino acid sequence?

How much information is there in the expression profile about the function of the gene? How much information is there in the function of the gene about the expression profile?

 Mutual information

25

Summary

     Entropy Conditional entropy Mutual information     There is no such thing as information content Information not defined for a single variable 2 random variables needed to talk about information Information in Y about X  I(X;Y) = I(Y;X)  I(X;Y) >= 0 I(X;Y) = 0 if and only if I(X;Y) > 0 otherwise  info in Y about X = info in X about Y   information never negative knowledge cannot harm X and Y statistically independent

26