Transcript TSBK01 Image Coding and Data Compression Lecture 2: Basic Information Theory Jörgen Ahlberg Div.

TSBK01 Image Coding and Data Compression
Lecture 2:
Basic Information Theory
Jörgen Ahlberg
Div. of Sensor Technology
Swedish Defence Research Agency (FOI)
Today
1.
2.
3.
4.
5.
What is information theory about?
Stochastic (information) sources.
Information and entropy.
Entropy for stochastic sources.
The source coding theorem.
Part 1: Information Theory
The
Claude Shannon: A Mathematical Theory of Communication
Bell System Technical Journal, 1948
Sometimes referred to as ”Shannon-Weaver”, since
the standalone publication has a foreword by Weaver.
Be careful!
Quotes about Shannon
 ”What is information? Sidestepping questions about
meaning, Shannon showed that it is a measurable
commodity”.
 ”Today, Shannon’s insight help shape virtually all systems
that store, process, or transmit information in digital form,
from compact discs to computers, from facsimile machines
to deep space probes”.
 ”Information theory has also infilitrated fields outside
communications, including linguistics, psychology,
economics, biology, even the arts”.
Change to an efficient
Change
representation,
to an efficient representation for,
Any source of information
i.e., data compression.
transmission, i.e., error control coding.
Source
Channel
Source
Channel
coder
coder
Channel
Channel
Source
Sink,
decoder
decoder
receiver
The channel is anything transmitting or storing information –
Recover
from channel
distortion.Uncompress
a radio link,
a cable,
a disk,
a CD, a piece of paper, …
Fundamental Entities
Source
C
H
Channel
Source
coder
R
Channel
coder
Channel
C
Channel
Source
Sink,
decoder
decoder
receiver
H: The information content of the source.
R: Rate from the source coder.
C: Channel capacity.
Fundamental Theorems
Source
C
H
Channel
Source
coder
R
Channel
coder
Channel
C
Channel
Source
Sink,
decoder
decoder
receiver
Shannon 1: Error-free transmission possible if R¸H and C¸R.
Shannon 2: Source
coding
and
channel
coding can be
Source coding
Channel
theorem
coding
(simplified)
theorem (simplified)
optimized independently, and binary symbols can be used as
intermediate format. Assumption: Arbitrarily long delays.
Part 2: Stochastic sources
 A source outputs symbols X1, X2, ...
 Each symbol take its value from an alphabet
Example 1: A text is a sequence of symbols
A (digitized) grayscale image is a
A Example
= (a1, a2,2:…).
each taking its value from the alphabet
sequence of symbols each taking its value
 Model:
P(X
)
assumed
be
known
for
A = (a, …,
z,1,…,X
A, …,NZ,
1, 2, …9, !,to
?, …).
from the alphabet A = (0,1) or A = (0, …, 255).
all combinations.
Source
X1, X2, …
Two Special Cases
1. The Memoryless Source


Each symbol independent of the previous
ones.
P(X1, X2, …, Xn) = P(X1) ¢ P(X2) ¢ … ¢ P(Xn)
2. The Markov Source


Each symbol depends on the previous one.
P(X1, X2, …, Xn) = P(X1) ¢ P(X2|X1) ¢ P(X3|X2) ¢ … ¢
P(Xn|Xn-1)
The Markov Source
 A symbol depends only on the previous
symbol, so the source can be modelled by a
state diagram.
0.7
b
0.5
a
1.0
0.3
0.2
c
0.3
A ternary source with
alphabet A = (a, b, c).
The Markov Source
 Assume we are in state a, i.e., Xk = a.
 The probabilities for the next symbol are:
0.7
b
P(Xk+1 = a | Xk = a) = 0.3
0.5
a
1.0
0.3
0.2
P(Xk+1 = b | Xk = a) = 0.7
P(Xk+1 = c | Xk = a) = 0
c
0.3
The Markov Source
 So, if Xk+1 = b, we know that Xk+2 will
equal c.
0.7
b
P(Xk+2 = a | Xk+1 = b) = 0
0.5
a
1.0
0.3
0.2
P(Xk+2 = b | Xk+1 = b) = 0
P(Xk+2 = c | Xk+1 = b) = 1
c
0.3
The Markov Source
 If all the states can be reached, the
stationary probabilities for the states can be
calculated from the given transition
probabilities.
Stationary probabilities? That’s the
 Markov models can
be usedto
represent
probabilities
i = P(Xk = ai) for any
k when Xk-1, Xmore
not one
given.
sources with dependencies
than
k-2, … are
step back.
– Use a state diagram with several symbols in
each state.
Analysis and Synthesis
 Stochastic models can be used for
analysing a source.
– Find a model that well represents the real-world
source, and then analyse the model instead of
the real world.
 Stochastic models can be used for
synthesizing a source.
– Use a random number generator in each step of
a Markov model to generate a sequence
simulating the source.
Show plastic slides!
Part 3: Information and Entropy
 Assume a binary memoryless source, e.g., a flip of
a coin. How much information do we receive when
we are told that the outcome is heads?
– If it’s a fair coin, i.e., P(heads) = P (tails) = 0.5, we say
that the amount of information is 1 bit.
– If we already know that it will be (or was) heads, i.e.,
P(heads) = 1, the amount of information is zero!
– If the coin is not fair, e.g., P(heads) = 0.9, the amount of
information is more than zero but less than one bit!
– Intuitively, the amount of information received is the
same if P(heads) = 0.9 or P (heads) = 0.1.
Self Information
 So, let’s look at it the way Shannon did.
 Assume a memoryless source with
– alphabet A = (a1, …, an)
– symbol probabilities (p1, …, pn).
 How much information do we get when
finding out that the next symbol is ai?
 According to Shannon the self information of
ai is
Why?
Assume two independent events A and B, with
probabilities P(A) = pA and P(B) = pB.
For both the events to happen, the probability is
pA ¢ pB. However, the amount of information
should be added, not multiplied.
Logarithms satisfy this!
No, we want the information to increase with
decreasing probabilities, so let’s use the negative
logarithm.
Self Information
Example 1:
Example 2:
Which logarithm? Pick the one you like! If you pick the natural log,
you’ll measure in nats, if you pick the 10-log, you’ll get Hartleys,
if you pick the 2-log (like everyone else), you’ll get bits.
Self Information
On average over all the symbols, we get:
H(X) is called the first order entropy of the source.
This can be regarded as the degree of uncertainty
about the following symbol.
Entropy
Example: Binary Memoryless Source
01101000…
BMS
Let
Then
1
The uncertainty (information) is greatest when
0
0.5
1
Entropy: Three properties
1. It can be shown that 0 · H · log N.
2. Maximum entropy (H = log N) is reached
when all symbols are equiprobable, i.e.,
pi = 1/N.
3. The difference log N – H is called the
redundancy of the source.
Part 4: Entropy for Memory Sources
 Assume a block of source symbols (X1, …,
Xn) and define the block entropy:
That is, the summation is done over all possible combinations of n symbols.
 The entropy for a memory source is defined
as:
That is, let the block length go towards infintity.
Divide by n to get the number of bits / symbol.
Entropy for a Markov Source
The entropy for a state Sk can be expressed as
Pkl is the transition probability from state k to state l.
Averaging over all states, we get the
entropy for the Markov source as
The Run-length Source
 Certain sources generate long runs or bursts of
equal symbols.

 Example:
1-
A
B
1-

 Probability for a burst of length r: P(r) = (1-)r-1¢
 Entropy: HR = - r=11 P(r) log P(r)
 If the average run length is , then HR/ = HM.
Part 5: The Source Coding Theorem
The entropy is the smallest number of bits
allowing error-free representation of the source.
Why is this? Let’s take a look on typical sequences!
Typical Sequences
 Assume a long sequence from a binary
memoryless source with P(1) = p.
 Among n bits, there will be approximately
w = n ¢ p ones.
 Thus, there is M = (n over w) such typical
sequences!
 Only these sequences are interesting. All
other sequences will appear with smaller
probability the larger is n.
How many are the typical
sequences?
bits/symbol
Enumeration needs log M bits, i.e,
bits per symbol!
How many bits do we need?
Thus, we need H(X) bits per symbol
to code any typical sequence!
The Source Coding Theorem
 Does tell us
– that we can represent the output from a source
X using H(X) bits/symbol.
– that we cannot do better.
 Does not tell us
– how to do it.
Summary
 The mathematical model of communication.
– Source, source coder, channel coder, channel,…
– Rate, entropy, channel capacity.
 Information theoretical entities
– Information, self-information, uncertainty, entropy.
 Sources
– BMS, Markov, RL
 The Source Coding Theorem