Transcript Error-Correcting Codes: Progress & Challenges Madhu Sudan MIT CSAIL Communication in presence of noise We are now ready We are not ready Noisy Channel Sender Receiver If information is digital, reliability is.

Error-Correcting Codes:
Progress & Challenges
Madhu Sudan
MIT CSAIL
Communication in presence of noise
We are now
ready
We are not
ready
Noisy
Channel
Sender
Receiver
If information is digital, reliability is critical
Shannon’s Model: Probabilistic Noise
Sender
Encode
(expand)
E:
§k
! §n
Receiver
Noisy
Channel
Decode
(compress?)
D : §n ! §k
Probabilistic Noise: E.g., every letter flipped to
random other letter of § w.p. p
Focus: Design good Encode/Decode algorithms.
Hamming Model: Worst-case error
Errors: Upto
t
worst-case errors
fE(x) : x 2 § k g
C
=
Focus: Code
(Note: Not encoding/decoding)
Goal: Design code so as to correct any pattern of
t errors.
Problems in Coding Theory, Broadly


Combinatorics: Design best possible errorcorrecting codes.
Probability/Algorithms: Design algorithms
correcting random/worst-case errors.
Part I (of III):
Combinatorial Results
Hamming Notions
jfijx =
g
¢(x;
y)
=
=
y
i
i
Hamming Distance:
f¢(x; y)g
2
¢(C)
=
min
x;y C
Distance of Code:
Code of distance 2t + 1 corrects t errors.
Main question:
Four Parameters: Length n, Message length k,
Distance d, alphabet q = j§ j.
How do they relate?
Want: # n, " k, " d, # q.
Asymptotically: Let R = k=n, ± = d=n
How do R; ±; q relate?
Simple results
fy 2 § n j¢(x; y) · rg
Ball(x;
r)
=
Ball:
¼ q cn
H
(±)
=
c
s.t.
Vol(q;
n;
±n)
Volume of Ball: q
jBall(x; r)j in § n .
Vol(q;
n;
r)
=
Entropy function:
Hamming (Packing) Bound:
(No code can have too many codewords)
So q k ¢ q Hq (±=2)n · q n
Simple results (contd.)
Gilbert-Varshamov (Greedy) Bound:
Let C : § k ! § n be maximal of distance d = ±n.
Then balls of radius d ¡ 1 around codewords cover § n
So q k ¢ q Hq (±n) ¸ q n .
Or : : : R ¸ 1 ¡ Hq (±).
Simple results (Summary)
For the best code:
¡
·
·
¡
1 Hq (±) R 1 Hq (±=2)
Which is right?
After fifty years of research … We still don’t know.
Binary case:

¡ ², ² ! 0.
=
1=2
±
Case of large distance:
~ 2)
(²2 ) · R · O(²
GV/Cherno®

LP Bound
Case of small (relative) distance:
No bound better than R · 1 ¡ (1 ¡ o(1)) ¢ H(±=2).
Hamming

Case of constant distance:
d
2
log n ¸ n ¡ k ¸ (1 ¡ o(1)) d2 log n
BCH
Hamming
Binary case (Closer look):

For general n; d
# Codewords ¸ 2n =Vol(2; n; d ¡ 1)


Can we do better? Twice as many codewords?
(won’t change asymptotics of R; ± )
Recent progress [Jiang-Vardy]:
# Codewords ¸ d ¢ 2n =Vol(2; n; d ¡ 1)
Proof idea of [Jiang-Vardy]:
Look at Hamming distance d ¡ 1 graph:
Vertices = f0; 1gn , u $ v , ¢(u; v) < d
Code = Independent set in this graph
GV Bound: I.S. size ¸ # vertices/degree.
Jiang-Vardy: Notice # triangles small.
Use [AKS] For graphs with no (small # of) triangles,
bound improves by factor of log degree.
Major questions in binary codes:


Give explicit construction meeting GV bound.
In particular, give codes of distance ± = 1=2 ¡ ²,
(² ! 0), of rate R = (²2 ).
± ! 0?
Is Hamming tight when
Design codes of distance ±,
with rate R = 1 ¡ c ¢ (1 + o(1)) ¢ ±log2 (1=±),
for some c < 1.
Hamming: c = 1=2

Is LP Bound tight?
Combinatorics (contd.): q-ary case

! 1 (and then ¯x q and let n ! 1)
and
let
q
±
Fix
1 ¡ ± ¡ O(1= log q) · R · 1 ¡ ± ¡ O(1=q)
GV bound

Plotkin
Surprising result (’80s):
Algebraic Geometry gives codes with R ¸ 1 ¡ ± ¡

(Also a negative¡surprise:
BCH codes only yield
¡
R = 1 d(1 1=q) logq n )
Not Hamming
p1
q ¡1
Major questions: q-ary case
² Have R = 1 ¡ ± ¡ f (q)
² Which is the fastest decaying function f (¢)
for which this holds?
p
² Give a natural explanation for why f (q) = O(1= q)
² Fix d, and let q ! 1 slowly, while n ! 1 fast.
How does
n¡ k
d log n
behave?
q
Does it approach 1 or 1 ?
2
Part II (of III):
Correcting Random Errors
Recall Shannon
² § -symmetric channel with error parameter p:
Transmits ¾ 2 § as ¾ with probability 1 ¡ p
0
and as ¾ 2 § ¡ f¾ g with probability p=(q ¡ 1).
² Shannon0 s Coding Theorem:
Can transmit at rate R = 1 ¡ Hq (p) ¡ ², 8² > 0.
If R = 1 ¡ Hq (p) ¡ ², then for every n,
there exist E : § Rn ! § n and D : § n ! § Rn
such that Pr[D(Channel(E(x))) = x] ¸ 1 ¡ exp(¡n).
² Converse Coding Theorem:
Can not transmit at rate R = 1 ¡ Hq (p)+², if ² > 0.
² So no mysteries?
Constructive versions
² Shannon0 s functions:
E picked at random, D brute force search.
² Can we get polynomial time computable E; D?
² [Forney 66]: Gave polynomial time computable E; D.
(Uses e± cient decoding of Reed-Solomon codes,
and composition of codes.)
² Didn0 t completely satisfy practical needs. Why?
² [Spielman 94 + Barg-Zemor 97]
Linear time computable E; D. (Still no satisfaction.)
² [Berrou et al. 92] Turbo codes + belief propagation
No theorems. (Much excitement/satisfaction.)
What is satisfaction?
Articulated by
[Luby,Mitzenmacher,Shokrollahi,Spielman ’96]
² Practical interest: In concrete settings of parameters.
¡
n = 10000, q = 2, p = :1. Desired error probability 10 6 .
What is largest k?
² For [Forney] and successors:
¡ Decoding complexity is exp(1=(1 ¡ H(p) ¡ (k=n))).
¡ Rate = 90% of capacity ) decoding time ¸ 2100 .
² The right question:
¡ Get decoding time n ¢ poly(1=²) (or even poly(n; 1=²))
where ² = 1 ¡ H(p) ¡ R.
Current state of the art

Luby et al.: Propose study of codes based on
irregular graphs (“Irregular LDPC Codes”).
LDPC Codes
1
0
1
0
n left vertices
n ¡ k right vertices
Codeword = 0/1 assignment to left
if neighborhood of right vertices
have even parity.
1
1
De¯nes E : f0; 1gk ! f0; 1gn .
0
1
0
Right vertices are parity checks.
Graph has low density.
Hence Low-Density-Parity-Check Codes.
LDPC Codes
1
0
1
0
1
1
0
1
0
Decoding Intuition:
Parity check fails
) some neighbor corrupted.
Few neighbors
) assigning blame works.
[Gallager 63 ... Sipser-Spielman 92]:
Correct (1) fraction errors.
Current hope:
Picking degrees carefully
will lead to code/algorithm
correcting p fraction random errors
Current state of the art




Luby et al.: Propose study of codes based on
irregular graphs (“Irregular LDPC Codes”).
No theorems so far for erroneous channels.
Strong analysis for (much) simpler case of
erasure channels (symbols are erased);
decoding time O(n log(1=²))
(Easy to get “composition” based algorithms with
¢ poly(1=²))
O(n
decoding time
)
Do have some proposals for errors as well (with
analysis by Luby et al., Richardson & Urbanke),
but none known to converge to Shannon limit.
Still open
Articulated by
[Luby,Mitzenmacher,Shokrollahi,Spielman ’96]
² The right question:
¡ Get decoding time n ¢ poly(1=²) (or even poly(n; 1=²))
where ² = 1 ¡ H(p) ¡ R.
Part III:
Correcting Adversarial Errors
Motivation:



As notions of communication/storage get more
complex, modeling error as oblivious (to
message/encoding/decoding) may be too
simplistic.
Need more general models of error +
encoding/decoding for such models.
Most pessimistic model: errors are worst-case.
Gap between worst-case & random errors


In Shannon model, with binary channel:
¡ 1=q fraction
 Can
upto 50%
(random)
errors.
(1correct
errors,
if channel
q-ary.)
In Hamming model, for binary channel:
 Code with more than n codewords has distance
at most 50%.
 So it corrects
at most 25% worst-case errors.
¡
1
( (1
2

1=q ) fraction errors in q-ary case.)
Need new approaches to bridge gap.
Approach: List-decoding



Main reason for gap between Shannon &
Hamming: The insistence on uniquely recovering
message.
List-decoding: Relaxed notion of recovery from
error. Decoder produces small list (of L)
codewords, such that it includes message.
Code is (p,L) list-decodable if it corrects p
fraction error with lists of size L.
List-decoding



Main reason for gap between Shannon &
Hamming: The insistence on uniquely recovering
message.
List-decoding [Elias ’57, Wozencraft ’58]: Relaxed
notion of recovery from error. Decoder produces
small list (of L) codewords, such that it includes
message.
Code is (p,L) list-decodable if it corrects p
fraction error with lists of size L.
What to do with list?



Probabilistic error: List has size one w.p. nearly 1
General channel: Need side information of only
O(log n) bits to disambiguate [Guruswami ’03]
 (Alt’ly if sender and receiver share O(log n) bits,
then they can disambiguate [Langberg ’04]).
Computationally bounded error:
 Model introduced by [Lipton, Ding Gopalan L.]
 List-decoding results can be extended (assuming
PKI and some memory at sender) [Micali et al.]
List-decoding: State of the art

[Zyablov-Pinsker/Blinovskii ¡
– late 80s]
There exist codes of rate 1 Hq (p) ¡ ²
that are (p; O(1))-list-decodable.


Matches Shannon’s converse perfectly! (So can’t
do better even for random error!)
But [ZP/B] non-constructive!
Algorithms for List-decoding



Not examined till ’88.
First results: [Goldreich-Levin] for “Hadamard’’ codes (nontrivial in their setting).
More recent work:


[S.’96, Shokrollahi-Wasserman ’98, Guruswami-S.’99, ParvareshVardy ’05, Guruswami-Rudra ’06] – Decode algebraic codes.
[Guruswami-Indyk ’00-’02]
codes.

– Decode graph-theoretic
[TaShma-Zuckerman ’02, Trevisan ’03]
for list-decoding.
– Propose new codes
Results in List-decoding

Q-ary
case:
² [Guruswami-Rudra
06] Codes of rate R ¡ ²
correcting 1 ¡ R fraction errors with q = f (²).
Converges to Shannon capacity!!

Binary
case:of rate ²c correcting
9 Codes
1
2
¡ ² fraction errors.
¡ c = 4: Guruswami et al. 2000
¡ c ! 3: Implied by Parvaresh-Vardy 05
Few lines about Guruswami-Rudra

Code = Collated Reed-Solomon Code +
Concatenation.FC
!1
Alphabet § =
;q
, C constant.
Code maps § K ! § N for N ¼ q=C.
Message: Degree C ¢ K polynomial over Fq .
q
Encoding:
First partition Fq into special sets S0 ; S1 ; : : : ; SN ,
with jS1 j = ¢ ¢ ¢ = jSN j = C.
Say S1 = f®0 ; : : : ; ®C ¡1 g, S2 = f®C ; : : : ; ®2C ¡1 g etc.
Encoding of P
hhP (® ); : : : ; P (®
i; hP (® ); : : : ; P (®
i¢¢¢i
)
)
0
C ¡1
C
2C ¡1
Few lines about Guruswami-Rudra

Special properties:
of K,Si s
² S = f® = ®; ° ¢ ®; : : : ; ° C ¡1 ¢ ®g.
i
iC
² ° satis¯es xq = ° x mod h(x)
for irreducible h of degree CK.

Is² this code combinatorially
good?
¡
Do Balls of radius (1 o(1)) ¢ (N ¡ K)
have few codewords?


Algorithmically good!! (uses ideas from
[S’96,GS’98,PV’05 + new ones].
Can concatenate to reduce alphabet size.
Few lines about Guruswami-Rudra

Warnings: K, N, partition all very special.
Alphabet § = FC ; q ! 1, C constant.
q
Code maps § K ! § N for N ¼ q=C.
Message: Degree C ¢ K polynomial over Fq .
Encoding: \\
\indent First partition $\F_q$ into
{\red special}
sets $S_0,S_1,\ldots,S_N$, \\
\indent \indent with $|S_1| = \cdots
= |S_N| = C$. \\
\indent Say $S_1 =
\{\alpha_1,\ldots,\alpha_C\}$,
$S_2 = \{\alpha_{C+1},\ldots,\alpha_{2C}\}$
etc.\\
Major open question
² Construct (p; O(1)) list-decodable binary code
of rate 1 ¡ H(p) ¡ ² with polytime list decoding..
² Note: If running time is poly(1=²) then this implies a
solution to the random error problem as well.
Conclusions


Many mysteries in combinatorial setting.
Significant progress in algorithmic setting, but
many important open questions as well.