Information Theory and Statistics: A TutorialInformation Theory and Statistics: A Tutorial is concerned with applications of information theory concepts in statistics, in the finite alphabet setting. The topics covered include large deviations, hypothesis testing, maximum likelihood estimation in exponential families, analysis of contingency tables, and iterative algorithms with an "information geometry" background. Also, an introduction is provided to the theory of universal coding, and to statistical inference via the minimum description length principle motivated by that theory. The tutorial does not assume the reader has an in-depth knowledge of Information Theory or statistics. As such, Information Theory and Statistics: A Tutorial, is an excellent introductory text to this highly-important topic in mathematics, computer science and electrical engineering. It provides both students and researchers with an invaluable resource to quickly get up to speed in the field. |
Contents
Preface | 1 |
Universal coding | 6 |
Iprojections | 23 |
Iterative algorithms | 43 |
Redundancy bounds | 75 |
Redundancy and the MDL principle | 89 |
Appendix A Summary of process concepts | 105 |
Common terms and phrases
algorithm alphabet arithmetic code arithmetic code determined assertion asymptotically asymptotically optimal chains of order Chapter cl(E code Cn coding process completes the proof contingency table converges convex convex set Csiszár D(PQ defined denotes distribution Q Dmin empirical distribution entropy equals ergodic process error probability expected length expected redundancy exponential family f-divergences fi(a finite follows given H(Pn Hence hypothesis I-projection of Q i.i.d. processes implies information divergence information theory iterative scaling Lemma length function linear family log Q(x log-linear family log-linear models log-sum inequality lumping property Markov chains maximum likelihood minimizer minimum n-code n-types nonnegative optimal outlier P₁ P₂ Pn(a prefix code probability distributions probability measure process Q proof of Theorem prove Pythagorean identity Qn(x random variables sample satisfies sequence sequential tests Shannon code stationary processes statistical subset Theorem 3.2 type 2 error universal codes vector y-marginal