# The Minimum Description Length Principle

MIT Press, 2007 - Computers - 703 pages

The minimum description length (MDL) principle is a powerful method of inductive inference, the basis of statistical modeling, pattern recognition, and machine learning. It holds that the best explanation, given a limited set of observed data, is the one that permits the greatest compression of the data. MDL methods are particularly well-suited for dealing with model selection, prediction, and estimation problems in situations where the models under consideration can be arbitrarily complex, and overfitting the data is a serious concern.This extensive, step-by-step introduction to the MDL Principle provides a comprehensive reference (with an emphasis on conceptual issues) that is accessible to graduate students and researchers in statistics, pattern classification, machine learning, and data mining, to philosophers interested in the foundations of statistics, and to researchers in other applied sciences that involve model selection, including biology, econometrics, and experimental psychology. Part I provides a basic introduction to MDL and an overview of the concepts in statistics and information theory needed to understand MDL. Part II treats universal coding, the information-theoretic notion on which MDL is built, and part III gives a formal treatment of MDL theory as a theory of inductive inference based on universal coding. Part IV provides a comprehensive overview of the statistical theory of exponential families with an emphasis on their information-theoretic properties. The text includes a number of summaries, paragraphs offering the reader a "fast track" through the material, and boxes highlighting the most important concepts.

### What people are saying -Write a review

#### LibraryThing Review

User Review  - mdreid - LibraryThing

I have not read this book from cover to cover (and doubt I will) but the sections I have read so far—universal codes, exponential families, and MDL in context—have all been excellent. I would ... Read full review

### Contents

 1 Learning Regularity and Compression 3 2 Probabilistic and Statistical Preliminaries 41 3 InformationTheoretic Preliminaries 79 4 InformationTheoretic Properties of Statistical Models 109 5 Crude TwoPart Code MDL 131 P A R T I I Universal Coding 165 6 Universal Coding with Countable Models 171 Normalized Maximum Likelihood 207
 P A R T I I I Refined MDL 403 14 MDL Model Selection 409 15 MDL Prediction and Estimation 459 16 MDL Consistency and Convergence 501 17 MDL in Context 523 P A R T I V Additional Background 597 18 The Exponential or Maximum Entropy Families 599 19 InformationTheoretic Properties of Exponential Families 623

 Bayes 231 Prequential Plugin 257 TwoPart 271 11 NMLWith Innite Complexity 295 12 Linear Regression 335
 References 651 List of Symbols 675 Subject Index 679 Copyright

### References to this book

 Philosophy of InformationLimited preview - 2008
All Book Search results &raquo;