The machine learning consultancy: [ Ссылка ]
Join my email list to get educational and useful articles (and nothing else!): [ Ссылка ]
Want to work together? See here: [ Ссылка ]
What's the safest distribution to pick in the absence of information? What about in the case where you have some, though only partial, information? The Principle of Maximum Entropy answers these questions well and as a result, is a frequent guiding rule for selecting distributions in the wild.
SOCIAL MEDIA
LinkedIn : [ Ссылка ]
Twitter : [ Ссылка ]
Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: [ Ссылка ]
Sources
Chapters 11-12 of [2] were primary sources - this is where I ironed out most of my intuition on this subject. Chapter 12 of [1] was helpful for understanding the relationship between the maximum entropy criteria and the form of the distribution that meets it. [3] was useful for a high level perspective and [4] was helpful for determining the list of maximum entropy distribution.
Also, thank you to Dr. Hanspeter Schmid of the University of Applied Sciences and Arts, Northwestern Switzerland. He helped me interpret some of the more technical details of [2] and prevented me from attaching an incorrect intuition to the continuous case - much appreciated!
[1] T. M. Cover and J. A. Thomas. Elements of Information Theory. 2nd edition. John Wiley, 2006.
[2] E. T. Jaynes. Probability theory: the logic of science. Cambridge university press, 2003.
[3] Principle of Maximum Entropy, Wikipedia, [ Ссылка ]
[4] Maximum Entropy Distribution, Wikipedia, [ Ссылка ]
Timestamps :
0:00 Intro
00:41 Guessing a Distribution and Maximum Entropy
04:16 Adding Information
06:40 An Example
08:00 The Continuous Case
10:26 The Shaky Continuous Foundation
