Excerpt

In a series of lectures given in 1956 to the Mobil Oil Company, Texas, Edwin Thompson Jaynes recalls discovering Claude Shannon’s seminal work on information theory in the Princeton University library. At that time, as a graduate student of statistical mechanics, he was stunned by Shannon’s novel derivation of the entropy expression and its varied applications, particularly since they were not restricted to thermodynamics. Jaynes immediately recognized Shannon’s work on entropy as having an importance to physics comparable to Dirac’s pioneering work in particle physics and viewed it as “. . . the most important work done by any scientist since the discovery of the Dirac equation” (Jaynes 1956). Shannon saw in the concept of entropy a possible application in the field of communication. His information-theoretic derivation of the entropy expression, subsequently known as Shannon entropy, can be found in his paper published in 1948 in The Bell System Technical Journal and it was this paper that so excited Jaynes. In the book that followed in 1949, The Mathematical Theory of Communication, Shannon expanded the idea of entropy as the key concept in information theory, which he is largely credited with founding. There followed an excess of articles in diverse areas using Shannon entropy. However, in 1956 he published “The Bandwagon,” a short piece in which he suggests moderation in the use of information theory in other fields.

Jaynes was particularly concerned about the bad reputation Shannon entropy had gained among physicists because of the lack of new results, but in 1956 he wrote: “I think the time has come now when physicists might find it worthwhile to take a sober second look at Information Theory and what it can do for them” (Jaynes 1956). Jaynes saw a direct link between entropy in statistical mechanics and information entropy, which he immediately identified as a significant shift in perspective. In this paper, he showed that entropy as expressed in the work of Boltzmann and Gibbs in the field of thermodynamics and Shannon entropy follow from the same underlying logic, thus establishing entropy as a general concept. Maximum entropy inference is introduced and in sections 3, 4, and 5 examples are given showing how the application of the principle of maximum entropy results in a conceptual shift in the approach to statistical mechanics. Jaynes proved that statistical mechanics can be interpreted in non-physical terms as statistical inference. In his 1957 paper he introduces the “principle of maximum entropy” and proposed the maximization of the entropy function H, subject to certain constraints, as a general principle of statistical inference applicable to a diverse range of areas. The principle has since proved particularly useful in providing the least biased probability distribution for a set of unknown prior probabilities. Jaynes’s view is that assigning equal probabilities in situations of total ignorance is the least prejudiced choice. However, when ignorance is not total, inferences should be based on all and only the information available. This is assured by choosing the probability distribution with maximum entropy.

Bibliography

Chliamovitch, G., O. Malaspinas, and B. Chopard. 2017. “Kinetic Theory Beyond the Stosszahlansatz.” Entropy 19 (8).

Jaynes, E. T. 1956. Probability Theory in Science and Engineering. Colloquium Lectures in Pure and Applied Science No. 4. New York, NY: Socony-Mobil Oil Co.

—. 1978. “Where Do we Stand on Maximum Entropy?” In Maximum Entropy Formalism Conference, Massachusetts Institute of Technology. Reprinted in E. T. Jaynes: Papers on Probability, Statistics and Statistical Physics.

—. 2002. Probability Theory: The Logic of Science. Edited by G. L. Bretthorst. St. Louis, MO: Washington University.

Shannon, C. E. 1948. “The Mathematical Theory of Communication.” Reprinted with corrections, The Bell System Technical Journal 27 (6): 379–423.

—. 1956. “The Bandwagon.” Published by Institute for Radio Engineers, Inc, New York, IRE Transactions on Information Theory 2 (1).

Shannon, C. E., and W. Weaver. 1949. The Mathematical Theory of Communication. Urbana, IL: University of Illinois Press.

BACK TO Foundational Papers in Complexity Science