Department of Computer Science, University College London
COMP0045 Probability Theory
and Stochastic Processes
Dr Guido Germano
Coursework 2020–2021
Choose one of the following questions for further reading and write a brief report/essay
of a few pages (2–3) with single-spaced lines, or twice as many slides (4–6) if you prefer
the latter. Writing more will not be penalised, but sheer length does not necessarily lead
to a higher mark: quality is more important.
LATEX is recommended because it is the system of choice for typesetting mathematic
and scientific texts, for which it gives the best results, but alternatively you may resort
to an editor like Word, Pages, LibreOce, etc., or to handwriting. Please upload both
the PDF and the source code (.tex, .bib, .docx, etc.). For handwriting JPG is acceptable
too.
For LATEX, take the standard document class article or the 001.tex slides template
for slides; you may use the latter for all questions, not only 2 e. The slides template
requires the UCL beamer theme by Simon Byrne and Maurizio Filippone, available from
https://github.com/UCL/ucl-beamer.
The marking criteria are similar to those for the dissertation about your summer
project, that are described in detail on the Moodle pages of COMP0076/77.
Questions
1. The Brandeis dice
In Section 6 of our lectures we have seen a fair die and a most unfair die. In his 1962
lectures for the Brandeis summer school, Jaynes introduced a die whose expected number
of spots per roll is not 3.5 like for a fair die, but 4.5, and used the maximum entropy
method to find the probabilities pi that face i comes up. In 2014, van Enk gave two
alternative solutions, one based on maximum entropy, one on Baryesian updating; both,
unlike Jaynes’ solution, yield error bars. Report and discuss.
Edwin T. Jaynes, Information theory and statistical mechanics, in Brandeis Univer-
sity Summer Institute Lectures in Theoretical Physics 1962, Volume 3, Statistical Physics,
edited by K.W. Ford, pages 181–218, Benjamin, New York, 1963, https://bayes.wustl.edu/
etj/articles/brandeis.pdf.
Steven van Enk, The Brandeis dice problem and statistical mechanics, Studies in His-
tory and Philosophy of Modern Physics 48 A, 1–6, 2014, DOI 10.1016/j.shpsb.2014.08.007
UCL COMP0045 AA 2020–2021 1 TURN OVER

2. Miscellaneous questions
Answer two of the following questions.
a) Show that the di↵erential entropy S(X) = EX(log fX) =
R +1
1 fX(x) log fX(x)dx
is not scale invariant, i.e. a transformation of X changes the result: S(aX) = S(X) +
log |a| and more in general S(g(X)) = S(X) + R +11 fX(x) log @g@x dx.
b) Show that one of the five continuous maximum entropy distributions introduced in
Section 54 Example 3 a–e of our lectures is actually the maximum entropy distribution
on that support with those conditions.
c) Introduce the exponential family of distributions. Is there a reason why all five contin-
uous maximum entropy distributions met in Section 54 Example 3 a–e of our lectures
belong to this family?
d) Explain with some detail why the moment-generating function MX(t) = EX

etX

does not always exist while the characteristic function 'X(⇠) = EX

ei⇠X

exists for
any random variable X.
e) Typeset in LATEX one page of the lecture notes that does not yet appear in the slides.
If you wish to do this question, please email me with subject “COMP0045 slides” to
be assigned a specific page.
UCL COMP0045 AA 2020–2021 2 END OF PAPER