Exercises for the course
Machine Learning 1
Winter semester 2020/21
Abteilung Maschinelles Lernen Institut fu ̈r Softwaretechnik und theoretische Informatik Fakult ̈at IV, Technische Universit ̈at Berlin Prof. Dr. Klaus-Robert Mu ̈ller Email: klaus-robert.mueller@tu-berlin.de
Exercise Sheet 13
Exercise 1: RBM with Ternary Hidden Units (20 + 10 P)
We consider a variant of the restricted Boltzmann machine with ternary hidden units h ∈ {−1, 0, 1}H . Input features remain binary, i.e. x ∈ {0, 1}d, like for the classical RBM. The probability model is given by:
1HH p(x,h|θ)=Zexp w⊤jx·hj+hjbj
j=1 j=1
where θ = (wj,bj)Hj=1 are the parameters of the model, and where and Z is the partition function that
normalizes the probability distribution to 1.
(a) Show that this modified RBM can also be expressed as a product of experts
1 H
(b) Show that the gradient of the log-likelihood assigned to some data point xn by the modified RBM has the form
∀Hj=1 : ∂logp(xn|θ) =xn ·σ(w⊤j xn +bj)−Ex∼p(x|θ)x·σ(w⊤j x+bj) ∂wj
∀Hj=1 : ∂logp(xn|θ) =σ(w⊤j xn +bj)−Ex∼p(x|θ)σ(w⊤j x+bj) ∂bj
Consider the product of experts:
with
where cosh is the hyperbolic cosine function.
p(x|θ) = Z gj(x,θj)=1+2cosh(w⊤j x+bj),
j=1
gj (x, θj ),
sinh(t) . 0.5 + cosh(t)
where σ(t) =
Exercise 2: Product of Gaussian Mixture Models (20 + 10 P)
1 H
p(x|θ) = Z
where each expert is a Gaussian mixture model in d-dimensions, and where each element of the mixture is
Gaussian with identity covariance:
H C112
the vector k ∈ {1,…,C}H) has center
k=1
1 H
mk = H
μjkj .
gj(x,θj)
j=1
αjk(2π)d/2 exp −2∥x−μjk∥ .
∀j=1 : gj(x,θj)=
(a) Show that p(x|θ) can be rewritten as a mixture of CH elements, where each mixture element (indexed by
(b) Give the centers mk of the mixture model equivalent to a product of two mixture models, where each mixture model in the product has 2 elements, where the first mixture has the two-dimensional centers
μ = 2 and μ = 4, and where the second mixture has the two-dimensional centers μ 11 0 12 0
μ =0. 22 4
Exercise 3: Programming (40 P)
Download the programming files on ISIS and follow the instructions.
= 0 and 21 2
j=1