2020/8/14 https://www.cse.unsw.edu.au/~cs9444/20T2/quiz/ans/quiz2_answers.html
COMP9444 Neural Networks and Deep Learning Quiz 2 (Probability and Backprop Variations)
This is an optional quiz to test your understanding of the material from Week 2.
1. Write the formula for a Gaussian distribution with mean μ and standard deviation σ.
P(x) = exp(-(x-μ)2/2σ2) / (sqrt(2π)σ)
2. Write the formula for Bayes’ Rule, in terms of a cause A and an effect B.
P(A|B) = P(B|A)P(A) / P(B)
3. Write the formula for the Entropy H( ) of a continuous probability distribution ()
H( ) = ∫ θ (θ) (-log (θ)) dθ
4. Write the formula for the Kullback-Leibler Divergence DKL( || ) between two continuous probability distributions () and ().
DKL( || ) = ∫ θ (θ) (log (θ) – log (θ)) dθ
5. Write the formulas for these Loss functions: Squared Error, Cross Entropy, Weight Decay. (remember to define any variables you use)
Assume is the actual output, is the target output and are the weights.
Squared Error: E = 1⁄2 Σ ( )2
Cross Entropy: E = Σ (- log( ) – (1- )log(1- ))
2
6. In the context of Supervised Learning, explain the difference between Maximum Likelihood estimation and Bayesian Inference.
In Maximum Likelihood estimation, the hypothesis ∈ is chosen which maximizes the conditional probability P(D | ) of the observed data D, conditioned on . In Bayesian Inference, the hypothesis ∈ is chosen which maximizes P(D | )P( ), where P( ) is the prior probability of .
7. Briefly explain the concept of Momentum, as an enhancement for Gradient Descent.
A running average of the differentials for each weight is maintained and used to update the weights as follows:
δw = αδw – η dE/dw w = w + δw
The constant α with 0 ≤ α < 1 is called the momentum.
Weight Decay: E = 1⁄2 Σ
https://www.cse.unsw.edu.au/~cs9444/20T2/quiz/ans/quiz2_answers.html 1/2
hhhh Hhh
h Hh
iz it iz iti it - i z i
jw it iz
qp
pp
jw j
qppqp qp
ppp
2020/8/14 https://www.cse.unsw.edu.au/~cs9444/20T2/quiz/ans/quiz2_answers.html
https://www.cse.unsw.edu.au/~cs9444/20T2/quiz/ans/quiz2_answers.html 2/2