2020/11/25 COMP9444 Exercises 4
COMP9444 Neural Networks and Deep Learning Term 3, 2020
Exercises 4: PyTorch This page was last updated: 10/09/2020 11:24:33
Download the zip file Ex4code.zip and unzip it. This will create a directory Ex4code with two very simple PyTorch programs, slope.py and xor.py.
1. Adjusting the Learning Rate and Momentum
The program slope.py solves the simplest possible machine learning task:
solve F(x) = A*x such that F(1)=1
a. Run the program by typing
python3 slope.py –lr 0.1
Try running the code using each of the following values for learning rate:
0.01, 0.1, 0.5, 1.0, 1.5, 1.9, 2.0, 2.1
Describe what happens in each case, in terms of the success and speed of the algorithm.
b. Now add momentum, by typing:
python3 slope.py –mom 0.1
Try running the code with learning rate kept at the default value of 1.9 but with momentum equal to 0.1, 0.2, etc. up to 0.9. For which value of momentum is the task solved in the fewest epochs? What happens when the momentum is 1.0? What happens when it is 1.1?
2. Learning the XOR Task
The program xor.py trains a 2-layer neural network on the XOR task.
a. Run the code ten times by typing
python3 xor.py
For how many runs does it reach the global minimum? For how many runs does it reach a local minimum?
b. Keeping the learning rate fixed at 0.1, can you find values of momentum (–mom) and initial weight size (–init) for which the code converges relatively quickly to the global minimum on virtually every run?
Make sure you attempt the questions yourself, before looking at the Sample Solutions. https://www.cse.unsw.edu.au/~cs9444/20T3/tut/Ex4_PyTorch.html 1/1