COMP9444
Neural Networks and Deep Learning
Typical Structure of a PyTorch Progam
COMP9444
⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T2
PyTorch
2 COMP9444 20T2 PyTorch
3
Defining a Model
Defining a Custom Model
class MyModel(torch.nn.Module):
Considerthefunction (x,y)→Axlog(y)+By2 import torch.nn as nn
class MyModel(nn.Module):
COMP9444
⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
3b. PyTorch
# create neural network according to model specification net = MyModel().to(device) # CPU or GPU
def __init__(self):
super(MyModel, self).__init__()
# define structure of the network here
def __init__(self):
super(MyModel, self).__init__()
self.A = nn.Parameter(torch.randn((1),requires_grad=True))
self.B = nn.Parameter(torch.randn((1),requires_grad=True))
def
forward(self, input):
# apply network and return output
COMP9444 20T2 PyTorch 1
train_loader = torch.utils.data.DataLoader(…)
test_loader = torch.utils.data.DataLoader(…)
# choose between SGD, Adam or other optimizer optimizer = torch.optim.SGD(net.parameters,…)
for epoch in range(1, epochs):
train(params, net, device, train_loader, optimizer) if epoch % 10 == 0:
def
forward(self, input):
output = self.A * input[:,0] * torch.log(input[:,1]) \
test(params, net, device, test_loader)
+ self.B * input[:,1] * input[:,1] return output
COMP9444 20T2 PyTorch 4
COMP9444 20T2 PyTorch 5
Building a Net from Individual Components
Defining a Sequential Network
class MyModel(torch.nn.Module):
class MyModel(torch.nn.Module):
COMP9444
⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T2
PyTorch
6
COMP9444 20T2
PyTorch 7
def __init__(self):
super(MyModel, self).__init__() self.in_to_hid = torch.nn.Linear(2,2) self.hid_to_out = torch.nn.Linear(2,1)
def
__init__(self, num_input, num_hid, num_out): super(MyModel, self).__init__()
self.main = nn.Sequential(
def
forward(self, input):
hid_sum = self.in_to_hid(input)
hidden = torch.tanh(hid_sum)
out_sum = self.hid_to_out(hidden)
output = torch.sigmoid(out_sum)
return output
nn.Linear(num_input, num_hid), nn.Tanh(),
nn.Linear(num_hid, num_out), nn.Sigmoid()
Sequential Components
Declaring Data Explicitly
Network Layers:
nn.Linear()
nn.Conv2d()
import torch.utils.data
Intermediate Operators:
nn.Dropout()
nn.BatchNorm()
input = torch.Tensor([[0,0],[0,1],[1,0],[1,1]])
target = torch.Tensor([[0],[1],[1],[0]])
Activation Functions:
nn.Tanh()
nn.Sigmoid()
nn.ReLU()
xdata = torch.utils.data.TensorDataset(input,target)
train_loader = torch.utils.data.DataLoader(xdata,batch_size=4)
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 ⃝c Alan Blair, 2017-20
def
)
forward(self, input):
output = self.main(input)
return output
COMP9444 20T2 PyTorch
8
COMP9444 20T2 PyTorch 9
Loading Data from a .csv File
Custom Datasets
import pandas as pd
from data import ImageFolder
dataset = ImageFolder(folder, transform)
df = pd.read_csv(“sonar.all-data.csv”)
df = df.replace(’R’,0)
df = df.replace(’M’,1)
data = torch.tensor(df.values,dtype=torch.float32) num_input = data.shape[1] – 1
import torchvision.datasets as dsets
input = data[:,0:num_input]
target = data[:,num_input:num_input+1]
mnistset = dsets.MNIST(…)
cifarset = dsets.CIFAR10(…)
celebset = dsets.CelebA(…)
dataset = torch.utils.data.TensorDataset(input,target)
COMP9444
⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T2
PyTorch
10
COMP9444 20T2
PyTorch 11
Choosing an Optimizer
Training
SGD stands for “Stochastic Gradient Descent”
def train(args, net, device, train_loader, optimizer):
optimizer = torch.optim.SGD( net.parameters(), lr=0.01, momentum=0.9,
for
batch_idx, (data,target)
optimizer.zero_grad()
output = net(data)
loss = …
loss.backward()
optimizer.step()
in enumerate(train_loader):
# zero the gradients
# apply network
# compute loss function
Adam = Adaptive Momentum (good for deep networks)
weight_decay=0.0001)
optimizer = torch.optim.Adam(net.parameters(),eps=0.000001,
lr=0.01, betas=(0.5,0.999),
# update gradients
# update weights
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
weight_decay=0.0001)
COMP9444 20T2 PyTorch
12
COMP9444 20T2 PyTorch 13
Loss Functions
Testing
loss = torch.sum((output-target)*(output-target))
loss = F.nll_loss(output,target)
loss = F.binary_cross_entropy(output,target)
loss = F.softmax(output,dim=1)
def test(args, model, device, test_loader):
loss = F.log_softmax(output,dim=1)
output = model(data)
test_loss += …
COMP9444
⃝c Alan Blair, 2017-20
COMP9444
⃝c Alan Blair, 2017-20
COMP9444 20T2
PyTorch
14
COMP9444 20T2
PyTorch 15
Computational Graphs
Controlling the Computational Graph
PyTorch automatically builds a computational graph, enabling it to backpropagate derivatives.
If we need to block the gradients from being backpropagated through a certain variable (or expression) A, we can exclude it from the computational graph by using:
Every Parameter includes .data and .grad components, for example: A.data
A.detach()
By default, loss.backward() discards the computational graph after
A.grad
optimizer.zero grad() sets all .grad components to zero.
computing the gradients.
If needed, we can force it to keep the computational graph by calling:
loss.backward() updates the .grad component of all Parameters by backpropagating gradients through the computational graph.
loss.backward(retain_graph=True)
optimizer.step() updates the .data components.
COMP9444 ⃝c Alan Blair, 2017-20
COMP9444 ⃝c Alan Blair, 2017-20
with torch.no_grad(): # suppress updating of gradients net.eval() # toggle batch norm, dropout test_loss = 0
for data, target in test_loader:
print(test_loss)
net.train() # toggle batch norm, dropout back again