FullyConnectedNets
Fully-Connected Neural Nets¶
In the previous homework you implemented a fully-connected two-layer neural network on CIFAR-10. The implementation was simple but not very modular since the loss and gradient were computed in a single monolithic function. This is manageable for a simple two-layer network, but would become impractical as we move to bigger models. Ideally we want to build networks using a more modular design so that we can implement different layer types in isolation and then snap them together into models with different architectures.
In this exercise we will implement fully-connected networks using a more modular approach. For each layer we will implement a forward and a backward function. The forward function will receive inputs, weights, and other parameters and will return both an output and a cache object storing data needed for the backward pass, like this:
def layer_forward(x, w):
“”” Receive inputs x and weights w “””
# Do some computations …
z = # … some intermediate value
# Do some more computations …
out = # the output
cache = (x, w, z, out) # Values we need to compute gradients
return out, cache
The backward pass will receive upstream derivatives and the cache object, and will return gradients with respect to the inputs and weights, like this:
def layer_backward(dout, cache):
“””
Receive derivative of loss with respect to outputs and cache,
and compute derivative with respect to inputs.
“””
# Unpack cache values
x, w, z, out = cache
# Use values in cache to compute derivatives
dx = # Derivative of loss with respect to x
dw = # Derivative of loss with respect to w
return dx, dw
After implementing a bunch of layers this way, we will be able to easily combine them to build classifiers with different architectures.
In addition to implementing fully-connected networks of arbitrary depth, we will also explore different update rules for optimization, and introduce Dropout as a regularizer and Batch Normalization as a tool to more efficiently optimize deep networks.
In [232]:
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver
%matplotlib inline
plt.rcParams[‘figure.figsize’] = (10.0, 8.0) # set default size of plots
plt.rcParams[‘image.interpolation’] = ‘nearest’
plt.rcParams[‘image.cmap’] = ‘gray’
# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2
def rel_error(x, y):
“”” returns relative error “””
return np.max(np.abs(x – y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))
The autoreload extension is already loaded. To reload it, use:
%reload_ext autoreload
In [233]:
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.iteritems():
print ‘%s: ‘ % k, v.shape
X_val: (1000, 3, 32, 32)
X_train: (49000, 3, 32, 32)
X_test: (1000, 3, 32, 32)
y_val: (1000,)
y_train: (49000,)
y_test: (1000,)
Affine layer: foward¶
Open the file cs231n/layers.py and implement the affine_forward function.
Once you are done you can test your implementaion by running the following:
In [234]:
# Test the affine_forward function
num_inputs = 2
input_shape = (4, 5, 6)
output_dim = 3
input_size = num_inputs * np.prod(input_shape)
weight_size = output_dim * np.prod(input_shape)
x = np.linspace(-0.1, 0.5, num=input_size).reshape(num_inputs, *input_shape)
w = np.linspace(-0.2, 0.3, num=weight_size).reshape(np.prod(input_shape), output_dim)
b = np.linspace(-0.3, 0.1, num=output_dim)
out, _ = affine_forward(x, w, b)
correct_out = np.array([[ 1.49834967, 1.70660132, 1.91485297],
[ 3.25553199, 3.5141327, 3.77273342]])
# Compare your output with ours. The error should be around 1e-9.
print ‘Testing affine_forward function:’
print ‘difference: ‘, rel_error(out, correct_out)
Testing affine_forward function:
difference: 9.76985004799e-10
Affine layer: backward¶
Now implement the affine_backward function and test your implementation using numeric gradient checking.
In [235]:
# Test the affine_backward function
x = np.random.randn(10, 2, 3)
w = np.random.randn(6, 5)
b = np.random.randn(5)
dout = np.random.randn(10, 5)
dx_num = eval_numerical_gradient_array(lambda x: affine_forward(x, w, b)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: affine_forward(x, w, b)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: affine_forward(x, w, b)[0], b, dout)
_, cache = affine_forward(x, w, b)
dx, dw, db = affine_backward(dout, cache)
# The error should be around 1e-10
print ‘Testing affine_backward function:’
print ‘dx error: ‘, rel_error(dx_num, dx)
print ‘dw error: ‘, rel_error(dw_num, dw)
print ‘db error: ‘, rel_error(db_num, db)
Testing affine_backward function:
dx error: 1.60905909466e-10
dw error: 9.29254103042e-11
db error: 7.95680052642e-12
ReLU layer: forward¶
Implement the forward pass for the ReLU activation function in the relu_forward function and test your implementation using the following:
In [236]:
# Test the relu_forward function
x = np.linspace(-0.5, 0.5, num=12).reshape(3, 4)
out, _ = relu_forward(x)
correct_out = np.array([[ 0., 0., 0., 0., ],
[ 0., 0., 0.04545455, 0.13636364,],
[ 0.22727273, 0.31818182, 0.40909091, 0.5, ]])
# Compare your output with ours. The error should be around 1e-8
print ‘Testing relu_forward function:’
print ‘difference: ‘, rel_error(out, correct_out)
Testing relu_forward function:
difference: 4.99999979802e-08
ReLU layer: backward¶
Now implement the backward pass for the ReLU activation function in the relu_backward function and test your implementation using numeric gradient checking:
In [237]:
x = np.random.randn(10, 10)
dout = np.random.randn(*x.shape)
dx_num = eval_numerical_gradient_array(lambda x: relu_forward(x)[0], x, dout)
_, cache = relu_forward(x)
dx = relu_backward(dout, cache)
# The error should be around 1e-12
print ‘Testing relu_backward function:’
print ‘dx error: ‘, rel_error(dx_num, dx)
Testing relu_backward function:
dx error: 3.27562012926e-12
“Sandwich” layers¶
There are some common patterns of layers that are frequently used in neural nets. For example, affine layers are frequently followed by a ReLU nonlinearity. To make these common patterns easy, we define several convenience layers in the file cs231n/layer_utils.py.
For now take a look at the affine_relu_forward and affine_relu_backward functions, and run the following to numerically gradient check the backward pass:
In [238]:
from cs231n.layer_utils import affine_relu_forward, affine_relu_backward
x = np.random.randn(2, 3, 4)
w = np.random.randn(12, 10)
b = np.random.randn(10)
dout = np.random.randn(2, 10)
out, cache = affine_relu_forward(x, w, b)
dx, dw, db = affine_relu_backward(dout, cache)
dx_num = eval_numerical_gradient_array(lambda x: affine_relu_forward(x, w, b)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: affine_relu_forward(x, w, b)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: affine_relu_forward(x, w, b)[0], b, dout)
print ‘Testing affine_relu_forward:’
print ‘dx error: ‘, rel_error(dx_num, dx)
print ‘dw error: ‘, rel_error(dw_num, dw)
print ‘db error: ‘, rel_error(db_num, db)
Testing affine_relu_forward:
dx error: 5.9286567627e-11
dw error: 5.49570462937e-11
db error: 7.82665623963e-12
Loss layers: Softmax and SVM¶
You implemented these loss functions in the last assignment, so we’ll give them to you for free here. You should still make sure you understand how they work by looking at the implementations in cs231n/layers.py.
You can make sure that the implementations are correct by running the following:
In [239]:
num_classes, num_inputs = 10, 50
x = 0.001 * np.random.randn(num_inputs, num_classes)
y = np.random.randint(num_classes, size=num_inputs)
dx_num = eval_numerical_gradient(lambda x: svm_loss(x, y)[0], x, verbose=False)
loss, dx = svm_loss(x, y)
# Test svm_loss function. Loss should be around 9 and dx error should be 1e-9
print ‘Testing svm_loss:’
print ‘loss: ‘, loss
print ‘dx error: ‘, rel_error(dx_num, dx)
dx_num = eval_numerical_gradient(lambda x: softmax_loss(x, y)[0], x, verbose=False)
loss, dx = softmax_loss(x, y)
# Test softmax_loss function. Loss should be 2.3 and dx error should be 1e-8
print ‘\nTesting softmax_loss:’
print ‘loss: ‘, loss
print ‘dx error: ‘, rel_error(dx_num, dx)
Testing svm_loss:
loss: 9.00215595269
dx error: 1.40215660067e-09
Testing softmax_loss:
loss: 2.30280117814
dx error: 7.36859896605e-09
Two-layer network¶
In the previous assignment you implemented a two-layer neural network in a single monolithic class. Now that you have implemented modular versions of the necessary layers, you will reimplement the two layer network using these modular implementations.
Open the file cs231n/classifiers/fc_net.py and complete the implementation of the TwoLayerNet class. This class will serve as a model for the other networks you will implement in this assignment, so read through it to make sure you understand the API. You can run the cell below to test your implementation.
In [240]:
N, D, H, C = 3, 5, 50, 7
X = np.random.randn(N, D)
y = np.random.randint(C, size=N)
std = 1e-2
model = TwoLayerNet(input_dim=D, hidden_dim=H, num_classes=C, weight_scale=std)
print ‘Testing initialization … ‘
W1_std = abs(model.params[‘W1’].std() – std)
b1 = model.params[‘b1’]
W2_std = abs(model.params[‘W2’].std() – std)
b2 = model.params[‘b2’]
assert W1_std < std / 10, 'First layer weights do not seem right'
assert np.all(b1 == 0), 'First layer biases do not seem right'
assert W2_std < std / 10, 'Second layer weights do not seem right'
assert np.all(b2 == 0), 'Second layer biases do not seem right'
print 'Testing test-time forward pass ... '
model.params['W1'] = np.linspace(-0.7, 0.3, num=D*H).reshape(D, H)
model.params['b1'] = np.linspace(-0.1, 0.9, num=H)
model.params['W2'] = np.linspace(-0.3, 0.4, num=H*C).reshape(H, C)
model.params['b2'] = np.linspace(-0.9, 0.1, num=C)
X = np.linspace(-5.5, 4.5, num=N*D).reshape(D, N).T
scores = model.loss(X)
correct_scores = np.asarray(
[[11.53165108, 12.2917344, 13.05181771, 13.81190102, 14.57198434, 15.33206765, 16.09215096],
[12.05769098, 12.74614105, 13.43459113, 14.1230412, 14.81149128, 15.49994135, 16.18839143],
[12.58373087, 13.20054771, 13.81736455, 14.43418138, 15.05099822, 15.66781506, 16.2846319 ]])
scores_diff = np.abs(scores - correct_scores).sum()
assert scores_diff < 1e-6, 'Problem with test-time forward pass'
print 'Testing training loss (no regularization)'
y = np.asarray([0, 5, 1])
loss, grads = model.loss(X, y)
correct_loss = 3.4702243556
assert abs(loss - correct_loss) < 1e-10, 'Problem with training-time loss'
model.reg = 1.0
loss, grads = model.loss(X, y)
correct_loss = 26.5948426952
assert abs(loss - correct_loss) < 1e-10, 'Problem with regularization loss'
for reg in [0.0, 0.7]:
print 'Running numeric gradient check with reg = ', reg
model.reg = reg
loss, grads = model.loss(X, y)
for name in sorted(grads):
f = lambda _: model.loss(X, y)[0]
grad_num = eval_numerical_gradient(f, model.params[name], verbose=False)
print '%s relative error: %.2e' % (name, rel_error(grad_num, grads[name]))
Testing initialization ...
Testing test-time forward pass ...
Testing training loss (no regularization)
Running numeric gradient check with reg = 0.0
W1 relative error: 1.83e-08
W2 relative error: 3.30e-10
b1 relative error: 6.19e-09
b2 relative error: 2.53e-10
Running numeric gradient check with reg = 0.7
W1 relative error: 3.12e-07
W2 relative error: 7.98e-08
b1 relative error: 1.56e-08
b2 relative error: 1.97e-09
Solver¶
In the previous assignment, the logic for training models was coupled to the models themselves. Following a more modular design, for this assignment we have split the logic for training models into a separate class.
Open the file cs231n/solver.py and read through it to familiarize yourself with the API. After doing so, use a Solver instance to train a TwoLayerNet that achieves at least 50% accuracy on the validation set.
In [241]:
model = TwoLayerNet(hidden_dim = 100, reg = 0.1)
# model = TwoLayerNet(hidden_dim = 100, reg = 0.5)
solver = None
##############################################################################
# TODO: Use a Solver instance to train a TwoLayerNet that achieves at least #
# 50% accuracy on the validation set. #
##############################################################################
# solver = Solver(model, data, optim_config={
# 'learning_rate': 1e-3,
# }, num_epochs = 2)
solver = Solver(model, data, optim_config={
'learning_rate': 1e-3,
}, num_epochs = 9)
solver.train()
print solver.val_acc_history
##############################################################################
# END OF YOUR CODE #
##############################################################################
(Iteration 1 / 4410) loss: 2.319738
(Epoch 0 / 9) train acc: 0.153000; val_acc: 0.127000
(Iteration 11 / 4410) loss: 2.262921
(Iteration 21 / 4410) loss: 2.159653
(Iteration 31 / 4410) loss: 1.950457
(Iteration 41 / 4410) loss: 1.915092
(Iteration 51 / 4410) loss: 1.989862
(Iteration 61 / 4410) loss: 2.020402
(Iteration 71 / 4410) loss: 1.940897
(Iteration 81 / 4410) loss: 1.822999
(Iteration 91 / 4410) loss: 1.816730
(Iteration 101 / 4410) loss: 1.837209
(Iteration 111 / 4410) loss: 1.650717
(Iteration 121 / 4410) loss: 1.781623
(Iteration 131 / 4410) loss: 1.730213
(Iteration 141 / 4410) loss: 1.652465
(Iteration 151 / 4410) loss: 1.760553
(Iteration 161 / 4410) loss: 1.673839
(Iteration 171 / 4410) loss: 1.759575
(Iteration 181 / 4410) loss: 1.644645
(Iteration 191 / 4410) loss: 1.666116
(Iteration 201 / 4410) loss: 1.924639
(Iteration 211 / 4410) loss: 1.782295
(Iteration 221 / 4410) loss: 1.646411
(Iteration 231 / 4410) loss: 1.714558
(Iteration 241 / 4410) loss: 1.692132
(Iteration 251 / 4410) loss: 1.511811
(Iteration 261 / 4410) loss: 1.723601
(Iteration 271 / 4410) loss: 1.776977
(Iteration 281 / 4410) loss: 1.858390
(Iteration 291 / 4410) loss: 1.774039
(Iteration 301 / 4410) loss: 1.426478
(Iteration 311 / 4410) loss: 1.845384
(Iteration 321 / 4410) loss: 1.789854
(Iteration 331 / 4410) loss: 1.645531
(Iteration 341 / 4410) loss: 1.652949
(Iteration 351 / 4410) loss: 1.803912
(Iteration 361 / 4410) loss: 1.574185
(Iteration 371 / 4410) loss: 1.613928
(Iteration 381 / 4410) loss: 1.641083
(Iteration 391 / 4410) loss: 1.594400
(Iteration 401 / 4410) loss: 1.499600
(Iteration 411 / 4410) loss: 1.480542
(Iteration 421 / 4410) loss: 1.601294
(Iteration 431 / 4410) loss: 1.604341
(Iteration 441 / 4410) loss: 1.646796
(Iteration 451 / 4410) loss: 1.534030
(Iteration 461 / 4410) loss: 1.533326
(Iteration 471 / 4410) loss: 1.535660
(Iteration 481 / 4410) loss: 1.548296
(Epoch 1 / 9) train acc: 0.463000; val_acc: 0.425000
(Iteration 491 / 4410) loss: 1.630025
(Iteration 501 / 4410) loss: 1.572768
(Iteration 511 / 4410) loss: 1.445413
(Iteration 521 / 4410) loss: 1.541172
(Iteration 531 / 4410) loss: 1.474246
(Iteration 541 / 4410) loss: 1.635412
(Iteration 551 / 4410) loss: 1.562905
(Iteration 561 / 4410) loss: 1.549470
(Iteration 571 / 4410) loss: 1.355848
(Iteration 581 / 4410) loss: 1.478852
(Iteration 591 / 4410) loss: 1.409039
(Iteration 601 / 4410) loss: 1.289073
(Iteration 611 / 4410) loss: 1.604687
(Iteration 621 / 4410) loss: 1.530044
(Iteration 631 / 4410) loss: 1.562123
(Iteration 641 / 4410) loss: 1.549260
(Iteration 651 / 4410) loss: 1.432657
(Iteration 661 / 4410) loss: 1.331559
(Iteration 671 / 4410) loss: 1.410692
(Iteration 681 / 4410) loss: 1.586152
(Iteration 691 / 4410) loss: 1.420657
(Iteration 701 / 4410) loss: 1.591942
(Iteration 711 / 4410) loss: 1.567296
(Iteration 721 / 4410) loss: 1.585252
(Iteration 731 / 4410) loss: 1.543941
(Iteration 741 / 4410) loss: 1.526780
(Iteration 751 / 4410) loss: 1.436072
(Iteration 761 / 4410) loss: 1.261689
(Iteration 771 / 4410) loss: 1.499828
(Iteration 781 / 4410) loss: 1.570915
(Iteration 791 / 4410) loss: 1.518846
(Iteration 801 / 4410) loss: 1.321185
(Iteration 811 / 4410) loss: 1.489962
(Iteration 821 / 4410) loss: 1.674558
(Iteration 831 / 4410) loss: 1.469799
(Iteration 841 / 4410) loss: 1.603955
(Iteration 851 / 4410) loss: 1.569082
(Iteration 861 / 4410) loss: 1.358193
(Iteration 871 / 4410) loss: 1.527113
(Iteration 881 / 4410) loss: 1.543378
(Iteration 891 / 4410) loss: 1.425420
(Iteration 901 / 4410) loss: 1.613125
(Iteration 911 / 4410) loss: 1.600631
(Iteration 921 / 4410) loss: 1.511697
(Iteration 931 / 4410) loss: 1.712064
(Iteration 941 / 4410) loss: 1.434962
(Iteration 951 / 4410) loss: 1.451377
(Iteration 961 / 4410) loss: 1.543924
(Iteration 971 / 4410) loss: 1.547489
(Epoch 2 / 9) train acc: 0.485000; val_acc: 0.457000
(Iteration 981 / 4410) loss: 1.391220
(Iteration 991 / 4410) loss: 1.542406
(Iteration 1001 / 4410) loss: 1.422547
(Iteration 1011 / 4410) loss: 1.664301
(Iteration 1021 / 4410) loss: 1.723590
(Iteration 1031 / 4410) loss: 1.551552
(Iteration 1041 / 4410) loss: 1.654569
(Iteration 1051 / 4410) loss: 1.604904
(Iteration 1061 / 4410) loss: 1.204552
(Iteration 1071 / 4410) loss: 1.560923
(Iteration 1081 / 4410) loss: 1.614699
(Iteration 1091 / 4410) loss: 1.220218
(Iteration 1101 / 4410) loss: 1.542579
(Iteration 1111 / 4410) loss: 1.284045
(Iteration 1121 / 4410) loss: 1.386605
(Iteration 1131 / 4410) loss: 1.721553
(Iteration 1141 / 4410) loss: 1.592495
(Iteration 1151 / 4410) loss: 1.565615
(Iteration 1161 / 4410) loss: 1.489961
(Iteration 1171 / 4410) loss: 1.361646
(Iteration 1181 / 4410) loss: 1.384437
(Iteration 1191 / 4410) loss: 1.448046
(Iteration 1201 / 4410) loss: 1.390661
(Iteration 1211 / 4410) loss: 1.412863
(Iteration 1221 / 4410) loss: 1.447514
(Iteration 1231 / 4410) loss: 1.108814
(Iteration 1241 / 4410) loss: 1.376537
(Iteration 1251 / 4410) loss: 1.654788
(Iteration 1261 / 4410) loss: 1.400324
(Iteration 1271 / 4410) loss: 1.391381
(Iteration 1281 / 4410) loss: 1.430033
(Iteration 1291 / 4410) loss: 1.462303
(Iteration 1301 / 4410) loss: 1.770055
(Iteration 1311 / 4410) loss: 1.521252
(Iteration 1321 / 4410) loss: 1.397367
(Iteration 1331 / 4410) loss: 1.341383
(Iteration 1341 / 4410) loss: 1.351241
(Iteration 1351 / 4410) loss: 1.379555
(Iteration 1361 / 4410) loss: 1.388974
(Iteration 1371 / 4410) loss: 1.425686
(Iteration 1381 / 4410) loss: 1.537995
(Iteration 1391 / 4410) loss: 1.449363
(Iteration 1401 / 4410) loss: 1.319210
(Iteration 1411 / 4410) loss: 1.319277
(Iteration 1421 / 4410) loss: 1.521891
(Iteration 1431 / 4410) loss: 1.479150
(Iteration 1441 / 4410) loss: 1.613436
(Iteration 1451 / 4410) loss: 1.419339
(Iteration 1461 / 4410) loss: 1.527739
(Epoch 3 / 9) train acc: 0.503000; val_acc: 0.458000
(Iteration 1471 / 4410) loss: 1.460143
(Iteration 1481 / 4410) loss: 1.302641
(Iteration 1491 / 4410) loss: 1.227460
(Iteration 1501 / 4410) loss: 1.317373
(Iteration 1511 / 4410) loss: 1.362677
(Iteration 1521 / 4410) loss: 1.456078
(Iteration 1531 / 4410) loss: 1.633912
(Iteration 1541 / 4410) loss: 1.535626
(Iteration 1551 / 4410) loss: 1.539447
(Iteration 1561 / 4410) loss: 1.452644
(Iteration 1571 / 4410) loss: 1.523952
(Iteration 1581 / 4410) loss: 1.553443
(Iteration 1591 / 4410) loss: 1.505596
(Iteration 1601 / 4410) loss: 1.365387
(Iteration 1611 / 4410) loss: 1.584877
(Iteration 1621 / 4410) loss: 1.406193
(Iteration 1631 / 4410) loss: 1.459195
(Iteration 1641 / 4410) loss: 1.354144
(Iteration 1651 / 4410) loss: 1.338928
(Iteration 1661 / 4410) loss: 1.418942
(Iteration 1671 / 4410) loss: 1.547180
(Iteration 1681 / 4410) loss: 1.390323
(Iteration 1691 / 4410) loss: 1.630900
(Iteration 1701 / 4410) loss: 1.363029
(Iteration 1711 / 4410) loss: 1.510090
(Iteration 1721 / 4410) loss: 1.579805
(Iteration 1731 / 4410) loss: 1.398198
(Iteration 1741 / 4410) loss: 1.422568
(Iteration 1751 / 4410) loss: 1.549703
(Iteration 1761 / 4410) loss: 1.598392
(Iteration 1771 / 4410) loss: 1.289773
(Iteration 1781 / 4410) loss: 1.250841
(Iteration 1791 / 4410) loss: 1.405500
(Iteration 1801 / 4410) loss: 1.470474
(Iteration 1811 / 4410) loss: 1.471360
(Iteration 1821 / 4410) loss: 1.481858
(Iteration 1831 / 4410) loss: 1.338778
(Iteration 1841 / 4410) loss: 1.401743
(Iteration 1851 / 4410) loss: 1.229749
(Iteration 1861 / 4410) loss: 1.453018
(Iteration 1871 / 4410) loss: 1.405095
(Iteration 1881 / 4410) loss: 1.519162
(Iteration 1891 / 4410) loss: 1.838334
(Iteration 1901 / 4410) loss: 1.476578
(Iteration 1911 / 4410) loss: 1.431684
(Iteration 1921 / 4410) loss: 1.511331
(Iteration 1931 / 4410) loss: 1.452015
(Iteration 1941 / 4410) loss: 1.263401
(Iteration 1951 / 4410) loss: 1.454320
(Epoch 4 / 9) train acc: 0.524000; val_acc: 0.473000
(Iteration 1961 / 4410) loss: 1.292333
(Iteration 1971 / 4410) loss: 1.430337
(Iteration 1981 / 4410) loss: 1.356667
(Iteration 1991 / 4410) loss: 1.289947
(Iteration 2001 / 4410) loss: 1.335689
(Iteration 2011 / 4410) loss: 1.254235
(Iteration 2021 / 4410) loss: 1.407755
(Iteration 2031 / 4410) loss: 1.369994
(Iteration 2041 / 4410) loss: 1.336487
(Iteration 2051 / 4410) loss: 1.363167
(Iteration 2061 / 4410) loss: 1.455742
(Iteration 2071 / 4410) loss: 1.320711
(Iteration 2081 / 4410) loss: 1.609578
(Iteration 2091 / 4410) loss: 1.473916
(Iteration 2101 / 4410) loss: 1.231268
(Iteration 2111 / 4410) loss: 1.443858
(Iteration 2121 / 4410) loss: 1.467306
(Iteration 2131 / 4410) loss: 1.632236
(Iteration 2141 / 4410) loss: 1.517211
(Iteration 2151 / 4410) loss: 1.378249
(Iteration 2161 / 4410) loss: 1.532184
(Iteration 2171 / 4410) loss: 1.401308
(Iteration 2181 / 4410) loss: 1.181670
(Iteration 2191 / 4410) loss: 1.381456
(Iteration 2201 / 4410) loss: 1.257514
(Iteration 2211 / 4410) loss: 1.426615
(Iteration 2221 / 4410) loss: 1.307758
(Iteration 2231 / 4410) loss: 1.313862
(Iteration 2241 / 4410) loss: 1.256748
(Iteration 2251 / 4410) loss: 1.353501
(Iteration 2261 / 4410) loss: 1.305433
(Iteration 2271 / 4410) loss: 1.267495
(Iteration 2281 / 4410) loss: 1.193422
(Iteration 2291 / 4410) loss: 1.605454
(Iteration 2301 / 4410) loss: 1.073342
(Iteration 2311 / 4410) loss: 1.344562
(Iteration 2321 / 4410) loss: 1.458204
(Iteration 2331 / 4410) loss: 1.339261
(Iteration 2341 / 4410) loss: 1.450693
(Iteration 2351 / 4410) loss: 1.603632
(Iteration 2361 / 4410) loss: 1.258094
(Iteration 2371 / 4410) loss: 1.479132
(Iteration 2381 / 4410) loss: 1.546170
(Iteration 2391 / 4410) loss: 1.409149
(Iteration 2401 / 4410) loss: 1.478375
(Iteration 2411 / 4410) loss: 1.329643
(Iteration 2421 / 4410) loss: 1.492849
(Iteration 2431 / 4410) loss: 1.608688
(Iteration 2441 / 4410) loss: 1.226119
(Epoch 5 / 9) train acc: 0.541000; val_acc: 0.500000
(Iteration 2451 / 4410) loss: 1.330464
(Iteration 2461 / 4410) loss: 1.395229
(Iteration 2471 / 4410) loss: 1.324528
(Iteration 2481 / 4410) loss: 1.135624
(Iteration 2491 / 4410) loss: 1.501870
(Iteration 2501 / 4410) loss: 1.460544
(Iteration 2511 / 4410) loss: 1.559089
(Iteration 2521 / 4410) loss: 1.421624
(Iteration 2531 / 4410) loss: 1.319807
(Iteration 2541 / 4410) loss: 1.230738
(Iteration 2551 / 4410) loss: 1.258506
(Iteration 2561 / 4410) loss: 1.223155
(Iteration 2571 / 4410) loss: 1.298995
(Iteration 2581 / 4410) loss: 1.351395
(Iteration 2591 / 4410) loss: 1.282634
(Iteration 2601 / 4410) loss: 1.334566
(Iteration 2611 / 4410) loss: 1.222974
(Iteration 2621 / 4410) loss: 1.154032
(Iteration 2631 / 4410) loss: 1.507005
(Iteration 2641 / 4410) loss: 1.482526
(Iteration 2651 / 4410) loss: 1.267671
(Iteration 2661 / 4410) loss: 1.554007
(Iteration 2671 / 4410) loss: 1.312191
(Iteration 2681 / 4410) loss: 1.338630
(Iteration 2691 / 4410) loss: 1.323107
(Iteration 2701 / 4410) loss: 1.154439
(Iteration 2711 / 4410) loss: 1.254514
(Iteration 2721 / 4410) loss: 1.591575
(Iteration 2731 / 4410) loss: 1.470424
(Iteration 2741 / 4410) loss: 1.686682
(Iteration 2751 / 4410) loss: 1.231132
(Iteration 2761 / 4410) loss: 1.596924
(Iteration 2771 / 4410) loss: 1.626200
(Iteration 2781 / 4410) loss: 1.489466
(Iteration 2791 / 4410) loss: 1.473520
(Iteration 2801 / 4410) loss: 1.129453
(Iteration 2811 / 4410) loss: 1.347902
(Iteration 2821 / 4410) loss: 1.230674
(Iteration 2831 / 4410) loss: 1.623829
(Iteration 2841 / 4410) loss: 1.379852
(Iteration 2851 / 4410) loss: 1.040788
(Iteration 2861 / 4410) loss: 1.356169
(Iteration 2871 / 4410) loss: 1.485796
(Iteration 2881 / 4410) loss: 1.546273
(Iteration 2891 / 4410) loss: 1.270810
(Iteration 2901 / 4410) loss: 1.307143
(Iteration 2911 / 4410) loss: 1.239869
(Iteration 2921 / 4410) loss: 1.486829
(Iteration 2931 / 4410) loss: 1.282943
(Epoch 6 / 9) train acc: 0.535000; val_acc: 0.477000
(Iteration 2941 / 4410) loss: 1.257630
(Iteration 2951 / 4410) loss: 1.416531
(Iteration 2961 / 4410) loss: 1.476723
(Iteration 2971 / 4410) loss: 1.516978
(Iteration 2981 / 4410) loss: 1.293609
(Iteration 2991 / 4410) loss: 1.316649
(Iteration 3001 / 4410) loss: 1.440714
(Iteration 3011 / 4410) loss: 1.551229
(Iteration 3021 / 4410) loss: 1.268180
(Iteration 3031 / 4410) loss: 1.285427
(Iteration 3041 / 4410) loss: 1.507909
(Iteration 3051 / 4410) loss: 1.194058
(Iteration 3061 / 4410) loss: 1.419716
(Iteration 3071 / 4410) loss: 1.278823
(Iteration 3081 / 4410) loss: 1.162273
(Iteration 3091 / 4410) loss: 1.303089
(Iteration 3101 / 4410) loss: 1.220389
(Iteration 3111 / 4410) loss: 1.300228
(Iteration 3121 / 4410) loss: 1.307898
(Iteration 3131 / 4410) loss: 1.469828
(Iteration 3141 / 4410) loss: 1.401164
(Iteration 3151 / 4410) loss: 1.277402
(Iteration 3161 / 4410) loss: 1.216145
(Iteration 3171 / 4410) loss: 1.229346
(Iteration 3181 / 4410) loss: 1.304570
(Iteration 3191 / 4410) loss: 1.301294
(Iteration 3201 / 4410) loss: 1.391827
(Iteration 3211 / 4410) loss: 1.507222
(Iteration 3221 / 4410) loss: 1.261636
(Iteration 3231 / 4410) loss: 1.142812
(Iteration 3241 / 4410) loss: 1.490284
(Iteration 3251 / 4410) loss: 1.397549
(Iteration 3261 / 4410) loss: 1.357512
(Iteration 3271 / 4410) loss: 1.182125
(Iteration 3281 / 4410) loss: 1.404025
(Iteration 3291 / 4410) loss: 1.264220
(Iteration 3301 / 4410) loss: 1.383975
(Iteration 3311 / 4410) loss: 1.241076
(Iteration 3321 / 4410) loss: 1.316902
(Iteration 3331 / 4410) loss: 1.445274
(Iteration 3341 / 4410) loss: 1.263965
(Iteration 3351 / 4410) loss: 1.338915
(Iteration 3361 / 4410) loss: 1.434232
(Iteration 3371 / 4410) loss: 1.171517
(Iteration 3381 / 4410) loss: 1.281230
(Iteration 3391 / 4410) loss: 1.296191
(Iteration 3401 / 4410) loss: 1.507836
(Iteration 3411 / 4410) loss: 1.278282
(Iteration 3421 / 4410) loss: 1.322937
(Epoch 7 / 9) train acc: 0.554000; val_acc: 0.498000
(Iteration 3431 / 4410) loss: 1.448609
(Iteration 3441 / 4410) loss: 1.308829
(Iteration 3451 / 4410) loss: 1.342272
(Iteration 3461 / 4410) loss: 1.240236
(Iteration 3471 / 4410) loss: 1.664111
(Iteration 3481 / 4410) loss: 1.298286
(Iteration 3491 / 4410) loss: 1.448136
(Iteration 3501 / 4410) loss: 1.381373
(Iteration 3511 / 4410) loss: 1.472371
(Iteration 3521 / 4410) loss: 1.341148
(Iteration 3531 / 4410) loss: 1.382441
(Iteration 3541 / 4410) loss: 1.128447
(Iteration 3551 / 4410) loss: 1.353595
(Iteration 3561 / 4410) loss: 1.367709
(Iteration 3571 / 4410) loss: 1.376994
(Iteration 3581 / 4410) loss: 1.167034
(Iteration 3591 / 4410) loss: 1.451596
(Iteration 3601 / 4410) loss: 1.457467
(Iteration 3611 / 4410) loss: 1.166876
(Iteration 3621 / 4410) loss: 1.276241
(Iteration 3631 / 4410) loss: 1.457309
(Iteration 3641 / 4410) loss: 1.442376
(Iteration 3651 / 4410) loss: 1.252569
(Iteration 3661 / 4410) loss: 1.361933
(Iteration 3671 / 4410) loss: 1.270811
(Iteration 3681 / 4410) loss: 1.178336
(Iteration 3691 / 4410) loss: 1.187737
(Iteration 3701 / 4410) loss: 1.439233
(Iteration 3711 / 4410) loss: 1.427057
(Iteration 3721 / 4410) loss: 1.226818
(Iteration 3731 / 4410) loss: 1.350572
(Iteration 3741 / 4410) loss: 1.426303
(Iteration 3751 / 4410) loss: 1.242823
(Iteration 3761 / 4410) loss: 1.452348
(Iteration 3771 / 4410) loss: 1.531706
(Iteration 3781 / 4410) loss: 1.664020
(Iteration 3791 / 4410) loss: 1.258655
(Iteration 3801 / 4410) loss: 1.450315
(Iteration 3811 / 4410) loss: 1.306280
(Iteration 3821 / 4410) loss: 1.439891
(Iteration 3831 / 4410) loss: 1.737188
(Iteration 3841 / 4410) loss: 1.297578
(Iteration 3851 / 4410) loss: 1.243508
(Iteration 3861 / 4410) loss: 1.489027
(Iteration 3871 / 4410) loss: 1.550849
(Iteration 3881 / 4410) loss: 1.219820
(Iteration 3891 / 4410) loss: 1.340278
(Iteration 3901 / 4410) loss: 1.354103
(Iteration 3911 / 4410) loss: 1.398382
(Epoch 8 / 9) train acc: 0.515000; val_acc: 0.449000
(Iteration 3921 / 4410) loss: 1.291152
(Iteration 3931 / 4410) loss: 1.225319
(Iteration 3941 / 4410) loss: 1.255579
(Iteration 3951 / 4410) loss: 1.477113
(Iteration 3961 / 4410) loss: 1.545517
(Iteration 3971 / 4410) loss: 1.206355
(Iteration 3981 / 4410) loss: 1.448442
(Iteration 3991 / 4410) loss: 1.409735
(Iteration 4001 / 4410) loss: 1.149757
(Iteration 4011 / 4410) loss: 1.224087
(Iteration 4021 / 4410) loss: 1.410009
(Iteration 4031 / 4410) loss: 1.356994
(Iteration 4041 / 4410) loss: 1.273410
(Iteration 4051 / 4410) loss: 1.133135
(Iteration 4061 / 4410) loss: 1.341896
(Iteration 4071 / 4410) loss: 1.516672
(Iteration 4081 / 4410) loss: 1.164957
(Iteration 4091 / 4410) loss: 1.377441
(Iteration 4101 / 4410) loss: 1.273577
(Iteration 4111 / 4410) loss: 1.425267
(Iteration 4121 / 4410) loss: 1.586978
(Iteration 4131 / 4410) loss: 1.351342
(Iteration 4141 / 4410) loss: 1.129888
(Iteration 4151 / 4410) loss: 1.096448
(Iteration 4161 / 4410) loss: 1.337608
(Iteration 4171 / 4410) loss: 1.191462
(Iteration 4181 / 4410) loss: 1.517677
(Iteration 4191 / 4410) loss: 1.202016
(Iteration 4201 / 4410) loss: 1.425942
(Iteration 4211 / 4410) loss: 1.322403
(Iteration 4221 / 4410) loss: 1.353571
(Iteration 4231 / 4410) loss: 1.278520
(Iteration 4241 / 4410) loss: 1.369579
(Iteration 4251 / 4410) loss: 1.470915
(Iteration 4261 / 4410) loss: 1.399251
(Iteration 4271 / 4410) loss: 1.323321
(Iteration 4281 / 4410) loss: 1.337231
(Iteration 4291 / 4410) loss: 1.265155
(Iteration 4301 / 4410) loss: 1.108870
(Iteration 4311 / 4410) loss: 1.516656
(Iteration 4321 / 4410) loss: 1.481227
(Iteration 4331 / 4410) loss: 1.174772
(Iteration 4341 / 4410) loss: 1.408258
(Iteration 4351 / 4410) loss: 1.258764
(Iteration 4361 / 4410) loss: 1.235203
(Iteration 4371 / 4410) loss: 1.357147
(Iteration 4381 / 4410) loss: 1.157298
(Iteration 4391 / 4410) loss: 1.656670
(Iteration 4401 / 4410) loss: 1.462702
(Epoch 9 / 9) train acc: 0.572000; val_acc: 0.491000
[0.127, 0.42499999999999999, 0.45700000000000002, 0.45800000000000002, 0.47299999999999998, 0.5, 0.47699999999999998, 0.498, 0.44900000000000001, 0.49099999999999999]
In [242]:
# Run this cell to visualize training loss and train / val accuracy
plt.subplot(2, 1, 1)
plt.title('Training loss')
plt.plot(solver.loss_history, 'o')
plt.xlabel('Iteration')
plt.subplot(2, 1, 2)
plt.title('Accuracy')
plt.plot(solver.train_acc_history, '-o', label='train')
plt.plot(solver.val_acc_history, '-o', label='val')
plt.plot([0.5] * len(solver.val_acc_history), 'k--')
plt.xlabel('Epoch')
plt.legend(loc='lower right')
plt.gcf().set_size_inches(15, 12)
plt.show()
Multilayer network¶
Next you will implement a fully-connected network with an arbitrary number of hidden layers.
Read through the FullyConnectedNet class in the file cs231n/classifiers/fc_net.py.
Implement the initialization, the forward pass, and the backward pass. For the moment don't worry about implementing dropout or batch normalization; we will add those features soon.
Initial loss and gradient check¶
As a sanity check, run the following to check the initial loss and to gradient check the network both with and without regularization. Do the initial losses seem reasonable?
For gradient checking, you should expect to see errors around 1e-6 or less.
In [246]:
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))
for reg in [0, 3.14]:
print 'Running check with reg = ', reg
model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
reg=reg, weight_scale=5e-2, dtype=np.float64)
loss, grads = model.loss(X, y)
print 'Initial loss: ', loss
for name in sorted(grads):
f = lambda _: model.loss(X, y)[0]
grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
print '%s relative error: %.2e' % (name, rel_error(grad_num, grads[name]))
Running check with reg = 0
Initial loss: 2.30249165892
W1 relative error: 1.26e-06
W2 relative error: 1.41e-07
W3 relative error: 8.57e-08
b1 relative error: 4.07e-08
b2 relative error: 1.96e-08
b3 relative error: 1.21e-10
Running check with reg = 3.14
Initial loss: 7.27401218691
W1 relative error: 5.88e-09
W2 relative error: 1.57e-08
W3 relative error: 5.40e-08
b1 relative error: 2.53e-08
b2 relative error: 1.08e-08
b3 relative error: 3.21e-10
As another sanity check, make sure you can overfit a small dataset of 50 images. First we will try a three-layer network with 100 units in each hidden layer. You will need to tweak the learning rate and initialization scale, but you should be able to overfit and achieve 100% training accuracy within 20 epochs.
In [247]:
# TODO: Use a three-layer Net to overfit 50 training examples.
num_train = 50
small_data = {
'X_train': data['X_train'][:num_train],
'y_train': data['y_train'][:num_train],
'X_val': data['X_val'],
'y_val': data['y_val'],
}
weight_scale = 1e-2
learning_rate = 1e-2
learning_rate = 1e-2 * 1.5
model = FullyConnectedNet([100, 100],
weight_scale=weight_scale, dtype=np.float64)
solver = Solver(model, small_data,
print_every=10, num_epochs=20, batch_size=25,
update_rule='sgd',
optim_config={
'learning_rate': learning_rate,
}
)
solver.train()
plt.plot(solver.loss_history, 'o')
plt.title('Training loss history')
plt.xlabel('Iteration')
plt.ylabel('Training loss')
plt.show()
(Iteration 1 / 40) loss: 2.235997
(Epoch 0 / 20) train acc: 0.180000; val_acc: 0.127000
(Epoch 1 / 20) train acc: 0.160000; val_acc: 0.112000
(Epoch 2 / 20) train acc: 0.440000; val_acc: 0.162000
(Epoch 3 / 20) train acc: 0.500000; val_acc: 0.153000
(Epoch 4 / 20) train acc: 0.600000; val_acc: 0.175000
(Epoch 5 / 20) train acc: 0.700000; val_acc: 0.163000
(Iteration 11 / 40) loss: 1.063033
(Epoch 6 / 20) train acc: 0.640000; val_acc: 0.172000
(Epoch 7 / 20) train acc: 0.880000; val_acc: 0.196000
(Epoch 8 / 20) train acc: 0.920000; val_acc: 0.208000
(Epoch 9 / 20) train acc: 0.920000; val_acc: 0.182000
(Epoch 10 / 20) train acc: 0.920000; val_acc: 0.186000
(Iteration 21 / 40) loss: 0.248051
(Epoch 11 / 20) train acc: 0.940000; val_acc: 0.182000
(Epoch 12 / 20) train acc: 1.000000; val_acc: 0.184000
(Epoch 13 / 20) train acc: 1.000000; val_acc: 0.181000
(Epoch 14 / 20) train acc: 1.000000; val_acc: 0.196000
(Epoch 15 / 20) train acc: 1.000000; val_acc: 0.196000
(Iteration 31 / 40) loss: 0.044978
(Epoch 16 / 20) train acc: 1.000000; val_acc: 0.188000
(Epoch 17 / 20) train acc: 1.000000; val_acc: 0.185000
(Epoch 18 / 20) train acc: 1.000000; val_acc: 0.183000
(Epoch 19 / 20) train acc: 1.000000; val_acc: 0.185000
(Epoch 20 / 20) train acc: 1.000000; val_acc: 0.186000
Now try to use a five-layer network with 100 units on each layer to overfit 50 training examples. Again you will have to adjust the learning rate and weight initialization, but you should be able to achieve 100% training accuracy within 20 epochs.
In [264]:
# TODO: Use a five-layer Net to overfit 50 training examples.
num_train = 50
small_data = {
'X_train': data['X_train'][:num_train],
'y_train': data['y_train'][:num_train],
'X_val': data['X_val'],
'y_val': data['y_val'],
}
learning_rate = 0.5e-1
weight_scale = 1
weight_scale = 1e-1
learning_rate = 1e-2 / 2 / 2 / 2 / 2 / 2
weight_scale = 0.08
weight_scale = 0.09
learning_rate = 0.0003
# learning_rate = 0.00025
# weight_scale = 0.085
# learning_rate = 1e-4
model = FullyConnectedNet([100, 100, 100, 100],
weight_scale=weight_scale, dtype=np.float64)
solver = Solver(model, small_data,
print_every=10, num_epochs=20, batch_size=50,
update_rule='sgd',
optim_config={
'learning_rate': learning_rate,
}
)
solver.train()
plt.plot(solver.loss_history, 'o')
plt.title('Training loss history')
plt.xlabel('Iteration')
plt.ylabel('Training loss')
plt.show()
(Iteration 1 / 20) loss: 82.351409
(Epoch 1 / 20) train acc: 0.080000; val_acc: 0.093000
(Epoch 2 / 20) train acc: 0.200000; val_acc: 0.082000
(Epoch 3 / 20) train acc: 0.300000; val_acc: 0.094000
(Epoch 4 / 20) train acc: 0.400000; val_acc: 0.101000
(Epoch 5 / 20) train acc: 0.460000; val_acc: 0.101000
(Epoch 6 / 20) train acc: 0.540000; val_acc: 0.100000
(Epoch 7 / 20) train acc: 0.560000; val_acc: 0.100000
(Epoch 8 / 20) train acc: 0.560000; val_acc: 0.108000
(Epoch 9 / 20) train acc: 0.700000; val_acc: 0.101000
(Epoch 10 / 20) train acc: 0.700000; val_acc: 0.102000
(Iteration 11 / 20) loss: 3.285188
(Epoch 11 / 20) train acc: 0.740000; val_acc: 0.099000
(Epoch 12 / 20) train acc: 0.720000; val_acc: 0.106000
(Epoch 13 / 20) train acc: 0.680000; val_acc: 0.105000
(Epoch 14 / 20) train acc: 0.880000; val_acc: 0.100000
(Epoch 15 / 20) train acc: 0.860000; val_acc: 0.100000
(Epoch 16 / 20) train acc: 0.880000; val_acc: 0.105000
(Epoch 17 / 20) train acc: 0.840000; val_acc: 0.093000
(Epoch 18 / 20) train acc: 0.920000; val_acc: 0.109000
(Epoch 19 / 20) train acc: 0.960000; val_acc: 0.108000
(Epoch 20 / 20) train acc: 0.980000; val_acc: 0.101000
Inline question:¶
Did you notice anything about the comparative difficulty of training the three-layer net vs training the five layer net?
Answer:¶
[FILL THIS IN]
Update rules¶
So far we have used vanilla stochastic gradient descent (SGD) as our update rule. More sophisticated update rules can make it easier to train deep networks. We will implement a few of the most commonly used update rules and compare them to vanilla SGD.
SGD+Momentum¶
Stochastic gradient descent with momentum is a widely used update rule that tends to make deep networks converge faster than vanilla stochstic gradient descent.
Open the file cs231n/optim.py and read the documentation at the top of the file to make sure you understand the API. Implement the SGD+momentum update rule in the function sgd_momentum and run the following to check your implementation. You should see errors less than 1e-8.
In [258]:
from cs231n.optim import sgd_momentum
N, D = 4, 5
w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
v = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)
config = {'learning_rate': 1e-3, 'velocity': v}
next_w, _ = sgd_momentum(w, dw, config=config)
expected_next_w = np.asarray([
[ 0.1406, 0.20738947, 0.27417895, 0.34096842, 0.40775789],
[ 0.47454737, 0.54133684, 0.60812632, 0.67491579, 0.74170526],
[ 0.80849474, 0.87528421, 0.94207368, 1.00886316, 1.07565263],
[ 1.14244211, 1.20923158, 1.27602105, 1.34281053, 1.4096 ]])
expected_velocity = np.asarray([
[ 0.5406, 0.55475789, 0.56891579, 0.58307368, 0.59723158],
[ 0.61138947, 0.62554737, 0.63970526, 0.65386316, 0.66802105],
[ 0.68217895, 0.69633684, 0.71049474, 0.72465263, 0.73881053],
[ 0.75296842, 0.76712632, 0.78128421, 0.79544211, 0.8096 ]])
print 'next_w error: ', rel_error(next_w, expected_next_w)
print 'velocity error: ', rel_error(expected_velocity, config['velocity'])
next_w error: 8.88234703351e-09
velocity error: 4.26928774328e-09
Once you have done so, run the following to train a six-layer network with both SGD and SGD+momentum. You should see the SGD+momentum update rule converge faster.
In [259]:
num_train = 4000
small_data = {
'X_train': data['X_train'][:num_train],
'y_train': data['y_train'][:num_train],
'X_val': data['X_val'],
'y_val': data['y_val'],
}
solvers = {}
for update_rule in ['sgd', 'sgd_momentum']:
print 'running with ', update_rule
model = FullyConnectedNet([100, 100, 100, 100, 100], weight_scale=5e-2)
solver = Solver(model, small_data,
num_epochs=5, batch_size=100,
update_rule=update_rule,
optim_config={
'learning_rate': 1e-2,
},
verbose=True)
solvers[update_rule] = solver
solver.train()
print
plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')
plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')
plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')
for update_rule, solver in solvers.iteritems():
plt.subplot(3, 1, 1)
plt.plot(solver.loss_history, 'o', label=update_rule)
plt.subplot(3, 1, 2)
plt.plot(solver.train_acc_history, '-o', label=update_rule)
plt.subplot(3, 1, 3)
plt.plot(solver.val_acc_history, '-o', label=update_rule)
for i in [1, 2, 3]:
plt.subplot(3, 1, i)
plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()
running with sgd
(Iteration 1 / 200) loss: 2.919069
(Epoch 0 / 5) train acc: 0.106000; val_acc: 0.085000
(Iteration 11 / 200) loss: 2.236698
(Iteration 21 / 200) loss: 2.123269
(Iteration 31 / 200) loss: 1.997131
(Epoch 1 / 5) train acc: 0.248000; val_acc: 0.248000
(Iteration 41 / 200) loss: 2.054081
(Iteration 51 / 200) loss: 1.955359
(Iteration 61 / 200) loss: 1.914537
(Iteration 71 / 200) loss: 1.980708
(Epoch 2 / 5) train acc: 0.319000; val_acc: 0.278000
(Iteration 81 / 200) loss: 1.995256
(Iteration 91 / 200) loss: 1.895241
(Iteration 101 / 200) loss: 1.781061
(Iteration 111 / 200) loss: 1.931299
(Epoch 3 / 5) train acc: 0.369000; val_acc: 0.306000
(Iteration 121 / 200) loss: 1.820317
(Iteration 131 / 200) loss: 1.734783
(Iteration 141 / 200) loss: 1.721730
(Iteration 151 / 200) loss: 1.804525
(Epoch 4 / 5) train acc: 0.440000; val_acc: 0.347000
(Iteration 161 / 200) loss: 1.529287
(Iteration 171 / 200) loss: 1.546395
(Iteration 181 / 200) loss: 1.799256
(Iteration 191 / 200) loss: 1.738598
(Epoch 5 / 5) train acc: 0.422000; val_acc: 0.332000
running with sgd_momentum
(Iteration 1 / 200) loss: 2.659168
(Epoch 0 / 5) train acc: 0.123000; val_acc: 0.125000
(Iteration 11 / 200) loss: 2.216596
(Iteration 21 / 200) loss: 1.966192
(Iteration 31 / 200) loss: 1.945459
(Epoch 1 / 5) train acc: 0.315000; val_acc: 0.275000
(Iteration 41 / 200) loss: 2.030965
(Iteration 51 / 200) loss: 1.934096
(Iteration 61 / 200) loss: 1.767673
(Iteration 71 / 200) loss: 1.629235
(Epoch 2 / 5) train acc: 0.405000; val_acc: 0.321000
(Iteration 81 / 200) loss: 1.702986
(Iteration 91 / 200) loss: 1.774102
(Iteration 101 / 200) loss: 1.729498
(Iteration 111 / 200) loss: 1.681033
(Epoch 3 / 5) train acc: 0.441000; val_acc: 0.321000
(Iteration 121 / 200) loss: 1.572395
(Iteration 131 / 200) loss: 1.478298
(Iteration 141 / 200) loss: 1.291229
(Iteration 151 / 200) loss: 1.694928
(Epoch 4 / 5) train acc: 0.476000; val_acc: 0.338000
(Iteration 161 / 200) loss: 1.690808
(Iteration 171 / 200) loss: 1.591263
(Iteration 181 / 200) loss: 1.693025
(Iteration 191 / 200) loss: 1.565157
(Epoch 5 / 5) train acc: 0.510000; val_acc: 0.356000
RMSProp and Adam¶
RMSProp [1] and Adam [2] are update rules that set per-parameter learning rates by using a running average of the second moments of gradients.
In the file cs231n/optim.py, implement the RMSProp update rule in the rmsprop function and implement the Adam update rule in the adam function, and check your implementations using the tests below.
[1] Tijmen Tieleman and Geoffrey Hinton. "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude." COURSERA: Neural Networks for Machine Learning 4 (2012).
[2] Diederik Kingma and Jimmy Ba, "Adam: A Method for Stochastic Optimization", ICLR 2015.
In [260]:
# Test RMSProp implementation; you should see errors less than 1e-7
from cs231n.optim import rmsprop
N, D = 4, 5
w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
cache = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)
config = {'learning_rate': 1e-2, 'cache': cache}
next_w, _ = rmsprop(w, dw, config=config)
expected_next_w = np.asarray([
[-0.39223849, -0.34037513, -0.28849239, -0.23659121, -0.18467247],
[-0.132737, -0.08078555, -0.02881884, 0.02316247, 0.07515774],
[ 0.12716641, 0.17918792, 0.23122175, 0.28326742, 0.33532447],
[ 0.38739248, 0.43947102, 0.49155973, 0.54365823, 0.59576619]])
expected_cache = np.asarray([
[ 0.5976, 0.6126277, 0.6277108, 0.64284931, 0.65804321],
[ 0.67329252, 0.68859723, 0.70395734, 0.71937285, 0.73484377],
[ 0.75037008, 0.7659518, 0.78158892, 0.79728144, 0.81302936],
[ 0.82883269, 0.84469141, 0.86060554, 0.87657507, 0.8926 ]])
print 'next_w error: ', rel_error(expected_next_w, next_w)
print 'cache error: ', rel_error(expected_cache, config['cache'])
next_w error: 9.52468751104e-08
cache error: 2.64779558072e-09
In [228]:
# Test Adam implementation; you should see errors around 1e-7 or less
from cs231n.optim import adam
N, D = 4, 5
w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
m = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)
v = np.linspace(0.7, 0.5, num=N*D).reshape(N, D)
config = {'learning_rate': 1e-2, 'm': m, 'v': v, 't': 5}
next_w, _ = adam(w, dw, config=config)
expected_next_w = np.asarray([
[-0.40094747, -0.34836187, -0.29577703, -0.24319299, -0.19060977],
[-0.1380274, -0.08544591, -0.03286534, 0.01971428, 0.0722929],
[ 0.1248705, 0.17744702, 0.23002243, 0.28259667, 0.33516969],
[ 0.38774145, 0.44031188, 0.49288093, 0.54544852, 0.59801459]])
expected_v = np.asarray([
[ 0.69966, 0.68908382, 0.67851319, 0.66794809, 0.65738853,],
[ 0.64683452, 0.63628604, 0.6257431, 0.61520571, 0.60467385,],
[ 0.59414753, 0.58362676, 0.57311152, 0.56260183, 0.55209767,],
[ 0.54159906, 0.53110598, 0.52061845, 0.51013645, 0.49966, ]])
expected_m = np.asarray([
[ 0.48, 0.49947368, 0.51894737, 0.53842105, 0.55789474],
[ 0.57736842, 0.59684211, 0.61631579, 0.63578947, 0.65526316],
[ 0.67473684, 0.69421053, 0.71368421, 0.73315789, 0.75263158],
[ 0.77210526, 0.79157895, 0.81105263, 0.83052632, 0.85 ]])
print 'next_w error: ', rel_error(expected_next_w, next_w)
print 'v error: ', rel_error(expected_v, config['v'])
print 'm error: ', rel_error(expected_m, config['m'])
next_w error: 1.13956917985e-07
v error: 4.20831403811e-09
m error: 4.21496319311e-09
Once you have debugged your RMSProp and Adam implementations, run the following to train a pair of deep networks using these new update rules:
In [261]:
learning_rates = {'rmsprop': 1e-4, 'adam': 1e-3}
for update_rule in ['adam', 'rmsprop']:
print 'running with ', update_rule
model = FullyConnectedNet([100, 100, 100, 100, 100], weight_scale=5e-2)
solver = Solver(model, small_data,
num_epochs=5, batch_size=100,
update_rule=update_rule,
optim_config={
'learning_rate': learning_rates[update_rule]
},
verbose=True)
solvers[update_rule] = solver
solver.train()
print
plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')
plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')
plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')
for update_rule, solver in solvers.iteritems():
plt.subplot(3, 1, 1)
plt.plot(solver.loss_history, 'o', label=update_rule)
plt.subplot(3, 1, 2)
plt.plot(solver.train_acc_history, '-o', label=update_rule)
plt.subplot(3, 1, 3)
plt.plot(solver.val_acc_history, '-o', label=update_rule)
for i in [1, 2, 3]:
plt.subplot(3, 1, i)
plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()
running with adam
(Iteration 1 / 200) loss: 2.600164
(Epoch 0 / 5) train acc: 0.152000; val_acc: 0.118000
(Iteration 11 / 200) loss: 1.971530
(Iteration 21 / 200) loss: 1.904616
(Iteration 31 / 200) loss: 1.833558
(Epoch 1 / 5) train acc: 0.376000; val_acc: 0.304000
(Iteration 41 / 200) loss: 1.762804
(Iteration 51 / 200) loss: 1.756276
(Iteration 61 / 200) loss: 1.799003
(Iteration 71 / 200) loss: 1.690885
(Epoch 2 / 5) train acc: 0.468000; val_acc: 0.340000
(Iteration 81 / 200) loss: 1.376211
(Iteration 91 / 200) loss: 1.494584
(Iteration 101 / 200) loss: 1.443118
(Iteration 111 / 200) loss: 1.423259
(Epoch 3 / 5) train acc: 0.519000; val_acc: 0.341000
(Iteration 121 / 200) loss: 1.429389
(Iteration 131 / 200) loss: 1.285626
(Iteration 141 / 200) loss: 1.314764
(Iteration 151 / 200) loss: 1.364215
(Epoch 4 / 5) train acc: 0.603000; val_acc: 0.389000
(Iteration 161 / 200) loss: 1.238301
(Iteration 171 / 200) loss: 1.100072
(Iteration 181 / 200) loss: 1.231007
(Iteration 191 / 200) loss: 1.239197
(Epoch 5 / 5) train acc: 0.579000; val_acc: 0.376000
running with rmsprop
(Iteration 1 / 200) loss: 2.412909
(Epoch 0 / 5) train acc: 0.155000; val_acc: 0.153000
(Iteration 11 / 200) loss: 2.058291
(Iteration 21 / 200) loss: 2.000020
(Iteration 31 / 200) loss: 1.934375
(Epoch 1 / 5) train acc: 0.362000; val_acc: 0.297000
(Iteration 41 / 200) loss: 1.815453
(Iteration 51 / 200) loss: 1.724382
(Iteration 61 / 200) loss: 1.752148
(Iteration 71 / 200) loss: 1.595303
(Epoch 2 / 5) train acc: 0.453000; val_acc: 0.343000
(Iteration 81 / 200) loss: 1.824375
(Iteration 91 / 200) loss: 1.524525
(Iteration 101 / 200) loss: 1.698573
(Iteration 111 / 200) loss: 1.586143
(Epoch 3 / 5) train acc: 0.449000; val_acc: 0.330000
(Iteration 121 / 200) loss: 1.553852
(Iteration 131 / 200) loss: 1.539025
(Iteration 141 / 200) loss: 1.550276
(Iteration 151 / 200) loss: 1.452384
(Epoch 4 / 5) train acc: 0.489000; val_acc: 0.351000
(Iteration 161 / 200) loss: 1.421751
(Iteration 171 / 200) loss: 1.456783
(Iteration 181 / 200) loss: 1.451436
(Iteration 191 / 200) loss: 1.491839
(Epoch 5 / 5) train acc: 0.506000; val_acc: 0.362000
Train a good model!¶
Train the best fully-connected model that you can on CIFAR-10, storing your best model in the best_model variable. We require you to get at least 50% accuracy on the validation set using a fully-connected net.
If you are careful it should be possible to get accuracies above 55%, but we don't require it for this part and won't assign extra credit for doing so. Later in the assignment we will ask you to train the best convolutional network that you can on CIFAR-10, and we would prefer that you spend your effort working on convolutional nets rather than fully-connected nets.
You might find it useful to complete the BatchNormalization.ipynb and Dropout.ipynb notebooks before completing this part, since those techniques can help you train powerful models.
In [ ]:
best_model = None
################################################################################
# TODO: Train the best FullyConnectedNet that you can on CIFAR-10. You might #
# batch normalization and dropout useful. Store your best model in the #
# best_model variable. #
################################################################################
pass
################################################################################
# END OF YOUR CODE #
################################################################################
Test you model¶
Run your best model on the validation and test sets. You should achieve above 50% accuracy on the validation set.
In [ ]:
y_test_pred = np.argmax(best_model.loss(X_test), axis=1)
y_val_pred = np.argmax(best_model.loss(X_val), axis=1)
print 'Validation set accuracy: ', (y_val_pred == y_val).mean()
print 'Test set accuracy: ', (y_test_pred == y_test).mean()