1-python_pytorch
COMP5329 – Deep Learning¶
Tutorial 1 – Python and PyTorch¶
Semester 1, 2021
Objectives:
Reviewing Python syntax
Get familiar with scientific computing libraries, such as NumPy.
Get started on PyTorch
Instructions:
Exercises to be completed on Python 3.7
We recommend using virtual environment or conda locally, or Google Colab on the cloud.
How To Install PyTorch¶
With virtual environment.
virtualenv -p python3 torch-cpu
source torch-cpu/bin/activate
pip install torch
With conda.
conda install -n torch-cpu pytorch torchvision -c pytorch
With Google Colab, PyTorch is pre-installed and ready to go.
Reviewing Python Syntax¶
In case you are not familiar with Python, here is a breif introduction for Python
Basic data type¶
number: Integer and float are basic number type in python. We can perform basic calculate with these type of data.
In [ ]:
x = 3
print(x, type(x))
3
In [ ]:
print(x + 1) # Addition
print(x – 1) # Subtraction
print(x * 2) # Multiplication
print(x ** 2) # Exponentiation
4
2
6
9
In [ ]:
x += 1
print(x)
x *= 2
print(x)
4
8
In [ ]:
y = 2.5
print(type(y))
print(y, y + 1, y * 2, y ** 2)
2.5 3.5 5.0 6.25
Boolean： Python implements all of the usual operators for Boolean logic, but uses English words rather than symbols (&&, ||, etc.):
In [ ]:
t, f = True, False
print(type(t))
In [ ]:
print(t and f) # Logical AND;
print(t or f) # Logical OR;
print(not t) # Logical NOT;
print(t != f) # Logical XOR;
False
True
False
True
String
In [ ]:
hello = ‘hello’ # String literals can use single quotes
world = “world” # or double quotes; it does not matter
print(hello, len(hello))
hello 5
In [ ]:
hw = hello + ‘ ‘ + world # String concatenation
print(hw)
hello world
In [ ]:
hw12 = ‘{} {} {}’.format(hello, world, 12) # string formatting
print(hw12)
hello world 12
Note: a String type of number is different with a normal number!
In [ ]:
print(“7” == 7)
print(7+1)
# uncomment the next line, try what happen
#print(“7” + 1)
False
8
Defining Python List¶
In [ ]:
x_list = [1, 2, 3, 4, 5, 6] #define a python list
print(“the result of print(x_list):”)
print(x_list)
print(“the result of print(x_list[0])”)
print(x_list[0]) #slicing
print(“the result of print(x_list[1])”)
print(x_list[1]) #slicing
print(“the result of print(x_list[-1])”)
print(x_list[-1]) #slicing
print(“the result of print(x_list[-2])”)
print(x_list[-2]) #slicing
print(“the result of print(x_list[1:-2])”)
print(x_list[1:-2]) #slicing
print(“the result of print(x_list[0::2])”)
print(x_list[0::2]) #odd element
# add new element to the list
x_list.append(7)
print(“the result of adding 7”)
print(x_list)
# you can add any type of element to the list!
x_list.append(“hello world”)
print(“the result of adding hello world”)
print(x_list)
# different with list, dict is a key-content pair
# in list, we use index to access the content corresponding that index
# in dict, we use a specific key (key can be most of the type of data such as int, float, string.
# However the key should be immutable, such that list cannot be a key of dict)
x_dic = {} #define python dictionary
x_dic[‘a’] = 1
x_dic[‘b’] = 2
print (x_dic)
print (x_dic[‘a’])
# set is a special case of list, that all duplicate elements will be removed.
x_set = set() #define python set
x_set.add(1)
x_set.add(2)
print (x_set)
x_set.add(1)
print (x_set)
the result of print(x_list):
[1, 2, 3, 4, 5, 6]
the result of print(x_list[0])
1
the result of print(x_list[1])
2
the result of print(x_list[-1])
6
the result of print(x_list[-2])
5
the result of print(x_list[1:-2])
[2, 3, 4]
the result of print(x_list[0::2])
[1, 3, 5]
the result of adding 7
[1, 2, 3, 4, 5, 6, 7]
the result of adding hello world
[1, 2, 3, 4, 5, 6, 7, ‘hello world’]
{‘a’: 1, ‘b’: 2}
1
{1, 2}
{1, 2}
Defining Python Matrix¶
For deep learning, matrix calculation is an important trick to speed up the neural network training progresses (which we will learn late 🙂 ).
For now, we need to learn how to use matrix/vector and calculate them in python.
In [ ]:
import random
#define a matrix with dimension M * N
def generate_matrix(m, n):
ls = []
for i in range(m): #generate outer dimension
sub_list = []
for j in range(n): #generate inner dimension
sub_list.append(random.randint(10, 20)) #random int in [10 to 20]
ls.append(sub_list)
return ls
generate_matrix(5, 5)
Out[ ]:
[[13, 19, 17, 16, 10],
[18, 12, 14, 16, 15],
[20, 14, 15, 19, 18],
[11, 11, 17, 18, 14],
[12, 20, 12, 13, 17]]
Matrix Multiplication¶
if we two matrix:
A =
1,2,3
4,5,6
and B =
7, 8
9, 10
10, 12
A is a 2×3 matrix and B is a 3×2 matrix
then their multiplication result is 2×2.
The result is:
58, 64
139， 154
58： (1, 2, 3) • (7, 9, 11) = 1×7 + 2×9 + 3×11 = 58 note: each result is calculated by dot product
64: (1, 2, 3) • (8, 10, 12) = 1×8 + 2×10 + 3×12 = 64
139: (4, 5, 6) • (7, 9, 11) = 4×7 + 5×9 + 6×11 = 139
153: (4, 5, 6) • (8, 10, 12) = 4×8 + 5×10 + 6×12 = 154
notes: given any two matrices m1 and m2, the size of m1 is H1 x W1, and the size of m2 is H2 x W2. For matrix multiplication of m1 and m2, e.g., m1 x m2, we need to make sure W1 == H2, and the size of the result will be H1 x W2. Like m2 x m1, we need to make sure W2 == H1, and the result will be H2 x W1.
As we see, m1 x m2 != m2 x m1.
In [ ]:
# matrix [i,j] * [j,k]
def matrix_multiply(m1, m2):
if len(m1[0]) != len(m2): #dimension illegal check
return None
result = []
for i in range(len(m1)):
sub = []
for j in range(len(m2[0])):
v = 0
for k in range(len(m2)):
v += m1[i][k] * m2[k][j]
sub.append(v)
result.append(sub)
return result
matrix1 = generate_matrix(3, 4)
matrix2 = generate_matrix(4, 5)
print(“the matrix 1 is:”)
print(matrix1)
print()
print(“the matrix 2 is:”)
print(matrix2)
print()
print(“the result of matrix multiplication”)
matrix_multiply(matrix1, matrix2)
the matrix 1 is:
[[18, 14, 20, 20], [11, 18, 19, 17], [20, 14, 13, 10]]
the matrix 2 is:
[[16, 15, 18, 15, 15], [12, 12, 20, 13, 16], [16, 13, 10, 16, 15], [14, 12, 19, 13, 12]]
the result of matrix multiplication
Out[ ]:
[[1056, 938, 1184, 1032, 1034],
[934, 832, 1071, 924, 942],
[836, 757, 960, 820, 839]]
Numpy Operations¶
with Numpy package, we can speed up the matrix multiplication without multiply and sum the elements one by one!
Numpy Array¶
In [ ]:
import numpy as np
#array(m * n)
def generate_numpy_matrix(m, n):
a = np.random.randint(10, size = m * n, dtype = np.int32) + 10
a = a.reshape(m, n) # how to do this by raw python ?
return a
matrix = generate_numpy_matrix(10, 5)
print(“the result of generated matrix”)
print (matrix)
print()
print(“the result of matrix[0] (the first row of the matrix)”)
print (matrix[0]) # first row
print()
print(“the result of matrix[:, 0] (the first column of the matrix)”)
print (matrix[:, 0]) # first column
print()
print(“the result of matrix[0::2]”)
print (matrix[0::2]) # odd rows
the result of generated matrix
[[19 13 15 17 10]
[16 13 18 11 13]
[11 15 16 19 18]
[10 19 19 17 14]
[16 15 18 10 15]
[11 16 11 10 14]
[19 12 13 11 12]
[15 13 14 15 15]
[18 11 15 15 16]
[10 14 10 19 19]]
the result of matrix[0] (the first row of the matrix)
[19 13 15 17 10]
the result of matrix[:, 0] (the first column of the matrix)
[19 16 11 10 16 11 19 15 18 10]
the result of matrix[0::2]
[[19 13 15 17 10]
[11 15 16 19 18]
[16 15 18 10 15]
[19 12 13 11 12]
[18 11 15 15 16]]
Numpy Matrix Multiplication¶
we can convert a matrix to an numpy matrix such that we can perform matrix calculation as simple as doing number calculation.
In [ ]:
matrix1 = generate_numpy_matrix(3, 4)
matrix2 = generate_numpy_matrix(4, 5)
print(“the result of matrix1”)
print (matrix1)
print()
print(“the result of matrix2”)
print (matrix2)
print()
print(“the result of np.matmul(m1,m2), which is doing matrix multiplication for two given matrix”)
result = np.matmul(matrix1, matrix2)
print (result)
print()
print(“the result of matrix1 @ matrix2, similar to np.matmul(matrix1, matrix2)”)
result = matrix1 @ matrix2 # for python3.5 or higher version
print (result)
print()
print(“the result of matrix1 * 3 + 2”)
result = matrix1 * 3 + 2 # how can we do the same operation by using raw python ?
print (result)
matrix3 = generate_numpy_matrix(3, 4)
result = matrix1 * matrix3 # dot multiplication
print()
print(“the result of matrix3”)
print (matrix3)
print(“the result of matrix1*matrix3”)
print (result)
the result of matrix1
[[19 18 17 11]
[14 15 16 16]
[16 12 16 13]]
the result of matrix2
[[16 18 18 15 13]
[16 18 14 18 14]
[10 17 16 17 18]
[18 13 19 12 10]]
the result of np.matmul(m1,m2), which is doing matrix multiplication for two given matrix
[[ 960 1098 1075 1030 915]
[ 912 1002 1022 944 840]
[ 842 945 959 884 794]]
the result of matrix1 @ matrix2, similar to np.matmul(matrix1, matrix2)
[[ 960 1098 1075 1030 915]
[ 912 1002 1022 944 840]
[ 842 945 959 884 794]]
the result of matrix1 * 3 + 2
[[59 56 53 35]
[44 47 50 50]
[50 38 50 41]]
the result of matrix3
[[18 13 12 18]
[12 13 13 14]
[12 17 12 17]]
the result of matrix1*matrix3
[[342 234 204 198]
[168 195 208 224]
[192 204 192 221]]
Other Numpy Operation¶
In [ ]:
matrix1 = generate_numpy_matrix(3, 4)
matrix2 = generate_numpy_matrix(3, 4)
print()
print(“the result of matrix1”)
print(matrix1)
print()
print(“the result of matrix2”)
print(matrix2)
print()
print(“the result of np.transpose(matrix1)”)
print (np.transpose(matrix1)) # matrix transpose
print()
print(“the result of matrix1.T”)
print (matrix1.T) #for python3.5 or higher version
print()
print(“the result of np.sum(matrix1)”)
print (np.sum(matrix1)) # sum
print()
print(“the result of np.sum(matrix1, axis = 1). What is the result if we set axis to 0?”)
print (np.sum(matrix1, axis = 1)) # column sum
print()
print(“the result of np.vstack([matrix1, matrix2])”)
print (np.vstack([matrix1, matrix2])) # combine two matrix in row
print()
print(“the result of np.hstack([matrix1, matrix2])”)
print (np.hstack([matrix1, matrix2])) # combine tow matrix in column
the result of matrix1
[[13 16 10 18]
[11 17 19 18]
[17 17 12 12]]
the result of matrix2
[[15 17 10 16]
[11 12 15 12]
[16 13 12 13]]
the result of np.transpose(matrix1)
[[13 11 17]
[16 17 17]
[10 19 12]
[18 18 12]]
the result of matrix1.T
[[13 11 17]
[16 17 17]
[10 19 12]
[18 18 12]]
the result of np.sum(matrix1)
180
the result of np.sum(matrix1, axis = 1). What is the result if we set axis to 0?
[57 65 58]
the result of np.vstack([matrix1, matrix2])
[[13 16 10 18]
[11 17 19 18]
[17 17 12 12]
[15 17 10 16]
[11 12 15 12]
[16 13 12 13]]
the result of np.hstack([matrix1, matrix2])
[[13 16 10 18 15 17 10 16]
[11 17 19 18 11 12 15 12]
[17 17 12 12 16 13 12 13]]
PyTorch: Getting started¶
In [ ]:
import torch
Tensors¶
Construct a 5×3 zero matrix:
In [ ]:
x = torch.zeros(5, 3)
print(x)
tensor([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]])
Construct a randomly initialized matrix:
In [ ]:
x = torch.rand(5, 3)
print(x)
tensor([[0.6598, 0.6122, 0.0784],
[0.3562, 0.7157, 0.6081],
[0.9704, 0.5284, 0.8483],
[0.9474, 0.3911, 0.8751],
[0.3879, 0.9093, 0.6586]])
Construct a tensor directly from python list:
In [ ]:
x = torch.tensor([5.5, 3])
print(x)
tensor([5.5000, 3.0000])
Construct a tensor directly from numpy matrix:
In [ ]:
np_matrix = generate_numpy_matrix(5, 3)
x = torch.tensor(np_matrix)
print(x)
print(x.device)
x_gpu = x.to(‘cuda’)
print(x_gpu.device)
tensor([[10, 15, 13],
[14, 15, 16],
[15, 15, 14],
[11, 16, 18],
[19, 11, 10]], dtype=torch.int32)
cpu
cuda:0
Operations¶
There are multiple syntaxes for operations. Thanks to the pythonic design of PyTorch, we can perfrom most numerical operations betwen tensors using python syntax.
In [ ]:
x = torch.rand(3, 3)
y = torch.rand(3, 3)
z = x+y
print(z)
# element-wise multiplication
z = x*y
print(z)
# matrix product
z = torch.mm(x, y)
print(z)
tensor([[1.1593, 1.5271, 1.4615],
[1.3420, 1.0367, 1.7000],
[1.5314, 0.8884, 1.4251]])
tensor([[0.3346, 0.5816, 0.5222],
[0.4351, 0.2679, 0.7217],
[0.5556, 0.1896, 0.4566]])
tensor([[1.5566, 1.2169, 1.7595],
[1.5137, 1.1334, 1.6872],
[1.0610, 0.8822, 1.2446]])
PyTorch tensors support numpy-like slicing.
In [ ]:
print(x[:, 1]) # print everything on 1st index of rank 1.
tensor([0.8013, 0.4902, 0.3562])
Reshaping a tensor is easy.
In [ ]:
x = torch.randn(4, 4)
y = x.view(16)
z = x.view(-1, 8) # the size -1 is inferred from other dimensions
print(x.size(), y.size(), z.size())
torch.Size([4, 4]) torch.Size([16]) torch.Size([2, 8])
Exercises¶
implementing y = $abs((w_1x + b_1)\bullet(w_2x + b_2))$
where x is a random input matrix, with shape [32, 256].
w, b are the random variables generated by PyTorch.
y is the output, with shape [32, 10]
In [ ]: