Introduction to
Artificial Intelligence
with Python
Neural Networks
Neural Networks
• •
Neurons are connected to and receive electrical signals from other neurons.
Neurons process input signals and can be activated.
artificial neural network
mathematical model for learning inspired by biological neural networks
Artificial Neural Networks
•
•
Model mathematical function from inputs to outputs based on the structure and parameters of the network.
Allows for learning the network’s parameters based on data.
(x1, x2)
h(x1, x2) = w0 + w1x1 + w2x2
h(x1, x2) = w0 + w1x1 + w2x2
h(x1, x2) = w0 + w1x1 + w2x2
step function
1
0
g(x) = 1 if x ≥ 0, else 0
w·x
output
step function
1
0
g(x) = 1 if x ≥ 0, else 0
w·x
output
logistic sigmoid
1
g(x) =
ex ex + 1
0
w·x
output
0
rectified linear unit (ReLU)
g(x) = max(0, x)
w·x
output
h(x1, x2) = g(w0 + w1x1 + w2x2)
h(x1, x2) = g(w0 + w1x1 + w2x2)
x1 x2
x1 x2
w1 w2
x1 x2
w1 w2
g(w0 + w1x1 + w2x2)
x1 x2
w0 w1
g(w0 + w1x1 + w2x2)
w2
x1 x2
w1 w2
g(w0 + w1x1 + w2x2)
Or
x
y
f(x, y)
0
0
0
0
1
1
1
0
1
1
1
1
x1 x2
w0 w1
g(w0 + w1x1 + w2x2)
w2
x1 x2
-1
1 1
g(-1 + 1×1 + 1×2)
-1 0 1
x1 x2
-1
1
0
0
g(-1 + 1×1 + 1×2)
0
1
-1 0 1
x1 x2
-1
1
1
1
g(-1 + 1×1 + 1×2)
0
1
-1 0 1
x1 x2
-1
1
1
1
g(-1 + 1×1 + 1×2)
1
1
-1 0 1
And
x
y
f(x, y)
0
0
0
0
1
0
1
0
0
1
1
1
x1 x2
-1
1 1
g(-1 + 1×1 + 1×2)
-1 0 1
x1 x2
-2
1 1
g(-2 + 1×1 + 1×2)
-1 0 1
x1 x2
-2
1
1
1
g(-2 + 1×1 + 1×2)
1
1
-1 0 1
x1 x2
-2
1
1
0
g(-2 + 1×1 + 1×2)
0
1
-1 0 1
humidity
probability of rain
pressure
advertising
sales
month
…
x1 x2
w1 w2
g(w0 + w1x1 + w2x2)
x1 x2 x3
w1 w2
w3
g(w0 + w1x1 + w2x2 + w3x3)
x1
x2 w2
x3 x4 x5
w1 w3
g ( ∑5 x i w i + w 0 ) i=1
w4 w5
x1 x2
w1 w2
g ( ∑n x i w i + w 0 ) i=1
…
wn-1 wn
xn-1 xn
gradient descent
algorithm for minimizing loss when training neural network
Gradient Descent
•
Start with a random choice of weights. Repeat:
•
Calculate the gradient based on all data points:
•
direction that will lead to decreasing loss.
•
Update weights according to the gradient.
Gradient Descent
• •
• •
Start with a random choice of weights. Repeat:
Calculate the gradient based on all data points: direction that will lead to decreasing loss.
Update weights according to the gradient.
Stochastic Gradient Descent
• •
• •
Start with a random choice of weights. Repeat:
Calculate the gradient based on one data point: direction that will lead to decreasing loss.
Update weights according to the gradient.
Mini-Batch Gradient Descent
• •
• •
Start with a random choice of weights. Repeat:
Calculate the gradient based on one small batch: direction that will lead to decreasing loss.
Update weights according to the gradient.
rainy sunny cloudy snowy
rainy sunny cloudy snowy
rainy sunny cloudy snowy
rainy sunny cloudy snowy
rainy 0.6 sunny
cloudy snowy
0.1
0.2 0.1
rainy 0.6 sunny
cloudy snowy
0.1
0.2 0.1
action 1 action 2 action 3 action 4
Perceptron
•
Only capable of learning linearly separable decision boundary.
multilayer neural network
artificial neural network with an input layer, an output layer, and at least one hidden layer
backpropagation
algorithm for training neural networks with hidden layers
Backpropagation
•
Start with a random choice of weights. Repeat:
•
Calculate error for output layer.
•
•
• •
For each layer, starting with output layer, and moving inwards towards earliest hidden layer:
Propagate error back one layer. Update weights.
deep neural networks
neural network with multiple hidden layers
Overfitting
dropout
temporarily removing units — selected at random — from a neural network to prevent over-reliance on certain units
TensorFlow
playground.tensorflow.org
computer vision
computational methods for analyzing and understanding digital images
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
0 255
…
…
image convolution
applying a filter that adds each pixel value of an image to its neighbors, weighted according to a kernel matrix
0
-1
0
-1
5
-1
0
-1
0
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
10
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
10
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
10
20
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
10
20
40
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
10
20
40
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
10
20
40
50
0
-1
0
-1
5
-1
0
-1
0
10
20
30
40
10
20
30
40
20
30
40
50
20
30
40
50
10
20
40
50
-1
-1
-1
-1
8
-1
-1
-1
-1
-1
-1
-1
-1
8
-1
-1
-1
-1
20
20
20
20
20
20
20
20
20
(20)(-1) + (20)(-1) + (20)(-1)
+ (20)(-1) + (20)(8) + (20)(-1)
+ (20)(-1) + (20)(-1) + (20)(-1)
0
-1
-1
-1
-1
8
-1
-1
-1
-1
20
20
20
50
50
50
50
50
50
(20)(-1) + (20)(-1) + (20)(-1)
+ (50)(-1) + (50)(8) + (50)(-1)
+ (50)(-1) + (50)(-1) + (50)(-1)
90
pooling
reducing the size of an input by sampling from regions in the input
max-pooling
pooling by choosing the maximum value in each region
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
50
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
50
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
50
110
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
50
110
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
50
110
20
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
50
110
20
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
50
110
20
40
30
40
80
90
20
50
100
110
0
10
20
30
10
20
40
30
50
110
20
40
convolutional neural network
neural networks that use convolution, usually for analyzing images
convolution pooling flattening
first convolution and pooling
second convolution and pooling
low-level features: edges, curves, shapes
high-level features: objects
input network output
input network output
feed-forward neural network
neural network that has connections only in one direction
recurrent neural network
neural network that generates output that feeds back into its own inputs
input network output
group of people walking in front of a building
input network
network
network
network
output output output output
input input input input
network
network
network
network output
She is in the library.
她在圖書館
input input input input
network
network
network
network
network
network
output output output
Neural Networks
Introduction to
Artificial Intelligence
with Python