Data Mining and Machine Learning
Learning MLP Weights using Error Back-Propagation
Peter Jančovič Slide 1
Data Mining and Machine Learning
Objectives
Outline of the MLP training
– The error function
– Optimisation by gradient decent
The Error Back-Propagation (EBP) – Calculating the derivatives
– Bringing everything together
– Summary of the EBP algorithm
– Practical considerations
Slide 2
Data Mining and Machine Learning
Feed-forward Neural Networks Multi-Layer Perceptron – Feed-Forward Neural Network
Input Layer (Input Units)
Artificial neuron
Hidden Layers (Hidden Units)
Slide 3
Output Layer (Output Units) Data Mining and Machine Learning
MLP training
To define an MLP must decide: – Number of layers
– Number of input units
– Number of hidden units
– Number of output units
Choosing the right numbers of layers and units is a
combination of experience and experimentation
Once these are defined, properties of the MLP are completely defined by the values of the weights
How do we choose the weight values?
Slide 4
Data Mining and Machine Learning
MLP training (continued)
MLP training needs a set of input vectors i with
corresponding target output vectors t(i)
Each input vector i is propagated through the
network to produce an output o(i)
The error E is the difference between the actual
output o(i) and the target output t(i) E oiti2 i
Objective of training is to learn the weights which minimise the average error over the training set
Slide 5
Data Mining and Machine Learning
Error Back-Propagation
i = i1,…,i5
Slide 6
Data Mining and Machine Learning
t(i) = t(i)1,…, t(i)3
MLP training (continued)
MLP training uses gradient descent
For each weight w
E w
calculate
Subtract a proportion of
E w
Error
E w
from w Slide 7
w
w E w
Data Mining and Machine Learning
MLP training (continued)
MLP weights learnt automatically from training data
Training uses an iterative computational technique called Error-Back-Propagation (EBP)
There are many variants of EBP
Slide 8
Data Mining and Machine Learning
Error-back propagation (EBP)
1. Choose initial values for the weights
2. Propagate each training sample i through the network to obtain o(i). Set E = |t(i) – o(i)|
3. EBP calculates E for each weight w by w
propagating the error back up through the network
4. When all training patterns have been seen, w is changed by an amount proportional to the average
E w
value of
5. Repeat until the change in error falls below a threshold
Slide 9
Data Mining and Machine Learning
Slide 10
Data Mining and Machine Learning
Slide 11
Data Mining and Machine Learning
Slide 12
Data Mining and Machine Learning
Slide 13
Data Mining and Machine Learning
Slide 14
Data Mining and Machine Learning
Slide 15
Data Mining and Machine Learning
Slide 16
Data Mining and Machine Learning
Slide 17
Data Mining and Machine Learning
Slide 18
Data Mining and Machine Learning
Slide 19
Data Mining and Machine Learning
Slide 20
Data Mining and Machine Learning
Summary MLP training
Error Back Propagation (EBP)
Slide 21
Data Mining and Machine Learning