W04L1-2-Newswires
Classifying newswires: a multi-class classification example¶
This notebook is based on the code samples found in Chapter 3, Section 5 of Deep Learning with Python and hosted on https://github.com/fchollet/deep-learning-with-python-notebooks.
Note that the original text features far more content, in particular further explanations and figures.
In [1]:
import tensorflow as tf
tf.config.experimental.list_physical_devices()
Out[1]:
[PhysicalDevice(name=’/physical_device:CPU:0′, device_type=’CPU’),
PhysicalDevice(name=’/physical_device:XLA_CPU:0′, device_type=’XLA_CPU’),
PhysicalDevice(name=’/physical_device:GPU:0′, device_type=’GPU’),
PhysicalDevice(name=’/physical_device:XLA_GPU:0′, device_type=’XLA_GPU’)]
In [2]:
physical_devices = tf.config.list_physical_devices(‘GPU’)
tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)
In [3]:
from tensorflow import keras
keras.__version__
Out[3]:
‘2.3.0-tf’
In the previous section we saw how to classify vector inputs into two mutually exclusive classes using a densely-connected neural network.
But what happens when you have more than two classes?
In this section, we will build a network to classify Reuters newswires into 46 different mutually-exclusive topics. Since we have many
classes, this problem is an instance of multi-class classification, and since each data point should be classified into only one
category, the problem is more specifically an instance of single-label, multi-class classification. If each data point could have
belonged to multiple categories (in our case, topics) then we would be facing a “multi-label, multi-class classification” problem.
The Reuters dataset¶
We will be working with the Reuters dataset, a set of short newswires and their topics, published by Reuters in 1986. It’s a very simple,
widely used toy dataset for text classification. There are 46 different topics; some topics are more represented than others, but each
topic has at least 10 examples in the training set.
Like IMDB and MNIST, the Reuters dataset comes packaged as part of Keras. Let’s take a look right away:
In [4]:
from tensorflow.keras.datasets import reuters
(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words=10000)
Like with the IMDB dataset, the argument num_words=10000 restricts the data to the 10,000 most frequently occurring words found in the
data.
We have 8,982 training examples and 2,246 test examples:
In [5]:
len(train_data)
Out[5]:
8982
In [6]:
len(test_data)
Out[6]:
2246
As with the IMDB reviews, each example is a list of integers (word indices):
In [7]:
train_data[10][:10]
Out[7]:
[1, 245, 273, 207, 156, 53, 74, 160, 26, 14]
Here’s how you can decode it back to words, in case you are curious:
In [8]:
word_index = reuters.get_word_index()
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
# Note that our indices were offset by 3
# because 0, 1 and 2 are reserved indices for “padding”, “start of sequence”, and “unknown”.
decoded_newswire = ‘ ‘.join([reverse_word_index.get(i – 3, ‘?’) for i in train_data[0]])
In [9]:
decoded_newswire
Out[9]:
‘? ? ? said as a result of its december acquisition of space co it expects earnings per share in 1987 of 1 15 to 1 30 dlrs per share up from 70 cts in 1986 the company said pretax net should rise to nine to 10 mln dlrs from six mln dlrs in 1986 and rental operation revenues to 19 to 22 mln dlrs from 12 5 mln dlrs it said cash flow per share this year should be 2 50 to three dlrs reuter 3′
The label associated with an example is an integer between 0 and 45: a topic index.
In [10]:
train_labels[10]
Out[10]:
3
Preparing the data¶
We can vectorize the data with the exact same code as in our previous example:
In [11]:
import numpy as np
def vectorize_sequences(sequences, dimension=10000):
results = np.zeros((len(sequences), dimension))
for i, sequence in enumerate(sequences):
results[i, sequence] = 1.
return results
# Our vectorized training data
x_train = vectorize_sequences(train_data)
# Our vectorized test data
x_test = vectorize_sequences(test_data)
To vectorize the labels we could just cast the label list as an integer tensor; we will look at this later on.
For now, we will use one-hot encoding.
One-hot encoding is a widely used format for categorical data, also called “categorical encoding”.
For a more detailed explanation of one-hot encoding, you can refer to Chapter 6, Section 1.
In our case, one-hot encoding of our labels consists in embedding each label as an all-zero vector with a 1 in the place of the label index, e.g.:
In [12]:
def to_one_hot(labels, dimension=46):
results = np.zeros((len(labels), dimension))
for i, label in enumerate(labels):
results[i, label] = 1.
return results
# Our vectorized training labels
one_hot_train_labels = to_one_hot(train_labels)
# Our vectorized test labels
one_hot_test_labels = to_one_hot(test_labels)
Note that there is a built-in way to do this in Keras:
In [13]:
from tensorflow.keras.utils import to_categorical
one_hot_train_labels = to_categorical(train_labels)
one_hot_test_labels = to_categorical(test_labels)
Building our network¶
This topic classification problem looks very similar to our previous movie review classification problem: in both cases, we are trying to
classify short snippets of text. There is however a new constraint here: the number of output classes has gone from 2 to 46, i.e. the
dimensionality of the output space is much larger.
In a stack of Dense layers like what we were using, each layer can only access information present in the output of the previous layer.
If one layer drops some information relevant to the classification problem, this information can never be recovered by later layers: each
layer can potentially become an information bottleneck. In our previous example, we were using 16-dimensional intermediate layers, but a
16-dimensional space may be too limited to learn to separate 46 different classes: such small layers may act as information bottlenecks,
permanently dropping relevant information.
For this reason we will use larger layers. Let’s go with 64 units:
In [14]:
from tensorflow.keras import models
from tensorflow.keras import layers
model = models.Sequential()
model.add(layers.Dense(64, activation=’relu’, input_shape=(10000,)))
model.add(layers.Dense(64, activation=’relu’))
model.add(layers.Dense(46, activation=’softmax’))
There are two other things you should note about this architecture:
We are ending the network with a Dense layer of size 46. This means that for each input sample, our network will output a
46-dimensional vector. Each entry in this vector (each dimension) will encode a different output class.
The last layer uses a softmax activation. It means that the network will
output a probability distribution over the 46 different output classes, i.e. for every input sample, the network will produce a
46-dimensional output vector where output[i] is the probability that the sample belongs to class i. The 46 scores will sum to 1.
The best loss function to use in this case is categorical_crossentropy. It measures the distance between two probability distributions:
in our case, between the probability distribution output by our network, and the true distribution of the labels. By minimizing the
distance between these two distributions, we train our network to output something as close as possible to the true labels.
In [15]:
model.compile(optimizer=’rmsprop’,
loss=’categorical_crossentropy’,
metrics=[‘accuracy’])
In [16]:
model.summary()
Model: “sequential”
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 64) 640064
_________________________________________________________________
dense_1 (Dense) (None, 64) 4160
_________________________________________________________________
dense_2 (Dense) (None, 46) 2990
=================================================================
Total params: 647,214
Trainable params: 647,214
Non-trainable params: 0
_________________________________________________________________
Validating our approach¶
Let’s set apart 1,000 samples in our training data to use as a validation set:
In [17]:
x_val = x_train[:1000]
partial_x_train = x_train[1000:]
y_val = one_hot_train_labels[:1000]
partial_y_train = one_hot_train_labels[1000:]
Now let’s train our network for 20 epochs:
In [18]:
history = model.fit(partial_x_train,
partial_y_train,
epochs=20,
batch_size=512,
validation_data=(x_val, y_val))
Epoch 1/20
16/16 [==============================] – 0s 28ms/step – loss: 2.5880 – accuracy: 0.5507 – val_loss: 1.6708 – val_accuracy: 0.6540
Epoch 2/20
16/16 [==============================] – 0s 15ms/step – loss: 1.3806 – accuracy: 0.7111 – val_loss: 1.2709 – val_accuracy: 0.7330
Epoch 3/20
16/16 [==============================] – 0s 15ms/step – loss: 1.0300 – accuracy: 0.7830 – val_loss: 1.1178 – val_accuracy: 0.7640
Epoch 4/20
16/16 [==============================] – 0s 14ms/step – loss: 0.8121 – accuracy: 0.8339 – val_loss: 1.0228 – val_accuracy: 0.7760
Epoch 5/20
16/16 [==============================] – 0s 14ms/step – loss: 0.6526 – accuracy: 0.8636 – val_loss: 0.9531 – val_accuracy: 0.8010
Epoch 6/20
16/16 [==============================] – 0s 13ms/step – loss: 0.5210 – accuracy: 0.8944 – val_loss: 0.9321 – val_accuracy: 0.8090
Epoch 7/20
16/16 [==============================] – 0s 13ms/step – loss: 0.4236 – accuracy: 0.9131 – val_loss: 0.8881 – val_accuracy: 0.8130
Epoch 8/20
16/16 [==============================] – 0s 13ms/step – loss: 0.3443 – accuracy: 0.9271 – val_loss: 0.8745 – val_accuracy: 0.8170
Epoch 9/20
16/16 [==============================] – 0s 16ms/step – loss: 0.2855 – accuracy: 0.9372 – val_loss: 0.8889 – val_accuracy: 0.8100
Epoch 10/20
16/16 [==============================] – 0s 13ms/step – loss: 0.2363 – accuracy: 0.9461 – val_loss: 0.9099 – val_accuracy: 0.8110
Epoch 11/20
16/16 [==============================] – 0s 14ms/step – loss: 0.2071 – accuracy: 0.9499 – val_loss: 0.9127 – val_accuracy: 0.8120
Epoch 12/20
16/16 [==============================] – 0s 13ms/step – loss: 0.1798 – accuracy: 0.9520 – val_loss: 0.9319 – val_accuracy: 0.8100
Epoch 13/20
16/16 [==============================] – 0s 13ms/step – loss: 0.1626 – accuracy: 0.9539 – val_loss: 1.0093 – val_accuracy: 0.8090
Epoch 14/20
16/16 [==============================] – 0s 13ms/step – loss: 0.1530 – accuracy: 0.9543 – val_loss: 0.9762 – val_accuracy: 0.8110
Epoch 15/20
16/16 [==============================] – 0s 13ms/step – loss: 0.1377 – accuracy: 0.9564 – val_loss: 1.0023 – val_accuracy: 0.8000
Epoch 16/20
16/16 [==============================] – 0s 13ms/step – loss: 0.1305 – accuracy: 0.9563 – val_loss: 0.9783 – val_accuracy: 0.8190
Epoch 17/20
16/16 [==============================] – 0s 13ms/step – loss: 0.1251 – accuracy: 0.9563 – val_loss: 1.0343 – val_accuracy: 0.7970
Epoch 18/20
16/16 [==============================] – 0s 13ms/step – loss: 0.1186 – accuracy: 0.9577 – val_loss: 1.0495 – val_accuracy: 0.8090
Epoch 19/20
16/16 [==============================] – 0s 16ms/step – loss: 0.1147 – accuracy: 0.9580 – val_loss: 1.0467 – val_accuracy: 0.8120
Epoch 20/20
16/16 [==============================] – 0s 12ms/step – loss: 0.1143 – accuracy: 0.9562 – val_loss: 1.0374 – val_accuracy: 0.8080
Let’s display its loss and accuracy curves:
In [19]:
%matplotlib inline
import matplotlib.pyplot as plt
loss = history.history[‘loss’]
val_loss = history.history[‘val_loss’]
epochs = range(1, len(loss) + 1)
plt.plot(epochs, loss, ‘bo’, label=’Training loss’)
plt.plot(epochs, val_loss, ‘b’, label=’Validation loss’)
plt.title(‘Training and validation loss’)
plt.xlabel(‘Epochs’)
plt.ylabel(‘Loss’)
plt.legend()
plt.show()
In [20]:
plt.clf() # clear figure
acc = history.history[‘accuracy’]
val_acc = history.history[‘val_accuracy’]
plt.plot(epochs, acc, ‘bo’, label=’Training acc’)
plt.plot(epochs, val_acc, ‘b’, label=’Validation acc’)
plt.title(‘Training and validation accuracy’)
plt.xlabel(‘Epochs’)
plt.ylabel(‘Loss’)
plt.legend()
plt.show()
It seems that the network starts overfitting after 8 epochs. We could train a new network from scratch for 8 epochs. Alternatively we can use Keras to train until the loss of the validation set increases. We can do this by adding a call back function that executes at the end of every epoch and forces to stop training:
In [21]:
model = models.Sequential()
model.add(layers.Dense(64, activation=’relu’, input_shape=(10000,)))
model.add(layers.Dense(64, activation=’relu’))
model.add(layers.Dense(46, activation=’softmax’))
In [22]:
model.compile(optimizer=’rmsprop’,
loss=’categorical_crossentropy’,
metrics=[‘accuracy’])
In [23]:
model.fit(partial_x_train,
partial_y_train,
epochs=20,
batch_size=512,
validation_data=(x_val, y_val),
callbacks=[keras.callbacks.EarlyStopping(monitor=’val_loss’)])
Epoch 1/20
16/16 [==============================] – 0s 17ms/step – loss: 2.6914 – accuracy: 0.5220 – val_loss: 1.7663 – val_accuracy: 0.6400
Epoch 2/20
16/16 [==============================] – 0s 13ms/step – loss: 1.4724 – accuracy: 0.6961 – val_loss: 1.3400 – val_accuracy: 0.7040
Epoch 3/20
16/16 [==============================] – 0s 13ms/step – loss: 1.0906 – accuracy: 0.7653 – val_loss: 1.1386 – val_accuracy: 0.7570
Epoch 4/20
16/16 [==============================] – 0s 14ms/step – loss: 0.8614 – accuracy: 0.8150 – val_loss: 1.0416 – val_accuracy: 0.7680
Epoch 5/20
16/16 [==============================] – 0s 13ms/step – loss: 0.6875 – accuracy: 0.8544 – val_loss: 1.0006 – val_accuracy: 0.7760
Epoch 6/20
16/16 [==============================] – 0s 14ms/step – loss: 0.5561 – accuracy: 0.8861 – val_loss: 0.9125 – val_accuracy: 0.8150
Epoch 7/20
16/16 [==============================] – 0s 14ms/step – loss: 0.4469 – accuracy: 0.9067 – val_loss: 0.9119 – val_accuracy: 0.8080
Epoch 8/20
16/16 [==============================] – 0s 13ms/step – loss: 0.3641 – accuracy: 0.9233 – val_loss: 0.8892 – val_accuracy: 0.8160
Epoch 9/20
16/16 [==============================] – 0s 12ms/step – loss: 0.3028 – accuracy: 0.9337 – val_loss: 0.8822 – val_accuracy: 0.8190
Epoch 10/20
16/16 [==============================] – 0s 13ms/step – loss: 0.2562 – accuracy: 0.9414 – val_loss: 0.8892 – val_accuracy: 0.8160
Out[23]:
In [24]:
results = model.evaluate(x_test, one_hot_test_labels)
results
71/71 [==============================] – 0s 2ms/step – loss: 0.9850 – accuracy: 0.7832
Out[24]:
[0.9850168824195862, 0.7831701040267944]
Our approach reaches an accuracy of ~78%. With a balanced binary classification problem, the accuracy reached by a purely random classifier
would be 50%, but in our case it would be closer to 19%. But is our data set balanced?
A simple count of the labels in the training set reveals that the data are very imbalanced, and category 3 is by far the most frequent:
In [25]:
from collections import Counter
c = Counter(train_labels)
c
Out[25]:
Counter({3: 3159,
4: 1949,
16: 444,
19: 549,
8: 139,
21: 100,
11: 390,
1: 432,
13: 172,
20: 269,
18: 66,
25: 92,
35: 10,
9: 101,
38: 19,
10: 124,
28: 48,
2: 74,
6: 48,
12: 49,
7: 16,
30: 45,
34: 50,
15: 20,
14: 26,
32: 32,
41: 30,
40: 36,
45: 18,
23: 41,
42: 13,
26: 24,
24: 62,
37: 19,
27: 15,
31: 39,
39: 24,
0: 55,
22: 15,
33: 11,
36: 49,
17: 39,
43: 21,
29: 19,
44: 12,
5: 17})
A majority baseline classifier would classify all newswires with the most popular category, in our case category 3. The accuracy of this majority baseline on the test data is still much lower than with our system:
In [26]:
float(np.sum(np.array(test_labels) == 3)) / len(test_labels)
Out[26]:
0.3619768477292965
Generating predictions on new data¶
We can verify that the predict method of our model instance returns a probability distribution over all 46 topics. Let’s generate topic
predictions for all of the test data:
In [27]:
predictions = model.predict(x_test)
Each entry in predictions is a vector of length 46:
In [28]:
predictions[0].shape
Out[28]:
(46,)
The coefficients in this vector sum to 1:
In [29]:
np.sum(predictions[0])
Out[29]:
1.0
The largest entry is the predicted class, i.e. the class with the highest probability:
In [30]:
np.argmax(predictions[0])
Out[30]:
3
A different way to handle the labels and the loss¶
We mentioned earlier that another way to encode the labels would be to cast them as an integer tensor, like such:
In [31]:
y_train = np.array(train_labels)
y_test = np.array(test_labels)
The only thing it would change is the choice of the loss function. Our previous loss, categorical_crossentropy, expects the labels to
follow a categorical encoding. With integer labels, we should use sparse_categorical_crossentropy:
In [32]:
model.compile(optimizer=’rmsprop’, loss=’sparse_categorical_crossentropy’, metrics=[‘acc’])
This new loss function is still mathematically the same as categorical_crossentropy; it just has a different interface.
On the importance of having sufficiently large intermediate layers¶
We mentioned earlier that since our final outputs were 46-dimensional, we should avoid intermediate layers with much less than 46 hidden
units. Now let’s try to see what happens when we introduce an information bottleneck by having intermediate layers significantly less than
46-dimensional, e.g. 4-dimensional.
In [33]:
model = models.Sequential()
model.add(layers.Dense(64, activation=’relu’, input_shape=(10000,)))
model.add(layers.Dense(4, activation=’relu’))
model.add(layers.Dense(46, activation=’softmax’))
model.compile(optimizer=’rmsprop’,
loss=’categorical_crossentropy’,
metrics=[‘accuracy’])
model.fit(partial_x_train,
partial_y_train,
epochs=20,
batch_size=128,
validation_data=(x_val, y_val),
callbacks=[keras.callbacks.EarlyStopping(monitor=’val_accuracy’, patience=3)])
Epoch 1/20
63/63 [==============================] – 0s 6ms/step – loss: 2.4506 – accuracy: 0.5068 – val_loss: 1.8073 – val_accuracy: 0.5460
Epoch 2/20
63/63 [==============================] – 0s 5ms/step – loss: 1.6431 – accuracy: 0.5646 – val_loss: 1.5939 – val_accuracy: 0.5810
Epoch 3/20
63/63 [==============================] – 0s 5ms/step – loss: 1.4448 – accuracy: 0.5921 – val_loss: 1.5170 – val_accuracy: 0.5890
Epoch 4/20
63/63 [==============================] – 0s 5ms/step – loss: 1.3232 – accuracy: 0.6159 – val_loss: 1.4598 – val_accuracy: 0.6120
Epoch 5/20
63/63 [==============================] – 0s 5ms/step – loss: 1.2185 – accuracy: 0.6398 – val_loss: 1.4275 – val_accuracy: 0.6510
Epoch 6/20
63/63 [==============================] – 0s 5ms/step – loss: 1.1288 – accuracy: 0.6862 – val_loss: 1.4288 – val_accuracy: 0.6620
Epoch 7/20
63/63 [==============================] – 0s 5ms/step – loss: 1.0481 – accuracy: 0.7149 – val_loss: 1.4047 – val_accuracy: 0.6840
Epoch 8/20
63/63 [==============================] – 0s 5ms/step – loss: 0.9872 – accuracy: 0.7269 – val_loss: 1.4309 – val_accuracy: 0.6850
Epoch 9/20
63/63 [==============================] – 0s 5ms/step – loss: 0.9380 – accuracy: 0.7325 – val_loss: 1.4263 – val_accuracy: 0.6960
Epoch 10/20
63/63 [==============================] – 0s 5ms/step – loss: 0.8990 – accuracy: 0.7425 – val_loss: 1.4358 – val_accuracy: 0.7000
Epoch 11/20
63/63 [==============================] – 0s 5ms/step – loss: 0.8642 – accuracy: 0.7506 – val_loss: 1.4783 – val_accuracy: 0.6980
Epoch 12/20
63/63 [==============================] – 0s 5ms/step – loss: 0.8335 – accuracy: 0.7573 – val_loss: 1.5581 – val_accuracy: 0.6870
Epoch 13/20
63/63 [==============================] – 0s 5ms/step – loss: 0.8075 – accuracy: 0.7620 – val_loss: 1.5645 – val_accuracy: 0.6920
Out[33]:
Our network now seems to peak at ~71% test accuracy, a 8% absolute drop. This drop is mostly due to the fact that we are now trying to
compress a lot of information (enough information to recover the separation hyperplanes of 46 classes) into an intermediate space that is
too low-dimensional. The network is able to cram most of the necessary information into these 8-dimensional representations, but not all
of it.
Further experiments¶
Try using larger or smaller layers: 32 units, 128 units…
We were using two hidden layers. Now try to use a single hidden layer, or three hidden layers.
Wrapping up¶
Here’s what you should take away from this example:
If you are trying to classify data points between N classes, your network should end with a Dense layer of size N.
In a single-label, multi-class classification problem, your network should end with a softmax activation, so that it will output a
probability distribution over the N output classes.
Categorical crossentropy is almost always the loss function you should use for such problems. It minimizes the distance between the
probability distributions output by the network, and the true distribution of the targets.
There are two ways to handle labels in multi-class classification: Encoding the labels via “categorical encoding” (also known as “one-hot encoding”) and using categorical_crossentropy as your loss
function.
Encoding the labels as integers and using the sparse_categorical_crossentropy loss function.
If you need to classify data into a large number of categories, then you should avoid creating information bottlenecks in your network by having
intermediate layers that are too small.
In [ ]: