COSC2779LabExercises_W6_2
¶
COSC 2779 | Deep Learning
¶
Week 6 Lab Exercises: **Transfer Learning**
¶
Introduction¶
In this tutorial, you will learn how to classify images of cats and dogs by using transfer learning from a pre-trained network.
A pre-trained model is a saved network that was previously trained on a large dataset, typically on a large-scale image-classification task. You either use the pretrained model as is or use transfer learning to customize this model to a given task.
In this tutorial, you will try two ways to customize a pretrained model:
Feature Extraction: Use the representations learned by a previous network to extract meaningful features from new samples. You simply add a new classifier, which will be trained from scratch, on top of the pretrained model so that you can repurpose the feature maps learned previously for the dataset.
Fine-Tuning: Unfreeze a few of the top layers of a frozen model base and jointly train both the newly-added classifier layers and the last layers of the base model. This allows us to “fine-tune” the higher-order feature representations in the base model in order to make them more relevant for the specific task.
Setting up the Notebook¶
Let’s first load the packages we need.
In [ ]:
import tensorflow as tf
AUTOTUNE = tf.data.experimental.AUTOTUNE
import numpy as np
import pandas as pd
import tensorflow_datasets as tfds
import pathlib
import shutil
import tempfile
from IPython import display
from matplotlib import pyplot as plt
import matplotlib.ticker as ticker
from sklearn.model_selection import train_test_split
We can use the tensor board to view the learning curves. Let’s first set it up.
In [ ]:
logdir = pathlib.Path(tempfile.mkdtemp())/”tensorboard_logs”
shutil.rmtree(logdir, ignore_errors=True)
# Load the TensorBoard notebook extension
%load_ext tensorboard
# Open an embedded TensorBoard viewer
%tensorboard –logdir {logdir}/models
We can also write our own function to plot the models training history ones training has completed.
In [ ]:
from itertools import cycle
def plotter(history_hold, metric = ‘binary_crossentropy’, ylim=[0.0, 1.0]):
cycol = cycle(‘bgrcmk’)
for name, item in history_hold.items():
y_train = item.history[metric]
y_val = item.history[‘val_’ + metric]
x_train = np.arange(0,len(y_val))
c=next(cycol)
plt.plot(x_train, y_train, c+’-‘, label=name+’_train’)
plt.plot(x_train, y_val, c+’–‘, label=name+’_val’)
plt.legend()
plt.xlim([1, max(plt.xlim())])
plt.ylim(ylim)
plt.xlabel(‘Epoch’)
plt.ylabel(metric)
plt.grid(True)
Loading Dataset¶
In this tutorial, you will use a dataset containing several thousand images of cats and dogs. Download and extract a zip file containing the images
In [ ]:
import os
_URL = ‘https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip’
path_to_zip = tf.keras.utils.get_file(‘cats_and_dogs.zip’, origin=_URL, extract=True)
PATH = os.path.join(os.path.dirname(path_to_zip), ‘cats_and_dogs_filtered’)
In [ ]:
train_dir = os.path.join(PATH, ‘train’)
test_dir = os.path.join(PATH, ‘validation’)
BATCH_SIZE = 32
IMG_SIZE = (160, 160)
Setup Data loading pipeline¶
Lets use the keras data loader pipeline to load the data.
We can also do some augmentation on the training data using the pipeline.
In [ ]:
from tensorflow.keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
width_shift_range=.2,
height_shift_range=.2,
rotation_range=30,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
validation_split=0.2)
train_generator = train_datagen.flow_from_directory(
train_dir,
target_size=IMG_SIZE,
batch_size=BATCH_SIZE,
class_mode=’binary’,
subset=’training’)
validation_generator = train_datagen.flow_from_directory(
train_dir,
target_size=IMG_SIZE,
batch_size=BATCH_SIZE,
class_mode=’binary’,
subset=’validation’)
test_datagen = ImageDataGenerator()
test_generator = test_datagen.flow_from_directory(
test_dir,
target_size=IMG_SIZE,
batch_size=BATCH_SIZE,
class_mode=’binary’)
Setup the model & train¶
You will create the base model from the MobileNet V2 model developed at Google. This is pre-trained on the ImageNet dataset, a large dataset consisting of 1.4M images and 1000 classes. ImageNet is a research training dataset with a wide variety of categories like jackfruit and syringe. This base of knowledge will help us classify cats and dogs from our specific dataset.
First, you need to pick which layer of MobileNet V2 you will use for feature extraction. The very last classification layer (on “top”, as most diagrams of machine learning models go from bottom to top) is not very useful. Instead, you will follow the common practice to depend on the very last layer before the flatten operation. This layer is called the “bottleneck layer”. The bottleneck layer features retain more generality as compared to the final/top layer.
No transfer learning¶
First, instantiate a MobileNet V2 model with no pre trained weights. In this instance the weights of the model will be initialised randomly.
By specifying the include_top=False argument, you load a network that doesn’t include the classification layers at the top, which is ideal for feature extraction.
In [ ]:
# Create the base model
IMG_SHAPE = IMG_SIZE + (3,)
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
include_top=False,
weights=None)
Lets now complete the model. We are going to use the functional API to build the model.
Since we need to do classification we have to plug a new layer (a fully connected) to the end of the base model. this can be done by doing a global average pooling and then a dense layer. We have also included a dropout to regularise.
This model expects pixel values in [-1,1], but at this point, the pixel values in your images are in [0-255]. To rescale them, use the preprocessing method included with the model on the input.
Note that we did not do any scaling in the data generator.
In [ ]:
preprocess_input = tf.keras.applications.mobilenet_v2.preprocess_input
inputs = tf.keras.Input(shape=IMG_SHAPE)
x = preprocess_input(inputs)
x = base_model(x, training=True)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.2)(x)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)
Now we can train the model.
In [ ]:
m_histories = {}
def get_callbacks(name):
return [
tf.keras.callbacks.TensorBoard(logdir/name, histogram_freq=1),
]
In [ ]:
base_learning_rate = 0.0001
model.compile(optimizer=tf.keras.optimizers.Adam(lr=base_learning_rate),
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=[‘accuracy’])
In [ ]:
m_histories[‘no_TL’] = model.fit(train_generator,
epochs=10,
verbose=0,
validation_data=validation_generator,
callbacks=get_callbacks(‘models/no_TL’))
Test the model on test data. This is a bad thing to do. You should not touch test data till you decide on the final model.
In [ ]:
model.evaluate(test_generator)
With transfer learning¶
To do transfere lerning we can use the pre-trained weights that is included with the model in tensorflow. The pre-loaded weights are trained on ImageNet.
In [ ]:
# Create the base model from the pre-trained model MobileNet V2
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
include_top=False,
weights=’imagenet’)
since we dont want to train these pre-loaded weight, we can freeze them.
In [ ]:
base_model.trainable = False
Now build the model and train
In [ ]:
preprocess_input = tf.keras.applications.mobilenet_v2.preprocess_input
inputs = tf.keras.Input(shape=IMG_SHAPE)
x = preprocess_input(inputs)
x = base_model(x, training=False)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.2)(x)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)
In [ ]:
model.compile(optimizer=tf.keras.optimizers.Adam(lr=base_learning_rate),
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=[‘accuracy’])
In [ ]:
m_histories[‘with_TL’] = model.fit(train_generator,
epochs=10,
validation_data=validation_generator,
callbacks=get_callbacks(‘models/with_TL’))
Test the model on test data
In [ ]:
model.evaluate(test_generator)
Did you observe an improvement when using transfer learning?
(Optional) Fine tuning the pre-trained weights¶
In the feature extraction experiment, you were only training a few layers on top of an MobileNet V2 base model. The weights of the pre-trained network were not updated during training.
One way to increase performance even further is to train (or “fine-tune”) the weights of the top layers of the pre-trained model alongside the training of the classifier you added. The training process will force the weights to be tuned from generic feature maps to features associated specifically with the dataset.
This should only be attempted after you have trained the top-level classifier with the pre-trained model set to non-trainable. If you add a randomly initialized classifier on top of a pre-trained model and attempt to train all layers jointly, the magnitude of the gradient updates will be too large (due to the random weights from the classifier) and your pre-trained model will forget what it has learned.
Also, you should try to fine-tune a small number of top layers rather than the whole MobileNet model. In most convolutional networks, the higher up a layer is, the more specialized it is. The first few layers learn very simple and generic features that generalize to almost all types of images. As you go higher up, the features are increasingly more specific to the dataset on which the model was trained. The goal of fine-tuning is to adapt these specialized features to work with the new dataset, rather than overwrite the generic learning.
All you need to do is unfreeze the base_model and set the bottom layers to be un-trainable. Then, you should recompile the model (necessary for these changes to take effect), and resume training.
In [ ]:
base_model.trainable = True
# Let’s take a look to see how many layers are in the base model
print(“Number of layers in the base model: “, len(base_model.layers))
# Fine-tune from this layer onwards
fine_tune_at = 100
# Freeze all the layers before the `fine_tune_at` layer
for layer in base_model.layers[:fine_tune_at]:
layer.trainable = False
Train the model and evaluate. As you are training a much larger model and want to readapt the pretrained weights, it is important to use a lower learning rate at this stage. Otherwise, your model could overfit very quickly.
In [ ]:
model.compile(optimizer=tf.keras.optimizers.Adam(lr=base_learning_rate/10),
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=[‘accuracy’])
In [ ]:
m_histories[‘with_TL_finetune’] = model.fit(train_generator,
epochs=10,
validation_data=validation_generator,
callbacks=get_callbacks(‘models/with_TL_finetune’))
In [ ]:
model.evaluate(test_generator)