Assessed Exercise for Deep Learning (M)¶
This exercise must be submitted as a colab notebook. Deadline Monday the 4th of March, 15:00.
You will create a classifier and test it on a collection of images for a new task. While you are welcome to build a full network from scratch, most of you will not have sufficient access to the data and compuational power required, so you are welcome to provide a solution based on transfer learning from a pre-trained network, adapted to your new task.
The Keras documentation has some examples of pretrained networks which are easy to integrate, see for example their ResNet50 example https://keras.io/applications/#classify-imagenet-classes-with-resnet50, or the InceptionV3 network https://keras.io/applications/#fine-tune-inceptionv3-on-a-new-set-of-classes. If you use the pretrained model, a good way to start is by freezing all layers up to the last layer before the output. Adapt the output layer to fit your classification problem. You might then unfreeze some earlier layers for further fine tuning.
You will need to create a training set (at least 100 images per class, potentially classifying e.g type of location, activity. If you are training a full network from scratch you would need orders of magnitude more, but this will work for transfer learning on an existing network). It might be sensible to start off testing and demonstrating your approach by using an existing dataset e.g. the flowers one. You can find other interesting datasets at:
• http://deeplearning.net/datasets
• UCI ML collection
• https://www.visualdata.io
• https://ai.google/tools/datasets
Students who put more effort into creating and analysing an interesting dataset will tend to do better in marks for sections 1. & 2. below.
Write a pre-processing step that will resize and crop the images to the right size ((224, 224) is default for ResNet50 and (299,299) is the default for Inception), and consider how you can apply data augmentation techniques to your new dataset, and design appropriate pre-processing functions.
In your submission you should have the following structure (share of the AX marks given in brackets at the end of each part):
1. Analysis of the problem. (15%)
2. Visualisation and analysis of the data type, quality and class distributions. You may want to design some data augmentation in your system. (20%)
3. Creation of multiple candidate network architectures. Include your justification of the design decisions. You should inlcude one very simple baseline model (e.g. a linear model, or a simple two layer Densely connected model). (15%)
4. Training. This should include code for hyperparameter search, regularisation methods. (15%)
5. Empirical evaluation of performance, and potentially visualisation and analysis of the trained network. This should make good use of graphs and tables of results, confusion matrices etc to represent the relative performance of the different models. Explain why you chose the metrics you use. (20%)
6. Report on the performance, discussing the suitability of the final network for use. (15%)
For each of the design decisions, make sure you describe in detail the motivation behind them.
Submission process
You should submit the colab notebook with all code needed to run your model and all visualisations of results in place (I don’t want to have to run 80 projects 🙂 ). This exercise must be submitted as a colab notebook. Deadline Monday the 4th of March, 15:00. If you have your own training data, make sure that any links to that are accessible by 3rd parties (me – I won’t share the links with anyone else).
Some code snippets which might be useful to get you started¶
In [0]:
#@title Run this to import the right things
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import collections
import math
import os
import random
from six.moves import urllib
import io
import shutil
from IPython.display import clear_output, Image, display, HTML
import tensorflow as tf
import tensorflow_hub as hub
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn.metrics as sk_metrics
import time
from tensorflow.python.keras.applications.resnet50 import preprocess_input
from tensorflow.python.keras.preprocessing.image import ImageDataGenerator
from keras.applications.inception_v3 import InceptionV3
from keras.applications.resnet50 import preprocess_input, decode_predictions, ResNet50
from keras.preprocessing import image
from keras import regularizers
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D, BatchNormalization, Dropout, Flatten
from keras import backend as K
Using TensorFlow backend.
In [0]:
#@title Load the Flowers data set and split into test & train sets [RUN ME]
FLOWERS_DIR = ‘./flower_photos’
TRAIN_FRACTION = 0.8
RANDOM_SEED = 2018
def download_images():
“””If the images aren’t already downloaded, save them to FLOWERS_DIR.”””
if not os.path.exists(FLOWERS_DIR):
DOWNLOAD_URL = ‘http://download.tensorflow.org/example_images/flower_photos.tgz’
print(‘Downloading flower images from %s…’ % DOWNLOAD_URL)
urllib.request.urlretrieve(DOWNLOAD_URL, ‘flower_photos.tgz’)
!tar xfz flower_photos.tgz
print(‘Flower photos are located in %s’ % FLOWERS_DIR)
download_images()
!rm ./flower_photos/LICENSE.txt
if not os.path.exists(‘./flowers’):
os.mkdir(‘./flowers’)
image_dir = FLOWERS_DIR
from sklearn.model_selection import train_test_split
# Read all flower images (.jpg) from a folder
# The function returns both the path of the flower image and the corresponding label
# which is defined by the name of the foler in which the image is
def read_images_from_dir(base_dir, folder):
path_folder = os.path.join(image_dir, folder)
files_directory = os.listdir(path_folder)
labels = []
images = []
for file in files_directory:
if file.endswith(‘.jpg’):
labels.append(folder)
images.append(os.path.join(path_folder, file))
return labels, images
def read_images(base_dir):
labels = []
images = []
folders = os.listdir(image_dir)
for folder in folders:
labels_folder, images_folder = read_images_from_dir(base_dir, folder)
labels.extend(labels_folder)
images.extend(images_folder)
return labels, images
labels, images = read_images(image_dir)
# Split the data into test and training sets
images_train, images_test, labels_train, labels_test = train_test_split(images, labels, test_size=0.3, random_state=8, stratify=labels)
# Create folders for test and training that can be passed to Keras Generators
train_folder = ‘./flowers/output/train’
test_folder = ‘./flowers/output/test’
output_folder = ‘./flowers/output’
def create_output_folders():
if not os.path.exists(output_folder):
print(‘Creating output directories’)
os.mkdir(output_folder)
if not os.path.exists(train_folder):
os.mkdir(train_folder)
for label in set(labels):
os.mkdir(train_folder + ‘/’ + label)
if not os.path.exists(test_folder):
os.mkdir(test_folder)
for label in set(labels):
os.mkdir(test_folder + ‘/’ + label)
def copy_files_to_train_and_validation_folders():
print(‘Copy training files to directory’)
for index, value in enumerate(images_train):
dest = os.path.join(train_folder, labels_train[index])
shutil.copy(value, dest)
print(‘Copy test files to directory’)
for index, value in enumerate(images_test):
shutil.copy(value, test_folder + ‘/’ + labels_test[index])
create_output_folders()
copy_files_to_train_and_validation_folders()
We now create some data generators for working with the flowers data
In [0]:
#images_train, images_test, labels_train, labels_test = train_test_split(images, labels, test_size=0.3, random_state=8, stratify=labels)
image_size = 224
data_generator = ImageDataGenerator(preprocessing_function=preprocess_input)
train_generator = data_generator.flow_from_directory(train_folder, target_size=(image_size, image_size), batch_size=24, class_mode=’categorical’)
validation_generator = data_generator.flow_from_directory(test_folder, target_size=(image_size, image_size), batch_size=24, class_mode=’categorical’)
Adapt the code below to work with the generators, then proceed with the experiments!
In [0]:
# create the base pre-trained model
#base_model = InceptionV3(weights=’imagenet’, include_top=False)
base_model = ResNet50(weights=’imagenet’, include_top=False)
lamb = 0.001
# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
# and a logistic layer — let’s say we have 200 classes
predictions = Dense(YOUR NUMBER OF CLASSES, activation=’softmax’)(x)
# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
layer.trainable = False
# compile the model (should be done *after* setting layers to non-trainable)
model.compile(optimizer=’rmsprop’, loss=’categorical_crossentropy’, metrics=[‘accuracy’] )
# train the model on the new data for a few epochs
model.fit_generator(FILL IN YOUR DETAILS…)
# at this point, the top layers are well trained and we can start fine-tuning
# convolutional layers from inception V3. We will freeze the bottom N layers
# and train the remaining top layers.
# let’s visualize layer names and layer indices to see how many layers
# we should freeze:
for i, layer in enumerate(base_model.layers):
print(i, layer.name)
# we chose to train the top 2 inception blocks, i.e. we will freeze
# the first 249 layers and unfreeze the rest:
for layer in model.layers[:249]:
layer.trainable = False
for layer in model.layers[249:]:
layer.trainable = True
# we need to recompile the model for these modifications to take effect
# we use SGD with a low learning rate
from keras.optimizers import SGD
model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss=’categorical_crossentropy’, metrics=[‘accuracy’])
# we train our model again (this time fine-tuning the top 2 inception blocks
# alongside the top Dense layers
model.fit_generator(FILL IN YOUR DETAILS…)