Assignment 2 Brief¶
Deadline: Tuesday, December 3, 2019 at 14:00 hrs¶
Number of marks available: 20¶
Scope: Sessions 6 to 9¶
1. Instructions¶
How and what to submit¶
A. Submit a Jupyter Notebook named COM4509-6509_Assignment_2_UCard_XXXXXXXXX.ipynb where XXXXXXXXX refers to your UCard number.
B. Upload the notebook file to MOLE before the deadline above.
C. NO DATA UPLOAD: Please do not upload the data files used. We have a copy already.
Assessment Criteria¶
• Being able to manipulate a dataset by generating sythetic data and extracting a particular subset.
• Being able to build and train different machine learning models with tunable hyperparameters to optimise given evaluation metric.
• Being able to compare different machine learning models and explain interesting results observed.
• Being able to follow examples in the lab and write code without the help of starter code.
Late submissions¶
We follow Department’s guidelines about late submissions, i.e., a deduction of 5% of the mark each working day the work is late after the deadline. NO late submission will be marked one week after the deadline because we will release a solution by then. Please read this link.
Use of unfair means¶
“Any form of unfair means is treated as a serious academic offence and action may be taken under the Discipline Regulations.” (from the MSc Handbook). Please carefully read this link on what constitutes Unfair Means if not sure.
2. Image classification and denoising¶
The CIFAR-10 dataset¶
In this assignment, we will work on the CIFAR-10 dataset collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton from the University of Toronto. This dataset consists of 60000 32×32 colour images in 10 classes, with 6000 images per class. Each image is a 3-channel colour images of 32×32 pixels in size. There are 50000 training images and 10000 test images.
Question 1: Data loading and manipulation (4 marks)¶
1a. Download both the training and test data of the CIFAR-10 dataset, e.g., by following the pytorch CIFAR10 tutorial. You can also download via other ways if you prefer.
1b. Add random noise to all training and test data to generate noisy dataset, e.g., by torch.randn(), with a scaling factor scale, e.g., original image + scale * torch.randn(), and normalise/standardise the pixel values to the original range, e.g., using np.clip(). You may choose any scale value between 0.2 and 0.5.
Note: Before generating the random noise, you MUST set the random seed to your UCard number XXXXXXXXX for reproducibility, e.g., using torch.manual_seed(). This seed needs to be used for all remaining code if there is randomness, for reproducibility.
1c. Extract a subset with only two classes: Cat and Dog and name it starting with CatDog.
1d. Show 10 pairs of original and noisy images of cats and 10 pairs of original and noisy images of dogs.
Question 1 Answer¶
In [1]:
# Write the code for your answer here. You can use multiple cells to improve readability.
import torch
import torchvision
import torchvision.transforms as transforms
transform = transforms.Compose(
[transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root=’./data’, train=True,
download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
shuffle=True, num_workers=2)
testset = torchvision.datasets.CIFAR10(root=’./data’, train=False,
download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
shuffle=False, num_workers=2)
classes = (‘plane’, ‘car’, ‘bird’, ‘cat’,’deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’)
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data\cifar-10-python.tar.gz
100%|██████████████████████████████████████████████████████████████▉| 170287104/170498071 [00:57<00:00, 5592060.57it/s] Extracting ./data\cifar-10-python.tar.gz to ./data Files already downloaded and verified 170500096it [01:10, 5592060.57it/s] In [2]: print(trainset) print(testset) Dataset CIFAR10 Number of datapoints: 50000 Root location: ./data Split: Train StandardTransform Transform: Compose( ToTensor() Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)) ) Dataset CIFAR10 Number of datapoints: 10000 Root location: ./data Split: Test StandardTransform Transform: Compose( ToTensor() Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)) ) In [ ]: Question 2: Dimensionality reduction, binary classification, and evaluation (6 marks)¶ This question uses the CatDog subset with no noise added. Training¶ 2a. Apply PCA on the training set to reduce the dimensionality. You need to study at least seven different values ($k_1, k_2, ..., k_7$) for the reduced dimensionality. Explain your choice. 2b. Train eight Naive Bayes Classifiers (NBC): one on the original features (raw pixels), and seven on PCA features with seven different dimensions in 2a, i.e., NBC on $k_1$ PCA features; NBC on $k_2$ PCA features; ..., NBC on $k_7$ PCA features. You will need to decide on what Naive Bayes classifier (Gaussian? Multinomial? etc.) to use and explain your choice. Testing and evaluation¶ 2c. Evalaute the eight Naive Bayes classifiers on the test set in terms of classification accuracy and visualise their performance using a bar graph. 2d. Plot the ROC Curves in true positive rates vs false positive rates for the eight Naive Bayes classifiers in one figure using eight different line/marker styles clearly labelled. 2e. Compute the area under the ROC curve values for the eight Naive Bayes classifiers and visualise using a bar graph. 2f. Describe at least three interesting observations from the evaluation results above. Each observation should have 3-5 sentences. In [12]: # Write the code for your answer here. You can use multiple cells to improve readability. Question 3: Noisy data and multiclass classification (6 marks)¶ Noisy CatDog subset.¶ 3a. Repeat 2a, 2b, and 2c on the noisy version of CatDog subset. Show the bar graph and compare it with that in 2c above. Multiclass classification using the original CIFAR-10 dataset (all 10 classes)¶ 3b. Apply PCA on the training set to reduce the dimensionality. You need to study at least three different values for the reduced dimensionality. Explain your choice. 3c. Train nine classifers: four Naive Bayes classifiers(one on the original features, and three on PCA features with three different dimensions in 3b); four Logistic Regression classifiers (one on the original features, and three on PCA features with three different dimensions in 3b); and one Convoluational Neural Network as defined in the pytorch CIFAR10 tutorial. 3d. Evalaute the nine classifiers on the test set. Summarise the classification accuracy, total training time, and total test time using three bar graphs. 3e. Show the confusion matrix for these nine classifiers (see Lab 8 - 1.4). 3f. Describe at least three interesting observations from the evaluation results above. Each observation should have 3-5 sentences. In [13]: # Write the code for your answer here. You can use multiple cells to improve readability. Question 4: Denoising Autoencoder (4 marks)¶ This question uses both the original and noisy CIFAR-10 datasets (all 10 classes). Read about denoising autoencoder at Wikepedia) and this short introduction or any other sources you like. 4a. Modify the autoencoder architecture in Lab 7 so that it takes colour images as input (i.e., 3 input channels). 4b. Training: feed the noisy training images as input to the autoencoder in 4a; use a loss function that computes the reconstruction error between the output of the autoencoder and the respective original images. 4c. Testing: evaluate the autoencoder trained in 4b on the test datasets (feed noisy images in and compute reconstruction errors on original clean images. Find the worstly denoised 30 images (those with the largest reconstruction errors) in the test set and show them in pairs with the original images (60 images to show in total). 4d. Choose at least two hyperparameters to vary. Study at least three different choices for each hyperparameter. When varying one hyperparameter, all the other hyperparameters can be fixed. Visualise the performance sensitivity with respect to these hyperparameters. 4e. Describe at least two interesting observations from the evaluation results above. Each observation should have 3-5 sentences. In [14]: # Write the code for your answer here. You can use multiple cells to improve readability.