代写代考 COM4513-6513] Assignment 1: Text Classification with Logistic Regression¶

assignment1

[COM4513-6513] Assignment 1: Text Classification with Logistic Regression¶
Instructor: ¶

The goal of this assignment is to develop and test two text classification systems:

Task 1: sentiment analysis, in particular to predict the sentiment of movie review, i.e. positive or negative (binary classification).
Task 2: topic classification, to predict whether a news article is about International issues, Sports or Business (multiclass classification).

For that purpose, you will implement:

Text processing methods for extracting Bag-Of-Word features, using (1) unigrams, bigrams and trigrams to obtain vector representations of documents. Two vector weighting schemes should be tested: (1) raw frequencies (3 marks; 1 for each ngram type); (2) tf.idf (1 marks).
Binary Logistic Regression classifiers that will be able to accurately classify movie reviews trained with (1) BOW-count (raw frequencies); and (2) BOW-tfidf (tf.idf weighted) for Task 1.
Multiclass Logistic Regression classifiers that will be able to accurately classify news articles trained with (1) BOW-count (raw frequencies); and (2) BOW-tfidf (tf.idf weighted) for Task 2.
The Stochastic Gradient Descent (SGD) algorithm to estimate the parameters of your Logistic Regression models. Your SGD algorithm should: Minimise the Binary Cross-entropy loss function for Task 1 (3 marks)
Minimise the Categorical Cross-entropy loss function for Task 2 (3 marks)
Use L2 regularisation (both tasks) (1 mark)
Perform multiple passes (epochs) over the training data (1 mark)
Randomise the order of training data after each pass (1 mark)
Stop training if the difference between the current and previous validation loss is smaller than a threshold (1 mark)
After each epoch print the training and development loss (1 mark)

Discuss how did you choose hyperparameters (e.g. learning rate and regularisation strength)? (2 marks; 0.5 for each model in each task).
After training the LR models, plot the learning process (i.e. training and validation loss in each epoch) using a line plot (1 mark; 0.5 for both BOW-count and BOW-tfidf LR models in each task) and discuss if your model overfits/underfits/is about right.
Model interpretability by showing the most important features for each class (i.e. most positive/negative weights). Give the top 10 for each class and comment on whether they make sense (if they don’t you might have a bug!). If we were to apply the classifier we’ve learned into a different domain such laptop reviews or restaurant reviews, do you think these features would generalise well? Can you propose what features the classifier could pick up as important in the new domain? (2 marks; 0.5 for BOW-count and BOW-tfidf LR models respectively in each task)

Data – Task 1¶
The data you will use for Task 1 are taken from here: http://www.cs.cornell.edu/people/pabo/movie-review-data/ and you can find it in the ./data_sentiment folder in CSV format:

data_sentiment/train.csv: contains 1,400 reviews, 700 positive (label: 1) and 700 negative (label: 0) to be used for training.
data_sentiment/dev.csv: contains 200 reviews, 100 positive and 100 negative to be used for hyperparameter selection and monitoring the training process.
data_sentiment/test.csv: contains 400 reviews, 200 positive and 200 negative to be used for testing.

Data – Task 2¶
The data you will use for Task 2 is a subset of the AG News Corpus and you can find it in the ./data_topic folder in CSV format:

data_topic/train.csv: contains 2,400 news articles, 800 for each class to be used for training.
data_topic/dev.csv: contains 150 news articles, 50 for each class to be used for hyperparameter selection and monitoring the training process.
data_topic/test.csv: contains 900 news articles, 300 for each class to be used for testing.

Submission Instructions¶
You should submit a Jupyter Notebook file (assignment1.ipynb) and an exported PDF version (you can do it from Jupyter: File->Download as->PDF via Latex).

You are advised to follow the code structure given in this notebook by completing all given funtions. You can also write any auxilliary/helper functions (and arguments for the functions) that you might need but note that you can provide a full solution without any such functions. Similarly, you can just use only the packages imported below but you are free to use any functionality from the Python Standard Library, NumPy, SciPy and Pandas. You are not allowed to use any third-party library such as Scikit-learn (apart from metric functions already provided), NLTK, Spacy, Keras etc..

Please make sure to comment your code. You should also mention if you’ve used Windows (not recommended) to write and test your code. There is no single correct answer on what your accuracy should be, but correct implementations usually achieve F1-scores around 80\% or higher. The quality of the analysis of the results is as important as the accuracy itself.

This assignment will be marked out of 20. It is worth 20\% of your final grade in the module.

The deadline for this assignment is 23:59 on Fri, 20 Mar 2020 and it needs to be submitted via MOLE. Standard departmental penalties for lateness will be applied. We use a range of strategies to detect unfair means, including Turnitin which helps detect plagiarism, so make sure you do not plagiarise.

import pandas as pd
import numpy as np
from collections import Counter
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import random

# fixing random seed for reproducibility
random.seed(123)
np.random.seed(123)

Load Raw texts and labels into arrays¶
First, you need to load the training, development and test sets from their corresponding CSV files (tip: you can use Pandas dataframes).

# fill in your code…

If you use Pandas you can see a sample of the data.

data_tr.head()

text label
0 note : some may consider portions of the follo… 1
1 note : some may consider portions of the follo… 1
2 every once in a while you see a film that is s… 1
3 when i was growing up in 1970s , boys in my sc… 1
4 the muppet movie is the first , and the best m… 1

The next step is to put the raw texts into Python lists and their corresponding labels into NumPy arrays:

# fill in your code…

Bag-of-Words Representation¶
To train and test Logisitc Regression models, you first need to obtain vector representations for all documents given a vocabulary of features (unigrams, bigrams, trigrams).

Text Pre-Processing Pipeline¶
To obtain a vocabulary of features, you should:

tokenise all texts into a list of unigrams (tip: using a regular expression)
remove stop words (using the one provided or one of your preference)
compute bigrams, trigrams given the remaining unigrams
remove ngrams appearing in less than K documents
use the remaining to create a vocabulary of unigrams, bigrams and trigrams (you can keep top N if you encounter memory issues).

stop_words = [‘a’,’in’,’on’,’at’,’and’,’or’,
‘to’, ‘the’, ‘of’, ‘an’, ‘by’,
‘as’, ‘is’, ‘was’, ‘were’, ‘been’, ‘be’,
‘are’,’for’, ‘this’, ‘that’, ‘these’, ‘those’, ‘you’, ‘i’,
‘it’, ‘he’, ‘she’, ‘we’, ‘they’ ‘will’, ‘have’, ‘has’,
‘do’, ‘did’, ‘can’, ‘could’, ‘who’, ‘which’, ‘what’,
‘his’, ‘her’, ‘they’, ‘them’, ‘from’, ‘with’, ‘its’]

N-gram extraction from a document¶
You first need to implement the extract_ngrams function. It takes as input:

x_raw: a string corresponding to the raw text of a document
ngram_range: a tuple of two integers denoting the type of ngrams you want to extract, e.g. (1,2) denotes extracting unigrams and bigrams.
token_pattern: a string to be used within a regular expression to extract all tokens. Note that data is already tokenised so you could opt for a simple white space tokenisation.
stop_words: a list of stop words
vocab: a given vocabulary. It should be used to extract specific features.

and returns:

a list of all extracted features.

See the examples below to see how this function should work.

def extract_ngrams(x_raw, ngram_range=(1,3), token_pattern=r’\b[A-Za-z][A-Za-z]+\b’, stop_words=[], vocab=set()):

# fill in your code…

extract_ngrams(“this is a great movie to watch”,
ngram_range=(1,3),
stop_words=stop_words)

(‘great’, ‘movie’),
(‘movie’, ‘watch’),
(‘great’, ‘movie’, ‘watch’)]

extract_ngrams(“this is a great movie to watch”,
ngram_range=(1,2),
stop_words=stop_words,
vocab=set([‘great’, (‘great’,’movie’)]))

[‘great’, (‘great’, ‘movie’)]

Note that it is OK to represent n-grams using lists instead of tuples: e.g. [‘great’, [‘great’, ‘movie’]]

Create a vocabulary of n-grams¶
Then the get_vocab function will be used to (1) create a vocabulary of ngrams; (2) count the document frequencies of ngrams; (3) their raw frequency. It takes as input:

X_raw: a list of strings each corresponding to the raw text of a document
ngram_range: a tuple of two integers denoting the type of ngrams you want to extract, e.g. (1,2) denotes extracting unigrams and bigrams.
token_pattern: a string to be used within a regular expression to extract all tokens. Note that data is already tokenised so you could opt for a simple white space tokenisation.
stop_words: a list of stop words
vocab: a given vocabulary. It should be used to extract specific features.
min_df: keep ngrams with a minimum document frequency.
keep_topN: keep top-N more frequent ngrams.

and returns:

vocab: a set of the n-grams that will be used as features.
df: a Counter (or dict) that contains ngrams as keys and their corresponding document frequency as values.
ngram_counts: counts of each ngram in vocab

Hint: it should make use of the extract_ngrams function.

def get_vocab(X_raw, ngram_range=(1,3), token_pattern=r’\b[A-Za-z][A-Za-z]+\b’, min_df=0, keep_topN=0, stop_words=[]):

# fill in your code…

return vocab, df, ngram_counts

Now you should use get_vocab to create your vocabulary and get document and raw frequencies of n-grams:

vocab, df, ngram_counts = get_vocab(X_tr_raw, ngram_range=(1,3), keep_topN=5000, stop_words=stop_words)
print(len(vocab))
print(list(vocab)[:100])
print(df.most_common()[:10])

[‘manages’, ‘questions’, ‘covered’, ‘body’, ‘ron’, ‘flair’, ‘drunken’, ‘approach’, ‘etc’, ‘allowing’, ‘lebowski’, ‘strong’, ‘model’, ‘category’, ‘family’, ‘couldn’, ‘argento’, ‘why’, ‘shown’, (‘doesn’, ‘work’), ‘ocean’, (‘lot’, ‘more’), ‘lou’, ‘attorney’, ‘kick’, ‘thinking’, ‘worth’, ‘larger’, (‘waste’, ‘time’), (‘back’, ‘forth’), ‘roles’, ‘adventures’, (‘million’, ‘dollars’), ‘critics’, ‘according’, (‘ghost’, ‘dog’), ‘outside’, ‘protect’, (‘last’, ‘time’), (‘but’, ‘so’), ‘creative’, ‘sell’, ‘pile’, ‘needless’, ‘immediately’, ‘screens’, ‘cards’, ‘blonde’, ‘meets’, ‘place’, ‘needs’, ‘needed’, ‘teacher’, ‘conceived’, ‘competition’, ‘powerful’, ‘expected’, (‘first’, ‘movie’), (‘but’, ‘least’), ‘gave’, ‘pleasures’, ‘spectacular’, ‘safe’, ‘wishes’, ‘stuff’, (‘there’, ‘something’), ‘robert’, ‘kid’, ‘latest’, (‘bad’, ‘guy’), ‘comet’, ‘street’, ‘intelligent’, ‘allow’, (‘tim’, ‘roth’), (‘production’, ‘design’), ‘living’, ‘abyss’, ‘clean’, (‘makes’, ‘him’), ‘aware’, ‘footage’, ‘vicious’, ‘sharon’, ‘genuinely’, ‘south’, ‘draw’, ‘wall’, (‘will’, ‘smith’), ‘romeo’, (‘scenes’, ‘but’), ‘sometimes’, ‘friend’, ‘millionaire’, ‘families’, ‘technique’, ‘spirit’, (‘not’, ‘going’), ‘horrifying’, ‘national’]

[(‘but’, 1334), (‘one’, 1247), (‘film’, 1231), (‘not’, 1170), (‘all’, 1117), (‘movie’, 1095), (‘out’, 1080), (‘so’, 1047), (‘there’, 1046), (‘like’, 1043)]

Then, you need to create vocabulary id -> word and id -> word dictionaries for reference:

# fill in your code…

Now you should be able to extract n-grams for each text in the training, development and test sets:

# fill in your code…

Vectorise documents¶

Next, write a function vectoriser to obtain Bag-of-ngram representations for a list of documents. The function should take as input:

X_ngram: a list of texts (documents), where each text is represented as list of n-grams in the vocab
vocab: a set of n-grams to be used for representing the documents

and return:

X_vec: an array with dimensionality Nx|vocab| where N is the number of documents and |vocab| is the size of the vocabulary. Each element of the array should represent the frequency of a given n-gram in a document.

def vectorise(X_ngram, vocab):

# fill in your code…

return X_vec

Finally, use vectorise to obtain document vectors for each document in the train, development and test set. You should extract both count and tf.idf vectors respectively:

Count vectors¶

# fill in your code…

X_tr_count.shape

(1400, 5000)

X_tr_count[:2,:50]

array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0.,
0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
[0., 0., 0., 1., 0., 0., 0., 2., 0., 0., 0., 0., 0., 0., 0., 1.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.,

TF.IDF vectors¶
First compute idfs an array containing inverted document frequencies (Note: its elements should correspond to your vocab)

# fill in your code…

Then transform your count vectors to tf.idf vectors:

# fill in your code…

X_tr_tfidf[1,:50]

array([0. , 0. , 0. , 2.24028121, 0. ,
0. , 0. , 5.67501654, 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
2.47354289, 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 2.56209629,
0. , 0. , 0. , 0. , 0. ])

Binary Logistic Regression¶
After obtaining vector representations of the data, now you are ready to implement Binary Logistic Regression for classifying sentiment.

First, you need to implement the sigmoid function. It takes as input:

z: a real number or an array of real numbers

and returns:

sig: the sigmoid of z

def sigmoid(z):

# fill in your code…

print(sigmoid(0))
print(sigmoid(np.array([-5., 1.2])))

[0.00669285 0.76852478]

Then, implement the predict_proba function to obtain prediction probabilities. It takes as input:

X: an array of inputs, i.e. documents represented by bag-of-ngram vectors ($N \times |vocab|$)
weights: a 1-D array of the model’s weights $(1, |vocab|)$

and returns:

preds_proba: the prediction probabilities of X given the weights

def predict_proba(X, weights):

# fill in your code…

return preds_proba

Then, implement the predict_class function to obtain the most probable class for each vector in an array of input vectors. It takes as input:

X: an array of documents represented by bag-of-ngram vectors ($N \times |vocab|$)
weights: a 1-D array of the model’s weights $(1, |vocab|)$

and returns:

preds_class: the predicted class for each x in X given the weights

def predict_class(X, weights):

# fill in your code…

return preds_class

To learn the weights from data, we need to minimise the binary cross-entropy loss. Implement binary_loss that takes as input:

X: input vectors
weights: model weights
alpha: regularisation strength

and return:

l: the loss score

def binary_loss(X, Y, weights, alpha=0.00001):

# fill in your code…

Now, you can implement Stochastic Gradient Descent to learn the weights of your sentiment classifier. The SGD function takes as input:

X_tr: array of training data (vectors)
Y_tr: labels of X_tr
X_dev: array of development (i.e. validation) data (vectors)
Y_dev: labels of X_dev
lr: learning rate
alpha: regularisation strength
epochs: number of full passes over the training data
tolerance: stop training if the difference between the current and previous validation loss is smaller than a threshold
print_progress: flag for printing the training progress (train/validation loss)

and returns:

weights: the weights learned
training_loss_history: an array with the average losses of the whole training set after each epoch
validation_loss_history: an array with the average losses of the whole development set after each epoch

def SGD(X_tr, Y_tr, X_dev=[], Y_dev=[], loss=”binary”, lr=0.1, alpha=0.00001, epochs=5, tolerance=0.0001, print_progress=True):

cur_loss_tr = 1.
cur_loss_dev = 1.
training_loss_history = []
validation_loss_history = []

# fill in your code…

return weights, training_loss_history, validation_loss_history

Train and Evaluate Logistic Regression with Count vectors¶
First train the model using SGD:

w_count, loss_tr_count, dev_loss_count = SGD(X_tr_count, Y_tr,
X_dev=X_dev_count,
Y_dev=Y_dev,
lr=0.0001,
alpha=0.001,
epochs=100)

/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:50: DeprecationWarning: elementwise == comparison failed; this will raise an error in the future.

程序代写 CS代考加微信: powcoder QQ: 1823890830 Email: powcoder@163.com

Related Posts