Subject Review
COMP90042
Natural Language Processing
Lecture 23
Semester 1 2021 Week 12 Jey Han Lau
COPYRIGHT 2021, THE UNIVERSITY OF MELBOURNE
1
COMP90042
L23
• •
•
Sentence segmentation Tokenisation
‣ Subword tokenisation Word normalisation
‣ Derivational vs. inflectional morphology
‣ Lemmatisation vs. stemming Stop words
•
Preprocessing
2
COMP90042
L23
• •
Derivation
Smoothing techniques
‣ Add-k
‣ Absolute discounting
‣ Katz Backoff
‣ Kneser-Ney smoothing ‣ Interpolation
N-gram Language Models
3
COMP90042
L23
Text Classification
• Buildingaclassificationsystem • Textclassificationtasks
‣ Topic classification
‣ Sentiment analysis
‣ Native language identification
• Algorithms
‣ Naive-Bayes, logistic regression, SVM ‣ kNN, neural networks
• Biasvs.variance
• Evaluationmetrics:Precision,recall,F1
4
COMP90042
L23
Part-of-Speech Tagging
• EnglishPOS
‣ Open vs. closed POS classes
• Tagsets
‣ Penn Treebank tags
• Automatictaggers ‣ Rule-based
‣ Statistical
– Unigram, classifier-based, HMM
5
COMP90042
L23
Hidden Markov Models Probabilistic formulation
‣ Parameters: emission and transition probabilities
• • •
•
Training
Viterbi algorithm
Generative vs. discriminative models
6
COMP90042
L23
• •
Formulation
Designing FF networks for NLP tasks
‣ Topic classification ‣ Language model
‣ POS tagging
• •
Word embeddings Convolutional networks
DL: Feed-forward Networks
7
COMP90042
L23
DL: Recurrent Networks
• Formulation
• RNNlanguagemodels • LSTM
‣ Functions of gates
‣ Variants
• DesigningRNNforNLPtasks
‣ Text classification: sentiment analysis ‣ POS tagging
8
COMP90042
L23
Lexical Semantics
• Definitionofwordsenses,glosses • Lexicalrelationships
‣ Synonymy, antonymy, hypernymy, meronymy • StructureofWordNet
• Wordsimilarity
‣ Path length, depth information, information content • Wordsensedisambiguation
‣ Supervised vs. unsupervised
9
COMP90042
L23
•
• • • •
Matrices for distributional semantics
‣ VSM, TF-IDF, word-word co-occurrence
Distributional Semantics
Association measures: PMI, PPMI Count-based methods: SVD Neural methods: skip-gram, CBOW Evaluation
‣ Word similarity, analogy
10
COMP90042
L23
• • •
Formulation with RNN ELMo
BERT
‣ Objectives
‣ Fine-tuning for downstream tasks Transformers
‣ Multi-head attention
•
Contextual Representation
11
COMP90042
L23
Discourse
• Motivationformodellingbeyondwords • Discoursesegmentation
‣ Text Tiling
• Discourseparsing
‣ Rhetorical structure theory • Anaphoraresolution
‣ Centering
‣ Supervised models
12
COMP90042
L23
Formal Language Theory & FSA
• Formallanguagetheoryasaframeworkfordefining language
• Regularlanguages
‣ Closure properties
• Finitestateacceptors
‣ Word morphology, weighted variant
• Finitestatetransducers
‣ Weighted variant, edit distance, morphological analysis
13
COMP90042
L23
• • • •
Center embedding
Basics of CFG
Syntactic constituent and its properties CFG parsing
‣ Chomsky normal form
‣ CYK
English sentence structure (Penn Treebank)
•
Context-Free Grammar
14
COMP90042
L23
Probabilistic Context-Free Grammar
• Ambiguityingrammars
• BasicsofprobabilisticCFGs • ProbabilityofaCFGtree
• Parsing
‣ Probabilistic CYK • Improvements
‣ Parent conditioning ‣ Head lexicalisation
15
COMP90042
L23
• • •
•
Notion of dependency between words Universal dependency
Properties of dependency trees
‣ Projectivity Parsing
‣ Transition-based ‣ Graph-based
Dependency Grammar
16
COMP90042
L23
Machine Translation
• StatisticalMT
‣ Language + translation model ‣ Alignments
• NeuralMT
‣ Encoder-decoder
‣ Beam search decoding ‣ Attention mechanism
• Evaluation:BLEU
17
COMP90042
L23
•
•
Named entity recognition
‣ NER tags, IOB tagging, models
• •
Relation extraction
‣ Rule-based, supervised, semi-supervised, distant supervision
‣ Unsupervised
Temporal expression extraction Event extraction
Information Extraction
18
COMP90042
L23
•
IR-based QA
‣ Question processing, answer type prediction ‣ Passage retrieval, answer extraction
•
• •
Reading comprehension
‣ Models: LSTM-based, BERT
Question Answering
Knowledge-based QA Hybrid QA: IBM Watson
19
COMP90042
L23
• •
Evolution of topic models LDA
‣ Sampling-based learning
‣ Hyper-parameters Evaluation:
‣ Word intrusion
‣ Topic coherence
•
Topic Modelling
20
COMP90042
L23
Summarisation
• Extractivesummarisation ‣ Single-document
– Unsupervised content selection ‣ Multi-document
– Maximum marginal relevance • Abstractivesummarisation
‣ Encoder-decoder models: copy mechanism • Evaluation:ROUGE
21
COMP90042
L23
• • •
•
Learning outcomes
Arguments against ethical checks Core NLP ethics concepts:
‣ bias, dual use, privacy
Discussion of applications/use cases
Ethics
22
COMP90042
L23
Exam
23
COMP90042
L23
Exam Structure
• Openbook
• 120points(40%forthesubject)
• Gradescope
• Time:120minuteswritingtime+15minutesreadingtime • 3parts:
‣ A: short answer questions ‣ B: method questions
‣ C: algorithm questions
24
COMP90042
L23
•
Several short questions
‣ 1-2 sentence answers for each
‣ Definitional, e.g. what is X?
‣ Conceptual, e.g. relate X and Y, purpose of Z?
‣ May call for an example illustrating a technique/ problem
Short Answer Questions
25
COMP90042
L23
• •
Longer answer
Focus on analysis and understanding
‣ Contrast different methods
‣ Analyse an algorithm/application
‣ Motivate a modelling technique
‣ Explain or derive mathematical equation
Method Questions
26
COMP90042
L23
Algorithmic Questions Perform algorithmic computations
‣ Numerical computations for algorithm on some given example data
‣ Present an outline of an algorithm on your own example
•
•
Not required to simplify maths (e.g. leaving fractions as log(5/4) is fine)
27
COMP90042
L23
What to Expect
Even coverage of topic from the semester
Be prepared for concepts that have not yet been assessed by homework / project
•
•
• •
Prescribed reading is fair game for topics mentioned in the lectures and workshops
Mock exam
28
COMP90042
L23
•
Final survey: https://forms.gle/ CYBfYuEh46mjGqG86
Questions?
29