Course Overview & Introduction
COMP90042
Natural Language Processing
Lecture 1
Semester 1 2021 Week 1 Jey Han Lau
COPYRIGHT 2021, THE UNIVERSITY OF MELBOURNE
1
COMP90042
L1
Prerequisites
• COMP90049“IntroductiontoMachineLearning”or
COMP30027 “Machine Learning”
‣ Modules → Welcome → Machine Learning Readings
• Pythonprogrammingexperience
• Noknowledgeoflinguisticsoradvancedmathematicsis assumed
• Caveats–Not“vanilla”computerscience
‣ Involves some basic linguistics, e.g., syntax and morphology
‣ Requires maths, e.g., algebra, optimisation, linear algebra, dynamic programming
2
COMP90042
L1
•
Expectations
‣ develop Python skills
‣ keep up with readings
‣ lecture/discussion board participation
•
Outcomes
‣ Practical familiarity with range of text analysis technologies
‣ Understanding of theoretical models underlying these tools
‣ Competence in reading research literature
Expectations and outcomes
3
COMP90042
L1
Assessment
• Assignments (25% total for 3 activities) ‣ 2 programming exercises
‣ Released in week 4 and 5; 1 week to complete ‣ 1 peer review of project report
‣ Released in week 11; 1.5 week to complete Project (35%)
‣ Released near Easter; 5 weeks to complete
•
• Hurdle >50% exam (20/40), and >50% for assignments + project (30/60)
•
Exam (40%)
‣ 2 hours, open book
‣ Covers content from lectures, workshop and prescribed reading
4
COMP90042
L1
Teaching Staff
Lecturer Head Tutor
Jey Han Lau Zenan Zhai
5
COMP90042
L1
• • • • •
Aili Shen
Fajri
Nathaniel Carpenter Shraey Bhatia
Yulia Otmakhova
Tutors
6
COMP90042
L1
Recommended Texts Texts:
‣ Jurafsky and Martin, Speech and Language Processing, 3rd ed., Prentice Hall. draft
‣ Eisenstein; Natural Language Processing, Draft 15/10/18
‣ Goldberg; A Primer on Neural Network Models for Natural Language Processing
•
•
Recommended for learning python:
‣ Steven Bird, Ewan Klein and Edward Loper, Natural Language Processing with Python, O’Reilly, 2009
7
COMP90042
L1
Contact hours ‣ Mon 16:15-17:15 Zoom
‣ Tue 15:15-16:15 Zoom Workshops: several across the week
‣ Worksheets & programming exercises
•
•
Lectures
•
Method of contact — ask questions on the Canvas discussion board
8
COMP90042
L1
•
• •
•
•
Trialing online zoom lectures for the first few weeks
Zoom Lectures
Gauge interest, participation rate and feasibility Preliminary version (v1) of lecture slides have
been published (Modules > Lectures > Slides) Lecture slides may be updated after the lectures to
incorporate poll/survey results
Lecture recordings will be available after each lecture
9
COMP90042
L1
10
COMP90042
L1
•
Making extensive use of python
‣ workshops feature programming challenges ‣ provided as interactive ‘notebooks’
‣ Modules → Using Jupyter Notebook and Python ‣ assignment and project in python
•
Using several great python libraries ‣ NLTK (basic text processing)
‣ Numpy, Scipy, Matplotlib (maths, plotting) ‣ Scikit-Learn (machine learning tools)
‣ keras, pytorch (deep learning)
Python
11
COMP90042
L1
•
Python New to Python?
‣ Expected to pick this up during the subject, on your own time
‣ Learning resources on worksheet
12
COMP90042
L1
•
•
•
Interdisciplinary study that involves linguistics, computer science and artificial intelligence.
Natural Language Processing
Aim of the study is to understand how to design algorithms to process and analyse human language data.
Closely related to computational linguistics, but computational linguistics aims to study language from a computational perspective to validate linguistic hypotheses.
13
COMP90042
L1
•
• •
•
•
Masses of information ‘trapped’ in unstructured text
Why process text?
How can we find or analyse this information?
Let computers automatically reason over this data?
First need to understand the structure, find important elements and relations, etc…
Over 1000s of languages….
14
COMP90042
L1
Talk To Transformer
15
COMP90042
L1
Why are you interested in NLP?
PollEv.com/jeyhanlau569
16
COMP90042
L1
17
COMP90042
L1
•
Intelligent conversational agent, e.g. TARS in Interstellar (2014)
‣ https://www.youtube.com/watch? v=wVEfFHzUby0
‣ Speech recognition
‣ Speech synthesis
‣ Natural language understanding
Motivating Applications (Sci-fi)
18
COMP90042
L1
Motivating Applications (Real-world)
•
•
Research behind Watson is not revolutionary
‣ But this is a transformative result in the history of AI
‣ Combines cutting-edge text processing components with large text collections and high performance computing
IBM ‘Watson’ system for Question Answering ‣ QA over large text collections
– Incorporating information extraction, and more
‣ https://www.youtube.com/watch?v=lI-M7O_bRNg
19
COMP90042
L1
20
COMP90042
L1
Course Overview
• Word,sequences,anddocuments • Textpreprocessing
• Languagemodels
• Textclassification
• Structurelearning
• Sequencetagging(e.g.part-of-speech)
• DeeplearningforNLP
• Feedforwardandrecurrentmodels
21
COMP90042
L1
Course Overview
• Semantics
• Howwordsformmeaning
• Syntax
• Howwordsarearranged
• Applications
• Machinetranslation
• Informationextraction • Questionanswering
22
COMP90042
L1
Models and Algorithms • Statemachines
‣ Formal models that consist of states, transitions between states, and input. E.g. finite-state automata.
• Formallanguagetheory
‣ Regular grammars, context-free grammars to explain
syntax
• Machinelearning
‣ Hidden Markov models for understanding sequences ‣ Logistic regressions, SVMs for classifying text
‣ Neural networks (deep learning)
23
COMP90042
L1
•
•
•
Alan Turing, famously proposed the Turing test, to assess whether a machine is intelligent
Language and Thought
The ability to process language can be seen as a litmus test for truly intelligent machines.
Because effective use of language is intertwined with our general cognitive abilities.
24
COMP90042
L1
•
•
•
Alan Turing predicted in 1950 that by 2000 a machine with 10 gigabytes of memory has 30% of fooling the human interrogator.
Are Machines Intelligent Yet?
The smartest conversational agent we have today are far away from being truly intelligent…
You can, however, build programs that can cheat the test. The ELIZA program is one such example.
25
COMP90042
L1
ELIZA
• User:Youarelikemyfatherinsomeways
• ELIZA:Whatresemblancedoyousee
• User:YouarenotveryaggressivebutIthinkyoudon’t want me to notice that
• ELIZA:WhatmakesyouthinkIamnotaggressive
• User:Youdon’targuewithme
• ELIZA:WhydoyouthinkIdon’targuewithyou
• User:Youareafraidofme
• ELIZA:DoesitpleaseyoutobelieveIamafraidofyou
26
COMP90042
L1
•
I made her duck:
‣ I cooked for her
‣ I cooked belonging to her
‣ I caused her to quickly lower her head or body
‣ I waved my magic wand and turned her into a
Challenges of Language: Ambiguity
•
Why so many possible interpretations?
27
COMP90042
L1
Challenges of Language: Ambiguity
• Duckcanmean:
‣ Noun:
‣ Verb: move head or body quickly down (e.g. to dodge something)
• Hercanbeadativepronoun(i.e.indirectobjecttoaverb)or possessive pronoun
• Makeissyntacticallyambiguous:
‣ Transitive (takes one object: duck)
‣ Ditransitive (1st object: her; 2nd object: duck)
‣ Can take a direct object and verb: object (her) is caused to perform the verbal action (duck)
28
COMP90042
L1
What are other challenges that made language processing difficult?
PollEv.com/jeyhanlau569
29
COMP90042
L1
30
COMP90042
L1
A brief history of NLP: 1950s • “ComputingMachineryandIntelligence”,AlanTuring
‣ Turing test: measure machine intelligence via a conversational test
• “SyntacticStructures”,NoamChomsky
‣ Formal language theory: uses algebra and set theory to
define formal languages as sequences of symbols
‣ Colourless green ideas sleep furiously – Sentence doesn’t make sense
– But its grammar is fine
– Highlights the difference between semantics (meaning) and syntax (sentence structure)
31
COMP90042
L1
1960-1970s ‣ Generative grammar
• Symbolicparadigm
– Discover a system of rules that generates grammatical
sentences
‣ Parsing algorithms
• Stochasticparadigm
‣ Bayesian method for optical character recognition and
authorship attribution
• Firstonlinecorpus:BrowncorpusofAmericanEnglish
‣ 1 million words, 500 documents from different genres (news, novels, etc)
32
COMP90042
L1
1970-1980s
• Stochasticparadigm
‣ Hidden Markov models, noisy channel decoding ‣ Speech recognition and synthesis
• Logic-basedparadigm
‣ More grammar systems (e.g. Lexical functional Grammar)
• Naturallanguageunderstanding
‣ Winograd’s SHRDLU
‣ Robot embedded in a toy blocks world
‣ Program takes natural language commands (move the red block to the left of the blue block)
‣ Motivates the field to study semantics and discourse
33
COMP90042
L1
1980-1990s
• Finite-statemachines
‣ Phonology, morphology and syntax
• Returnofempiricism
‣ Probabilistic models developed by IBM for speech
recognition
‣ Inspired other data-driven approaches on part-of- speech tagging, parsing, and semantics
‣ Empirical evaluation based on held-out data, quantitative metrics, and comparison with state-of- the-art
34
COMP90042
L1 1990-2000s: Rise of Machine Learning
• Bettercomputationalpower
• GraduallesseningofthedominanceofChomskyan
theories of linguistics
• Morelanguagecorporadeveloped
‣ Penn Treebank, PropBank, RSTBank, etc
‣ Corpora with various forms of syntactic, semantic
and discourse annotations
• Bettermodelsadaptedfromthemachinelearning community: support vector machines, logistic regression
35
COMP90042
L1
2000s: Deep Learning
• Emergenceofverydeepneuralnetworks(i.e.networkswith many many layers)
• Startedfromthecomputervisioncommunityforimage classification
• Advantage:usesrawdataasinput(e.g.justwordsand documents), without the need to develop hand-engineered features
• Computationallyexpensive:reliesonGPUtoscaleforlarge models and training data
• ContributedtotheAIwavewenowexperience: ‣ Home assistants and chatbots
36
COMP90042
L1
•
Are NLP problems solved?
‣ Machine translation still is far from perfect ‣ NLP models still can’t reason over text
‣ Not quite close to passing the Turing Test
–
Amazon Alexa Prize: https:// www.youtube.com/watch? v=WTGuOg7GXYU
Future of NLP
37