Course Overview & Introduction
COMP90042
Natural Language Processing Lecture 1
COPYRIGHT 2020, THE UNIVERSITY OF MELBOURNE
1
COMP90042
L1
Prerequisites
• COMP90049“IntroductiontoMachineLearning”or
COMP30027 “Machine Learning”
‣ Modules → Welcome → Machine Learning Readings
• Pythonprogrammingexperience
• Noknowledgeoflinguisticsoradvancedmathematicsis assumed
• Caveats–Not“vanilla”computerscience
‣ Involves some basic linguistics, e.g., syntax and morphology
‣ Requires maths, e.g., algebra, optimisation, linear algebra, dynamic programming
2
COMP90042
L1
•
Expectations
‣ develop Python skills
‣ keep up with readings ‣ classroom participation
Expectations and outcomes
• Outcomes
‣ Practical familiarity with range of text analysis
technologies
‣ Understanding of theoretical models underlying these tools
‣ Competence in reading research literature
3
COMP90042
L1
Assessment: Assignments and Exam
•
•
•
•
Assignments (20% total = 6-7% each)
‣ Small activities building on workshop
‣ Released every few weeks, given 2-3 weeks to complete
Project (30% total)
‣ Released near Easter & due near end of semester
Exam (50%)
‣ two hour, closed book
‣ covers content from lectures, workshop and prescribed reading
Hurdle >50% exam, and >50% for (assignment + project)
4
COMP90042
L1
Teaching Staff
Lecturer Head Tutor
Jey Han Lau Zenan Zhai
5
COMP90042
L1
• • • • • • •
Aili Shen Biaoyan Fang Dalin Wang Fajri
Haonan Li Jun Wang Nitika Mathur
Tutors
6
COMP90042
L1
Recommended Texts • Texts:
‣ Jurafsky and Martin, Speech and Language Processing, 3rd ed., Prentice Hall. draft
‣ Eisenstein; Natural Language Processing, Draft 15/10/18 ‣ Goldberg; A Primer on Neural Network Models for
Natural Language Processing
•
•
Recommended for learning python:
‣ Steven Bird, Ewan Klein and Edward Loper, Natural
Language Processing with Python, O’Reilly, 2009
Reading links or lecture slides will be posted to Canvas
7
COMP90042
L1
8
COMP90042
L1
•
Contact hours
‣ Mon 09:00-10:00 Glyn Davis (B117)
Lectures
‣ Mon 16:15-17:15 Law GM15 (David P. Durham)
• Workshops:severalacrosstheweek
‣ Bring any questions you have to your tutors
‣ May run office hour, if there is sufficient demand
• Firstmethodofcontact—askquestionsonthe Canvas discussion board
9
COMP90042
L1
•
Making extensive use of python
‣ workshops feature programming challenges ‣ provided as interactive ‘notebooks’
‣ homework and project in python
• Usingseveralgreatpythonlibraries ‣ NLTK (text processing)
‣ Numpy, Scipy, Matplotlib (maths, plotting) ‣ Scikit-Learn (machine learning tools)
Python
10
COMP90042
L1
•
Python New to Python?
‣ Expected to pick this up during the subject, on your own time
‣ Learning resources on worksheet
11
COMP90042
L1
https://talktotransformer.com/
12
COMP90042
L1
•
•
•
Interdisciplinary study that involves linguistics, computer science and artificial intelligence.
Natural Language Processing
Aim of the study is to understand how to design algorithms to process and analyse human language data.
Closely related to computational linguistics, but computational linguistics aims to study language from a computational perspective to validate linguistic hypotheses.
13
COMP90042
L1
Why process text?
• Masses of information ‘trapped’ in unstructured text
‣ How can we find this information?
‣ Let computers automatically reason over this data?
‣ First need to understand the structure, find important elements and relations, etc…
‣ Over 1000s of languages….
• Challenges
‣ Search, displaying results
‣ Information extraction ‣ Translation
‣ Question answering ‣…
14
COMP90042
L1
•
Intelligent conversational agent, e.g. TARS in Interstellar (2014)
‣ https://www.youtube.com/watch? v=wVEfFHzUby0
‣ Speech recognition
‣ Natural language understanding
‣ Speech synthesis
Motivating Applications
15
COMP90042
L1
Motivating Applications IBM ‘Watson’ system for Question Answering
‣ QA over large text collections
– Incorporating information extraction, and more
‣ https://www.youtube.com/watch?v=FC3IryWr4c8
‣ https://www.youtube.com/watch?v=lI-M7O_bRNg
(from 3:30-4:30)
• ResearchbehindWatsonisnotrevolutionary
‣ But this is a transformative result in the history of AI
‣ Combines cutting-edge text processing components with large text collections and high performance computing
•
16
COMP90042
L1
17
COMP90042
L1
Course Overview
• Word,sequences,anddocuments • Textpreprocessing
• Languagemodels
• Textclassification
• Structurelearning
• Sequencetagging(e.g.part-of-speech)
• DeeplearningforNLP
• Feedforwardandrecurrentmodels
18
COMP90042
L1
Course Overview
• Semantics
• Howwordsformmeaning
• Syntax
• Howwordsarearranged
• Applications
• Machinetranslation
• Informationextraction • Questionanswering
19
COMP90042
L1
Models and Algorithms • Statemachines
‣ Formal models that consist of states, transitions between states, and input. E.g. finite-state automata.
• Formalrulesystems
‣ Regular grammars, context-free grammars to explain
syntax
• Machinelearning
‣ Hidden Markov models for understanding sequences ‣ Logistic regressions, SVMs for classifying text
‣ Neural networks (deep learning)
20
COMP90042
L1
•
I made her duck:
‣ I cooked for her
‣ I cooked belonging to her
‣ I caused her to quickly lower her head or body
‣ I waved my magic wand and turned her into a
Ambiguity in Language
•
Why so many possible interpretations? Language is hard!
21
COMP90042
L1
Ambiguity in Language
• Duckcanmean:
‣ Noun:
‣ Verb: move head or body quickly down (e.g. to dodge something)
• Hercanbeadativepronoun(i.e.indirectobjecttoaverb)or possessive pronoun
• Makeissyntacticallyambiguous:
‣ Transitive (takes one object: duck)
‣ Ditransitive (1st object: her; 2nd object: duck)
‣ Can take a direct object and verb: object (her) is caused to perform the verbal action (duck)
22
COMP90042
L1
•
•
•
The ability to process language can be seen as a litmus test for truly intelligent machines.
Language and Thought
Because effective use of language is intertwined with our general cognitive abilities.
Alan Turing, famously proposed the Turing test, to assess whether a machine is intelligent
23
COMP90042
L1
Turing Test
• 3participants:2humansanda computer
• Oneofthehumansisaninterrogator
Computer
Human •
Test involves a conversation between 2 parties
Interrogator (Human)
• Theroleoftheinterrogatoristo determine which participant is the machine by asking a series of questions
• Machineisintelligentifitcanfoolthe interrogator into thinking that he/she is talking to a human
24
COMP90042
L1
•
•
•
Alan Turing predicted that by the end of 20th century a machine with 10 gigabytes of memory has 30% of fooling the human interrogator.
Are Machines Intelligent Yet?
The smartest conversational agent we have today are far away from being truly intelligent…
You can, however, build programs that can cheat the test. The ELIZA program is one such example.
25
COMP90042
L1
ELIZA
• User:Youarelikemyfatherinsomeways
• ELIZA:Whatresemblancedoyousee
• User:YouarenotveryaggressivebutIthinkyoudon’t want me to notice that
• ELIZA:WhatmakesyouthinkIamnotaggressive
• User:Youdon’targuewithme
• ELIZA:WhydoyouthinkIdon’targuewithyou
• User:Youareafraidofme
• ELIZA:DoesitpleaseyoutobelieveIamafraidofyou
26
COMP90042
L1
A brief history of NLP: 1950s • “ComputingMachineryandIntelligence”,AlanTuring
‣ Turing test: measure machine intelligence via a conversational test
• “SyntacticStructures”,NoamChomsky
‣ Formal language theory: uses algebra and set theory to
define formal languages as sequences of symbols
‣ Colourless green ideas sleep furiously – Sentence doesn’t make sense
– But its grammar seems fine
– Highlights the difference between semantics (meaning) and syntax (sentence structure)
27
COMP90042
L1
1960-1970s ‣ Generative grammar
• Symbolicparadigm
– Discover a system of rules that generates grammatical
sentences
‣ Parsing algorithms
• Stochasticparadigm
‣ Bayesian method for optical character recognition and
authorship attribution
• Firstonlinecorpus:BrowncorpusofAmericanEnglish
‣ 1 million words, 500 documents from different genres (news, novels, etc)
28
COMP90042
L1
1970-1980s
• Stochasticparadigm
‣ Hidden Markov models, noisy channel decoding ‣ Speech recognition and synthesis
• Logic-basedparadigm
‣ More grammar systems (e.g. Lexical functional Grammar)
• Naturallanguageunderstanding
‣ Winograd’s SHRDLU
‣ Robot embedded in a toy blocks world
‣ Program takes natural language commands (move the red block to the left of the blue block)
‣ Motivates the field to study semantics and discourse
29
COMP90042
L1
1980-1990s
• Finite-statemachines
‣ Phonology, morphology and syntax
• Returnofempiricism
‣ Probabilistic models developed by IBM for speech
recognition
‣ Inspired other data-driven approaches on part-of- speech tagging, parsing, and semantics
‣ Empirical evaluation based on held-out data, quantitative metrics, and comparison with state-of- the-art
30
COMP90042
L1 1990-2000s: Rise of Machine Learning
• Bettercomputationalpower
• GraduallesseningofthedominanceofChomskyan
theories of linguistics
• Morelanguagecorporadeveloped
‣ Penn Treebank, PropBank, RSTBank, etc
‣ Corpora with various forms of syntactic, semantic
and discourse annotations
• Bettermodelsadaptedfromthemachinelearning community: support vector machines, logistic regression
31
COMP90042
L1
2000s: Deep Learning
• Emergenceofverydeepneuralnetworks(i.e.networkswith many many layers)
• Startedfromthecomputervisioncommunityforimage classification
• Advantage:usesrawdataasinput(e.g.justwordsand documents), without the need to develop hand-engineered features
• Computationallyexpensive:reliesonGPUtoscaleforlarge models and training data
• ContributedtotheAIwavewenowexperience: ‣ Home assistants and chatbots
32
COMP90042
L1
•
Are NLP problems solved?
‣ Machine translation still is far from perfect ‣ NLP models still can’t reason over text
‣ Not quite close to passing the Turing Test
–
Amazon Alexa Prize: https:// www.youtube.com/watch? v=WTGuOg7GXYU
Future of NLP
33