NICTA Copyright 2014
Semantics
Lizhen Qu
NICTA Copyright 2014
Overview of the NLP Lectures
• Introduction to natural language processing (NLP).
• Regular expressions, sentence splitting,
tokenization, part-of-speech tagging.
• Language models.
• Vector semantics.
• Parsing.
• Semantics.
– Lexical semantics.
2
NICTA Copyright 2014
Semantic Analysis
• Meaning representation: formal structures to represent the
meanings of linguistic utterances.
• Semantic analysis: the process of constructing and assigning
meaning representations to linguistic inputs.
3
I have a car.
semantic
analysis
Meaning Representations
NICTA Copyright 2014
Lexical Semantics
• Lexical semantics: linguistic study of word meaning.
• Key questions:
– What is the meaning of words?
• Most words have more than one senses.
– How are the meanings of different words related?
• Specific relations between senses.
– Vehicle is more general than car.
• Semantic fields.
– travel is related to flight.
4
NICTA Copyright 2014
Terminology
• Wordforms: any form of a word.
– good, better, best, decide, decided.
• Lexeme: an abstract representation of a word and
corresponds to a set of all its forms.
– RUN = {run, runs, ran and running}.
• Lemma: a particular grammatical form that is used to
represent a lexeme. This is often the base form.
– book, car, fly
• Word sense: a discrete representation of one aspect of
the meaning of a word.
5
NICTA Copyright 2014
Lemmatization
• Map a wordform to a lemma.
– Countries -> country
– Europe -> Europe
• Tools:
– http://nlp.stanford.edu/software/corenlp.shtml
– http://textanalysisonline.com/nltk-wordnet-lemmatizer
6
NICTA Copyright 2014
WordNet
• Very large lexical database of English.
– 117k nouns, 115k verbs, 21k adjectives, and 4k adverbs.
• Synset: the set of near-synonyms for a sense.
– 82k noun synsets, 13k verb synsets, 18k adj. synsets, 3k
adv. synsets.
– Avg. # synsets : noun 1.24, verb 2.17, adj. 1.4, adv. 1.25
• Website : http://wordnet.princeton.edu/
• WordNet in other languages: German, Italian …
7
NICTA Copyright 2014
Knowledge Bases (Open Linked Data)
8
Entity
Graph
Economic
Graph OpenIE
(Ollie,
Reverb)
(Bob_Dylan, compose,
Like_a_rolling_stone”)
(The_Dark_Night, directedBy,
Christopher_Nolan)
NICTA Copyright 2014
Overview of the NLP Lectures
• Introduction to natural language processing (NLP).
• Regular expressions, sentence splitting,
tokenization, part-of-speech tagging.
• Language models.
• Vector semantics.
• Parsing.
• Semantics.
– Lexical semantics.
– Compositional semantics.
9
NICTA Copyright 2014
Compositional Semantics
• Principle of compositionality.
• Examples:
10
The meaning of a complex expression is determined by
its structure and the meanings of its constituents.
OR
NICTA Copyright 2014
Challenges for NNs
• Variable lengths of sentences.
• Variable syntactic structures.
11
Dog bites man.
The man saw the dog with the telescope.
NICTA Copyright 2014
Recurrent Neural Networks
12
RNN
x
NICTA Copyright 2014
Recurrent Neural Networks
13
RNN
x
y
NICTA Copyright 2014
Recurrent Neural Networks
14
fW
x
y
ht = fW(ht�1,xt)
yt = g✓(ht)
old state input new state
label at time t
g✓(ht)
AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9
NICTA Copyright 2014
Vallina RNN for Word Prediction
15
h
t
= tanh(W
h
h
t�1 +Wxxt)
p(Y = y0|xAAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9
NICTA Copyright 2014
Vallina RNN for Word Prediction
16
fW
Dog
h
t
= tanh(W
h
h
t�1 +Wxxt)
p(Y = y0|xAAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==
h1
AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==
h2
AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==
h3
AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==
g✓(h1)
AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9zMpPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj29+HJv6AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9zMpPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj29+HJv6AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9zMpPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj29+HJv6AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9zMpPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj29+HJv6
g✓(h2)
AAACDHicbVDLSsNAFJ3UV62vqEs3g1Wom5IUwboruHFZwdpCW8JkOmmGTiZh5kYoIT/gxl9x40LFrR/gzr9x+hC09cCFwzn3cu89fiK4Bsf5sgorq2vrG8XN0tb2zu6evX9wp+NUUdaisYhVxyeaCS5ZCzgI1kkUI5EvWNsfXU389j1TmsfyFsYJ60dkKHnAKQEjefbJ0Mt6EYHQD7IehAxInld+hDD3slp+hj277FSdKfAyceekjOZoevZnbxDTNGISqCBad10ngX5GFHAqWF7qpZolhI7IkHUNlSRiup9Nv8nxqVEGOIiVKQl4qv6eyEik9TjyTefkTr3oTcT/vG4KQb2fcZmkwCSdLQpSgSHGk2jwgCtGQYwNIVRxcyumIVGEggmwZEJwF19eJq1a9bLq3pyXG/V5GkV0hI5RBbnoAjXQNWqiFqLoAT2hF/RqPVrP1pv1PmstWPOZQ/QH1sc3f6Kb+w==AAACDHicbVDLSsNAFJ3UV62vqEs3g1Wom5IUwboruHFZwdpCW8JkOmmGTiZh5kYoIT/gxl9x40LFrR/gzr9x+hC09cCFwzn3cu89fiK4Bsf5sgorq2vrG8XN0tb2zu6evX9wp+NUUdaisYhVxyeaCS5ZCzgI1kkUI5EvWNsfXU389j1TmsfyFsYJ60dkKHnAKQEjefbJ0Mt6EYHQD7IehAxInld+hDD3slp+hj277FSdKfAyceekjOZoevZnbxDTNGISqCBad10ngX5GFHAqWF7qpZolhI7IkHUNlSRiup9Nv8nxqVEGOIiVKQl4qv6eyEik9TjyTefkTr3oTcT/vG4KQb2fcZmkwCSdLQpSgSHGk2jwgCtGQYwNIVRxcyumIVGEggmwZEJwF19eJq1a9bLq3pyXG/V5GkV0hI5RBbnoAjXQNWqiFqLoAT2hF/RqPVrP1pv1PmstWPOZQ/QH1sc3f6Kb+w==AAACDHicbVDLSsNAFJ3UV62vqEs3g1Wom5IUwboruHFZwdpCW8JkOmmGTiZh5kYoIT/gxl9x40LFrR/gzr9x+hC09cCFwzn3cu89fiK4Bsf5sgorq2vrG8XN0tb2zu6evX9wp+NUUdaisYhVxyeaCS5ZCzgI1kkUI5EvWNsfXU389j1TmsfyFsYJ60dkKHnAKQEjefbJ0Mt6EYHQD7IehAxInld+hDD3slp+hj277FSdKfAyceekjOZoevZnbxDTNGISqCBad10ngX5GFHAqWF7qpZolhI7IkHUNlSRiup9Nv8nxqVEGOIiVKQl4qv6eyEik9TjyTefkTr3oTcT/vG4KQb2fcZmkwCSdLQpSgSHGk2jwgCtGQYwNIVRxcyumIVGEggmwZEJwF19eJq1a9bLq3pyXG/V5GkV0hI5RBbnoAjXQNWqiFqLoAT2hF/RqPVrP1pv1PmstWPOZQ/QH1sc3f6Kb+w==AAACDHicbVDLSsNAFJ3UV62vqEs3g1Wom5IUwboruHFZwdpCW8JkOmmGTiZh5kYoIT/gxl9x40LFrR/gzr9x+hC09cCFwzn3cu89fiK4Bsf5sgorq2vrG8XN0tb2zu6evX9wp+NUUdaisYhVxyeaCS5ZCzgI1kkUI5EvWNsfXU389j1TmsfyFsYJ60dkKHnAKQEjefbJ0Mt6EYHQD7IehAxInld+hDD3slp+hj277FSdKfAyceekjOZoevZnbxDTNGISqCBad10ngX5GFHAqWF7qpZolhI7IkHUNlSRiup9Nv8nxqVEGOIiVKQl4qv6eyEik9TjyTefkTr3oTcT/vG4KQb2fcZmkwCSdLQpSgSHGk2jwgCtGQYwNIVRxcyumIVGEggmwZEJwF19eJq1a9bLq3pyXG/V5GkV0hI5RBbnoAjXQNWqiFqLoAT2hF/RqPVrP1pv1PmstWPOZQ/QH1sc3f6Kb+w==
g✓(h3)
AAACDHicbVBNS8NAEN3Ur1q/oh69LFahXkqigvVW8OKxgrWFtoTNdtMu3WzC7kQoIX/Ai3/FiwcVr/4Ab/4bN20FbX0w8Hhvhpl5fiy4Bsf5sgpLyyura8X10sbm1vaOvbt3p6NEUdakkYhU2yeaCS5ZEzgI1o4VI6EvWMsfXeV+654pzSN5C+OY9UIykDzglICRPPto4KXdkMDQD9IuDBmQLKv8CMPMS8+yE+zZZafqTIAXiTsjZTRDw7M/u/2IJiGTQAXRuuM6MfRSooBTwbJSN9EsJnREBqxjqCQh07108k2Gj43Sx0GkTEnAE/X3REpCrcehbzrzO/W8l4v/eZ0Eglov5TJOgEk6XRQkAkOE82hwnytGQYwNIVRxcyumQ6IIBRNgyYTgzr+8SJqn1cuqe3NertdmaRTRATpEFeSiC1RH16iBmoiiB/SEXtCr9Wg9W2/W+7S1YM1m9tEfWB/fgSib/A==AAACDHicbVBNS8NAEN3Ur1q/oh69LFahXkqigvVW8OKxgrWFtoTNdtMu3WzC7kQoIX/Ai3/FiwcVr/4Ab/4bN20FbX0w8Hhvhpl5fiy4Bsf5sgpLyyura8X10sbm1vaOvbt3p6NEUdakkYhU2yeaCS5ZEzgI1o4VI6EvWMsfXeV+654pzSN5C+OY9UIykDzglICRPPto4KXdkMDQD9IuDBmQLKv8CMPMS8+yE+zZZafqTIAXiTsjZTRDw7M/u/2IJiGTQAXRuuM6MfRSooBTwbJSN9EsJnREBqxjqCQh07108k2Gj43Sx0GkTEnAE/X3REpCrcehbzrzO/W8l4v/eZ0Eglov5TJOgEk6XRQkAkOE82hwnytGQYwNIVRxcyumQ6IIBRNgyYTgzr+8SJqn1cuqe3NertdmaRTRATpEFeSiC1RH16iBmoiiB/SEXtCr9Wg9W2/W+7S1YM1m9tEfWB/fgSib/A==AAACDHicbVBNS8NAEN3Ur1q/oh69LFahXkqigvVW8OKxgrWFtoTNdtMu3WzC7kQoIX/Ai3/FiwcVr/4Ab/4bN20FbX0w8Hhvhpl5fiy4Bsf5sgpLyyura8X10sbm1vaOvbt3p6NEUdakkYhU2yeaCS5ZEzgI1o4VI6EvWMsfXeV+654pzSN5C+OY9UIykDzglICRPPto4KXdkMDQD9IuDBmQLKv8CMPMS8+yE+zZZafqTIAXiTsjZTRDw7M/u/2IJiGTQAXRuuM6MfRSooBTwbJSN9EsJnREBqxjqCQh07108k2Gj43Sx0GkTEnAE/X3REpCrcehbzrzO/W8l4v/eZ0Eglov5TJOgEk6XRQkAkOE82hwnytGQYwNIVRxcyumQ6IIBRNgyYTgzr+8SJqn1cuqe3NertdmaRTRATpEFeSiC1RH16iBmoiiB/SEXtCr9Wg9W2/W+7S1YM1m9tEfWB/fgSib/A==AAACDHicbVBNS8NAEN3Ur1q/oh69LFahXkqigvVW8OKxgrWFtoTNdtMu3WzC7kQoIX/Ai3/FiwcVr/4Ab/4bN20FbX0w8Hhvhpl5fiy4Bsf5sgpLyyura8X10sbm1vaOvbt3p6NEUdakkYhU2yeaCS5ZEzgI1o4VI6EvWMsfXeV+654pzSN5C+OY9UIykDzglICRPPto4KXdkMDQD9IuDBmQLKv8CMPMS8+yE+zZZafqTIAXiTsjZTRDw7M/u/2IJiGTQAXRuuM6MfRSooBTwbJSN9EsJnREBqxjqCQh07108k2Gj43Sx0GkTEnAE/X3REpCrcehbzrzO/W8l4v/eZ0Eglov5TJOgEk6XRQkAkOE82hwnytGQYwNIVRxcyumQ6IIBRNgyYTgzr+8SJqn1cuqe3NertdmaRTRATpEFeSiC1RH16iBmoiiB/SEXtCr9Wg9W2/W+7S1YM1m9tEfWB/fgSib/A==
NICTA Copyright 2014
Long Short Term Memory and GRU [1,2,5]
17
LSTM
x
y
p(Y = y0|xAAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9
NICTA Copyright 2014
Recursive Neural Networks [3,4]
18
left child right child parent
label of parent node
hp = fW(xlc,xrc)
yp = g✓(hp)
fW
xlc
yp
xrc
g✓(hp)
AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5
NICTA Copyright 2014
Recursive Neural Networks [3,4]
19
hp = fW(xlc,xrc)
yp = g✓(hp)
yp
fW
Dog bites
hdog bites
man
hdog bites man
fW
yp
g✓(hp)
AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5
g✓(hp)
AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5
NICTA Copyright 2014
Sequence Classification
20
fW
fW
Dog
fW
bites
h0
AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==
h1
AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==
h2
AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==
h3
AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==
s✓(H)
AAACCHicbVBNS8NAEN34WetX1KOXxSLUS0lEsN4KXnqsYGyhDWGz3bRLNx/sToQScvXiX/HiQcWrP8Gb/8ZNG0FbHww83pthZp6fCK7Asr6MldW19Y3NylZ1e2d3b988OLxTcSopc2gsYtnziWKCR8wBDoL1EslI6AvW9SfXhd+9Z1LxOLqFacLckIwiHnBKQEueiZWXDUICYz/IBjBmQPK8/iO08zPsmTWrYc2Al4ldkhoq0fHMz8EwpmnIIqCCKNW3rQTcjEjgVLC8OkgVSwidkBHraxqRkCk3m32S41OtDHEQS10R4Jn6eyIjoVLT0NedxY1q0SvE/7x+CkHTzXiUpMAiOl8UpAJDjItY8JBLRkFMNSFUcn0rpmMiCQUdXlWHYC++vEyc88ZVw765qLWaZRoVdIxOUB3Z6BK1UBt1kIMoekBP6AW9Go/Gs/FmvM9bV4xy5gj9gfHxDUSfmjY=AAACCHicbVBNS8NAEN34WetX1KOXxSLUS0lEsN4KXnqsYGyhDWGz3bRLNx/sToQScvXiX/HiQcWrP8Gb/8ZNG0FbHww83pthZp6fCK7Asr6MldW19Y3NylZ1e2d3b988OLxTcSopc2gsYtnziWKCR8wBDoL1EslI6AvW9SfXhd+9Z1LxOLqFacLckIwiHnBKQEueiZWXDUICYz/IBjBmQPK8/iO08zPsmTWrYc2Al4ldkhoq0fHMz8EwpmnIIqCCKNW3rQTcjEjgVLC8OkgVSwidkBHraxqRkCk3m32S41OtDHEQS10R4Jn6eyIjoVLT0NedxY1q0SvE/7x+CkHTzXiUpMAiOl8UpAJDjItY8JBLRkFMNSFUcn0rpmMiCQUdXlWHYC++vEyc88ZVw765qLWaZRoVdIxOUB3Z6BK1UBt1kIMoekBP6AW9Go/Gs/FmvM9bV4xy5gj9gfHxDUSfmjY=AAACCHicbVBNS8NAEN34WetX1KOXxSLUS0lEsN4KXnqsYGyhDWGz3bRLNx/sToQScvXiX/HiQcWrP8Gb/8ZNG0FbHww83pthZp6fCK7Asr6MldW19Y3NylZ1e2d3b988OLxTcSopc2gsYtnziWKCR8wBDoL1EslI6AvW9SfXhd+9Z1LxOLqFacLckIwiHnBKQEueiZWXDUICYz/IBjBmQPK8/iO08zPsmTWrYc2Al4ldkhoq0fHMz8EwpmnIIqCCKNW3rQTcjEjgVLC8OkgVSwidkBHraxqRkCk3m32S41OtDHEQS10R4Jn6eyIjoVLT0NedxY1q0SvE/7x+CkHTzXiUpMAiOl8UpAJDjItY8JBLRkFMNSFUcn0rpmMiCQUdXlWHYC++vEyc88ZVw765qLWaZRoVdIxOUB3Z6BK1UBt1kIMoekBP6AW9Go/Gs/FmvM9bV4xy5gj9gfHxDUSfmjY=AAACCHicbVBNS8NAEN34WetX1KOXxSLUS0lEsN4KXnqsYGyhDWGz3bRLNx/sToQScvXiX/HiQcWrP8Gb/8ZNG0FbHww83pthZp6fCK7Asr6MldW19Y3NylZ1e2d3b988OLxTcSopc2gsYtnziWKCR8wBDoL1EslI6AvW9SfXhd+9Z1LxOLqFacLckIwiHnBKQEueiZWXDUICYz/IBjBmQPK8/iO08zPsmTWrYc2Al4ldkhoq0fHMz8EwpmnIIqCCKNW3rQTcjEjgVLC8OkgVSwidkBHraxqRkCk3m32S41OtDHEQS10R4Jn6eyIjoVLT0NedxY1q0SvE/7x+CkHTzXiUpMAiOl8UpAJDjItY8JBLRkFMNSFUcn0rpmMiCQUdXlWHYC++vEyc88ZVw765qLWaZRoVdIxOUB3Z6BK1UBt1kIMoekBP6AW9Go/Gs/FmvM9bV4xy5gj9gfHxDUSfmjY=
y ˆhi = maxt=1,…,T hi,t
AAACE3icbZDLSsNAFIYnXmu9VV26GSyCYAmJCNaFUHDjskJjC00Ik+m0GTq5MHMilpCHcOOruHGh4taNO9/G6WWhrQcGPv7/HM6cP0gFV2BZ38bS8srq2nppo7y5tb2zW9nbv1NJJilzaCIS2QmIYoLHzAEOgnVSyUgUCNYOhtdjv33PpOJJ3IJRyryIDGLe55SAlvzKqRsSyMPCz3mBr7AbkQc/B012DZumiWu4VeBQuzUo/ErVMq1J4UWwZ1BFs2r6lS+3l9AsYjFQQZTq2lYKXk4kcCpYUXYzxVJCh2TAuhpjEjHl5ZOjCnyslR7uJ1K/GPBE/T2Rk0ipURTozohAqOa9sfif182gX/dyHqcZsJhOF/UzgSHB44Rwj0tGQYw0ECq5/iumIZGEgs6xrEOw509eBOfMvDTt2/Nqoz5Lo4QO0RE6QTa6QA10g5rIQRQ9omf0it6MJ+PFeDc+pq1LxmzmAP0p4/MHQKab8g==AAACE3icbZDLSsNAFIYnXmu9VV26GSyCYAmJCNaFUHDjskJjC00Ik+m0GTq5MHMilpCHcOOruHGh4taNO9/G6WWhrQcGPv7/HM6cP0gFV2BZ38bS8srq2nppo7y5tb2zW9nbv1NJJilzaCIS2QmIYoLHzAEOgnVSyUgUCNYOhtdjv33PpOJJ3IJRyryIDGLe55SAlvzKqRsSyMPCz3mBr7AbkQc/B012DZumiWu4VeBQuzUo/ErVMq1J4UWwZ1BFs2r6lS+3l9AsYjFQQZTq2lYKXk4kcCpYUXYzxVJCh2TAuhpjEjHl5ZOjCnyslR7uJ1K/GPBE/T2Rk0ipURTozohAqOa9sfif182gX/dyHqcZsJhOF/UzgSHB44Rwj0tGQYw0ECq5/iumIZGEgs6xrEOw509eBOfMvDTt2/Nqoz5Lo4QO0RE6QTa6QA10g5rIQRQ9omf0it6MJ+PFeDc+pq1LxmzmAP0p4/MHQKab8g==AAACE3icbZDLSsNAFIYnXmu9VV26GSyCYAmJCNaFUHDjskJjC00Ik+m0GTq5MHMilpCHcOOruHGh4taNO9/G6WWhrQcGPv7/HM6cP0gFV2BZ38bS8srq2nppo7y5tb2zW9nbv1NJJilzaCIS2QmIYoLHzAEOgnVSyUgUCNYOhtdjv33PpOJJ3IJRyryIDGLe55SAlvzKqRsSyMPCz3mBr7AbkQc/B012DZumiWu4VeBQuzUo/ErVMq1J4UWwZ1BFs2r6lS+3l9AsYjFQQZTq2lYKXk4kcCpYUXYzxVJCh2TAuhpjEjHl5ZOjCnyslR7uJ1K/GPBE/T2Rk0ipURTozohAqOa9sfif182gX/dyHqcZsJhOF/UzgSHB44Rwj0tGQYw0ECq5/iumIZGEgs6xrEOw509eBOfMvDTt2/Nqoz5Lo4QO0RE6QTa6QA10g5rIQRQ9omf0it6MJ+PFeDc+pq1LxmzmAP0p4/MHQKab8g==AAACE3icbZDLSsNAFIYnXmu9VV26GSyCYAmJCNaFUHDjskJjC00Ik+m0GTq5MHMilpCHcOOruHGh4taNO9/G6WWhrQcGPv7/HM6cP0gFV2BZ38bS8srq2nppo7y5tb2zW9nbv1NJJilzaCIS2QmIYoLHzAEOgnVSyUgUCNYOhtdjv33PpOJJ3IJRyryIDGLe55SAlvzKqRsSyMPCz3mBr7AbkQc/B012DZumiWu4VeBQuzUo/ErVMq1J4UWwZ1BFs2r6lS+3l9AsYjFQQZTq2lYKXk4kcCpYUXYzxVJCh2TAuhpjEjHl5ZOjCnyslR7uJ1K/GPBE/T2Rk0ipURTozohAqOa9sfif182gX/dyHqcZsJhOF/UzgSHB44Rwj0tGQYw0ECq5/iumIZGEgs6xrEOw509eBOfMvDTt2/Nqoz5Lo4QO0RE6QTa6QA10g5rIQRQ9omf0it6MJ+PFeDc+pq1LxmzmAP0p4/MHQKab8g==
ĥ = hT
AAACB3icbVDLSsNAFJ34rPUVdenCwSK4KokI1oVQcOOyQmMLTQiT6aQZOpmEmYlQQpZu/BU3LlTc+gvu/BsnbUBtPTBw5px7ufeeIGVUKsv6MpaWV1bX1msb9c2t7Z1dc2//TiaZwMTBCUtEP0CSMMqJo6hipJ8KguKAkV4wvi793j0Rkia8qyYp8WI04jSkGCkt+eaRGyGVuzFSURDmUVHAK/jz87u+2bCa1hRwkdgVaYAKHd/8dIcJzmLCFWZIyoFtpcrLkVAUM1LU3UySFOExGpGBphzFRHr59JACnmhlCMNE6McVnKq/O3IUSzmJA11Z7ijnvVL8zxtkKmx5OeVppgjHs0FhxqBKYJkKHFJBsGITTRAWVO8KcYQEwkpnV9ch2PMnLxLnrHnZtG/PG+1WlUYNHIJjcApscAHa4AZ0gAMweABP4AW8Go/Gs/FmvM9Kl4yq5wD8gfHxDXRemcg=AAACB3icbVDLSsNAFJ34rPUVdenCwSK4KokI1oVQcOOyQmMLTQiT6aQZOpmEmYlQQpZu/BU3LlTc+gvu/BsnbUBtPTBw5px7ufeeIGVUKsv6MpaWV1bX1msb9c2t7Z1dc2//TiaZwMTBCUtEP0CSMMqJo6hipJ8KguKAkV4wvi793j0Rkia8qyYp8WI04jSkGCkt+eaRGyGVuzFSURDmUVHAK/jz87u+2bCa1hRwkdgVaYAKHd/8dIcJzmLCFWZIyoFtpcrLkVAUM1LU3UySFOExGpGBphzFRHr59JACnmhlCMNE6McVnKq/O3IUSzmJA11Z7ijnvVL8zxtkKmx5OeVppgjHs0FhxqBKYJkKHFJBsGITTRAWVO8KcYQEwkpnV9ch2PMnLxLnrHnZtG/PG+1WlUYNHIJjcApscAHa4AZ0gAMweABP4AW8Go/Gs/FmvM9Kl4yq5wD8gfHxDXRemcg=AAACB3icbVDLSsNAFJ34rPUVdenCwSK4KokI1oVQcOOyQmMLTQiT6aQZOpmEmYlQQpZu/BU3LlTc+gvu/BsnbUBtPTBw5px7ufeeIGVUKsv6MpaWV1bX1msb9c2t7Z1dc2//TiaZwMTBCUtEP0CSMMqJo6hipJ8KguKAkV4wvi793j0Rkia8qyYp8WI04jSkGCkt+eaRGyGVuzFSURDmUVHAK/jz87u+2bCa1hRwkdgVaYAKHd/8dIcJzmLCFWZIyoFtpcrLkVAUM1LU3UySFOExGpGBphzFRHr59JACnmhlCMNE6McVnKq/O3IUSzmJA11Z7ijnvVL8zxtkKmx5OeVppgjHs0FhxqBKYJkKHFJBsGITTRAWVO8KcYQEwkpnV9ch2PMnLxLnrHnZtG/PG+1WlUYNHIJjcApscAHa4AZ0gAMweABP4AW8Go/Gs/FmvM9Kl4yq5wD8gfHxDXRemcg=AAACB3icbVDLSsNAFJ34rPUVdenCwSK4KokI1oVQcOOyQmMLTQiT6aQZOpmEmYlQQpZu/BU3LlTc+gvu/BsnbUBtPTBw5px7ufeeIGVUKsv6MpaWV1bX1msb9c2t7Z1dc2//TiaZwMTBCUtEP0CSMMqJo6hipJ8KguKAkV4wvi793j0Rkia8qyYp8WI04jSkGCkt+eaRGyGVuzFSURDmUVHAK/jz87u+2bCa1hRwkdgVaYAKHd/8dIcJzmLCFWZIyoFtpcrLkVAUM1LU3UySFOExGpGBphzFRHr59JACnmhlCMNE6McVnKq/O3IUSzmJA11Z7ijnvVL8zxtkKmx5OeVppgjHs0FhxqBKYJkKHFJBsGITTRAWVO8KcYQEwkpnV9ch2PMnLxLnrHnZtG/PG+1WlUYNHIJjcApscAHa4AZ0gAMweABP4AW8Go/Gs/FmvM9Kl4yq5wD8gfHxDXRemcg=
g✓(ĥ)
AAACDnicbVBNS8NAEN34WetX1KOXxaLUS0lEsN4KXjxWsLbQlLLZbpqlm03YnQgl5B948a948aDi1bM3/42btoi2Phh4vDfDzDw/EVyD43xZS8srq2vrpY3y5tb2zq69t3+n41RR1qKxiFXHJ5oJLlkLOAjWSRQjkS9Y2x9dFX77ninNY3kL44T1IjKUPOCUgJH69smwn3kRgdAPMg9CBiTPq15I4EcN8/wU9+2KU3MmwIvEnZEKmqHZtz+9QUzTiEmggmjddZ0EehlRwKlgedlLNUsIHZEh6xoqScR0L5v8k+NjowxwECtTEvBE/T2RkUjrceSbzuJIPe8V4n9eN4Wg3su4TFJgkk4XBanAEOMiHDzgilEQY0MIVdzcimlIFKFgIiybENz5lxdJ66x2WXNvziuN+iyNEjpER6iKXHSBGugaNVELUfSAntALerUerWfrzXqfti5Zs5kD9AfWxzd3oZ0XAAACDnicbVBNS8NAEN34WetX1KOXxaLUS0lEsN4KXjxWsLbQlLLZbpqlm03YnQgl5B948a948aDi1bM3/42btoi2Phh4vDfDzDw/EVyD43xZS8srq2vrpY3y5tb2zq69t3+n41RR1qKxiFXHJ5oJLlkLOAjWSRQjkS9Y2x9dFX77ninNY3kL44T1IjKUPOCUgJH69smwn3kRgdAPMg9CBiTPq15I4EcN8/wU9+2KU3MmwIvEnZEKmqHZtz+9QUzTiEmggmjddZ0EehlRwKlgedlLNUsIHZEh6xoqScR0L5v8k+NjowxwECtTEvBE/T2RkUjrceSbzuJIPe8V4n9eN4Wg3su4TFJgkk4XBanAEOMiHDzgilEQY0MIVdzcimlIFKFgIiybENz5lxdJ66x2WXNvziuN+iyNEjpER6iKXHSBGugaNVELUfSAntALerUerWfrzXqfti5Zs5kD9AfWxzd3oZ0XAAACDnicbVBNS8NAEN34WetX1KOXxaLUS0lEsN4KXjxWsLbQlLLZbpqlm03YnQgl5B948a948aDi1bM3/42btoi2Phh4vDfDzDw/EVyD43xZS8srq2vrpY3y5tb2zq69t3+n41RR1qKxiFXHJ5oJLlkLOAjWSRQjkS9Y2x9dFX77ninNY3kL44T1IjKUPOCUgJH69smwn3kRgdAPMg9CBiTPq15I4EcN8/wU9+2KU3MmwIvEnZEKmqHZtz+9QUzTiEmggmjddZ0EehlRwKlgedlLNUsIHZEh6xoqScR0L5v8k+NjowxwECtTEvBE/T2RkUjrceSbzuJIPe8V4n9eN4Wg3su4TFJgkk4XBanAEOMiHDzgilEQY0MIVdzcimlIFKFgIiybENz5lxdJ66x2WXNvziuN+iyNEjpER6iKXHSBGugaNVELUfSAntALerUerWfrzXqfti5Zs5kD9AfWxzd3oZ0XAAACDnicbVBNS8NAEN34WetX1KOXxaLUS0lEsN4KXjxWsLbQlLLZbpqlm03YnQgl5B948a948aDi1bM3/42btoi2Phh4vDfDzDw/EVyD43xZS8srq2vrpY3y5tb2zq69t3+n41RR1qKxiFXHJ5oJLlkLOAjWSRQjkS9Y2x9dFX77ninNY3kL44T1IjKUPOCUgJH69smwn3kRgdAPMg9CBiTPq15I4EcN8/wU9+2KU3MmwIvEnZEKmqHZtz+9QUzTiEmggmjddZ0EehlRwKlgedlLNUsIHZEh6xoqScR0L5v8k+NjowxwECtTEvBE/T2RkUjrceSbzuJIPe8V4n9eN4Wg3su4TFJgkk4XBanAEOMiHDzgilEQY0MIVdzcimlIFKFgIiybENz5lxdJ66x2WXNvziuN+iyNEjpER6iKXHSBGugaNVELUfSAntALerUerWfrzXqfti5Zs5kD9AfWxzd3oZ0X
NICTA Copyright 2014
REFERENCES
• [1] Sepp Hochreiter and Juergen Schmidhuber. Long short-term
memory. In Neural computation, 1997.
• [2] https://www.tensorflow.org/tutorials/recurrent .
• [3] Kai Sheng Tai, Richard Socher, and Christopher D. Manning.
Improved Semantic Representations from Tree-Structured Long
Short-Term Memory Networks. In ACL, 2015.
• [4] Irsoy, Ozan and Claire Cardie. Deep recursive neural networks
for compositionality in language. In NIPS, 2014.
• [5] Chung, Junyoung, Caglar Gulcehre, KyungHyun Cho, and
Yoshua Bengio. “Empirical evaluation of gated recurrent neural
networks on sequence modeling.” In NIPS Deep Learning and
Representation Learning Workshop, 2014.
• [6] https://github.com/aymericdamien/TensorFlow-Examples.
• [7] http://web.stanford.edu/class/cs20si/syllabus.html.
21
NICTA Copyright 2014
Learning Resources
• NLP conferences.
– ACL, EMNLP, COLING, NAACL, EACL …
• NLP online courses.
– http://web.stanford.edu/class/cs224n/
– http://www.cs.columbia.edu/~cs4705/
• ML online courses.
– https://www.coursera.org/course/ml
– https://www.cs.ox.ac.uk/people/nando.defreitas/
machinelearning/
https://www.youtube.com/watch?
v=SGZ6BttHMPw&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNm
UBH
22