程序代写代做代考 database deep learning NICTA Copyright 2014

NICTA Copyright 2014

Semantics

Lizhen Qu

NICTA Copyright 2014

Overview of the NLP Lectures

•  Introduction to natural language processing (NLP).

•  Regular expressions, sentence splitting,
tokenization, part-of-speech tagging.

•  Language models.

•  Vector semantics.

•  Parsing.

•  Semantics.
–  Lexical semantics.
2

NICTA Copyright 2014

Semantic Analysis
•  Meaning representation: formal structures to represent the

meanings of linguistic utterances.
•  Semantic analysis: the process of constructing and assigning

meaning representations to linguistic inputs.

3

I have a car.

semantic
analysis

Meaning Representations

NICTA Copyright 2014

Lexical Semantics

•  Lexical semantics: linguistic study of word meaning.

•  Key questions:

–  What is the meaning of words?
•  Most words have more than one senses.

–  How are the meanings of different words related?
•  Specific relations between senses.

–  Vehicle is more general than car.
•  Semantic fields.

–  travel is related to flight.

4

NICTA Copyright 2014

Terminology

•  Wordforms: any form of a word.
–  good, better, best, decide, decided.

•  Lexeme: an abstract representation of a word and
corresponds to a set of all its forms.
–  RUN = {run, runs, ran and running}.

•  Lemma: a particular grammatical form that is used to
represent a lexeme. This is often the base form.
–  book, car, fly

•  Word sense: a discrete representation of one aspect of
the meaning of a word.

5

NICTA Copyright 2014

Lemmatization

•  Map a wordform to a lemma.
–  Countries -> country
–  Europe -> Europe

•  Tools:
–  http://nlp.stanford.edu/software/corenlp.shtml
–  http://textanalysisonline.com/nltk-wordnet-lemmatizer

6

NICTA Copyright 2014

WordNet

•  Very large lexical database of English.
–  117k nouns, 115k verbs, 21k adjectives, and 4k adverbs.

•  Synset: the set of near-synonyms for a sense.
–  82k noun synsets, 13k verb synsets, 18k adj. synsets, 3k

adv. synsets.
–  Avg. # synsets : noun 1.24, verb 2.17, adj. 1.4, adv. 1.25

•  Website : http://wordnet.princeton.edu/
•  WordNet in other languages: German, Italian …

7

NICTA Copyright 2014

Knowledge Bases (Open Linked Data)

8

Entity
Graph

Economic
Graph OpenIE

(Ollie,
Reverb)

(Bob_Dylan, compose,
Like_a_rolling_stone”)

(The_Dark_Night, directedBy,
Christopher_Nolan)

NICTA Copyright 2014

Overview of the NLP Lectures

•  Introduction to natural language processing (NLP).

•  Regular expressions, sentence splitting,
tokenization, part-of-speech tagging.

•  Language models.

•  Vector semantics.

•  Parsing.

•  Semantics.
–  Lexical semantics.
–  Compositional semantics.

9

NICTA Copyright 2014

Compositional Semantics

•  Principle of compositionality.

•  Examples:

10

The meaning of a complex expression is determined by
its structure and the meanings of its constituents.

OR

NICTA Copyright 2014

Challenges for NNs

•  Variable lengths of sentences.

•  Variable syntactic structures.

11

Dog bites man.

The man saw the dog with the telescope.

NICTA Copyright 2014

Recurrent Neural Networks

12

RNN

x

NICTA Copyright 2014

Recurrent Neural Networks

13

RNN

x

y

NICTA Copyright 2014

Recurrent Neural Networks

14

fW

x

y
ht = fW(ht�1,xt)

yt = g✓(ht)

old state input new state

label at time t

g✓(ht)
AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9

NICTA Copyright 2014

Vallina RNN for Word Prediction

15

h

t

= tanh(W
h

h

t�1 +Wxxt)

p(Y = y0|xAAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9

NICTA Copyright 2014

Vallina RNN for Word Prediction

16

fW

Dog

h

t

= tanh(W
h

h

t�1 +Wxxt)

p(Y = y0|xAAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==

h1
AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==

h2
AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==

h3
AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==

g✓(h1)
AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9zMpPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj29+HJv6AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9zMpPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj29+HJv6AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9zMpPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj29+HJv6AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9zMpPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj29+HJv6

g✓(h2)
AAACDHicbVDLSsNAFJ3UV62vqEs3g1Wom5IUwboruHFZwdpCW8JkOmmGTiZh5kYoIT/gxl9x40LFrR/gzr9x+hC09cCFwzn3cu89fiK4Bsf5sgorq2vrG8XN0tb2zu6evX9wp+NUUdaisYhVxyeaCS5ZCzgI1kkUI5EvWNsfXU389j1TmsfyFsYJ60dkKHnAKQEjefbJ0Mt6EYHQD7IehAxInld+hDD3slp+hj277FSdKfAyceekjOZoevZnbxDTNGISqCBad10ngX5GFHAqWF7qpZolhI7IkHUNlSRiup9Nv8nxqVEGOIiVKQl4qv6eyEik9TjyTefkTr3oTcT/vG4KQb2fcZmkwCSdLQpSgSHGk2jwgCtGQYwNIVRxcyumIVGEggmwZEJwF19eJq1a9bLq3pyXG/V5GkV0hI5RBbnoAjXQNWqiFqLoAT2hF/RqPVrP1pv1PmstWPOZQ/QH1sc3f6Kb+w==AAACDHicbVDLSsNAFJ3UV62vqEs3g1Wom5IUwboruHFZwdpCW8JkOmmGTiZh5kYoIT/gxl9x40LFrR/gzr9x+hC09cCFwzn3cu89fiK4Bsf5sgorq2vrG8XN0tb2zu6evX9wp+NUUdaisYhVxyeaCS5ZCzgI1kkUI5EvWNsfXU389j1TmsfyFsYJ60dkKHnAKQEjefbJ0Mt6EYHQD7IehAxInld+hDD3slp+hj277FSdKfAyceekjOZoevZnbxDTNGISqCBad10ngX5GFHAqWF7qpZolhI7IkHUNlSRiup9Nv8nxqVEGOIiVKQl4qv6eyEik9TjyTefkTr3oTcT/vG4KQb2fcZmkwCSdLQpSgSHGk2jwgCtGQYwNIVRxcyumIVGEggmwZEJwF19eJq1a9bLq3pyXG/V5GkV0hI5RBbnoAjXQNWqiFqLoAT2hF/RqPVrP1pv1PmstWPOZQ/QH1sc3f6Kb+w==AAACDHicbVDLSsNAFJ3UV62vqEs3g1Wom5IUwboruHFZwdpCW8JkOmmGTiZh5kYoIT/gxl9x40LFrR/gzr9x+hC09cCFwzn3cu89fiK4Bsf5sgorq2vrG8XN0tb2zu6evX9wp+NUUdaisYhVxyeaCS5ZCzgI1kkUI5EvWNsfXU389j1TmsfyFsYJ60dkKHnAKQEjefbJ0Mt6EYHQD7IehAxInld+hDD3slp+hj277FSdKfAyceekjOZoevZnbxDTNGISqCBad10ngX5GFHAqWF7qpZolhI7IkHUNlSRiup9Nv8nxqVEGOIiVKQl4qv6eyEik9TjyTefkTr3oTcT/vG4KQb2fcZmkwCSdLQpSgSHGk2jwgCtGQYwNIVRxcyumIVGEggmwZEJwF19eJq1a9bLq3pyXG/V5GkV0hI5RBbnoAjXQNWqiFqLoAT2hF/RqPVrP1pv1PmstWPOZQ/QH1sc3f6Kb+w==AAACDHicbVDLSsNAFJ3UV62vqEs3g1Wom5IUwboruHFZwdpCW8JkOmmGTiZh5kYoIT/gxl9x40LFrR/gzr9x+hC09cCFwzn3cu89fiK4Bsf5sgorq2vrG8XN0tb2zu6evX9wp+NUUdaisYhVxyeaCS5ZCzgI1kkUI5EvWNsfXU389j1TmsfyFsYJ60dkKHnAKQEjefbJ0Mt6EYHQD7IehAxInld+hDD3slp+hj277FSdKfAyceekjOZoevZnbxDTNGISqCBad10ngX5GFHAqWF7qpZolhI7IkHUNlSRiup9Nv8nxqVEGOIiVKQl4qv6eyEik9TjyTefkTr3oTcT/vG4KQb2fcZmkwCSdLQpSgSHGk2jwgCtGQYwNIVRxcyumIVGEggmwZEJwF19eJq1a9bLq3pyXG/V5GkV0hI5RBbnoAjXQNWqiFqLoAT2hF/RqPVrP1pv1PmstWPOZQ/QH1sc3f6Kb+w==

g✓(h3)
AAACDHicbVBNS8NAEN3Ur1q/oh69LFahXkqigvVW8OKxgrWFtoTNdtMu3WzC7kQoIX/Ai3/FiwcVr/4Ab/4bN20FbX0w8Hhvhpl5fiy4Bsf5sgpLyyura8X10sbm1vaOvbt3p6NEUdakkYhU2yeaCS5ZEzgI1o4VI6EvWMsfXeV+654pzSN5C+OY9UIykDzglICRPPto4KXdkMDQD9IuDBmQLKv8CMPMS8+yE+zZZafqTIAXiTsjZTRDw7M/u/2IJiGTQAXRuuM6MfRSooBTwbJSN9EsJnREBqxjqCQh07108k2Gj43Sx0GkTEnAE/X3REpCrcehbzrzO/W8l4v/eZ0Eglov5TJOgEk6XRQkAkOE82hwnytGQYwNIVRxcyumQ6IIBRNgyYTgzr+8SJqn1cuqe3NertdmaRTRATpEFeSiC1RH16iBmoiiB/SEXtCr9Wg9W2/W+7S1YM1m9tEfWB/fgSib/A==AAACDHicbVBNS8NAEN3Ur1q/oh69LFahXkqigvVW8OKxgrWFtoTNdtMu3WzC7kQoIX/Ai3/FiwcVr/4Ab/4bN20FbX0w8Hhvhpl5fiy4Bsf5sgpLyyura8X10sbm1vaOvbt3p6NEUdakkYhU2yeaCS5ZEzgI1o4VI6EvWMsfXeV+654pzSN5C+OY9UIykDzglICRPPto4KXdkMDQD9IuDBmQLKv8CMPMS8+yE+zZZafqTIAXiTsjZTRDw7M/u/2IJiGTQAXRuuM6MfRSooBTwbJSN9EsJnREBqxjqCQh07108k2Gj43Sx0GkTEnAE/X3REpCrcehbzrzO/W8l4v/eZ0Eglov5TJOgEk6XRQkAkOE82hwnytGQYwNIVRxcyumQ6IIBRNgyYTgzr+8SJqn1cuqe3NertdmaRTRATpEFeSiC1RH16iBmoiiB/SEXtCr9Wg9W2/W+7S1YM1m9tEfWB/fgSib/A==AAACDHicbVBNS8NAEN3Ur1q/oh69LFahXkqigvVW8OKxgrWFtoTNdtMu3WzC7kQoIX/Ai3/FiwcVr/4Ab/4bN20FbX0w8Hhvhpl5fiy4Bsf5sgpLyyura8X10sbm1vaOvbt3p6NEUdakkYhU2yeaCS5ZEzgI1o4VI6EvWMsfXeV+654pzSN5C+OY9UIykDzglICRPPto4KXdkMDQD9IuDBmQLKv8CMPMS8+yE+zZZafqTIAXiTsjZTRDw7M/u/2IJiGTQAXRuuM6MfRSooBTwbJSN9EsJnREBqxjqCQh07108k2Gj43Sx0GkTEnAE/X3REpCrcehbzrzO/W8l4v/eZ0Eglov5TJOgEk6XRQkAkOE82hwnytGQYwNIVRxcyumQ6IIBRNgyYTgzr+8SJqn1cuqe3NertdmaRTRATpEFeSiC1RH16iBmoiiB/SEXtCr9Wg9W2/W+7S1YM1m9tEfWB/fgSib/A==AAACDHicbVBNS8NAEN3Ur1q/oh69LFahXkqigvVW8OKxgrWFtoTNdtMu3WzC7kQoIX/Ai3/FiwcVr/4Ab/4bN20FbX0w8Hhvhpl5fiy4Bsf5sgpLyyura8X10sbm1vaOvbt3p6NEUdakkYhU2yeaCS5ZEzgI1o4VI6EvWMsfXeV+654pzSN5C+OY9UIykDzglICRPPto4KXdkMDQD9IuDBmQLKv8CMPMS8+yE+zZZafqTIAXiTsjZTRDw7M/u/2IJiGTQAXRuuM6MfRSooBTwbJSN9EsJnREBqxjqCQh07108k2Gj43Sx0GkTEnAE/X3REpCrcehbzrzO/W8l4v/eZ0Eglov5TJOgEk6XRQkAkOE82hwnytGQYwNIVRxcyumQ6IIBRNgyYTgzr+8SJqn1cuqe3NertdmaRTRATpEFeSiC1RH16iBmoiiB/SEXtCr9Wg9W2/W+7S1YM1m9tEfWB/fgSib/A==

NICTA Copyright 2014

Long Short Term Memory and GRU [1,2,5]

17

LSTM

x

y

p(Y = y0|xAAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9DPJT7NXqZtOcAi8SqyR1VKLj1T6dQUzTkEVABVGqb5kJuBmRwKlgedVJFUsIHZMh62sakZApN5t+k+MTrQxwEEtdEeCp+nsiI6FSk9DXncWdat4rxP+8fgpBy814lKTAIjpbFKQCQ4yLaPCAS0ZBTDQhVHJ9K6YjIgkFHWBVh2DNv7xI7LPmZdO6Oa+3W2UaFXSIjlADWegCtdE16iAbUfSAntALejUejWfjzXiftS4Z5cwB+gPj4xvkLpw9

NICTA Copyright 2014

Recursive Neural Networks [3,4]

18

left child right child parent

label of parent node

hp = fW(xlc,xrc)

yp = g✓(hp)
fW

xlc

yp

xrc

g✓(hp)
AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5

NICTA Copyright 2014

Recursive Neural Networks [3,4]

19

hp = fW(xlc,xrc)
yp = g✓(hp)

yp

fW

Dog bites

hdog bites

man

hdog bites man

fW

yp
g✓(hp)

AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5

g✓(hp)
AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5AAACDHicbVDLSsNAFJ34rPVVdelmsAp1UxIRrLuCG5cVjC00IUymk3bo5MHMjVBCfsCNv+LGhYpbP8Cdf+OkjaCtBy4czrmXe+/xE8EVmOaXsbS8srq2Xtmobm5t7+zW9vbvVJxKymwai1j2fKKY4BGzgYNgvUQyEvqCdf3xVeF375lUPI5uYZIwNyTDiAecEtCSVzseepkTEhj5QebAiAHJ88aPMMq9LMlPsVerm01zCrxIrJLUUYmOV/t0BjFNQxYBFUSpvmUm4GZEAqeC5VUnVSwhdEyGrK9pREKm3Gz6TY5PtDLAQSx1RYCn6u+JjIRKTUJfdxZ3qnmvEP/z+ikELTfjUZICi+hsUZAKDDEuosEDLhkFMdGEUMn1rZiOiCQUdIBVHYI1//Iisc+al03r5rzebpVpVNAhOkINZKEL1EbXqINsRNEDekIv6NV4NJ6NN+N91rpklDMH6A+Mj2/eFpw5

NICTA Copyright 2014

Sequence Classification

20

fW

fW

Dog

fW

bites

h0
AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G7qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDd3GReQ==

h1
AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIYL0VvHisYLTQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDtg4sDDPv8WYnTKUw6LrfTmVtfWNzq7pd29nd2z+oHx49mCTTjPsskYnuhtRwKRT3UaDk3VRzGoeSP4aTm8J/fOLaiETd4zTl/ZiOlIgEo2ilIIgpjsMoH88G3qDecJvuHGSVeCVpQInOoP4VDBOWxVwhk9SYnuem2M+pRsEkn9WCzPCUsgkd8Z6lisbc9PN55hk5s8qQRIm2TyGZq783chobM41DO1lkNMteIf7n9TKMWv1cqDRDrtjiUJRJggkpCiBDoTlDObWEMi1sVsLGVFOGtqaaLcFb/vIq8S+a103v7rLRbpVtVOEETuEcPLiCNtxCB3xgkMIzvMKbkzkvzrvzsRitOOXOMfyB8/kDePSReg==

h2
AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiRFsO4KblxWMLbQhDKZTtqhk0mYh1BCf8ONCxW3fo07/8ZJm4W2Hhg4nHMv98yJMs6Udt1vp7KxubW9U92t7e0fHB7Vj08eVWokoT5JeSr7EVaUM0F9zTSn/UxSnESc9qLpbeH3nqhULBUPepbRMMFjwWJGsLZSECRYT6I4n8yHrWG94TbdBdA68UrSgBLdYf0rGKXEJFRowrFSA8/NdJhjqRnhdF4LjKIZJlM8pgNLBU6oCvNF5jm6sMoIxam0T2i0UH9v5DhRapZEdrLIqFa9QvzPGxgdt8OcicxoKsjyUGw40ikqCkAjJinRfGYJJpLZrIhMsMRE25pqtgRv9cvrxG81b5re/VWj0y7bqMIZnMMleHANHbiDLvhAIINneIU3xzgvzrvzsRytOOXOKfyB8/kDeneRew==

h3
AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQqWHcFNy4rGFtoQplMJ+3QySTMQyihv+HGhYpbv8adf+OkzUJbDwwczrmXe+ZEGWdKu+63U1lb39jcqm7Xdnb39g/qh0ePKjWSUJ+kPJW9CCvKmaC+ZprTXiYpTiJOu9HktvC7T1QqlooHPc1omOCRYDEjWFspCBKsx1Gcj2eDy0G94TbdOdAq8UrSgBKdQf0rGKbEJFRowrFSfc/NdJhjqRnhdFYLjKIZJhM8on1LBU6oCvN55hk6s8oQxam0T2g0V39v5DhRappEdrLIqJa9QvzP6xsdt8KcicxoKsjiUGw40ikqCkBDJinRfGoJJpLZrIiMscRE25pqtgRv+curxL9o3jS9+6tGu1W2UYUTOIVz8OAa2nAHHfCBQAbP8ApvjnFenHfnYzFaccqdY/gD5/MHe/qRfA==

s✓(H)
AAACCHicbVBNS8NAEN34WetX1KOXxSLUS0lEsN4KXnqsYGyhDWGz3bRLNx/sToQScvXiX/HiQcWrP8Gb/8ZNG0FbHww83pthZp6fCK7Asr6MldW19Y3NylZ1e2d3b988OLxTcSopc2gsYtnziWKCR8wBDoL1EslI6AvW9SfXhd+9Z1LxOLqFacLckIwiHnBKQEueiZWXDUICYz/IBjBmQPK8/iO08zPsmTWrYc2Al4ldkhoq0fHMz8EwpmnIIqCCKNW3rQTcjEjgVLC8OkgVSwidkBHraxqRkCk3m32S41OtDHEQS10R4Jn6eyIjoVLT0NedxY1q0SvE/7x+CkHTzXiUpMAiOl8UpAJDjItY8JBLRkFMNSFUcn0rpmMiCQUdXlWHYC++vEyc88ZVw765qLWaZRoVdIxOUB3Z6BK1UBt1kIMoekBP6AW9Go/Gs/FmvM9bV4xy5gj9gfHxDUSfmjY=AAACCHicbVBNS8NAEN34WetX1KOXxSLUS0lEsN4KXnqsYGyhDWGz3bRLNx/sToQScvXiX/HiQcWrP8Gb/8ZNG0FbHww83pthZp6fCK7Asr6MldW19Y3NylZ1e2d3b988OLxTcSopc2gsYtnziWKCR8wBDoL1EslI6AvW9SfXhd+9Z1LxOLqFacLckIwiHnBKQEueiZWXDUICYz/IBjBmQPK8/iO08zPsmTWrYc2Al4ldkhoq0fHMz8EwpmnIIqCCKNW3rQTcjEjgVLC8OkgVSwidkBHraxqRkCk3m32S41OtDHEQS10R4Jn6eyIjoVLT0NedxY1q0SvE/7x+CkHTzXiUpMAiOl8UpAJDjItY8JBLRkFMNSFUcn0rpmMiCQUdXlWHYC++vEyc88ZVw765qLWaZRoVdIxOUB3Z6BK1UBt1kIMoekBP6AW9Go/Gs/FmvM9bV4xy5gj9gfHxDUSfmjY=AAACCHicbVBNS8NAEN34WetX1KOXxSLUS0lEsN4KXnqsYGyhDWGz3bRLNx/sToQScvXiX/HiQcWrP8Gb/8ZNG0FbHww83pthZp6fCK7Asr6MldW19Y3NylZ1e2d3b988OLxTcSopc2gsYtnziWKCR8wBDoL1EslI6AvW9SfXhd+9Z1LxOLqFacLckIwiHnBKQEueiZWXDUICYz/IBjBmQPK8/iO08zPsmTWrYc2Al4ldkhoq0fHMz8EwpmnIIqCCKNW3rQTcjEjgVLC8OkgVSwidkBHraxqRkCk3m32S41OtDHEQS10R4Jn6eyIjoVLT0NedxY1q0SvE/7x+CkHTzXiUpMAiOl8UpAJDjItY8JBLRkFMNSFUcn0rpmMiCQUdXlWHYC++vEyc88ZVw765qLWaZRoVdIxOUB3Z6BK1UBt1kIMoekBP6AW9Go/Gs/FmvM9bV4xy5gj9gfHxDUSfmjY=AAACCHicbVBNS8NAEN34WetX1KOXxSLUS0lEsN4KXnqsYGyhDWGz3bRLNx/sToQScvXiX/HiQcWrP8Gb/8ZNG0FbHww83pthZp6fCK7Asr6MldW19Y3NylZ1e2d3b988OLxTcSopc2gsYtnziWKCR8wBDoL1EslI6AvW9SfXhd+9Z1LxOLqFacLckIwiHnBKQEueiZWXDUICYz/IBjBmQPK8/iO08zPsmTWrYc2Al4ldkhoq0fHMz8EwpmnIIqCCKNW3rQTcjEjgVLC8OkgVSwidkBHraxqRkCk3m32S41OtDHEQS10R4Jn6eyIjoVLT0NedxY1q0SvE/7x+CkHTzXiUpMAiOl8UpAJDjItY8JBLRkFMNSFUcn0rpmMiCQUdXlWHYC++vEyc88ZVw765qLWaZRoVdIxOUB3Z6BK1UBt1kIMoekBP6AW9Go/Gs/FmvM9bV4xy5gj9gfHxDUSfmjY=

y ˆhi = maxt=1,…,T hi,t
AAACE3icbZDLSsNAFIYnXmu9VV26GSyCYAmJCNaFUHDjskJjC00Ik+m0GTq5MHMilpCHcOOruHGh4taNO9/G6WWhrQcGPv7/HM6cP0gFV2BZ38bS8srq2nppo7y5tb2zW9nbv1NJJilzaCIS2QmIYoLHzAEOgnVSyUgUCNYOhtdjv33PpOJJ3IJRyryIDGLe55SAlvzKqRsSyMPCz3mBr7AbkQc/B012DZumiWu4VeBQuzUo/ErVMq1J4UWwZ1BFs2r6lS+3l9AsYjFQQZTq2lYKXk4kcCpYUXYzxVJCh2TAuhpjEjHl5ZOjCnyslR7uJ1K/GPBE/T2Rk0ipURTozohAqOa9sfif182gX/dyHqcZsJhOF/UzgSHB44Rwj0tGQYw0ECq5/iumIZGEgs6xrEOw509eBOfMvDTt2/Nqoz5Lo4QO0RE6QTa6QA10g5rIQRQ9omf0it6MJ+PFeDc+pq1LxmzmAP0p4/MHQKab8g==AAACE3icbZDLSsNAFIYnXmu9VV26GSyCYAmJCNaFUHDjskJjC00Ik+m0GTq5MHMilpCHcOOruHGh4taNO9/G6WWhrQcGPv7/HM6cP0gFV2BZ38bS8srq2nppo7y5tb2zW9nbv1NJJilzaCIS2QmIYoLHzAEOgnVSyUgUCNYOhtdjv33PpOJJ3IJRyryIDGLe55SAlvzKqRsSyMPCz3mBr7AbkQc/B012DZumiWu4VeBQuzUo/ErVMq1J4UWwZ1BFs2r6lS+3l9AsYjFQQZTq2lYKXk4kcCpYUXYzxVJCh2TAuhpjEjHl5ZOjCnyslR7uJ1K/GPBE/T2Rk0ipURTozohAqOa9sfif182gX/dyHqcZsJhOF/UzgSHB44Rwj0tGQYw0ECq5/iumIZGEgs6xrEOw509eBOfMvDTt2/Nqoz5Lo4QO0RE6QTa6QA10g5rIQRQ9omf0it6MJ+PFeDc+pq1LxmzmAP0p4/MHQKab8g==AAACE3icbZDLSsNAFIYnXmu9VV26GSyCYAmJCNaFUHDjskJjC00Ik+m0GTq5MHMilpCHcOOruHGh4taNO9/G6WWhrQcGPv7/HM6cP0gFV2BZ38bS8srq2nppo7y5tb2zW9nbv1NJJilzaCIS2QmIYoLHzAEOgnVSyUgUCNYOhtdjv33PpOJJ3IJRyryIDGLe55SAlvzKqRsSyMPCz3mBr7AbkQc/B012DZumiWu4VeBQuzUo/ErVMq1J4UWwZ1BFs2r6lS+3l9AsYjFQQZTq2lYKXk4kcCpYUXYzxVJCh2TAuhpjEjHl5ZOjCnyslR7uJ1K/GPBE/T2Rk0ipURTozohAqOa9sfif182gX/dyHqcZsJhOF/UzgSHB44Rwj0tGQYw0ECq5/iumIZGEgs6xrEOw509eBOfMvDTt2/Nqoz5Lo4QO0RE6QTa6QA10g5rIQRQ9omf0it6MJ+PFeDc+pq1LxmzmAP0p4/MHQKab8g==AAACE3icbZDLSsNAFIYnXmu9VV26GSyCYAmJCNaFUHDjskJjC00Ik+m0GTq5MHMilpCHcOOruHGh4taNO9/G6WWhrQcGPv7/HM6cP0gFV2BZ38bS8srq2nppo7y5tb2zW9nbv1NJJilzaCIS2QmIYoLHzAEOgnVSyUgUCNYOhtdjv33PpOJJ3IJRyryIDGLe55SAlvzKqRsSyMPCz3mBr7AbkQc/B012DZumiWu4VeBQuzUo/ErVMq1J4UWwZ1BFs2r6lS+3l9AsYjFQQZTq2lYKXk4kcCpYUXYzxVJCh2TAuhpjEjHl5ZOjCnyslR7uJ1K/GPBE/T2Rk0ipURTozohAqOa9sfif182gX/dyHqcZsJhOF/UzgSHB44Rwj0tGQYw0ECq5/iumIZGEgs6xrEOw509eBOfMvDTt2/Nqoz5Lo4QO0RE6QTa6QA10g5rIQRQ9omf0it6MJ+PFeDc+pq1LxmzmAP0p4/MHQKab8g==

ĥ = hT
AAACB3icbVDLSsNAFJ34rPUVdenCwSK4KokI1oVQcOOyQmMLTQiT6aQZOpmEmYlQQpZu/BU3LlTc+gvu/BsnbUBtPTBw5px7ufeeIGVUKsv6MpaWV1bX1msb9c2t7Z1dc2//TiaZwMTBCUtEP0CSMMqJo6hipJ8KguKAkV4wvi793j0Rkia8qyYp8WI04jSkGCkt+eaRGyGVuzFSURDmUVHAK/jz87u+2bCa1hRwkdgVaYAKHd/8dIcJzmLCFWZIyoFtpcrLkVAUM1LU3UySFOExGpGBphzFRHr59JACnmhlCMNE6McVnKq/O3IUSzmJA11Z7ijnvVL8zxtkKmx5OeVppgjHs0FhxqBKYJkKHFJBsGITTRAWVO8KcYQEwkpnV9ch2PMnLxLnrHnZtG/PG+1WlUYNHIJjcApscAHa4AZ0gAMweABP4AW8Go/Gs/FmvM9Kl4yq5wD8gfHxDXRemcg=AAACB3icbVDLSsNAFJ34rPUVdenCwSK4KokI1oVQcOOyQmMLTQiT6aQZOpmEmYlQQpZu/BU3LlTc+gvu/BsnbUBtPTBw5px7ufeeIGVUKsv6MpaWV1bX1msb9c2t7Z1dc2//TiaZwMTBCUtEP0CSMMqJo6hipJ8KguKAkV4wvi793j0Rkia8qyYp8WI04jSkGCkt+eaRGyGVuzFSURDmUVHAK/jz87u+2bCa1hRwkdgVaYAKHd/8dIcJzmLCFWZIyoFtpcrLkVAUM1LU3UySFOExGpGBphzFRHr59JACnmhlCMNE6McVnKq/O3IUSzmJA11Z7ijnvVL8zxtkKmx5OeVppgjHs0FhxqBKYJkKHFJBsGITTRAWVO8KcYQEwkpnV9ch2PMnLxLnrHnZtG/PG+1WlUYNHIJjcApscAHa4AZ0gAMweABP4AW8Go/Gs/FmvM9Kl4yq5wD8gfHxDXRemcg=AAACB3icbVDLSsNAFJ34rPUVdenCwSK4KokI1oVQcOOyQmMLTQiT6aQZOpmEmYlQQpZu/BU3LlTc+gvu/BsnbUBtPTBw5px7ufeeIGVUKsv6MpaWV1bX1msb9c2t7Z1dc2//TiaZwMTBCUtEP0CSMMqJo6hipJ8KguKAkV4wvi793j0Rkia8qyYp8WI04jSkGCkt+eaRGyGVuzFSURDmUVHAK/jz87u+2bCa1hRwkdgVaYAKHd/8dIcJzmLCFWZIyoFtpcrLkVAUM1LU3UySFOExGpGBphzFRHr59JACnmhlCMNE6McVnKq/O3IUSzmJA11Z7ijnvVL8zxtkKmx5OeVppgjHs0FhxqBKYJkKHFJBsGITTRAWVO8KcYQEwkpnV9ch2PMnLxLnrHnZtG/PG+1WlUYNHIJjcApscAHa4AZ0gAMweABP4AW8Go/Gs/FmvM9Kl4yq5wD8gfHxDXRemcg=AAACB3icbVDLSsNAFJ34rPUVdenCwSK4KokI1oVQcOOyQmMLTQiT6aQZOpmEmYlQQpZu/BU3LlTc+gvu/BsnbUBtPTBw5px7ufeeIGVUKsv6MpaWV1bX1msb9c2t7Z1dc2//TiaZwMTBCUtEP0CSMMqJo6hipJ8KguKAkV4wvi793j0Rkia8qyYp8WI04jSkGCkt+eaRGyGVuzFSURDmUVHAK/jz87u+2bCa1hRwkdgVaYAKHd/8dIcJzmLCFWZIyoFtpcrLkVAUM1LU3UySFOExGpGBphzFRHr59JACnmhlCMNE6McVnKq/O3IUSzmJA11Z7ijnvVL8zxtkKmx5OeVppgjHs0FhxqBKYJkKHFJBsGITTRAWVO8KcYQEwkpnV9ch2PMnLxLnrHnZtG/PG+1WlUYNHIJjcApscAHa4AZ0gAMweABP4AW8Go/Gs/FmvM9Kl4yq5wD8gfHxDXRemcg=

g✓(ĥ)
AAACDnicbVBNS8NAEN34WetX1KOXxaLUS0lEsN4KXjxWsLbQlLLZbpqlm03YnQgl5B948a948aDi1bM3/42btoi2Phh4vDfDzDw/EVyD43xZS8srq2vrpY3y5tb2zq69t3+n41RR1qKxiFXHJ5oJLlkLOAjWSRQjkS9Y2x9dFX77ninNY3kL44T1IjKUPOCUgJH69smwn3kRgdAPMg9CBiTPq15I4EcN8/wU9+2KU3MmwIvEnZEKmqHZtz+9QUzTiEmggmjddZ0EehlRwKlgedlLNUsIHZEh6xoqScR0L5v8k+NjowxwECtTEvBE/T2RkUjrceSbzuJIPe8V4n9eN4Wg3su4TFJgkk4XBanAEOMiHDzgilEQY0MIVdzcimlIFKFgIiybENz5lxdJ66x2WXNvziuN+iyNEjpER6iKXHSBGugaNVELUfSAntALerUerWfrzXqfti5Zs5kD9AfWxzd3oZ0XAAACDnicbVBNS8NAEN34WetX1KOXxaLUS0lEsN4KXjxWsLbQlLLZbpqlm03YnQgl5B948a948aDi1bM3/42btoi2Phh4vDfDzDw/EVyD43xZS8srq2vrpY3y5tb2zq69t3+n41RR1qKxiFXHJ5oJLlkLOAjWSRQjkS9Y2x9dFX77ninNY3kL44T1IjKUPOCUgJH69smwn3kRgdAPMg9CBiTPq15I4EcN8/wU9+2KU3MmwIvEnZEKmqHZtz+9QUzTiEmggmjddZ0EehlRwKlgedlLNUsIHZEh6xoqScR0L5v8k+NjowxwECtTEvBE/T2RkUjrceSbzuJIPe8V4n9eN4Wg3su4TFJgkk4XBanAEOMiHDzgilEQY0MIVdzcimlIFKFgIiybENz5lxdJ66x2WXNvziuN+iyNEjpER6iKXHSBGugaNVELUfSAntALerUerWfrzXqfti5Zs5kD9AfWxzd3oZ0XAAACDnicbVBNS8NAEN34WetX1KOXxaLUS0lEsN4KXjxWsLbQlLLZbpqlm03YnQgl5B948a948aDi1bM3/42btoi2Phh4vDfDzDw/EVyD43xZS8srq2vrpY3y5tb2zq69t3+n41RR1qKxiFXHJ5oJLlkLOAjWSRQjkS9Y2x9dFX77ninNY3kL44T1IjKUPOCUgJH69smwn3kRgdAPMg9CBiTPq15I4EcN8/wU9+2KU3MmwIvEnZEKmqHZtz+9QUzTiEmggmjddZ0EehlRwKlgedlLNUsIHZEh6xoqScR0L5v8k+NjowxwECtTEvBE/T2RkUjrceSbzuJIPe8V4n9eN4Wg3su4TFJgkk4XBanAEOMiHDzgilEQY0MIVdzcimlIFKFgIiybENz5lxdJ66x2WXNvziuN+iyNEjpER6iKXHSBGugaNVELUfSAntALerUerWfrzXqfti5Zs5kD9AfWxzd3oZ0XAAACDnicbVBNS8NAEN34WetX1KOXxaLUS0lEsN4KXjxWsLbQlLLZbpqlm03YnQgl5B948a948aDi1bM3/42btoi2Phh4vDfDzDw/EVyD43xZS8srq2vrpY3y5tb2zq69t3+n41RR1qKxiFXHJ5oJLlkLOAjWSRQjkS9Y2x9dFX77ninNY3kL44T1IjKUPOCUgJH69smwn3kRgdAPMg9CBiTPq15I4EcN8/wU9+2KU3MmwIvEnZEKmqHZtz+9QUzTiEmggmjddZ0EehlRwKlgedlLNUsIHZEh6xoqScR0L5v8k+NjowxwECtTEvBE/T2RkUjrceSbzuJIPe8V4n9eN4Wg3su4TFJgkk4XBanAEOMiHDzgilEQY0MIVdzcimlIFKFgIiybENz5lxdJ66x2WXNvziuN+iyNEjpER6iKXHSBGugaNVELUfSAntALerUerWfrzXqfti5Zs5kD9AfWxzd3oZ0X

NICTA Copyright 2014

REFERENCES
•  [1] Sepp Hochreiter and Juergen Schmidhuber. Long short-term

memory. In Neural computation, 1997.
•  [2] https://www.tensorflow.org/tutorials/recurrent .
•  [3] Kai Sheng Tai, Richard Socher, and Christopher D. Manning.

Improved Semantic Representations from Tree-Structured Long
Short-Term Memory Networks. In ACL, 2015.

•  [4] Irsoy, Ozan and Claire Cardie. Deep recursive neural networks
for compositionality in language. In NIPS, 2014.

•  [5] Chung, Junyoung, Caglar Gulcehre, KyungHyun Cho, and
Yoshua Bengio. “Empirical evaluation of gated recurrent neural
networks on sequence modeling.” In NIPS Deep Learning and
Representation Learning Workshop, 2014.

•  [6] https://github.com/aymericdamien/TensorFlow-Examples.
•  [7] http://web.stanford.edu/class/cs20si/syllabus.html.

21

NICTA Copyright 2014

Learning Resources

•  NLP conferences.
–  ACL, EMNLP, COLING, NAACL, EACL …

•  NLP online courses.
–  http://web.stanford.edu/class/cs224n/
–  http://www.cs.columbia.edu/~cs4705/

•  ML online courses.
–  https://www.coursera.org/course/ml
–  https://www.cs.ox.ac.uk/people/nando.defreitas/

machinelearning/
https://www.youtube.com/watch?
v=SGZ6BttHMPw&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNm
UBH

22