Question Answering
COMP90042
Natural Language Processing
Lecture 19
Semester 1 2021 Week 10 Jey Han Lau
COPYRIGHT 2021, THE UNIVERSITY OF MELBOURNE
1
COMP90042
L19
•
•
Definition: question answering (“QA”) is the task of automatically determining the answer for a natural language question
Introduction
Mostly focus on “factoid” questions
2
COMP90042
L19
Factoid Questions
Factoid questions, have short precise answers:
• • • •
What war involved the battle of Chapultepec? What is the date of Boxing Day?
What are some fragrant white climbing roses? What are tannins?
3
COMP90042
L19
Non-factoid Questions
General non-factoid questions require a longer answer, critical analysis, summary, calculation and more:
• •
Why is the date of Australia Day contentious? What is the angle 60 degrees in radians?
4
COMP90042
L19
Why do we focus on
factoid questions in NLP?
• They are easier
• They have an objective answer
• Current NLP technologies cannot handle non-factoid answers
• There’s less demand for systems to automatically answer non-factoid questions
5
COMP90042
L19
•
Information retrieval-based QA
‣ Given a query, search relevant documents
‣ Find answers within these relevant documents
•
Knowledge-based QA
‣ Builds semantic representation of the query ‣ Query database of facts to find answers
2 Key Approaches
6
COMP90042
L19
• • •
IR-based QA Knowledge-based QA Hybrid QA
Outline
7
COMP90042
L19
IR-based QA
8
COMP90042
L19
IR-based Factoid QA: TREC-QA
1. UsequestiontomakequeryforIRengine
2. Finddocument,andpassagewithindocument 3. Extractshortanswerstring
9
COMP90042
L19
Question Processing
• Findkeypartsofquestionthatwillhelpretrieval
‣ Discard non-content words/symbols (wh-word, ?, etc) ‣ Formulate as tf-idf query, using unigrams or bigrams ‣ Identify entities and prioritise match
• Mayreformulatequestionusingtemplates
‣ E.g. “Where is Federation Square located?”
‣ Query = “Federation Square located”
‣ Query = “Federation Square is located [in/at]”
• Predictexpectedanswertype(here=LOCATION)
10
COMP90042
L19
•
•
Knowing the type of answer can help in:
‣ finding the right passage containing the answer ‣ finding the answer string
Answer Types
Treat as classification
‣ given question, predict
answer type
‣ key feature is question
headword
‣ What are the animals on the Australian coat of arms?
‣ Generally not a difficult task
11
COMP90042
L19
12
COMP90042
L19
Retrieval
• Findtopndocumentsmatchingquery(standardIR)
• Nextfindpassages(paragraphsorsentences)in these documents (also driven by IR)
• Shouldcontain:
‣ many instances of the question keywords
‣ several named entities of the answer type
‣ close proximity of these terms in the passage ‣ high ranking by IR engine
• Re-rankIRoutputstofindbestpassage(e.g.,using supervised learning)
13
COMP90042
L19
•
Find a concise answer to the question, as a span in the passage
‣ “Who is the federal MP for Melbourne?”
‣ The Division of Melbourne is an Australian Electoral Division in Victoria, represented since the 2010 election by Adam Bandt, a member of the Greens.
‣ “How many Australian PMs have there been since 2013?”
‣ Australia has had five prime ministers in five years. No wonder Merkel needed a cheat sheet at the G-20.
Answer Extraction
14
COMP90042
L19
•
•
•
•
Use a neural network to extract answer
AKA reading comprehension task
But deep learning models require lots of data
How?
Do we have enough data to train comprehension models?
15
COMP90042
L19
MCTest
•
•
•
Crowdworkers write fictional stories, questions and answers
500 stories, 2000 questions
Multiple choice questions
16
COMP90042
L19
SQuAD
• UseWikipediapassages
• Firstsetofcrowdworkerscreatequestions(givenpassage) • Secondsetofcrowdworkerslabeltheanswer
• 150Kquestions(!)
• Secondversionincludesunanswerablequestions
17
COMP90042
L19
Reading Comprehension
• Givenaquestionandcontextpassage,predictwhere
the answer span starts and end in passage? • Compute:
pstart(i): prob. of token i is the starting token pend(i): prob. of token i is the ending token
‣
‣
18
COMP90042
L19
• •
LSTM-Based Model
Feed question tokens to a bidirectional LSTM
Aggregate LSTM outputs via weighted sum to produce q, the final question embedding
19
COMP90042
L19
• •
Process passage in a similar way, using another bidirectional LSTM More than just word embeddings as input
‣ A feature to denote whether the word matches a question word
‣ POS feature
‣ Weighted question embedding: produced by attending to each question words
LSTM-Based Model
20
COMP90042
L19
LSTM-Based Model
{p1, . . . , pm}: one vector for each passage token from bidirectional LSTM
•
• Tocomputestartandendprobabilityforeachtoken:
pstart(i) ∝ exp(piWsq) pend(i) ∝ exp(piWeq)
21
COMP90042
L19
•
Bert-Based Model Fine-tune BERT to predict answer span
p (i) ∝ exp(S⊺T′) start ⊺ i
p (i) ∝ exp(E T′) end i
22
COMP90042
L19
Why BERT works better than LSTM?
• It has more parameters
• It’s pre-trained and so already “knows” language before it’s adapted to the task
• Multi-head attention is the secret sauce
• Self-attention architecture allows fine-grained analysis between words in question and context paragraph
23
COMP90042
L19
Knowledge-based QA
24
COMP90042
L19
QA over structured KB
•
•
‣ E.g.
‣ Link “Ada Lovelace” with the correct entity in the KB to find triple (Ada Lovelace, birth-year, 1815)
Many large knowledge bases
‣ Freebase, DBpedia, Yago, …
Can we support natural language queries?
25
COMP90042
L19
•
•
•
Converting natural language sentence into triple is not trivial
But…
Entity linking also an important component ‣ Ambiguity: “When was Lovelace born?”
Can we simplify this two-step process?
26
COMP90042
L19
Semantic Parsing
• ConvertquestionsintologicalformstoqueryKBdirectly ‣ Predicate calculus
‣ Programming query (e.g. SQL)
27
COMP90042
L19
•
Text-to-text problem:
‣ Input = natural language sentence ‣ Output = string in logical form
•
Encoder-decoder model (lecture 17!)
How to Build a Semantic Parser?
28
COMP90042
L19
Hybrid QA
29
COMP90042
L19
•
•
Why not use both text-based and knowledge- based resources for QA?
Hybrid Methods
IBM’s Watson which won the game show Jeopardy! uses a wide variety of resources to answer questions
‣ THEATRE: A new play based on this Sir Arthur Conan Doyle canine classic opened on the London stage in 2007.
‣ The Hound Of The Baskervilles
30
COMP90042
L19
•
• •
Generate lots of candidate answers from text- based and knowledge-based sources
Core Idea of Watson
Use a rich variety of evidence to score them
Many components in the system, most trained separately
31
COMP90042
L19
Watson
32
COMP90042
L19
Watson
• Use web documents (Wikipedia e.g.) and knowledgebases
• Answer extraction: extract all NPs, or use article title if Wikipedia document
33
COMP90042
L19
Watson
• Use many sources of evidence to score candidate answers
• Search for extra evidence: passages that support the candidate answers
• Candidate answer should really match answer type
34
COMP90042
L19
Watson
• Merge similar answers:
• JFK, John F. Kennedy, John
Fitzgerald Kennedy
• Use synonym lexicons, Wikipedia
35
COMP90042
L19
•
IR: Mean Reciprocal Rank for systems returning matching passages or answer strings
‣ E.g. system returns 4 passages for a query, first correct passage is the 3rd passage
‣ MRR = 1⁄3
MCTest: Accuracy
SQuAD: Exact match of string against gold answer
• •
QA Evaluation
36
COMP90042
L19
•
IR-based QA: search textual resources to answer questions
‣ Reading comprehension: assumes question+passage
•
•
Knowledge-based QA: search structured resources to answer questions
A Final Word
Hot area: many new approaches & evaluation datasets being created all the time (narratives, QA, commonsense reasoning, etc)
37
COMP90042
L19
Reading • JM3 Ch. 23.2, 23.4, 23.6
38