7b: Language Processing
Discussion: Translation, Transformers and ChatBots
Attention Mechanism and Neural Machine Translation
These articles explain how an attention mechanism can be used for sequence-to-sequence prediction,
and how stacked LSTMs, combined with attention and word vectors, can be used for multi-lingual
neural machine translation:
Cho, K., 2015. Introduction to Neural Machine Translation with GPUs (part 3), NVIDIA Developer
Blog.
Wu, Y. et al., 2016. Google’s Neural Machine Translation System: Bridging the Gap between
Human and Machine Translation, arxiv 1609.08144.
Transformers and GPT-3
Transformers like GPT-3 have recently achieved very impressive results on text prediction, as
described in these resources:
Bloem, P., 2019. Transformers from Scratch.
Brown, T., 2020. Language Models are Few-Shot Learners, arxiv 2005.14165.
GPT-3: Language Models are Few-Shot Learners (Paper Explained by Yannic Kilcher):
Recommended discussion
In the Ed discussion board, write a brief post in the category Lessons – Week 7 outlining your
thoughts on the following three points. Then reply to at least one other person’s post and read
through others to consider multiple viewpoints.
1. Do you believe GPT-3 is really producing “original” material, or is it just “mashing” existing texts
together in a deep way?
2. What will be the implications of systems like GPT-3 on issues such as journalism, commentary,
copyright, authorship, plagiarism, education?
3. Until recently, major companies were still using traditional scripting for their chat-bots, rather
than deep neural networks. One motivation for this might be to avoid problems such as those
manifested by Microsoft’s Tay system in 2016. Do you think chatbots in the future might be
based on transformers and other deep learning architectures? If so, how could they avoid
becoming inappropriately rude, o�ensive, misleading, deceptive or impertinent?
An error occurred.
Try watching this video on www.youtube.com, or enable JavaScript if it is disabled in your
browser.
Week 7 Thursday video