CS计算机代考程序代写 deep learning algorithm COMP5046

2
BERT
How the BERT is working (Brief)

2
BERT
Pre-training and Transfer Learning in NLP
BERT: Input Representation

2
BERT
Word-piece Token Embedding
Word Structure and subword models
Common words end up being a part of the subword vocabulary, while rarer words are split into (sometimes intuitive, sometimes not) components.
In the worst case, words are split into as many subwords as they have characters
Common words
Variations misspellings novel items
Word
Vocab mapping
computer
Computer
play
play
coooooool
coo##ooo#ool
laern
la##ern##
Transformerify
Transformer##ify

2
BERT
Pretraining for BERT
Remember GPT? Pretraining the Language Model

2
BERT
Pretraining: Masked Language Model (LM)
Mask out k% of the input words, and then predict the masked words (use k=15%)
store gallon
The man went to the [MASK] to buy a [MASK] of milk
• Too little masking: Too expensive to train
• Too much masking: Not enough context

2
BERT
Pre-training and Transfer Learning in NLP
BERT: Masked language Model
With those 15%

2
BERT
Pre-training and Transfer Learning in NLP
BERT: Next Sentence Prediction

2
BERT
Pre-training and Transfer Learning in NLP
BERT: Input Representation

2
BERT
Model Details
• Data: Wikipedia (2.5B words) + BookCorpus (800M words)
• Batch Size: 131,072 words (1024 sequences * 128 length or
256 sequences *512 length
• Training Time: 1M steps (~40 epochs)
• Optimizer: AdamW, 1e-4 learning rate, linear decay
• BERT-Base: 12-layer, 768-hidden, 12-head
• BERT-Large: 24-layer, 1024-hidden, 16-head
• Trained on 4×4 or 8×8 TPU slice for 4 days

2
BERT
Pre-training and Transfer Learning in NLP
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2
BERT
Pre-training and Transfer Learning in NLP
The two steps of how BERT is developed. Download the model pre- trained in step 1 (trained on un-annotated data), and only worry about fine-tuning it for step 2
Pretraining Fine-Tuning

2
BERT
Fine-Tuning Procedure

2
BERT
Fine-Tuning Procedure
The following shows a number of ways to use BERT for different tasks.

2
BERT
Accuracy… Performance
https://rajpurkar.github.io/SQuAD-explorer/

2
BERT
Effect of Pre-training Task
• Masked LM (compared to left-to-right LM) is very important on some tasks, Next Sentence Prediction is important on other tasks.
• Left-to-right model does very poorly on word-level task (SQuAD), although this is mitigated by Bi-LSTM

2
BERT
Effect of Model Size
• Big models help a lot
• Going from 110M→340M params helps even on datasets with
3,600 labelled examples
• Improvements have not asymptoted

2
BERT
Resource! Resource!
TPUS… and Resources..
BERT-Base: 4 Cloud TPUs (16 TPU chips total) in 4 days BERT-Large: 16 Cloud TPUs (64 TPU chips total)

2
BERT Questions
• Why did no one think of this before?
• Better Question: Why wasn’t contextual pre-training popular
before 2018 with ELMo?
• Good Results on pre-training is > 1,000 x to 100,000 more expensive than supervised training.

2
BERT Questions
• The model must be learning more than “contextual embeddings”
• Alternate interpretation: Predicting missing words (or next words) requires learning many types of language understanding features
– Syntax, semantics, pragmatics, coreference, etc.
• Implication: Pre-trained model is much bigger than it needs to
be to solve specific task
• Task-specific model distillation words very well

2
BERT
More advanced pre-trained model

3
Post BERT

3
Post BERT Pretrained Model Map

3
Post BERT RoBERTa
A Robustly Optimized BERT Pretraining Approach (Liu et al, University of Washington and Facebook, 2019)
• Trained BERT for more epochs and/or on more data
– Showed that more epochs alone helps, even on same data
– More data also helps
• Improved masking and pre-training data slightly

3
Post BERT XLNet
Generalized Autoregressive Pretraining for Language Understanding (Yang et al, CMU and Google, 2019)
Innovation 1: Relative Position Embedding
• Sentence: Caren ate a hot dog
• Absolute Attention: “How much should dog attend to hot (in any position), and how much should dog in position 4 attend to the word in position 3?”
• Relative Attention: “How much should dog attend to hot (in any position) and how much should dog attend to the previous word?”

3
Post BERT XLNet
Generalized Autoregressive Pretraining for Language Understanding (Yang et al, CMU and Google, 2019)
Innovation 2: Permutation Language Modeling
• In a left-to-right language model, every word is predicted based on all of the words to its left
• Instead: Randomly permute the order for every training sentence
• Equivalent to masking, but many more predictions per sentence
• Can be done efficiently with Transformers

3
Post BERT XLNet
Generalized Autoregressive Pretraining for Language Understanding (Yang et al, CMU and Google, 2019)
• Also used more data and bigger models, but showed that innovations improved on BERT even with same data and model size
• XLNet results:

3
Post BERT ALBERT
A Lite BERT for Self-supervised Learning of Language Representations (Lan et al, Google and TTI Chicago, 2019)
Innovation 1: Factorized Embedding Parameterisation
• Use small embedding size (e.g., 128) and then project it to Transformer hidden size (e.g., 1024) with parameter matrix
VS.

3
Post BERT ALBERT
Innovation #2: Cross-layer parameter sharing
• Share all parameters between Transformer layers Results:
ALBERT is light in terms of parameters, not speed

3
Post BERT T5
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al, Google, 2019)
Ablated many aspects of pre-training:
• Model Size
• Amount of Training data
• Domain/cleanness of training data
• Pre-training objective details (e.g. span length of masked text)
• Ensembling
• Finetuning recipe (e.g. only allowing certain layers to finetune)
• Multi-task training

3
Post BERT T5
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al, Google, 2019)
Conclusions:
• Scaling up model size and amount of training data helps a lot
• Best model is 11B parameters (BERT-Large is 330M), trained on
120B words of cleaned common crawl text
• Exact masking/corruptions strategy does not matter that much
• Mostly negative results for better finetuning and multi-task strategies

3
Post BERT ELECTRA
Pre-training Text Encoders as Discriminators Rather Than Generators (Clark et al, 2020)
Train model to discriminate locally plausible text from real text:

3
Post BERT ELECTRA
Pre-training Text Encoders as Discriminators Rather Than Generators (Clark et al, 2020)
Difficult to match SOTA results with less compute

3
Post BERT LongFormer
The Long-Document Transformer (Peters et al., 2020)
Why?
• •
Traditional Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length
To address this, Longformer uses an attention pattern that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer.

3
Post BERT
LongFormer
The Long-Document Transformer (Peters et al., 2020)

3
Post BERT
Applying Models to Production Services
• BERT and other pre-trained language models are extremely large and expensive
• How are companies applying them to low-latency production services?

3
Post BERT
Applying Models to Production Services
• BERT and other pre-trained language models are extremely large and expensive
• How are companies applying them to low-latency production services?
The Answer is ‘Distillation’ (Model Compression)

3
Post BERT
Distillation (Model Compression)
The idea has been around for a long time (from 2006)
• Model Compression (Bucila et al. 2006)
• Distilling the Knowledge in a Neural Network (Hinton et al., 2015)

4
Multimodal Pretrained Model

4
Multimodal Pretrained Model Video Representation Learning
Supervised Learning: Large labelled data with CNN
• Expensive to collect labelled data
• Small corresponding label vocabs: not able to represent the
nuances of actions (e.g. difference “sipping” – “drinking” –
“gulping”)
• Represent short video clips (a few seconds long)
Unsupervised Learning: Learning density models from video
• Single static stochastic variable, decoded into a sequence
learning using RNN (VAE-style loss or GAN-style loss)
• Temporal stochastic variable (SV2P/SVCLP) or GAN-based
(SAVP/MoCoGAN)
• What if not using explicit stochastic latent variable

4
Multimodal Pretrained Model
VideoBERT (ICCV 2019)
A Joint Model for Video and Language Representation Learning

4
Multimodal Pretrained Model
ViLBERT (NIPS 2019)
A Joint Model for Video and Language Representation Learning

4
Multimodal Pretrained Model ViLBERT (NIPS 2019)
Results across all transfer tasks
• improves performance over a single-stream model
• result in improved visiolinguistic representations
• Finetuning from ViLBERT is a powerful strategy for vision and
language tasks

5
Before we finish this lecture…

0
Before we finish the lecture… The current ML/DL-based NLP trends
1) Assume that we collect data from Royal Prince Alfred Hospital
ME

0
Before we finish the lecture… The current ML/DL-based NLP trends
1) Assume that we collect data from Royal Prince Alfred Hospital
ME
Training with RNN, Transformer, or even pretrained model…
2) Train and Test on data from the same hospital

0
Before we finish the lecture… The current ML/DL-based NLP trends
1) Assume that we collect data from Royal Prince Alfred Hospital
ME
Training with RNN, Transformer, or even pretrained model…
2) Train and Test on data from the same hospital

0
Before we finish the lecture… The current ML/DL-based NLP trends
“Indeed, we can publish papers showing the algorithms are comparable to human medical experts in spotting certain conditions”

0
Before we finish the lecture… The current ML/DL-based NLP trends
What if we use this same model to another hospital?

0
Before we finish the lecture… The current ML/DL-based NLP trends
Assume you take that same DL-based NLP model, to St Vincent’s Private Hospital,
with an older testing machine, and the technician there uses a slightly different testing protocol
ME

0
Before we finish the lecture… The current ML/DL-based NLP trends
Assume you take that same DL-based NLP model, to St Vincent’s Private Hospital,
with an older testing machine, and the technician there uses a slightly different testing protocol
ME
Data drifts to cause the performance of DL-based NLP model to degrade significantly

0
Before we finish the lecture…
The current ML/DL-based NLP trends
In contrast, the Doctor just can walk down the street and diagnose the patient

0
Before we finish the lecture… The current ML/DL-based NLP trends
When a system is not performing well,
many teams instinctually try to improve the code
(try different model/component or change hyperparameters)
However, for many practical applications,
it is more effective instead to focus on improving the data

0
Before we finish the lecture… The current ML/DL-based NLP trends
When a system is not performing well,
many teams instinctually try to improve the code
(try different model/component or change hyperparameters)
However, for many practical applications,
it is more effective instead to focus on improving the data
Everyone jokes about ML/DL is 80% data preparation, but no one seems to care
Prof. Andrew Ng (March, 2021)

0
Before we finish the lecture… The current ML/DL-based NLP trends
There is unprecedented competition around beating the benchmarks. If Google has BERT then OpenAI has GPT-3.
However, these fancy models take up only 20% of business problems. What differentiates a good deployment is the quality of data.
“Data Dispersion”

/
Reference
Reference for this lecture
• Deng, L., & Liu, Y. (Eds.). (2018). Deep Learning in Natural Language Processing. Springer.
• Rao, D., & McMahan, B. (2019). Natural Language Processing with PyTorch: Build Intelligent Language Applications Using Deep Learning. ” O’Reilly Media, Inc.”.
• Manning, C. D., Manning, C. D., & Schütze, H. (1999). Foundations of statistical natural language processing. MIT press.
• Manning, C 2018, Natural Language Processing with Deep Learning, lecture notes, Stanford University
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008).
• Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
• Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
• Miller, A., Fisch, A., Dodge, J., Karimi, A. H., Bordes, A., & Weston, J. (2016). Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126.
• Drawings
• http://jalammar.github.io/illustrated-bert/
• http://jalammar.github.io/illustrated-transformer/