NLP-progress/sentiment_analysis.md at master · sebastianruder/NLP-progress
Skip to content
In this repository
All GitHub
↵
Jump to
↵
No suggested jump to results
In this repository
All GitHub
↵
Jump to
↵
In this user
All GitHub
↵
Jump to
↵
In this repository
All GitHub
↵
Jump to
↵
Loading
Dashboard
Pull requests
Issues
Marketplace
Explore
Codespaces
Sponsors
Settings
Michaelyulai
Sign out
New repository
Import repository
New gist
New organization
Loading
Sorry, something went wrong.
{{ message }}
sebastianruder
/NLP-progress
Type # for issues and pull requests, > for commands, and ? for help
Type # for issues, pull requests, and projects, > for commands, and ? for help
Type # for issues, pull requests, and projects, / for files, and > for commands
We’ve encountered an error and some results aren’t available at this time. Type a new search or try again later.
No results matched your search
Search for issues and pull requests
#
Search for issues, pull requests, discussions, and projects
#
Search for organizations, repositories, and users
@
Search for projects
!
Search for files
/
Activate command mode
>
Search your issues, pull requests, and discussions
# author:@me
Search your issues, pull requests, and discussions
# author:@me
Filter to pull requests
# is:pr
Filter to issues
# is:issue
Filter to discussions
# is:discussion
Filter to projects
# is:project
Filter to open issues, pull requests, and discussions
# is:open
sebastianruder
/
NLP-progress
Public
Unwatch
Stop ignoring
Watch
Notifications
Participating and @mentions
Only receive notifications from this repository when participating or @mentioned.
All Activity
Notified of all notifications on this repository.
Ignore
Never be notified.
Custom
Select events you want to be notified of in addition to participating and @mentions.
Get push notifications on iOS or Android.
Custom
Custom
Select events you want to be notified of in addition to participating and @mentions.
Issues
Pull requests
Releases
Discussions
Security alerts
Apply
Cancel
1.3k
Unstar
19.4k
Star
19.4k
Fork
Fork NLP-progress
Loading
If this dialog fails to load, you can visit the fork page directly.
3.3k
Code
Issues
29
Pull requests
11
Actions
Projects
0
Wiki
Security
Insights
More
Code
Issues
Pull requests
Actions
Projects
Wiki
Security
Insights
Open in github.dev
Permalink
master
Switch branches/tags
Branches
Tags
Could not load branches
Nothing to show
Loading
{{ refName }}
default
View all branches
Could not load tags
Nothing to show
{{ refName }}
default
Loading
View all tags
NLP-progress/english/sentiment_analysis.md
Go to file
Go to file
T
Go to line
L
Copy path
Copy permalink
Cannot retrieve contributors at this time
Sentiment analysis
IMDb
SST
Yelp
SemEval
Aspect-based sentiment analysis
Sentihood
SemEval-2014 Task 4
Sentiment classification with user and product information
Subjectivity analysis
SUBJ
207 lines (154 sloc)
19.1 KB
Raw
Blame
Loading
Open with Desktop
View raw
Copy raw contents
Copy raw contents
Loading
Copy raw contents
Copy raw contents
View blame
Edit file
Delete file
Sentiment analysis
Sentiment analysis is the task of classifying the polarity of a given text.
IMDb
The IMDb dataset is a binary
sentiment analysis dataset consisting of 50,000 reviews from the Internet Movie Database (IMDb) labeled as positive or
negative. The dataset contains an even number of positive and negative reviews. Only highly polarizing reviews are considered.
A negative review has a score ≤ 4 out of 10, and a positive review has a score ≥ 7 out of 10. No more than 30 reviews are
included per movie. Models are evaluated based on accuracy.
Model Accuracy Paper / Source
XLNet (Yang et al., 2019) 96.21 XLNet: Generalized Autoregressive Pretraining for Language Understanding
BERT_large+ITPT (Sun et al., 2019) 95.79 How to Fine-Tune BERT for Text Classification?
BERT_base+ITPT (Sun et al., 2019) 95.63 How to Fine-Tune BERT for Text Classification?
ULMFiT (Howard and Ruder, 2018) 95.4 Universal Language Model Fine-tuning for Text Classification
Block-sparse LSTM (Gray et al., 2017) 94.99 GPU Kernels for Block-Sparse Weights
oh-LSTM (Johnson and Zhang, 2016) 94.1 Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings
Virtual adversarial training (Miyato et al., 2016) 94.1 Adversarial Training Methods for Semi-Supervised Text Classification
BCN+Char+CoVe (McCann et al., 2017) 91.8 Learned in Translation: Contextualized Word Vectors
SST
The Stanford Sentiment Treebank
contains 215,154 phrases with fine-grained sentiment labels in the parse trees
of 11,855 sentences in movie reviews. Models are evaluated either on fine-grained
(five-way) or binary classification based on accuracy.
Fine-grained classification (SST-5, 94,2k examples):
Model Accuracy Paper / Source
BCN+Suffix BiLSTM-Tied+CoVe (Brahma, 2018) 56.2 Improved Sentence Modeling using Suffix Bidirectional LSTM
BCN+ELMo (Peters et al., 2018) 54.7 Deep contextualized word representations
BCN+Char+CoVe (McCann et al., 2017) 53.7 Learned in Translation: Contextualized Word Vectors
Binary classification (SST-2, 56.4k examples):
Model Accuracy Paper / Source Code
XLNet-Large (ensemble) (Yang et al., 2019) 96.8 XLNet: Generalized Autoregressive Pretraining for Language Understanding Official
MT-DNN-ensemble (Liu et al., 2019) 96.5 Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding Official
Snorkel MeTaL(ensemble) (Ratner et al., 2018) 96.2 Training Complex Models with Multi-Task Weak Supervision Official
MT-DNN (Liu et al., 2019) 95.6 Multi-Task Deep Neural Networks for Natural Language Understanding Official
Bidirectional Encoder Representations from Transformers (Devlin et al., 2018) 94.9 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Official
Block-sparse LSTM (Gray et al., 2017) 93.2 GPU Kernels for Block-Sparse Weights Offical
bmLSTM (Radford et al., 2017) 91.8 Learning to Generate Reviews and Discovering Sentiment Unoffical
Single layer bilstm distilled from BERT (Tang et al., 2019) 90.7 Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
BCN+Char+CoVe (McCann et al., 2017) 90.3 Learned in Translation: Contextualized Word Vectors Official
Neural Semantic Encoder (Munkhdalai and Yu, 2017) 89.7 Neural Semantic Encoders
BLSTM-2DCNN (Zhou et al., 2017) 89.5 Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling
Yelp
The Yelp Review dataset
consists of more than 500,000 Yelp reviews. There is both a binary and a fine-grained (five-class)
version of the dataset. Models are evaluated based on error (1 – accuracy; lower is better).
Fine-grained classification:
Model Error Paper / Source
XLNet (Yang et al., 2019) 27.80 XLNet: Generalized Autoregressive Pretraining for Language Understanding
BERT_large+ITPT (Sun et al., 2019) 28.62 How to Fine-Tune BERT for Text Classification?
BERT_base+ITPT (Sun et al., 2019) 29.42 How to Fine-Tune BERT for Text Classification?
ULMFiT (Howard and Ruder, 2018) 29.98 Universal Language Model Fine-tuning for Text Classification
DPCNN (Johnson and Zhang, 2017) 30.58 Deep Pyramid Convolutional Neural Networks for Text Categorization
CNN (Johnson and Zhang, 2016) 32.39 Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings
Char-level CNN (Zhang et al., 2015) 37.95 Character-level Convolutional Networks for Text Classification
Binary classification:
Model Error Paper / Source
XLNet (Yang et al., 2019) 1.55 XLNet: Generalized Autoregressive Pretraining for Language Understanding
BERT_large+ITPT (Sun et al., 2019) 1.81 How to Fine-Tune BERT for Text Classification?
BERT_base+ITPT (Sun et al., 2019) 1.92 How to Fine-Tune BERT for Text Classification?
ULMFiT (Howard and Ruder, 2018) 2.16 Universal Language Model Fine-tuning for Text Classification
DPCNN (Johnson and Zhang, 2017) 2.64 Deep Pyramid Convolutional Neural Networks for Text Categorization
CNN (Johnson and Zhang, 2016) 2.90 Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings
Char-level CNN (Zhang et al., 2015) 4.88 Character-level Convolutional Networks for Text Classification
SemEval
SemEval (International Workshop on Semantic Evaluation) has a specific task for Sentiment analysis.
Latest year overview of such task (Task 4) can be reached at: http://www.aclweb.org/anthology/S17-2088
SemEval-2017 Task 4 consists of five subtasks, each offered for both Arabic and English:
Subtask A: Given a tweet, decide whether it expresses POSITIVE, NEGATIVE or NEUTRAL
sentiment.
Subtask B: Given a tweet and a topic, classify the sentiment conveyed towards that
topic on a two-point scale: POSITIVE vs. NEGATIVE.
Subtask C: Given a tweet and a topic, classify the sentiment conveyed in the
tweet towards that topic on a five-point scale: STRONGLYPOSITIVE, WEAKLYPOSITIVE,
NEUTRAL, WEAKLYNEGATIVE, and STRONGLYNEGATIVE.
Subtask D: Given a set of tweets about a topic, estimate the distribution of tweets
across the POSITIVE and NEGATIVE classes.
Subtask E: Given a set of tweets about a topic, estimate the distribution of tweets
across the five classes: STRONGLYPOSITIVE, WEAKLYPOSITIVE, NEUTRAL, WEAKLYNEGATIVE, and STRONGLYNEGATIVE.
Subtask A results:
Model F1-score Paper / Source
LSTMs+CNNs ensemble with multiple conv. ops (Cliche. 2017) 0.685 BB twtr at SemEval-2017 Task 4: Twitter Sentiment Analysis with CNNs and LSTMs
Deep Bi-LSTM+attention (Baziotis et al., 2017) 0.677 DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis
Aspect-based sentiment analysis
Sentihood
Sentihood is a dataset for targeted aspect-based sentiment analysis (TABSA), which aims
to identify fine-grained polarity towards a specific aspect. The dataset consists of 5,215 sentences,
3,862 of which contain a single target, and the remainder multiple targets.
Dataset mirror: https://github.com/uclmr/jack/tree/master/data/sentihood
Model Aspect (F1) Sentiment (acc) Paper / Source Code
QACG-BERT (Wu and Ong, 2020) 89.7 93.8 Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis Official
Sun et al. (2019) 87.9 93.6 Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence Official
Liu et al. (2018) 78.5 91.0 Recurrent Entity Networks with Delayed Memory Update for Targeted Aspect-based Sentiment Analysis Official
SenticLSTM (Ma et al., 2018) 78.2 89.3 Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM
LSTM-LOC (Saeidi et al., 2016) 69.3 81.9 Sentihood: Targeted aspect based sentiment analysis dataset for urban neighbourhoods
SemEval-2014 Task 4
The SemEval-2014 Task 4 contains two domain-specific datasets for laptops and restaurants, consisting of over 6K sentences with fine-grained aspect-level human annotations.
The task consists of the following subtasks:
Subtask 1: Aspect term extraction
Subtask 2: Aspect term polarity
Subtask 3: Aspect category detection
Subtask 4: Aspect category polarity
Preprocessed dataset: https://github.com/songyouwei/ABSA-PyTorch/tree/master/datasets/semeval14
https://github.com/howardhsu/BERT-for-RRC-ABSA (with both subtask 1 and subtask 2)
Subtask 1 results (SemEval-2014 Task 4 for Laptop and SemEval-2016 Task 5 for Restaurant):
Model Laptop (F1) Restaurant (F1) Paper / Source Code
ACE + fine-tune (Wang et al., 2020) 87.4 81.3 Automated Concatenation of Embeddings for Structured Prediction Official
BERT-PT (Hu, Xu, et al., 2019) 84.26 77.97 BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis official
DE-CNN (Hu, Xu, et al., 2018) 81.59 74.37 Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction official
MIN (Li, Xin, et al., 2017) 77.58 73.44 [Deep Multi-Task Learning for Aspect Term Extraction with Memory Interaction]
RNCRF (Wang, Wenya. et al., 2016) 78.42 69.74 Recursive Neural Conditional Random Fields for Aspect-based Sentiment Analysis official
Subtask 2 results:
Model Restaurant (acc) Laptop (acc) Paper / Source Code
BERT-ADA (Rietzler, Alexander, et al., 2019) 87.89 80.23 Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification official
LCF-BERT (Zeng, Yang, et al., 2019) 87.14 82.45 LCF: A Local Context Focus Mechanism for Aspect-Based Sentiment Classification official / Link
BERT-PT (Hu, Xu, et al., 2019) 84.95 78.07 BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis official
AOA (Huang, Binxuan, et al., 2018) 81.20 74.50 Aspect Level Sentiment Classification with Attention-over-Attention Neural Networks Link
TNet (Li, Xin, et al., 2018) 80.79 76.01 Transformation Networks for Target-Oriented Sentiment Classification Official / Link
RAM (Chen, Peng, et al., 2017) 80.23 74.49 Recurrent Attention Network on Memory for Aspect Sentiment Analysis Link
MemNet (Tang, Duyu, et al., 2016) 80.95 72.21 Aspect Level Sentiment Classification with Deep Memory Network Official / Link
IAN (Ma, Dehong, et al., 2017) 78.60 72.10 Interactive Attention Networks for Aspect-Level Sentiment Classification Link
ATAE-LSTM (Wang, Yequan, et al. 2016) 77.20 68.70 Attention-based lstm for aspect-level sentiment classification Link
TD-LSTM (Tang, Duyu, et al., 2016) 75.63 68.13 Effective LSTMs for Target-Dependent Sentiment Classification Official / Link
Sentiment classification with user and product information
This is the same task on sentiment classification, where the given text is a review, but we are also additionally given (a) the user who wrote the text, and (b) the product which the text is written for. There are three widely used datasets, introduced by Tang et. al (2015): IMDB, Yelp 2013, and Yelp 2014. Evaluation is done using both accuracy and RMSE, but for brevity, we only provide the accuracy here. Please look at the papers for the RMSE values.
Model IMDB (acc) Yelp 2013 (acc) Yelp 2014 (acc) Paper / Source Code
BiLSTM+CHIM (Amplayo, 2019) 56.4 67.8 69.2 Rethinking Attribute Representation and Injection for Sentiment Classification Link
BiLSTM + linear-basis-cust (Kim, et al., 2019) – 67.1 – Categorical Metadata Representation for Customized Text Classification Link
CMA (Ma, et al., 2017) 54.0 66.4 67.6 Cascading Multiway Attention for Document-level Sentiment Classification –
DUPMN (Long, et al., 2018) 53.9 66.2 67.6 Dual Memory Network Model for Biased Product Review Classification –
HCSC (Amplayo, et al., 2018) 54.2 65.7 – Cold-Start Aware User and Product Attention for Sentiment Classification Link
NSC (Chen, et al., 2016) 53.3 65.0 66.7 Neural Sentiment Classification with User and Product Attention Link
UPDMN (Dou, 2017) 46.5 63.9 61.3 Capturing User and Product Information for Document Level Sentiment Analysis with Deep Memory Network –
UPNN (Tang, et al., 2016) 43.5 59.6 60.8 Learning Semantic Representations of Users and Products for Document Level Sentiment Classification Link
Subjectivity analysis
A related task to sentiment analysis is the subjectivity analysis with the goal of labeling an opinion as either subjective or objective.
SUBJ
Subjectivity dataset includes 5,000 subjective and 5,000 objective processed sentences.
Model Accuracy Paper / Source
AdaSent (Zhao et al., 2015) 95.50 Self-Adaptive Hierarchical Sentence Model
CNN+MCFA (Amplayo et al., 2018) 94.80 Translations as Additional Contexts for Sentence Classification
Byte mLSTM (Radford et al., 2017) 94.60 Learning to Generate Reviews and Discovering Sentiment
USE (Cer et al., 2018) 93.90 Universal Sentence Encoder
Fast Dropout (Wang and Manning, 2013) 93.60 Fast Dropout Training
Go back to the README
Go
© 2021 GitHub, Inc.
Terms
Privacy
Security
Status
Docs
Contact GitHub
Pricing
API
Training
Blog
About
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session.
You signed out in another tab or window. Reload to refresh your session.