World Wide Web (2014) 17:847–867 DOI 10.1007/s11280-013-0247-z
Sentiment analysis on microblog utilizing appraisal theory Peter Korenek & Marián Šimko
Received: 8 November 2012 / Revised: 5 July 2013 / Accepted: 22 July 2013 / Published online: 13 August 2013 # Springer Science+Business Media New York 2013
Abstract People and companies selling goods or providing services have always desired to know what people think about their products. The number of opinions on the Web has significantly increased with the emergence of microblogs. In this paper we present a novel method for sentiment analysis of a text that allows the recognition of opinions in microblogs which are connected to a particular target or an entity. This method differs from other approaches in utilizing appraisal theory, which we employ for the analysis of microblog posts. The results of the experiments we performed on Twitter showed that our method improves sentiment classi- fication and is feasible even for such specific content as presented on microblogs.
Keywords Microblog.Opinionmining.Appraisaltheory.Twitter.Sentimentanalysis 1 Introduction
Social media perceptibly enhance communication on the Web and continue to spread among people of all age groups. Their users appreciate such a form of communication, since it is often free, quick and available for everyone. The amount of data produced by web users when exchanging information and communicating is vast and ever-increasing. Using logs, auxiliary records and metadata related to communication multiplies the growth of “explicit” social data. When considering the preservation of various types of social media, we face the issue of the high dynamics of social media in contrast to the need to store what is relevant. But how to preserve social media records with respect to effectiveness and low expenses? It may vary from one social media type to another.
One of the recently emerged forms of web communication is microblog. The main differ- ences between microblogs and other social networks are the limitation of post-length to approximately 140 characters, and the one-way “follows” relationship established between users. These features have made microblogs extremely popular, and enabled microblogging services to become platforms for an instant exchange of information, where users can express their feelings, opinions and ideas to anyone who may be interested. Due to the limitation of the length, new forms of expressing ideas were developed in all languages. Microblog posts contain slang expressions, abbreviations, multiplications of phonemes in words or emoticons. They also
P. Korenek : M. Šimko (*)
Institute of Informatics and Software Engineering, Faculty of Informatics and Information Technologies, Slovak University of Technology in Bratislava, Ilkovičova, 84216 Bratislava, Slovakia
e-mail: simko@fiit.stuba.sk
P. Korenek
e-mail: pkorenek@gmail.com
848 World Wide Web (2014) 17:847–867
have special features, e.g., words starting with a symbol “#” that marks the main topic of a post. Using such expressions makes it easier and much quicker to express what the author thinks and how she or he feels.
With the growing popularity of microblogs—and Twitter as the most popular microblogging social network in particular—the interest to analyse and study the network and the content it produces has increased. Microblog data are being used with advantage in various areas related to the Web itself, e.g. web search [19], as well as in real world applications, e.g., to identify current trends in the world [11], to track important world events such as elections [10] or to help emergency services to get recent information about incidents [1].
A microblog is a valuable source of recent and topical data. Archiving microblog-based data together with the knowledge that can be derived is important for future generations. Open issues related to social media preservation relate to microblogs as well. When considering small-in- size microblog posts, it is important to understand the context of the posts, which is crucial for ex post processing. In that case, the combined and united information representing the context, including additional important information and metadata, should be preserved. A very impor- tant aspect related to the microblog is sentiment and opinions implicitly present in posts. It is important not only when considering filtering data to be preserved according to specified criteria, but also as being a part of the context or metadata that are archived.
Opinions and ideas that users write in their microblogs are very valuable. Companies all around the world pay large amounts of money for surveys whose aim is to retrieve the opinions of customers on their products, to find out what they consider as positive or negative about their services [3]. This information can be used to improve their products and services, so that they could increase their profit. Governments and political parties want to know what people who have elected them think about their activities as well [30]. There is also a non-commercial use of opinion mining, e.g., when we want to find common interests between groups of people, or just want to find a new friend who is interested in the same topics as we are [4].
Microblogs as such are a very attractive source for sentiment analysis and opinion mining due to the amount and topicality of the available content. On the other hand, they pose interesting research challenges that arise from the specificities of a microblog (informal style, length of a post, the resulting use of special symbols in posts, etc.) that typically may reflect in the lower performance of state-of-the-art approaches.
The aim of our work is to improve sentiment classification of a microblog by employing a psychological theory called the appraisal theory. The basic concept of this theory was intro- duced by M. Arnold in the 1940s [26]. The motivation that stands behind this idea is the knowledge that emotions arise from an appraisal of concrete situations that produce specific reactions on various people. The theory lays basis for structured sentiment extraction that is based on appraisal expression, a basic grammatical unit by which an opinion is expressed [8].
The appraisal theory helps us to understand the feelings of microbloggers better, especially at moments when they are creating their microblog posts. In comparison to other methods, this theory allows a deeper and fine-grained analysis of the texts that microbloggers write. We follow the work of Bloom who laid basis for sentiment analysis based on the appraisal theory and proposed a method for sentiment classification on a microblog [8]. In line with advancing sentiment analysis, the contribution of our work lies in providing a descriptive characteristic for social data to be preserved, which is potentially useful for prospective processing of “sealed” social data when the current context and environment will not be available.
The rest of the paper is structured as follows. In Section 2 we discuss the related work. In Section 3 we briefly introduce basic concepts of the appraisal theory and indicate the way we employ it in our work. In Section 4 we present our method for sentiment classification. Section 5
World Wide Web (2014) 17:847–867 849 provides the evaluation of the method and discussion about the results. In Section 6 we
conclude our work.
2 Related work
Opinion mining is not a new field of research. Before the arrival of blog-like articles there was an extensive research in mining emotions from user generated texts such as forums, reviews or short stories published on the Web.
State-of-the-art in opinion mining can be found in an exhaustive survey by Pang and Lee [23]. The authors described the motivation, the basic concepts and the most common features used for opinion mining. They also explained the main platform-independent approaches to sentiment classification and application of knowledge about opinions in various areas of life. Sentiment analysis and opinion mining are closely connected to the field of natural language processing. The first approaches only used dictionary methods—comparing words from texts to the dictionary of terms without any deeper analysis, e.g., [24]. Later, new features like emoticons, n-grams or various word valuations appeared. These features were used as input to classifiers.
One of the research directions in the field relates to works based on linguistic processing of the underlying content covering various domains. Thet et al. created a sentiment classifier based on a lexical and syntactic analysis of movie reviews [29]. They divided all the sentences in the reviews into single clauses and calculated a sentiment score for each clause. They created a dependency tree of clauses using grammatical relationships between words. To calculate a score of a clause they assigned values to all objects, subjects, verbs and predicates. A sum of scores of clauses created the overall score for the review. The classification accuracy ranges from 66 % to 91 %. Tchalakova et al. devised a method for classifying product reviews using distinctive (maximally occurring) phrases as features [28]. They did not use standard sentiment lexicons of phrases, but extracted these phrases from an annotated document set and marked them with an annotation according to statistical results. This approach achieved 81 % accuracy in the classification of reviews. Zhang et al. presented a three step algorithm for opinion mining in blogs [33]. Their algorithm was able to retrieve documents from a dataset that contains a query, to classify document to opinion- ative and non-opinionative and rate the document’s relevance to a query. The authors claim that their work is from 28 % to 32 % better than other works on TREC 2006 Blog Track Data [32]. Abelovský created a classifier of texts based on neuron networks and a statistical approach which was based on the dependency of the occurrence of words in texts with similar polarity (PMI—Pointwise Mutual Information) [2]. The first step of his algorithm is the identification of the attributes of words in sentences. Afterwards, a statistical occurrence of combination of words with a list of positive and negative words is created. This information is used for the identification of subjective words in texts. The author modified PMI calculation in such a way, as to try to separate objects (words) in texts into sentences and he applied identified dependencies in relation to objects in sentences and not to the whole texts. The approach was verified on product and movie reviews with the accuracy ranging from 80 % to 86 %.
The emergence of blogs and microblogs stirred up the research in the field of sentiment analysis. The reason is clear—microblog and blog posts consist of opinions and emotions that the authors want to present to other people. In recent years, microblogs have become particularly widespread among people who tend to share their actual thoughts and moods.
850 World Wide Web (2014) 17:847–867
Microblog posts express feelings, opinions and introduce new forms of text expressions, enabling instant communication that is usually spontaneous and often not thought-through. Go et al. continued in the work of Pang and Lee by devising a method specific to microblogs [12]. According to their research, they were the first researchers who looked into the problem of sentiment classification on Twitter. They used an occurrence count of words from a dictionary of terms tagged with orientation (positive or negative) and emotions count as features for classification. In the pre-processing phase they removed forwarded (retweeted) posts and also posts containing both positive and negative emoticons. They used SVM and the Naïve Bayes classifier to classify tweets to corresponding orientation class. They achieved the overall accuracy of about 82 % for both classifiers. Barbosa and Feng divided the process of sentiment classification on microblogs into two steps [5]. In the first step they created a classifier for categorising posts into two classes—“objective” and “subjective”. The second step is a usual sentiment classification into classes “positive” and “negative”. The authors claim that the detection of n-grams is not suitable for opinion detection on microblogs because of their short length. The authors use part of speech (POS) tagging—they assign a part of speech tag to every term in the text and for adjectives they assign a level of subjectivity (weak or strong) and polarity of a word (positive or negative). They classify the subjectivity using URL, a count of positive words, strong subjective words, capitals in words and verbs as features and achieve the overall accuracy
of about 80 %.
Pak and Paroubek presented an approach based on linguistic analysis to classify
microblogs according to sentiment [21]. They assembled a dataset using positive and negative emoticons as search terms. The authors suggested that the pronoun for first person (“I”) and adjectives that are connected to this pronoun often occur in subjective texts. They used the presence of n-grams and the position of n-grams as features for classification. The evaluation of their method showed that the presence of bigrams is best for sentiment classification of microblog posts. Parikh et al. classified microblog posts using unigrams and multigrams [25]. They found out that the exchange of words with type expressions and removal of punctuation improves classification up to 20 % in comparison to not pre- processed data. They used the Naïve Bayes classifier for classification with the accuracy of about 58 %. Based on their work, they recommend using POS tagging and grammar check on microblogs.
The aforementioned works, similarly to the majority of state-of-the art, operate only with one attribute of opinions – orientation (the text can be positive, negative or neutral). Bollen et al. extended the analysed attributes with a six dimensional vector of moods (tension, depression, anger, vigour, fatigue, confusion) which can be extracted from the text using a psychometric instrument called Profile of Mood States (POMS) [9]. They analysed the impact of world global events on the mood in microblog posts. They identified a correlation between cultural, political and other events with the mood level of analysed posts. Another approach that extends the knowledge about emotions was presented by Whitelaw et al. [31]. They used a special psychological theory called appraisal theory for sentiment identification on movie reviews retrieved from the IMDB web portal. This theory says that emotions are consequences of how an author appraised a particular situation. The theory is used in sentiment analysis to discover what an author feels and what the situation that caused this feeling is. The authors created their own appraisal dictionary by a semi-automatic method – they manually annotated a large amount of words and searched for similar words in the dictionary using the created service. The final accuracy of the method varied from 77 % to 90 %.
Bloom in his thesis continued with the research of the use of the appraisal theory in text analysis and created a tool called FLAG (Functional Local Appraisal Grammar Extractor)
World Wide Web (2014) 17:847–867 851
[8]. This tool is used to search for areas in texts (primary longer texts, e.g., blogs) that are suitable for syntax analysis and the consequent lookup of entities in a text. Local grammars are used to find the entities. The method uses attitude, focus and also basic graduation of words from the appraisal theory. Bloom created a dictionary for the appraisal theory based on the dictionary created by Martin and White [20]. This dictionary contains words with appropriate categories known from the appraisal theory. When the extended dictionary was used, the accuracy of sentiment identification increased to about 50 %, which is better by tens of per cent in comparison to other approaches using the same datasets.
Microblog posts are usually classified according to orientation. Sentiment is always connected to an object and that is the reason why opinions should be mined from a text with a connection to an object. The target of a post can be an object, an event, or a person. Eventually, the author of a post expresses her or his opinion about a general topic. The differentiation of microblog posts containing a simple statement like “I feel great” from posts like “That new film Pirates of Caribbean was great and I like it” can noticeably extend the knowledge about the relationship between a user and various entities. If targets are extracted from every post, they can be connected with opinions and these connections can be aggregated to a comprehensive model of user sentiment related with the targets.
An approach taking into consideration targets (i.e., what the opinion is about) in sentences was presented by Jiang et al. [16]. They presented a three-step process for retrieving sentiment from microblog posts. The first step is subjectivity classification as in [5], the second step is orientation classification and the last step is the identification of targets in posts. The authors paid attention to the fact that sometimes it is hard to univocally identify opinions from such short texts. To improve this identification they created a graph of connections between microblog posts based on similarity and relativeness by utilizing microblog-specific features (retweet, reply, mention). The accuracy of target identification was reported to be about 68 %. The accuracy of sentiment classification was about 79 %. Extending sentiment classification with target identification raised the accuracy of sentiment classification to 85 %.
Tan and Lee [27] continued the research of Jiang et al. [16] and analysed the relationships between the authors of microblogs and identified the targets from her or his microblog posts. The authors verified experimentally that information about relationships between users can improve the classification of microblog posts according to their orientation. The authors also empirically proved that if two users share opinions, there is a high possibility that a connection between these users in a social network exists. Experiments were performed on a dataset created from posts that contain positive, negative or neutral opinions on politicians. According to the results, the accuracy of the classification varies by topics, but using relationship information, the accuracy increases by approximately 5–10 %. It is obvious from the results that the ‘follower/followee’ relationship is better for sentiment identification than the ‘mention’ relationship. The described works prove that the incorpo- ration of microblog features can increase the accuracy of sentiment identification and classification. Jiang et al. specialized in target identification in microblogs, but they did not take into consideration the relationship between users [16]. On the contrary, Tan and Lee specialized in the field of user relationships but did not analyse more general relationships between users, topics and sentiment [27]. The existing approaches suggest that the attribute of orientation may not be sufficient for sentiment identification and classification. We consider the appraisal theory as an important basis for broadening the knowledge about sentiment, because it can increase the accuracy of sentiment classification in texts as shown by Bloom [8]. We consider his research as the most elaborated work in the field which employs appraisal theory for sentiment classification to date.
852 World Wide Web (2014) 17:847–867
To conclude, there are only several researches that classify microblog posts by their orientation or also identify targets [27, 16]. Some approaches also use the fact that microblogs are a part of social network with various relationships and take them into consideration for sentiment identification and classification [27].
In our work we focus on the identification of sentiment that is connected to a particular target of a microblog post. We consider the appraisal theory as one of the most promising approaches in this field for further improvement of the results. To the best of our knowledge, there is a gap in the use of the appraisal theory to classify sentiments on microblogs. We introduce an approach using the appraisal theory to classify opinions that are connected to targets in microblog posts. We identify targets using hybrid patterns that we devised especially for microblogs. Our aim is to explore the suitability and the applicability of the theory in relation to the specifics of user-generated microblog content. The contribution of our work is a novel combination of structured sentiment analysis (target identification) by utilizing the appraisal theory applied to microblog.
3 Employing appraisal theory
In many works, opinions are classified according to their polarity as positive, negative or neutral. The appraisal theory shifts sentiment classification further and considers the ap- praisal expression – a basic grammatical unit by which an opinion is expressed [8]. Besides polarity, appraisal expressions also cover additional attributes of opinions that extend the basic description of the expressed opinion. The basic attributes attitude, engagement, graduation are complemented with polarity/orientation (see Figure 1) [20].
Attitude expresses the current state of a person at the time she or he wrote a text. It has multiple subcategories: affect, which represents the feelings of the author (happy, sad), appreciation, which talks about the opinion that a person has about the inner or outer qualities of an object (ugly, beautiful, shy, etc.), and judgment, which describes the behav- iour of somebody in a social context (heroic, feeble-minded).
Engagement determines the position of text proposal. It reflects probability or possibility (perhaps, seems) in most cases.
Graduation expresses that the meaning of a term is gradated by some adjective (or other part of speech) and its meaning is strengthened or weakened (very, few). It has two subclasses – force and focus. Force is about graduating (raise or lower) the interpersonal impact (slightly,
Appraisal
Attitude Engagement Graduation Polarity Orientation
Figure 1 The tree of appraisal theory categories (first two levels), based on [20]
Negative Positive
Marked Unmarked
Focus
Force
Affect Appreciation Judgement
World Wide Web (2014) 17:847–867 853
very, completely) and focus is graduating (blur or sharpen) the focus of term’s semantic categorizations (true friend, pure folly).
Orientation recognizes if a term is positive or negative. Note that some works refer to this attribute as to polarity (e.g., [22]). In contrast, the polarity attribute represents the fact that a sentimental term can be unmarked or marked. That means that the term’s orientation can be influenced by an expression that negates the meaning of the term. If a term contains such a negation (not, never), it is labelled as marked.
The main advantage of using the appraisal theory in sentiment classification is that it allows us to take a look deeper inside the mind of authors who write texts and find out their real feelings using linguistic and psychological analysis of their texts. It is not only a static comparison of words from text with a predefined dictionary, but also the analysis of types of words used and their categorization according to their effect on other words in text.
To take advantage of the appraisal theory, we need a dictionary of terms and phrases that are tagged with attributes from the theory. To build such a dictionary we started with words that are categorized by Martin and White [20]. We collected about eight hundred words this way. To extend this dictionary, we used WordNet to find synonyms to words identified in the previous step. For each word we created sets of similar words and manually categorized them according to their part of speech and the usual position of the word in sentences.
An example of entries in our appraisal dictionary is presented in Table 1. Each entry contains a word, a part-of-speech tag, a category and subcategories according to the appraisal theory, and appraisal value. The higher the value, the stronger the membership to given category. The appraisal value involves polarity and orientation, i.e., negative values represent negative orientation. The appraisal values were set based on analysis of large corpora of sentiment word occurrences.
4 Method for structured sentiment analysis based on appraisal theory
We propose the method for sentiment analysis based on the principles of the appraisal theory, oriented on targets and adapted for a microblog. The main difference from the previous works which use the appraisal theory is that we use the whole known basic appraisal tree and we particularly consider microblog specifics. In this section we describe our method which consists of the following steps in detail (see Figure 2):
Table 1
Word
Biggest Frustrated Harmonious Hate Incredible Nasty Obviously Patient Powerful Unclear
Part of speech
adjective adjective adjective adjective adjective adjective adverb adjective adjective adjective
Main category
graduation attitude attitude attitude attitude attitude engagement attitude attitude attitude
Second level category
force
affect appreciation affect judgment appreciation N/A judgment judgment appreciation
Other level categories Value maximization 1
unhappiness, misery −5 composition, balance 2 unhappiness, antipathy −5 admiration, normality 3 reaction, quality −3 N/A 1 admiration, tenacity 2 admiration, capacity 3 composition, complexity −1
An example of appraisal dictionary entries
854
World Wide Web (2014) 17:847–867
1
Pre-processing
2
Hybrid patterns
Target identification
3
Appraisal score computation
4
Sentiment analysis
Classifier
Positive Negative Neutral
Microblog posts classified
Microblog posts
Figure 2
Appraisal d i ct i o n ar y
1. 2. 3. 4.
microblog post pre-processing, target identification,
appraisal score computation, sentiment analysis.
Overview of the proposed method
In the first step we pre-process a microblog post to prepare it for further processing. Then we identify the targets of the post, i.e., what the opinions are about. We utilize hybrid patterns that we designed to address microblog-specific features, the language used in particular. In the third step we compute the score for targets expressing the overall appraisal of the post. We utilize the assembled appraisal dictionary and principles from appraisal theory. Finally, we analyse sentiment by extracting features and classifying microblog posts into positive, negative or neutral orientation classes. We employ selected general and microblog-specific features.
4.1 Microblog post pre-processing
The reason why we have to carefully pre-process the data is that microblog posts have specific content that differs from standard texts, which makes the analysis more difficult [25]. There are many abbreviations, slang words or microblog-specific words that appear in the text and that we have to cope with. The aim of this step is to prepare microblog posts for further processing. Pre- processing consists of the following steps:
– – – –
tokenisation—splitting texts into sentences and words,
slang removal—substitution of slang words for regular words or phrases, part of speech tagging—adding a POS tag to each word in a sentence, lemmatisation—converting a word into its basic lemma form.
During pre-processing we also count emoticons, slang words and microblog-specific features like “#”, “RT” or “@”. These counts are later used as attributes for sentiment classification.
4.2 Target identification
Having pre-processed the microblog posts, we continue with target identification. The aim of this step is to find exact targets in posts, i.e., to what entities opinions are bound. In our method, we utilize POS tags provided in the previous step to find entities that are candidates for targets. Usually these are nouns – subjects, objects or pronouns or their combination that represent the main idea of a post. To resolve which terms are targets in a sentence we use hybrid patterns—patterns based on combined lexical, syntactic and microblog-specific attributes of text.
World Wide Web (2014) 17:847–867 855
Hybrid patterns are necessary due to the specifics of microblog posts. The length of a post and language used in particular make posts less suitable for traditional statistical or more comprehensive linguistic approaches used in other domains (e.g., [13, 14]). Our motivation for using hybrid patterns was to improve the identification of targets in microblog posts. Statistical methods based on PMI are often used for this task (e.g., [16]). Such methods calculate occurrence frequency of all nouns and using this statistics, they identify some nouns as targets. Such approach may not be accurate, because a word can be identified as a target even if it is mentioned in the sentence incidentally. Contrarily, a word that appeared fewer times may not be identified as target, even though it actually is a target. The length of a microblog post makes this task really difficult. When analysing microblog posts, it is important to know what kind of sentence we analyse. The same combination of words can have a completely different meaning when used in a normal sentence or in a question. Also, even a simple comma can change the meaning of a sentence. We identify important words in the text by utilizing selected indicators—such as the position of nouns or verbs in a sentence, the position of words like ‘when’, ‘why’, ‘where’, and the division of sentences to sub- sentences—and assign them sentiment words from the post. We believe that by employing hybrid patterns, we are able to identify targets more accurately and to assign correct sentiment words to them.
Hybrid patterns do not rely on lists of common single words like statistical approaches often do. Note that we do not refuse the statistical approach. We also use it in our patterns, but not for single words, but for the detection of n-grams. Hybrid patterns allow us to find single targets with their position in the text. We create n-grams from nouns, adjectives and verbs from a sentence but do not know if obtained coupled words are meaningful. We use the statistical approach based on the amount of mutual information between words (PMI) to estimate if a coupled word is meaningful. According to the PMI ratio, we decide if we identify coupled words as a target or not.
We have analysed large amount of microblogs and we have observed several patterns repeating in microblog posts. Table 2 shows the patterns that we specified to match the targets in posts. Our aim was to take advantage of microblog idiosyncrasies in order to overcome traditional statistical approaches. We identified two groups of patterns: structural patterns and position patterns. Structural patterns are used to match the structure of a microblog post and divide posts into smaller parts or utilise external resources. Structural patterns can be applied recursively. We devised the position patterns by observing the most common post types on microblogs. They have roots in the nature of microblog communi- cation and reflect the intentions of microbloggers, who often write subjective messages. The position patterns deal with finding the position of the author or the target in a sentence.
By applying a pattern, we either split a microblog post into smaller parts or get a list of identified targets for a sentence. Usually more than one logical sentence appears in a microblog
Table 2 Structural and position hybrid patterns for target identifi- cation in microblog posts
Structural patterns
Position patterns
S1 Multisentence pattern P1
S2 RT pattern P2
S3 Condition pattern P3
S4 Link pattern P4
Question pattern
Pattern “I verb subject” Pattern “pronoun verb subject” Pattern “@# verb subject”
P5 No verb pattern
856 World Wide Web (2014) 17:847–867
post, so we need to combine the patterns. We arranged the patterns in logical order in which we match the patterns against a post.
Now we are going to describe the patterns in more detail. A microblog post example matching the pattern follows each pattern description.
Multisentence pattern This pattern splits a sentence by searching for punctuation and/or conjunctions. If a post has more sentences, there can be multiple targets in the sentence or all sentences have the same target. We perform an independent analysis of targets for all sub- sentences using other patterns. All identified targets are assigned to the original post.
Example: “Love my new I0S5 @Apple updates. Just when I think it can’t get any better somehow it simplifies my life more. That’s right-it’s an Apple”
RT pattern This pattern detects one of the most important features in microblogs—forwarding a post often referred to as retweet (term adopted from the most popular microblogging service Twitter). A retweet has two forms—official and unofficial. When using the official form, an author shares a post without modifying the original post. A mark “RT” is added in front of the original post to the beginning of the new post. To resolve what a target of a sentence in the post is, we remove this mark and match other patterns against the original post. In the case of unofficial retweets, an author not only adds the RT mark before the original post but she or he also adds her or his own comment before this mark. These comments contain an opinion or some kind of sentiment the author expresses about the original post. We split the post at the RT mark into two parts to resolve what the target of the post is. We match the remaining patterns on both parts of the post and afterwards we join both target sets together.
Example:“Interesting bookcase..RT @venturebeat: #google releases an infinite digital bookcase”
Condition pattern The condition pattern is used to match sentences containing “if[−then[−else]]” logic. Using this pattern we divide the sentence into the condition and the consequence parts. Usually we further analyse only the consequence part, because in most cases, this is the part containing an opinion on a topic. The topic may or may not occur in the condition.
Example: “@FishMama: If you made a purchase, just wait for the @apple survey! hate going b/c of the bad #custserv ”
Link pattern If we detect a link in a microblog post, we analyse its position in sentence. If the link is mentioned only peripherally, we ignore it. Otherwise we process the web resource’s content to identify the entities which the link points at.
Example: “Google Play Music coming to Europe on Nexus launch day http://bit.ly/ XMyLYG #android”
Question pattern If we identify a question mark in a post, it is not clear if the author is asking a question or is trying to state something. For example, the author can criticise some defect: “Why does not an iPhone have a longer battery life?”. In this case, the author points
World Wide Web (2014) 17:847–867 857
to the weakness of iPhones—too short battery life. But the sentence “Does iPhone have longer battery life than BlackBerry??” represents a typical question. To distinguish these two types, we analyse the occurrence of modal verbs, interrogative pronouns and the occurrence of engagement words according to the appraisal theory. We found out that the combination of certain interrogative pronouns (mainly the ones starting with ‘W’) with the combination of negative polarity and engagement words typically represents a negative statement about the subjects of sentences in posts.
Example: “@apple why is my iPhone battery so crappy? #fail”
Pattern “I verb subject” This pattern matches first person statements by identifying the pronoun “I”. Posts matching this pattern usually contain opinions of the author or their sentiment. Microblog posts of this type contain many adjectives. If a text matches this pattern, we mark the last noun (or personal pronoun) before an adverbial as a target and we mark the adjectives before the target but after the verb as an opinion.
Example: “I really hate dealing with the brain dead people at the @apple store. For such good products, customer service sucks.”
Pattern “pronoun verb subject” This pattern shows one of the basic ways in which authors write their posts. They use a personal pronoun such as “You” often followed by mention (@) or hash tag (#). One or more verbs usually follow these words. Afterwards, there are typically some adjectives and nouns or an adverbial of time/place. We mark the last noun (or personal pronoun) in a post as a target of the post—similarly to the previous pattern.
Example: “@apple you know the issues they have with dst and alarms”
Pattern “@# verb subject” This is the last pattern from the set of patterns that deals with the position of an author. Instead of pronouns in this pattern we identify a noun, a hash tag or a mention that is located at the beginning of a sentence as a main actor of the post. It is a third person actor.
Example: “@apple and @nintendo need to partner up for the application store”
No verb pattern This pattern is used for microblog posts, where no verbs are present. Usually these posts contain only pronouns, adjectives or interjections. The last noun in the sentence is marked as a target.
Example: “#samsung, #google android for reveal: live event web log by #engadet” 4.3 Appraisal score computation
Knowing what the targets of a microblog posts are, we compare the terms collocated with a target with the appraisal dictionary containing the terms aligned according to the appraisal theory. This is important because very often we can find many sentimental words in one sentence, but only some of these words (that are connected with a target) determine the
858 World Wide Web (2014) 17:847–867
sentiment in a post. The appraisal score builds on the basic notion of orientation of the microblog post and enhances it with a fine-grained representation of opinions resulting from the appraisal theory.
The way we calculate the appraisal score for a target (targetScore) is described by the (slightly simplified) pseudo-algorithm that follows:
Every word collocated with a target (appraisal candidate) is looked up in the appraisal dictionary. According to its appraisal category, we assign a numerical value to it, which represents its attitude. If the word has a neighbour that belongs to the graduation category, the value of attitude is multiplied by the appraisal value of this word (from graduation category). If a sentence contains engagement, this is summed over the appraisals. If there is a negation in the sentence, which belongs to the appraisal expression, the target score is multiplied by minus one.
The final appraisal score of the sentence is the sum of its targets’ appraisal scores. Finally, the appraisal score of a microblog post is the sum of the sentences’ appraisal scores.
Let us now consider an example that shall clarify the steps described above. Consider the post below. The first and fourth row are the post content, the second and fifth row contain labels if the word is an appraisal word (A), or if the word has been identified to be a target (tx). The third and sixth row contain appraisal values (the parentheses indicate that the appraisal value was not used in the example).
1.
We apply the three steps of our method sequentially as follows: In the pre-processing step, we split the post into four sentences:
c1. i’m amazed at how siri has changed the way i use my phone c2. i had no idea it would be this good, and it’s still in beta
c3. good job @apple
c4. wow
World Wide Web (2014) 17:847–867 859
2. For each sentence we match patterns to identify targets. c1 matches the pattern P2 and potential targets t1=“siri” and t2=“phone” are extracted. c2 matches the pattern S1 which splits c2 into:
c21. i had no idea it would be this good c22. and it’s still in beta
c21 matches the pattern P2 and potential target t31=“it” is extracted. c22 matches the pattern P3 and potential target t32=“it” is extracted. c3 matches the pattern P4 and potential target t4=“apple” is extracted. c4 does not match any pattern.
3. Having identified the targets, the appraisal score is computed for each target. First, appraisal words are looked up and are aligned with targets:
& “amaze” is aligned with t1
& no appraisal word is aligned with t2
& “good” and “no” are aligned with t31
& no appraisal word is aligned with t32
& “good” is aligned with t4
After that, the appraisal scores are computed:
& targetScore(t1)=3 because the appraisal value of “amaze” is 3
& targetScore(t2)=0 since no appraisal is detected to be aligned with t2
& targetScore(t31)=2 is computed as the sum of two appraisal scores: −1 for “no” and
3 for “good”
& targetScore(t32)=0 since no appraisal is detected to be aligned with t2
& targetScore(t4)=3 because the appraisal value of “good” is 3
The appraisal score for the whole post is 3+0+(2+0)+3=8, suggesting a relatively strong positive orientation that has a potential to get the post classified as positive in the following step of the method.
The example shows how the post is pre-processed, how hybrid patterns are matched and how the appraisal score is computed. No advanced elements such as graduation are included due to the complexity of the demonstration. Note that the need for hybrid pattern utilization resulted from the difficulty to perform proper natural language processing as a result of language idiosyncrasies present on a microblog. The described computation, usually applied to not as “linguistically nice” posts as in our example, aims to estimate appraisal character- istics rather than determine them exactly.
4.4 Sentiment analysis
In the final step of our method we analyse the sentiment of microblog posts by classifying them according to their orientation into positive, negative and neutral. We consider the microblog- specific features, the linguistic features and the appraisal features of microblog posts. A full list of features is presented in Table 3. We believe that microblog-specific features like forwarding the post (retweet), mention or hashtag improve the sentiment classification.
When we classify a microblog post according to a particular target, we consider only the sentences that contain the target. The results of classification are groups of positive, negative or neutral posts according to targets specified.
860
Table 3 The summary of microblog post features used for classification
Feature
HashTags Emoticon Mention IsForwarded
URL
Slang
Caps Punctuation Appraisal score
World Wide Web (2014) 17:847–867 Description
Count of HashTags
Count of emoticons
Count of mentions
Flag if the microblog post is forwarded (retweet) or not
Count of URLs
Count of slangs normalized by count of words
Count of words with capitals
The main type of punctuation
Appraisal score computed as described in previous section
5 Evaluation
We selected Twitter to evaluate our approach. Twitter is the most popular microblogging service and covers a wide range of users that represent a general sample of microbloggers. Note that we did not employ any Twitter-specific features that cannot be found in other microblogging services. Very similar results are expected for other general-purpose microblogging services.
Our aim was to test the following hypotheses:
1. Utilising the appraisal theory for sentiment analysis on a microblog increases the orienta- tion classification accuracy.
2. Structured sentiment analysis outperforms sentiment analysis with no target identification.
To evaluate our method we implemented the method in Java, taking advantage of the existing tools: GATE,1 the advanced language processing tool that we used for tokenisation, POS tagging and lemmatisation; NoSlang,2 the dictionary of slang terms that we used for slang identification; and JavaML,3 a library for text classification.
We used the existing ST02 dataset4 of manually annotated microblog posts with orientation, which was created for the queries “apple”, “google”, “microsoft”, “twitter” in October 2011. This dataset originally contained more than five thousand posts. Table 4 shows the distribution of the posts according to their orientation.
The microblog post that fits the condition of the query is the one that contains a hashtag, a mention or a normal word that matches the term being searched. The author of the dataset marked every post with labels “positive”, “negative”, “neutral” or “irrelevant”. In our work we ignored irrelevant posts, because it was not our concern to classify them into the basic three classes of orientation. Unfortunately, many microblog posts that were labelled by the author were no more accessible for public download. Finally, we gathered approximately 2,600 posts (all belonging to classes positive, negative or neutral).
1 http://gate.ac.uk/
2 http://www.noslang.com/dictionary/
3 http://java-ml.sourceforge.net/
4 http://www.sananalytics.com/lab/twitter-sentiment/
World Wide Web (2014) 17:847–867 Table 4 Distribution of words in
the dataset4 by orientation
5.1 Orientation classification
Topic Positive
Apple 191 Google 218 Microsoft 93 Twitter 68
Neutral Negative Irrelevant
581 377 164 604 61 498 671 138 513 647 78 611
861
Twitter term
@apple #google #microsoft #twitter
To evaluate our first hypothesis we classified the posts using the SVM classifier while employing 10-fold cross validation on the dataset. In each fold we chose data in the way that every training set and testing set contained microblog posts of all three types of orientation. The results are shown in Table 5. The rows of the table contain the count of microblog posts that were manually labelled as positive, negative or neutral. The columns contain the count of posts assigned an orientation class by our method.
The classification accuracy for all classes was 87.57 % (calculated as a count of a correctly identified orientation class of microblog posts divided by the overall count of posts). The differences between particular folds were about 1–2 %. The accuracy across all folds is depicted in Figure 3.
The most difficult part of the sentiment analysis was to identify non-personal posts, because these sentences also contain some terms from the appraisal dictionary. A typical example of such post is an offer: “Buy the best iPhone in our shop” or “win a best game, click on link”. The reason is simple—every offer propagates its product as the best product in the area, so we generally can say that it is an appraisal, but not personal. If the sentence above looked like “I bought the best phone I ever had in that shop” it would be a real opinion of a person who bought a great mobile. But when the seller says it is the best, it obviously might not be true. Offers on microblogs are a special category of microblog posts that our algorithm cannot handle (one could say that it is spam). We also cannot recognize sentiment in microblog posts that have too many grammatical errors—we cannot find words from these sentences in the appraisal dictio- nary and also not in slang dictionaries.
In order to determine the contribution of our work with respect to state-of-the-art, we compared the results of our method to the results of Go et al. [12]. The authors in their work used a dictionary of n-grams to identify sentiment words in the text. They used a count of identified sentiment words, together with other features to classify microblog posts using SVM and Naïve Bayes. The authors released their work as a web service,5 making it easy to use. For each microblog post from our dataset we used the service to obtain the class and then we calculated the accuracy of classification. According to the obtained results, 25 % of the incorrectly classified microblog posts were positive ones marked as neutral, 40 % were negative posts marked as neutral. The remaining ones were neutral posts from which 75 % were marked as positive. The algorithm of Go et al. achieved 79.12 % overall accuracy. Our method is better by more than 8 % when applied to the same data. The visual comparison of the results is depicted in Figure 4.
The presented results are obtained using specific types of queries to retrieve microblog posts. It is important to note that we do not expect different results when using other—non- technological or more abstract types of queries (e.g., “art”). Our assumption here is that there is no significant difference in the microbloggers’ writing styles when writing about different
5 https://sites.google.com/site/twittersentimenthelp/api
862
Table 5 The results of classification
World Wide Web (2014) 17:847–867 Positive Negative Neutral Sum
247 2 40 289 4 326 26 356 144 107 1702 1,953 395 435 1768 2,598
96.00% 94.00% 92.00% 90.00% 88.00% 86.00% 84.00% 82.00% 80.00% 78.00% 76.00%
Positive Negative Neutral Sum
topics. The writing style of a microblog post is related rather to the intention users have on the microblog [15] than to topic of the post. The proposed method is domain independent and it has no constraints resulting in a different performance when applied on microblog posts retrieved by other types of queries.
We performed an additional small experiment to assess this assumption. We created a small dataset containing tweets retrieved using query “art” representing a generic type of posts. We retrieved and manually labelled 100 posts from Twitter altogether (17 were labelled as positive, 12 as negative, 64 as neutral and 7 as irrelevant—consistently with the ST02 dataset). We applied our method and obtained the overall accuracy of 87.10 %. This result corresponds with our previous findings. It shows that the method behaves similarly when applied to other types of queries.
We believe that the main reason for the higher accuracy of our method when compared to Go et al. is the advanced post analysis—in terms of both comprehensive content analysis based on the appraisal theory and target identification. Target identification improves the location of important—subject bearing—words in a text and the appraisal theory allows making a more precise evaluation of sentiment towards targets. Our approach addresses to a certain extent a lack of semantics problem identified by the Go et al., which hinders sentiment analysis. Identification of targets constitutes a step towards semantic analysis of microblog posts. The influence of the target identification on the overall results is discussed in detail in the following section.
5.2 Target identification
The goal of our next experiment was to evaluate the contribution of target identification to sentiment analysis on microblogs. First, we evaluated the accuracy of target identification. The author of the ST02 dataset provided every microblog post also with its targets. In order to verify the target identification in microblog posts, we compared these targets with the targets identified by our algorithm. Our algorithm also identifies multi-words as targets. If
Figure 3
Spread of classification accuracy over all folds
Positive
Negative
Neutral All
World Wide Web (2014) 17:847–867
863
100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00%
0.00%
Our method Go et al.
Figure 4
Comparison of our approach with approach of Go et al. [12]
positive
negative
neutral
all
this term contains a target which was marked by the author of the dataset, then we marked this target as correct. The overall accuracy of target identification was 85.18 %.
Table 6 presents the statistics on coverage of hybrid pattern matches in the ST02 dataset. It shows that patterns were matched 8,003 times. The most often matching pattern was No verb pattern, occurring 1.211 times per tweet in average.
We compared the classification employing target analysis to the classification without it (all opinions belong to the whole post) utilizing the same ST02 dataset. We consider a sentiment related to the whole post as an average of its target’s sentiments. It is based on a simplified assumption of this experiment that in a microblog post, typically, one major opinion is expressed. The results of the comparison are shown in Figure 5.
The results show that employing target identification significantly improves the overall accuracy of sentiment classification on a microblog. We further investigated the impact of target identification. We split the dataset into two parts: posts where target identification was successful and post where it was not. The parts contained 2,213 and 385 posts, respectively. We obtained the overall classification accuracy of 88.93 % for the part with correctly identified targets and
Table6 Hybridpatternscoverage in the ST02 dataset
Pattern
S1 Multisentence pattern
S2 RT pattern
S3 Condition pattern
S4 Link pattern
P1 Question pattern
P2 Pattern “You verb subject”
P3 Pattern “I verb subject”
P4 Pattern “@# verb subject”
P5 No verb pattern
Sum
Matches Matches [%]
Average match count per tweet
0.162 0.136 0.022 0.425 0.136 0.156 0.178 0.654 1.211 3.080
422 353 58 1,105
354 406 462
1,698 3,145 8,003
5.27 4.41 0.72
13.81 4.42 5.07 5.77
21.22
39.30 100.00
864
World Wide Web (2014) 17:847–867 With TA
100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00%
0.00%
Without TA
Figure 5
Comparison of classification with target analysis and without it
positive
negative
neutral
all
80.00 % for the part where targets were identification was incorrect. This confirms our findings that target identification correlates with the overall sentiment classification accuracy.
The obtained results are very reasonable. We could expect an even higher improvement if the accuracy of target identification was higher. This remains an open problem for our further research. We see a potential in more comprehensive linguistic processing of microblog posts while focusing on specific linguistic features on a microblog, resulting in higher match count of patterns.
We conducted several experiments in order to evaluate our method. We compared our classification results with a publicly available manually labelled dataset. We obtained a very reasonable accuracy due to the utilisation of the appraisal theory and the incorporation of target identification, employing microblog-specific pattern matching. The results suggest that opinion mining based on the notion of appraisal (realised by appraisal expressions) improves the accuracy of sentiment analysis in microblog posts, which are more difficult to process in comparison to traditional text, due to the specifics of microblog posts such as small length, specific language, a large number of slang expressions and unusual sentence-level constructions.
6 Conclusions
Social media preservation poses challenges on how and in what form we should archive social media records for further generations. It is important to preserve relevant information without losing the information about the context of social communication. Along with user data, it is useful to preserve additional derived information such as sentiment or opinions, which reflect the state of the time period.
In this paper we presented a novel method for sentiment analysis on a microblog. We utilise the appraisal theory to identify sentiment that is connected to a main target of a microblog post. We created an appraisal dictionary by following the works of Martin and White [20] and Bloom [8] to contain more annotated terms. Compared to previous works, we have used all basic categories of the appraisal theory. To the best of our knowledge, this is the first time when the appraisal theory is applied to the analysis of microblogs. This task was a challenge in comparison to the “traditional” content analysis because of the short length and specific language of microblog posts.
World Wide Web (2014) 17:847–867 865
We conducted several experiments using a large publicly available dataset in order to evaluate our method. In the first one, we evaluated the accuracy of classification of microblog posts according to their orientation. We achieved the accuracy of 87.57 %, making our method more accurate than state-of-the-art [12] when tested on the same dataset. In comparison to other works, our method is independent from the topics of microblogs. We have shown that despite the microblog’s weaknesses (in terms of quantity of content to analyse), our approach can effectively classify random post related to different topics. We have also shown that target identification in microblog posts increases the accuracy of sentiment analysis. Moreover, there is room for further improvements, since we were able to identify only 85.18 % of targets correctly, mainly due to the complicated syntactical structure of microblog posts. However, there is a reasonable chance to further improve the accuracy of sentiment analysis if target identification performs better, e.g., by extending and further improving the utilised patterns. This ranks among the challenges of our future work.
We performed all experiments utilising Twitter as the most popular microblogging service today. Since we did not employ any particular feature of Twitter that cannot be applied for other microblogging services, the results can be generalised for all general-purpose microblogs.
In addition, our future work will cover the detection of neutral microblog posts that are similar to offer postings, i.e., the posts which contain some appraisal but do not contain clearly oriented sentiment and which form a relatively big portion of all microblog posts. A potential improvement of the method may be expected by further enrichment of the appraisal dictionary.
Microblog posts represent the evidence of human history written by people themselves. The employment of the appraisal theory has allowed us to draw some statistical conclusions about the types of sentiment that the authors of microblogs express via the texts they write. Besides the traditional utilisation in determining opinions on products, people or events, which is particu- larly useful when considering preservation of relevant microblog records and their context, the application of our method reaches beyond the tasks related to social media preservation. Sentiment and opinions are potentially interesting information when building a user model, especially when considering open information spaces, where the user model is limited to the interests’ estimations of users [6, 17]. We obtain information about what a user likes or dislikes and in what way. We know if she or he admires, hates, judges or fears something. This information is very valuable when we want to improve user experience on the Web, e.g., by increasing accuracy of information retrieval tasks [18], by recommending relevant products, services or just by offering articles on the Web that match or fit to user interests better [7]. This particularly applies to microbloggers and opens new ways how to make microblogging networks even more social by discovering new potential relationships between their users.
Acknowledgments This work was partially supported by the Scientific Grant Agency of Slovak Republic, grant No. VG1/0675/11, the Slovak Research and Development Agency under the contract No. APVV-0208- 10 and it is the partial result of the Research & Development Operational Programme for the project Research of methods for acquisition, analysis and personalized conveying of information and knowledge, ITMS 26240220039, co-funded by the ERDF.
References
1. Abel, F., Hauff, C., Houben, G. J., Stronkman, R., Tao, K.: Semantics + Filtering + Search = Twitcident exploring information in social web streams. In: Proceedings of the 23rd ACM conference on Hypertext and social media, HyperText ′12. ACM, pp. 285–294 (2012)
866 World Wide Web (2014) 17:847–867
2. Abelovský, P.: Opinion mining. Diploma thesis. Faculty of informatics and Information Technologies, Slovak University of Technology in Bratislava (2011)
3. Archak, N., Ghose, A., and Ipeirotis, P. G.: Show me the money!: Deriving the pricing power of product features by mining consumer reviews. In: Proc. of the 13th ACM SIGKDD int. conf. on Knowledge discovery and data mining, KDD ′07. ACM, pp. 56–65 (2007)
4. Bae, Y., and Lee, H.: A sentiment analysis of audiences on twitter: Who is the positive or negative audience of popular twitterers? In: Proc. of the 5th Int. Conf. on Convergence and hybrid information technology, ICHIT′11, LNCS 6935, Springer, pp. 732–739 (2011)
5. Barbosa, L., Feng, J.: Robust sentiment detection on twitter from biased and noisy data. In: Proc. of the 23rd Int. Conf. on Computational Linguistics, COLING ′10. Association for Computational Linguistics, pp. 36–44 (2010)
6. Barla, M., Bieliková, M.: Ordinary web pages as a source for metadata acquisition for open corpus user modeling. In: Proc. of IADIS WWW/Internet 2010. IADIS Press, pp. 227–233 (2010)
7. Bieliková, M., Kompan, M., Zeleník, D.: Effective hierarchical vector-based news representation for personalized recommendation. Comput. Sci. Inf. Syst. 9(1), 303–322 (2012)
8. Bloom, K.: Sentiment analysis based on appraisal theory and functional local grammars. Dissertation thesis, Illinois Institute of Technology (2011)
9. Bollen, J., Pepe, A., Mao, H.: Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. In: Proc. of the Fifth Int. AAAI Conf. on Weblogs and Social Media, ICWSM 2011, pp. 450– 453 (2011)
10. Diakopoulos, N. A., Shamma, D. A.: Characterizing debate performance via aggregated twitter sentiment. In: Proc. of the 28th int. Conf. on Human factors in computing systems, ACM, pp. 1195–1198 (2010)
11. Dong, A.: Time is of the essence: Improving recency ranking using Twitter data. In: Proc. of the 19th int. Conf. on World Wide Web. ACM, pp. 331–340 (2010)
12. Go, A., Bhayani, R., Huang, L.: Twitter sentiment classification using distant supervision. In: CS224N Project Report, Stanford (2009)
13. Guo, H., Zhu, H., Guo, Z., Su, Z. Domain customization for aspect-oriented opinion analysis with multi- level latent sentiment clues. In: Proc. of the 20th ACM int. conf. on Information and knowledge management. ACM, pp. 2493–2496 (2011)
14. Hu, M., Liu, B.: Mining opinion features in customer reviews. In Proc. of the National Conf. on Artificial Intelligence. AAAI Press, MIT Press, pp. 755–760 (2004)
15. Java, A., Song, X., Finin, T., Tseng, B.: Why we twitter: understanding microblogging usage and communities. In: Proc. of the 9th WebKDD and 1st SNA-KDD 2007 workshop on Web mining and social network analysis. ACM, pp. 56–65 (2007)
16. Jiang, L., Yu, M., Zhou, M., Liu, X., Zhao, T.: Target-dependent twitter sentiment classification. In: Proc. of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, HLT ′11, vol. 1..Association for Computational Linguistics Stroudsburg, pp. 151–160 (2011)
17. Kanta, M., Šimko, M., Bieliková, M.: Trend-aware user modeling with location-aware trends. In: Proc. of 7th Int. Workshop on Semantic and Social Media adaptation and Personalization, SMAP 2012. IEEE CS, pp. 23–28 (2012)
18. Kramár, T., Barla, M., Bieliková, M. Disambiguating search by leveraging a social context based on the stream of user’s activity. In: Proc. of 18th int. conf. on User Modeling, Adaptation, and Personalization, UMAP 2010. Springer, pp. 387–392 (2010)
19. Majer, T., Šimko, M.: Leveraging microblogs for resource ranking. In: Proc. of 38th Conf. on Current Trends in Theory and Practice of Computer Science, SOFSEM 2012. Springer, pp. 518– 529 (2012)
20. Martin, J.R., White, P.R.R.: The Language of Evaluation: Appraisal in English. Palgrave Macmillan, London (2005)
21. Pak, A., Paroubek, P.: Twitter as a corpus for sentiment analysis and opinion mining. In: Proc. of the Seventh Int. Conf. on Language Resources and Evaluation, LREC′10, pp. 1320–1326 (2010)
22. Pandey. V., Iver, C.: Sentiment analysis of microblogs (2009) Available online: http://www.stanford.edu/ class/cs229/proj2009/PandeyIyer.pdf. Accessed July 5th, 2013
23. Pang B., Lee, L.: Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2), Now Publishers Inc., pp. 1–135 (2008)
24. Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up? Sentiment classification using machine learning techniques. In: Proc. of the Conf. on Empirical Methods in Natural Language Processing, EMNLP-02. Association for Computational Linguistics, pp. 79–86 (2002)
25. Parikh, R., Movassate, M.: Sentiment analysis of user-generated twitter updates using various classifica- tion techniques. CS224N Final Report, pp. 1–18 (2009)
World Wide Web (2014) 17:847–867 867
26. Reisenzein, R.: Arnold’s theory of emotion in historical perspective. Cognit. Emot. 20(7), 920–951 (2006). Psychology press
27. Tan, C., Lee, L., Tang, J., Jiang, L., Zhou, M., Li, P.: User-level sentiment analysis incorporating social networks. In: Proc. of the 17th ACM SIGKDD Int. Conf. on Knowledge discovery and data mining, ACM, pp. 1397–1405 (2011)
28. Tchalakova, M., Gerdemann, D., Meurers, D. Automatic sentiment classification of product reviews using maximal phrases based analysis. In: Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis, WASSA ′11, Association for Computational Linguistics, pp. 111–117 (2011)
29. Thet, T. T., Na, J. C., Khoo, C. S. G., Shakthikumar, S.: Sentiment analysis of movie reviews on discussion boards using a linguistic approach. In: Proc. of the 1st Int. CIKM workshop on Topic-sentiment analysis for mass opinion, TSA ′09, ACM, pp. 81–84 (2009)
30. Tumasjan, A., Sprenger, T.O., Sandner, P.G., Welpe, I. M.: Predicting elections with Twitter: What 140 characters reveal about political sentiment. In: 4th Int. AAAI Conf. on Weblogs and Social Media, ICWSM 2010, Association for the Advancement of Artificial Intelligence, pp. 178–185 (2010)
31. Whitelaw, C., Garg, N., Argamon, S.: Using appraisal groups for sentiment analysis. In: Proc. of the 14th ACM Int. Conf. on Information and knowledge management, CIKM’05. ACM Press, pp. 625–631 (2005)
32. Zhang, W., Yu, C.: UIC at TREC 2006 Blog Track. In Proc. of 15th Text REtrieval Conference, TREC 2006. (2006) Available online: http://trec.nist.gov/pubs/trec15/papers/ uillinois-chicago.blog.final.pdf.
Accessed July 5th, 2013
33. Zhang, W., Yu, C., Meng, W.: Opinion retrieval from blogs. In: Proc. of the sixteenth ACM Conf. on
Information and knowledge management, ACM, pp. 831–840 (2007)