Opinion Mining Sudeshna Sarkar 24th and 26th October 2007 Bing Liu Janyce Wiebe, U.
Download
Report
Transcript Opinion Mining Sudeshna Sarkar 24th and 26th October 2007 Bing Liu Janyce Wiebe, U.
Opinion Mining
Sudeshna Sarkar
24th and 26th October 2007
Bing Liu
Janyce Wiebe, U. Pittsburgh
Claire Cardie, Cornell U.
Ellen Riloff, U. Utah
Josef Ruppenhofer, U. Pittsburgh
Introduction – facts and opinions
Two main types of information on the Web.
Facts and Opinions
Current search engines search for facts (assume they
are true)
Facts can be expressed with topic keywords.
Search engines do not search for opinions
Opinions are hard to express with a few keywords
How do people think of Motorola Cell phones?
Current search ranking strategy is not appropriate for
opinion retrieval/search.
MIAS - June 14, 2007
3
Introduction – user generated content
Word-of-mouth on the Web
One can express personal experiences and opinions on almost
anything, at review sites, forums, discussion groups, blogs ...,
(called the user generated content.)
They contain valuable information
Web/global scale
No longer limited to your circle of friends
Our interest: to mine opinions expressed in the user-generated
content
An intellectually very challenging problem.
Practically very useful.
MIAS - June 14, 2007
4
Introduction – Applications
Businesses and organizations: product and service benchmarking.
Market intelligence.
Business spends a huge amount of money to find consumer
sentiments and opinions.
Consultants, surveys and focused groups, etc
Individuals: interested in other’s opinions when
Purchasing a product or using a service,
Finding opinions on political topics,
Many other decision making tasks.
Ads placements: Placing ads in user-generated content
Place an ad when one praises an product.
Place an ad from a competitor if one criticizes an product.
Opinion retrieval/search: providing general search for opinions.
MIAS - June 14, 2007
5
Question Answering
Opinion question answering:
Q: What is the international reaction
to the reelection of Robert Mugabe
as President of Zimbabwe?
A: African observers generally
approved of his victory while
Western Governments denounced
it.
MIAS - June 14, 2007
6
More motivation
Product review mining: What features of the ThinkPad T43 do
customers like and which do they dislike?
Review classification: Is a review positive or negative toward the movie?
Tracking sentiments toward topics over time: Is anger ratcheting up or
cooling down?
Etc.
MIAS - June 14, 2007
7
Two types of evaluation
Direct Opinions: sentiment expressions on some objects,
e.g., products, events, topics, persons
E.g., “the picture quality of this camera is great”
Subjective
Comparisons: relations expressing similarities or
differences of more than one object. Usually expressing
an ordering.
E.g., “car x is cheaper than car y.”
Objective or subjective.
MIAS - June 14, 2007
8
Opinion search
(Liu, Web Data Mining book, 2007)
Can you search for opinions as conveniently as general
Web search?
Whenever you need to make a decision, you may want
some opinions from others,
Wouldn’t it be nice? you can find them on a search
system instantly, by issuing queries such as
Opinions: “Motorola cell phones”
Comparisons: “Motorola vs. Nokia”
Cannot be done yet!
MIAS - June 14, 2007
9
Typical opinion search queries
Find the opinion of a person or organization (opinion
holder) on a particular object or a feature of an object.
E.g., what is Bill Clinton’s opinion on abortion?
Find positive and/or negative opinions on a particular
object (or some features of the object), e.g.,
customer opinions on a digital camera,
public opinions on a political topic.
Find how opinions on an object change with time.
How object A compares with Object B?
Gmail vs. Yahoo mail
MIAS - June 14, 2007
10
Find the opinion of a person on X
In some cases, the general search engine can handle it,
i.e., using suitable keywords.
Bill Clinton’s opinion on abortion
Reason:
One person or organization usually has only one
opinion on a particular topic.
The opinion is likely contained in a single document.
Thus, a good keyword query may be sufficient.
MIAS - June 14, 2007
11
Find opinions on an object X
We use product reviews as an example:
Searching for opinions in product reviews is different
from general Web search.
E.g., search for opinions on “Motorola RAZR V3”
General Web search for a fact: rank pages according to
some authority and relevance scores.
The user views the first page (if the search is perfect).
One fact = Multiple facts
Opinion search: rank is desirable, however
reading only the review ranked at the top is dangerous
because it is only the opinion of one person.
One opinion Multiple opinions
MIAS - June 14, 2007
12
Search opinions (contd)
Ranking:
produce two rankings
Or, one ranking but
Positive opinions and negative opinions
Some kind of summary of both, e.g., # of each
The top (say 30) reviews should reflect the natural distribution
of all reviews (assume that there is no spam), i.e., with the
right balance of positive and negative reviews.
Questions:
Should the user reads all the top reviews? OR
Should the system prepare a summary of the reviews?
MIAS - June 14, 2007
13
Reviews are similar to surveys
Reviews can be regarded as traditional surveys.
In traditional survey, returned survey forms are treated
as raw data.
Analysis is performed to summarize the survey
results.
E.g., % against or for a particular issue, etc.
In opinion search,
Can a summary be produced?
What should the summary be?
MIAS - June 14, 2007
14
Roadmap
Opinion mining – the abstraction
Domain level sentiment classification
Sentence level sentiment analysis
Feature-based sentiment analysis and
summarization
Comparative sentence and relation extraction
Summary
MIAS - June 14, 2007
15
Opinion mining – the abstraction
(Hu and Liu, KDD-04)
Basic components of an opinion
Opinion holder: A person or an organization that holds an specific
opinion on a particular object.
Object: on which an opinion is expressed
Opinion: a view, attitude, or appraisal on an object from an opinion
holder.
Objectives of opinion mining: many ...
We use consumer reviews of products to develop the
ideas. Other opinionated contexts are similar.
MIAS - June 14, 2007
16
Object/entity
Definition (object): An object O is an entity which can be
a product, person, event, organization, or topic. O is
represented as a tree or taxonomy of components (or
parts), sub-components, and so on.
An opinion can be expressed on any node or attribute of
the node.
To simplify our discussion, we use “features” to represent
both components and attributes.
Each node represents a component and is associated with a set of
attributes.
O is the root node (which also has a set of attributes)
The term “feature” should be understood in a broad sense
Product feature, topic or sub-topic, event or sub-event, etc
Note: the object O itself is also a feature.
MIAS - June 14, 2007
17
A model of a review
An object is represented with a finite set of features,
F = {f1, f2, …, fn}.
Each feature fi in F can be expressed with a finite set of words or
phrases Wi, which are synonyms.
That is to say: we have a set of corresponding synonym sets W = {W1,
W2, …, Wn} for the features.
Model of a review: An opinion holder j comments on a subset of the
features Sj F of an object O.
For each feature fk Sj that j comments on, he/she
chooses a word or phrase from Wk to describe the feature,
expresses a positive, negative or neutral opinion on fk.
MIAS - June 14, 2007
18
Opinion mining tasks
At the document (or review) level:
Task: sentiment classification of reviews
Classes: positive, negative, and neutral
Assumption: each document (or review) focuses on a single object O
(not true in many discussion posts) and contains opinion from a
single opinion holder.
At the sentence level:
Task 1: identifying subjective/opinionated sentences
Classes: objective and subjective (opinionated)
Task 2: sentiment classification of sentences
Classes: positive, negative and neutral.
Assumption: a sentence contains only one opinion
not true in many cases.
Then we can also consider clauses.
MIAS - June 14, 2007
19
Opinion mining tasks (contd)
At the feature level:
Task 1: Identifying and extracting object features that have been
commented on in each review.
Task 2: Determining whether the opinions on the features are positive,
negative or neutral in the review.
Task 3: Grouping feature synonyms.
Produce a feature-based opinion summary of multiple reviews
(more on this later).
Opinion holders: identify holders is also useful, e.g., in news articles,
etc, but they are usually known in user generated content, i.e., the
authors of the posts.
MIAS - June 14, 2007
20
More at the feature level
Problem 1: Both F and W are unknown.
We need to perform all three tasks:
Problem 2: F is known but W is unknown.
All three tasks are needed. Task 3 is easier. It
becomes the problem of matching discovered features
with the set of given features F.
Problem 3: W is known (F is known too).
Only task 2 is needed.
F: the set of features
W: synonyms of each feature
MIAS - June 14, 2007
21
Roadmap
Opinion mining – the abstraction
Document level sentiment classification
Sentence level sentiment analysis
Feature-based sentiment analysis and summarization
Comparative Sentence and relation extraction
Summary
MIAS - June 14, 2007
22
Sentiment classification
Classify documents (e.g., reviews) based on the overall sentiments
expressed by authors,
Positive, negative, and (possibly) neutral
Since in our model an object O itself is also a feature, then
sentiment classification essentially determines the opinion
expressed on O in each document (e.g., review).
Similar but different from topic-based text classification.
In topic-based text classification, topic words are important.
In sentiment classification, sentiment words are more important,
e.g., great, excellent, horrible, bad, worst, etc.
MIAS - June 14, 2007
23
Unsupervised review classification
(Turney, ACL-02)
Data: reviews from epinions.com on automobiles, banks,
movies, and travel destinations.
The approach: Three steps
Step 1:
Part-of-speech tagging
Extracting two consecutive words (two-word phrases)
from reviews if their tags conform to some given
patterns, e.g., (1) JJ, (2) NN.
MIAS - June 14, 2007
24
Step 2: Estimate the semantic orientation of the
extracted phrases
Use Pointwise mutual information
P( word1 word 2 )
PMI ( word1 , word 2 ) log 2
P( word1 ) P( word 2 )
Semantic orientation (SO):
SO(phrase) = PMI(phrase, “excellent”)
PMI(phrase, “poor”)
Using AltaVista near operator to do search to find the
number of hits to compute PMI and SO.
MIAS - June 14, 2007
25
Step 3: Compute the average SO of all phrases
classify the review as recommended if average SO is
positive, not recommended otherwise.
Final classification accuracy:
automobiles - 84%
banks - 80%
movies - 65.83
travel destinations - 70.53%
MIAS - June 14, 2007
26
Sentiment classification using machine
learning methods (Pang et al, EMNLP-02)
The paper applied several machine learning techniques to
classify movie reviews into positive and negative.
Three classification techniques were tried:
Naïve Bayes
Maximum entropy
Support vector machine
Pre-processing settings: negation tag, unigram (single
words), bigram, POS tag, position.
SVM: the best accuracy 83% (unigram)
MIAS - June 14, 2007
27
MIAS - June 14, 2007
28
Roadmap
Opinion mining – the abstraction
Document level sentiment classification
Sentence level sentiment analysis
Feature-based sentiment analysis and
summarization
Comparative sentence and relation
extraction
Summary
MIAS - June 14, 2007
29
Sentence-level sentiment analysis
Document-level sentiment classification is too coarse for most
applications.
Let us move to the sentence level.
Much of the work on sentence level sentiment analysis focus on
identifying subjective sentences in news articles.
Classification: objective and subjective.
All techniques use some forms of machine learning.
E.g., using a naïve Bayesian classifier with a set of data
features/attributes extracted from training sentences (Wiebe et al.
ACL-99).
MIAS - June 14, 2007
30
Using learnt patterns (Rilloff
and Wiebe, EMNLP-
03)
A bootstrapping approach.
A high precision classifier is used to automatically identify some
subjective and objective sentences.
A set of patterns are then learned from these identified subjective
and objective sentences.
Two high precision (low recall) classifiers were used,
a high precision subjective classifier
A high precision objective classifier
Based on manually collected lexical items, single words and ngrams, which are good subjective clues.
Syntactic templates are provided to restrict the kinds of patterns to be
discovered, e.g., <subj> passive-verb.
The learned patterns are then used to extract more subject and
objective sentences (the process can be repeated).
MIAS - June 14, 2007
31
Subjectivity and polarity (orientation)
(Yu and Hazivassiloglou, EMNLP-03)
For subjective or opinion sentence identification, three
methods were tried:
Sentence similarity.
Naïve Bayesian classification.
Multiple naïve Bayesian (NB) classifiers.
For opinion orientation (positive, negative or neutral) (also
called polarity) classification, it uses a similar method to
(Turney, ACL-02), but
with more seed words (rather than two) and based on
log-likelihood ratio (LLR).
For classification of each word, it takes average of LLR
scores of words in the sentence and use cutoffs to
decide positive, negative or neutral.
MIAS - June 14, 2007
32
Other related work
Consider gradable adjectives (Hatzivassiloglou and
Wiebe, Coling-00)
Semi-supervised learning with the initial training set
identified by some strong patterns and then applying NB
or self-training (Wiebe and Riloff, CicLing 05)
Finding strength of opinions at the clause level (Wilson
etal, AAAI-04)
Sum up orientations of opinion words in a sentence (or
within some word window) Kim and Hovy, Coling-04
MIAS - June 14, 2007
33
Let us go further?
Sentiment classifications at both document and sentence (or clause)
level are useful, but
They do not find what the opinion holder liked and disliked.
An negative sentiment on an object
does not mean that the opinion holder dislikes everything about the
object.
A positive sentiment on an object
does not mean that the opinion holder likes everything about the
object.
We need to go to the feature level.
MIAS - June 14, 2007
34
But before we go further
Let us discuss Opinion Words or Phrases (also called polar words,
opinion bearing words, etc). E.g.,
Positive: beautiful, wonderful, good, amazing,
Negative: bad, poor, terrible, cost someone an arm and a leg
(idiom).
They are instrumental for opinion mining
Three main ways to compile such a list:
Manual approach: not a bad idea, only an one- time effort
Corpus-based approaches
Dictionary-based approaches
Important to note:
Some opinion words are context independent. (eg, good)
Some are context dependent. (eg, long)
MIAS - June 14, 2007
35
Corpus-based approaches
Rely on syntactic or co-occurrence patterns in large corpuses.
(Hazivassiloglou and McKeown, ACL-97; Turney, ACL-02; Yu and
Hazivassiloglou, EMNLP-03; Kanayama and Nasukawa, EMNLP-06; Ding and
Liu, 2007)
Can find domain (not context) dependent orientations (positive,
negative, or neutral).
(Turney, ACL-02) and (Yu and Hazivassiloglou, EMNLP-03) are
similar.
Assign opinion orientations (polarities) to words/phrases.
(Yu and Hazivassiloglou, EMNLP-03) is different from (Turney,
ACL-02) in that
using more seed words (rather than two) and using loglikelihood ratio (rather than PMI).
MIAS - June 14, 2007
36
Corpus-based approaches (contd)
Use constraints (or conventions) on connectives to identify opinion words
(Hazivassiloglou and McKeown, ACL-97; Kanayama and Nasukawa,
EMNLP-06; Ding and Liu, SIGIR-07). E.g.,
Conjunction: conjoined adjectives usually have the same orientation
(Hazivassiloglou and McKeown, ACL-97).
E.g., “This car is beautiful and spacious.” (conjunction)
AND, OR, BUT, EITHER-OR, and NEITHER-NOR have similar
constraints
Learning using
log-linear model: determine if two conjoined adjectives are of the same or
different orientations.
Clustering: produce two sets of words: positive and negative
Corpus: 21 million word 1987 Wall Street Journal corpus.
MIAS - June 14, 2007
37
Dictionary-based approaches
Typically use WordNet’s synsets and hierarchies to
acquire opinion words
Start with a small seed set of opinion words
Use the set to search for synonyms and antonyms in
WordNet (Hu and Liu, KDD-04; Kim and Hovy, COLING04).
Manual inspection may be used afterward.
Use additional information (e.g., glosses) from WordNet
(Andreevskaia and Bergler, EACL-06) and learning (Esuti
and Sebastiani, CIKM-05).
Weakness of the approach: Do not find domain and/or
context dependent opinion words, e.g., small, long, fast.
MIAS - June 14, 2007
38
Roadmap
Opinion mining – the abstraction
Document level sentiment classification
Sentence level sentiment analysis
Feature-based sentiment analysis and
summarization
Comparative sentence and relation extraction
Summary
MIAS - June 14, 2007
39
Feature-based opinion mining and
summarization (Hu and Liu, KDD-04)
Again focus on reviews (easier to work in a concrete
domain!)
Objective: find what reviewers (opinion holders) liked
and disliked
Product features and opinions on the features
Since the number of reviews on an object can be large,
an opinion summary should be produced.
Desirable to be a structured summary.
Easy to visualize and to compare.
Analogous to but different from multi-document
summarization.
MIAS - June 14, 2007
40
The tasks
Recall the three tasks in our model.
Task 1: Extracting object features that have been
commented on in each review.
Task 2: Determining whether the opinions on the
features are positive, negative or neutral.
Task 3: Grouping feature synonyms.
Summary
Task 2 may not be needed depending on the format of
reviews.
MIAS - June 14, 2007
41
Different review format
Format 1 - Pros, Cons and detailed review: The reviewer is
asked to describe Pros and Cons separately and also
write a detailed review. Epinions.com uses this format.
Format 2 - Pros and Cons: The reviewer is asked to
describe Pros and Cons separately. C|net.com used to
use this format.
Format 3 - free format: The reviewer can write freely, i.e.,
no separation of Pros and Cons. Amazon.com uses this
format.
MIAS - June 14, 2007
42
Format 1
Format 2
Format 3
GREAT Camera., Jun 3, 2004
Reviewer: jprice174 from Atlanta, Ga.
I did a lot of research last year before I bought
this camera... It kinda hurt to leave behind my
beloved nikon 35mm SLR, but I was going to
Italy, and I needed something smaller, and
digital.
The pictures coming out of this camera are
amazing. The 'auto' feature takes great
pictures most of the time. And with digital,
you're not wasting film if the picture doesn't
come out.
MIAS - June 14, 2007
43
Feature-based Summary (Hu and Liu, KDD-04)
GREAT Camera., Jun 3, 2004
Reviewer: jprice174 from Atlanta, Ga.
I did a lot of research last year before
I bought this camera... It kinda hurt to
leave behind my beloved nikon
35mm SLR, but I was going to Italy,
and I needed something smaller, and
digital.
The pictures coming out of this
camera are amazing. The 'auto'
feature takes great pictures most of
the time. And with digital, you're not
wasting film if the picture doesn't
come out. …
Feature Based Summary:
….
Feature2: battery life
…
Feature1: picture
Positive: 12
The pictures coming out of this camera are
amazing.
Overall this is a good camera with a really good
picture clarity.
…
Negative: 2
The pictures come out hazy if your hands
shake even for a moment during the entire
process of taking a picture.
Focusing on a display rack about 20 feet away
in a brightly lit room during day time, pictures
produced by this camera were blurry and in a
shade of orange.
MIAS - June 14, 2007
44
Visual summarization & comparison
Summary of
reviews of
Digital camera
1
+
_
Picture
Comparison of
reviews of
Battery
Zoom
Size
Weight
+
Digital camera 1
Digital camera 2
_
MIAS - June 14, 2007
45
Feature extraction from Pros and Cons of
Format 1 (Liu et al WWW-03; Hu and Liu, AAAI-CAAW-05)
Observation: Each sentence segment in Pros or Cons contains only
one feature. Sentence segments can be separated by commas,
periods, semi-colons, hyphens, ‘&’’s, ‘and’’s, ‘but’’s, etc.
Pros in Example 1 can be separated into 3 segments:
great photos
<photo>
easy to use
<use>
very small
<small> <size>
Cons can be separated into 2 segments:
battery usage
<battery>
included memory is stingy
<memory>
MIAS - June 14, 2007
46
Extraction using label sequential rules
Label sequential rules (LSR) are a special kind of sequential
patterns, discovered from sequences.
LSR Mining is supervised (Liu’s Web mining book 2006).
The training data set is a set of sequences, e.g.,
“Included memory is stingy”
is turned into a sequence with POS tags.
{included, VB}{memory, NN}{is, VB}{stingy, JJ}
then turned into
{included, VB}{$feature, NN}{is, VB}{stingy, JJ}
MIAS - June 14, 2007
47
Using LSRs for extraction
Based on a set of training sequences, we can mine label
sequential rules, e.g.,
{easy, JJ }{to}{*, VB} {easy, JJ}{to}{$feature, VB}
[sup = 10%, conf = 95%]
Feature Extraction
Only the right hand side of each rule is needed.
The word in the sentence segment of a new review that
matches $feature is extracted.
We need to deal with conflict resolution also (multiple
rules are applicable.
MIAS - June 14, 2007
48
Extraction of features of formats 2 and 3
Reviews of these formats are usually complete
sentences
e.g., “the pictures are very clear.”
Explicit feature: picture
“It is small enough to fit easily in a coat pocket or purse.”
Implicit feature: size
Extraction: Frequency based approach
Frequent features
Infrequent features
MIAS - June 14, 2007
49
Frequency based approach
(Hu and Liu, KDD-04)
Frequent features: those features that have been talked about by many
reviewers.
Use sequential pattern mining
Why the frequency based approach?
Different reviewers tell different stories (irrelevant)
When product features are discussed, the words that they
use converge.
They are main features.
Sequential pattern mining finds frequent phrases.
Froogle has an implementation of the approach (no POS restriction).
MIAS - June 14, 2007
50
Using part-of relationship and the Web
(Popescu and Etzioni, EMNLP-05)
Improved (Hu and Liu, KDD-04) by removing those frequent noun
phrases that may not be features: better precision (a small drop in
recall).
It identifies part-of relationship
Each noun phrase is given a pointwise mutual information score
between the phrase and part discriminators associated with the
product class, e.g., a scanner class.
The part discriminators for the scanner class are, “of scanner”,
“scanner has”, “scanner comes with”, etc, which are used to find
components or parts of scanners by searching on the Web: the
KnowItAll approach, (Etzioni et al, WWW-04).
MIAS - June 14, 2007
51
Infrequent features extraction
How to find the infrequent features?
Observation: the same opinion word can be used to describe
different features and objects.
“The pictures are absolutely amazing.”
“The software that comes with it is amazing.”
Frequent
features
Infrequent
features
Opinion words
MIAS - June 14, 2007
52
Identify feature synonyms
Liu et al (WWW-05) made an attempt using only WordNet.
Carenini et al (K-CAP-05) proposed a more sophisticated method
based on several similarity metrics, but it requires a taxonomy of
features to be given.
The system merges each discovered feature to a feature node in
the taxonomy.
The similarity metrics are defined based on string similarity,
synonyms and other distances measured using WordNet.
Experimental results based on digital camera and DVD reviews
show promising results.
Many ideas in information integration are applicable.
MIAS - June 14, 2007
53
Identify opinion orientation on feature
For each feature, we identify the sentiment or opinion orientation
expressed by a reviewer.
We work based on sentences, but also consider,
A sentence may contain multiple features.
Different features may have different opinions.
E.g., The battery life and picture quality are great (+), but the view
founder is small (-).
Almost all approaches make use of opinion words and phrases. But
note again:
Some opinion words have context independent orientations, e.g.
great.
Some other opinion words have context dependent orientations,
e.g., “small”
Many ways to use them.
MIAS - June 14, 2007
54
Aggregation of opinion words
(Hu and Liu, KDD-04; Ding and Liu, SIGIR-07)
Input: a pair (f, s), where f is a feature and s is a sentence that
contains f.
Output: whether the opinion on f in s is positive, negative, or neutral.
Two steps:
Step 1: split the sentence if needed based on BUT words (but,
except that, etc).
Step 2: work on the segment sf containing f. Let the set of
opinion words in sf be w1, .., wn. Sum up their orientations (1, -1,
0), and assign the orientation to (f, s) accordingly.
wi .o
n
In (Ding and Liu, SIGIR-07), step 2 is changed to
i 1
d ( wi , f )
with better results. wi.o is the opinion orientation of wi. d(wi, f) is the
distance from f to wi.
MIAS - June 14, 2007
55
Context dependent opinions
Popescu and Etzioni (2005) used
constraints of connectives in (Hazivassiloglou and McKeown, ACL97), and some additional constraints, e.g., morphological
relationships, synonymy and antonymy, and
relaxation labeling to propagate opinion orientations to words and
features.
Ding and Liu (2007) used
constraints of connectives both at intra-sentence and inter-sentence
levels, and
additional constraints of, e.g., TOO, BUT, NEGATION.
to directly assign opinions to (f, s) with good results (> 0.85 of F-score).
MIAS - June 14, 2007
56
Roadmap
Opinion mining – the abstraction
Document level sentiment classification
Sentence level sentiment analysis
Feature-based sentiment analysis and
summarization
Comparative sentence and relation extraction
Summary
MIAS - June 14, 2007
57
Extraction of Comparatives
Comparative sentence mining
Identify comparative sentences
Extract comparative relations from them
MIAS - June 14, 2007
58
Linguistic Perspective
Comparative sentences use morphemes like
more/most, -er/-est, less/least, as
than and as are used to make a `standard’ against
which an entire entity is compared
Limitations
Limited coverage
“In market capital, Intel is way ahead of AMD.”
Non-comparatives with comparative words
“In the context of speed, faster means better.”
MIAS - June 14, 2007
59
Types of Comparatives
Gradable
Non-Equal Gradable: Relations of the type greater or
less than
Equative: Relations of type equal to
Keywords like better, ahead, beats, etc
“Optics of camera A is better than that of camera B”
Keywords and phrases like equal to, same as, both, all
“Camera A and camera B both come in 7MP”
Superlative: Relations of the type greater or less than
all others
Keywords and phrases like best, most, better than all
“Camera A is the cheapest camera available in the market.”
MIAS - June 14, 2007
60
Types of Comparatives: non-gradable
Non-gradable: Sentences that compare features of two
or more objects, but do not grade them. Sentences
which imply:
Object A is similar to or different from Object B with
regard to some features
Object A has feature F1, Object B has feature F2
Object A has feature F, but Object B does not have
MIAS - June 14, 2007
61
Comparative Relation: gradable
Definition: A gradable comparative relation captures the
essence of gradable comparative sentence and is
represented with the following:
(relationWord, features, entityS1, entityS2, type)
relationWord: The keyword used to express a
comparative relation in a sentence.
features: a set of features being compared.
entityS1 and entityS2: Sets of entities being
compared.
type: non-equal gradable, equative or superlative
MIAS - June 14, 2007
62
Examples: Comparative relations
“car X has better controls than car Y”
(relationWord = better, features = controls, entityS1 = carX, entityS2
= carY, type = non-equal-gradable)
“car X and car Y have equal mileage”
(relationWord = equal, features = mileage, entityS1 = carX, entityS2
= carY, type = equative)
“car X is cheaper than both car Y and car Z”
(relationWord = cheaper, features = null, entityS1 = carX, entityS2 =
{carY, carZ}, type = non-equal-gradable)
“company X produces a variety of cars, but still best cars come from
company Y”
(relationWord = best, features = cars, entityS1 = companyY, entityS2
= companyX, type = superlative)
MIAS - June 14, 2007
63
Tasks
Given a collection of evaluative texts
Task 1: Identify comparative sentences
Task 2: Categorize different types of comparative
sentences.
Task 3: Extract comparative relations from the
sentences
MIAS - June 14, 2007
64
Identify comparative sentences
Keyword strategy
An observation: Its is easy to find a small set of
keywords that covers almost all comparative
sentences, i.e., with a very high recall and a
reasonable precision
A list of 83 keywords used in comparative sentences
compiled by (Jinal and Liu, Sigir-06) including
Words with POS tags of JJR, JJS, RBR, RBS
POS tags are used as keyword instead of individual
words
Exceptions: more, less, most, least
Other indicative word like beat, exceed, ahead, etc.
Phrases like in the lead, on par with, etc.
MIAS - June 14, 2007
65
2-step learning strategy
Step 1: Extract sentences which contain at least one
keyword (recall = 98%, precision = 32% on our data set
of gradables)
Step 2: Use Naïve Bayes classifier to classify sentences
into two classes
Comparative and non-comparative
Attributes: class sequential rules (CSRs) generated
from sentences in step 1
MIAS - June 14, 2007
66
1.
2.
3.
Sequence data preparation
Use words within a radius r of a keyword to form a sequence
(words are replaced with POS tags)
…
CSR generation
Use different minimum supports for different keywords
13 manual rules, which were hard to generate automatically
Learning using a NB classifier
Use CSRs and manual rules as attributes to build a final
classifier
MIAS - June 14, 2007
67
Classify different types of comparatives
Classify comparative sentences into three types: nonequal gradable, equative and superlative
SVM learner gives the best result
Asstribute set is the set of keywords
If the sentence has a particular keywords in the
attribute set, the corresponding value is 1, and 0
otherwise.
MIAS - June 14, 2007
68
Extraction of comparative relations
Assumptions
There is only one relation in a sentence
Entities and features are nominals
Adjectival comparatives
Does not deal with adverbial comparatives
3 steps
Sequence data generation
Label sequential rule (LSR) generation
Build a sequential cover/extractor from LSRs
MIAS - June 14, 2007
69
Sequence data generation
Label Set = {$entityS1, $entityS2, $feature}
Three labels are used as pivots to generate sequences.
Radius of 4 for optimal results
Following words are also added
Distance words = {l1, l2, l3, l4, r1, r2, r3, r4}
Special words #start and #end are used to mark the
start and the end of a sentence.
MIAS - June 14, 2007
70
Sequence data generation example
The comparative sentence
“Canon/NNP has/VBZ better/JJR optics/NNS”
has $entityS1 “Canon” and $feature “optics”
Sequences are:
<{#start>{l1}{$entityS1, NNP){r1}{has, VBZ}{r2}{better,
JJR}{r3}{$Feature, NNS}{r4}{#end}>
<{#start>{l4}{$entityS1, NNP){l3}{has, VBZ}{l2}{better,
JJR}{l1}{$Feature, NNS}{r1}{#end}>
MIAS - June 14, 2007
71
Build a sequential cover from LSRs
LSR: {*, NN}{VBZ} { $entityS1, NN}{VBZ}
Select the LSR rule with the highest confidence. Replace
the matched elements in the sentences that satisfy the
rules with the labels in the rule.
Recalculate the confidence of each remaining rule based
on the modified data from step 1.
Repeat step 1 and 2 until no rule left with confidence
higher than minconf value (they sued 90%)
MIAS - June 14, 2007
72
Experimental Results (Jindal and Liu, AAAI 06)
Identifying Gradable Comparative Sentences
Precision = 82% and recall = 81%
Classification into three gradable types
SVM gave accuracy of 96%
Extraction of comparative relations
LSR: F-score = 72%
MIAS - June 14, 2007
73
Summary
Two types of evaluations
Direct opinions: We studied
The problem abstraction
Sentiment analysis at document level, sentence level and
feature level
Comparisons:
Very hard problems, but very useful
The current techniques are still in their infancy.
Industrial applications are coming up
MIAS - June 14, 2007
74
END
MIAS - June 14, 2007
75
Manual and Automatic
Subjectivity and Sentiment
Analysis
Outline
Corpus Annotation
Pure NLP
Lexicon development
Recognizing Contextual Polarity in Phrase-Level
Sentiment Analysis
Applications
Product review mining
MIAS - June 14, 2007
77
Corpus Annotation
Wiebe, Wilson, Cardie. Language Resources and Evaluation 39
(1-2), 2005
MIAS - June 14, 2007
78
Overview
Fine-grained: expression-level rather than sentence or document
level
The photo quality was the best that I have seen in a camera.
The photo quality was the best that I have seen in a camera.
Annotate
expressions of opinions, evaluations, emotions
material attributed to a source, but presented objectively
MIAS - June 14, 2007
79
Overview
Fine-grained: expression-level rather than sentence or document
level
The photo quality was the best that I have seen in a camera.
The photo quality was the best that I have seen in a camera.
Annotate
expressions of opinions, evaluations, emotions, beliefs
material attributed to a source, but presented objectively
MIAS - June 14, 2007
80
Overview
Opinions, evaluations, emotions, speculations are
private states.
They are expressed in language by subjective
expressions.
Private state: state that is not open to objective
observation or verification.
Quirk, Greenbaum, Leech, Svartvik (1985). A
Comprehensive Grammar of the English Language.
MIAS - June 14, 2007
81
Overview
Focus on three ways private states are expressed in
language
Direct subjective expressions
Expressive subjective elements
Objective speech events
MIAS - June 14, 2007
82
Direct Subjective Expressions
Direct mentions of private states
The United States fears a spill-over from the antiterrorist campaign.
Private states expressed in speech events
“We foresaw electoral fraud but not daylight robbery,”
Tsvangirai said.
MIAS - June 14, 2007
83
Expressive Subjective Elements [Banfield
1982]
“We foresaw electoral fraud but not daylight robbery,”
Tsvangirai said
The part of the US human rights report about China is full of
absurdities and fabrications
MIAS - June 14, 2007
84
Objective Speech Events
Material attributed to a source, but presented as
objective fact
The government, it added, has amended the Pakistan Citizenship Act 10 of
1951 to enable women of Pakistani descent to claim Pakistani nationality for
their children born to foreign husbands.
MIAS - June 14, 2007
85
MIAS - June 14, 2007
86
Nested Sources
“The report is full of absurdities,’’ Xirao-Nima said the next day.
MIAS - June 14, 2007
87
Nested Sources
(Writer)
“The report is full of absurdities,’’ Xirao-Nima said the next day.
MIAS - June 14, 2007
88
Nested Sources
(Writer, Xirao-Nima)
“The report is full of absurdities,’’ Xirao-Nima said the next day.
MIAS - June 14, 2007
89
Nested Sources
(Writer Xirao-Nima)
(Writer Xirao-Nima)
“The report is full of absurdities,’’ Xirao-Nima said the next day.
MIAS - June 14, 2007
90
Nested Sources
(Writer)
(Writer Xirao-Nima)
(Writer Xirao-Nima)
“The report is full of absurdities,’’ Xirao-Nima said the next day.
MIAS - June 14, 2007
91
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event
anchor: the entire sentence
source: <writer>
implicit: true
Direct subjective
anchor: said
source: <writer, Xirao-Nima>
intensity: high
expression intensity: neutral
attitude type: negative
target: report
Expressive subjective element
anchor: full of absurdities
source: <writer, Xirao-Nima>
intensity: high
attitude type: negative
MIAS - June 14, 2007
92
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event
anchor: the entire sentence
source: <writer>
implicit: true
Direct subjective
anchor: said
source: <writer, Xirao-Nima>
intensity: high
expression intensity: neutral
attitude type: negative
target: report
Expressive subjective element
anchor: full of absurdities
source: <writer, Xirao-Nima>
intensity: high
attitude type: negative
MIAS - June 14, 2007
93
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event
anchor: the entire sentence
source: <writer>
implicit: true
Direct subjective
anchor: said
source: <writer, Xirao-Nima>
intensity: high
expression intensity: neutral
attitude type: negative
target: report
Expressive subjective element
anchor: full of absurdities
source: <writer, Xirao-Nima>
intensity: high
attitude type: negative
MIAS - June 14, 2007
94
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event
anchor: the entire sentence
source: <writer>
implicit: true
Direct subjective
anchor: said
source: <writer, Xirao-Nima>
intensity: high
expression intensity: neutral
attitude type: negative
target: report
Expressive subjective element
anchor: full of absurdities
source: <writer, Xirao-Nima>
intensity: high
attitude type: negative
MIAS - June 14, 2007
95
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event
anchor: the entire sentence
source: <writer>
implicit: true
Direct subjective
anchor: said
source: <writer, Xirao-Nima>
intensity: high
expression intensity: neutral
attitude type: negative
target: report
Expressive subjective element
anchor: full of absurdities
source: <writer, Xirao-Nima>
intensity: high
attitude type: negative
MIAS - June 14, 2007
96
“The report is full of absurdities,” Xirao-Nima said the next day.
Objective speech event
anchor: the entire sentence
source: <writer>
implicit: true
Direct subjective
anchor: said
source: <writer, Xirao-Nima>
intensity: high
expression intensity: neutral
attitude type: negative
target: report
Expressive subjective element
anchor: full of absurdities
source: <writer, Xirao-Nima>
intensity: high
attitude type: negative
MIAS - June 14, 2007
97
“The US fears a spill-over’’, said Xirao-Nima, a professor of
foreign affairs at the Central University for Nationalities.
MIAS - June 14, 2007
98
(Writer)
“The US fears a spill-over’’, said Xirao-Nima, a professor of
foreign affairs at the Central University for Nationalities.
MIAS - June 14, 2007
99
(writer, Xirao-Nima)
“The US fears a spill-over’’, said Xirao-Nima, a professor of
foreign affairs at the Central University for Nationalities.
MIAS - June 14, 2007
100
(writer, Xirao-Nima, US)
“The US fears a spill-over’’, said Xirao-Nima, a professor of
foreign affairs at the Central University for Nationalities.
MIAS - June 14, 2007
101
(Writer)
(writer, Xirao-Nima, US)
(writer, Xirao-Nima)
“The US fears a spill-over’’, said Xirao-Nima, a professor of
foreign affairs at the Central University for Nationalities.
MIAS - June 14, 2007
102
“The US fears a spill-over’’, said Xirao-Nima, a professor of
foreign affairs at the Central University for Nationalities.
Objective speech event
anchor: the entire sentence
source: <writer>
implicit: true
Objective speech event
anchor: said
source: <writer, Xirao-Nima>
Direct subjective
anchor: fears
source: <writer, Xirao-Nima, US>
intensity: medium
expression intensity: medium
attitude type: negative
target: spill-over
MIAS - June 14, 2007
103
The report has been strongly criticized and condemned by
many countries.
MIAS - June 14, 2007
104
The report has been strongly criticized and condemned by
many countries.
Objective speech event
anchor: the entire sentence
source: <writer>
implicit: true
Direct subjective
anchor: strongly criticized and condemned
source: <writer, many-countries>
intensity: high
expression intensity: high
attitude type: negative
target: report
MIAS - June 14, 2007
105
As usual, the US state Department published its annual report on
human rights practices in world countries last Monday.
And as usual, the portion about China contains little truth and
many absurdities, exaggerations and fabrications.
MIAS - June 14, 2007
106
As usual, the US state Department published its annual report on
human rights practices in world countries last Monday.
And as usual, the portion about China contains little truth and
many absurdities, exaggerations and fabrications.
Objective
anchor :
source :
implicit :
speech event
the entire 1st sentence
<writer>
true
Direct subjective
anchor : the entire 2nd sentence
source : <writer>
implicit : true
intensity : high
expression intensity : medium
attitude type : negative
target : report
Expressive subjective element
anchor : And as usual
source : <writer>
intensity : low
attitude type : negative
Expressive subjective element
anchor : little truth
source : <writer>
intensity : medium
attitude type : negative
Expressive subjective element
anchor : many absurdities, exaggerations,
and fabrications
source : <writer>
intensity : medium
attitude type : negative
MIAS - June 14, 2007
107
Corpus
www.cs.pitt.edu/mqpa/databaserelease (version 2)
English language versions of articles from the world press (187 news
sources)
Also includes contextual polarity annotations (later)
Themes of the instructions:
No rules about how particular words should be annotated.
Don’t take expressions out of context and think about what they
could mean, but judge them as they are used in that sentence.
MIAS - June 14, 2007
108
Agreement
Inter-annotator agreement studies performed on various
aspects of the scheme
Kappa is a measure of the degree of nonrandom
agreement between observers and/or measurements of
a specific categorical variable
Kappa values range between .70 and .80
MIAS - June 14, 2007
109
Agreement
Annotator 1
Annotator 2
Two council street
wardens who helped lift a
14-ton bus off an injured
schoolboy are to be
especially commended
for their heroic actions.
Two council street
wardens who helped lift a
14-ton bus off an injured
schoolboy are to be
especially commended
for their heroic actions.
Nathan Thomson and
Neville Sharpe will
receive citations from the
mayor of Croydon later
this month.
Nathan Thomson and
Neville Sharpe will
receive citations from the
mayor of Croydon later
this month.
MIAS - June 14, 2007
110
Agreement
Inter-annotator agreement studies performed on various
aspects of the scheme
Kappa is a measure of the degree of nonrandom
agreement between observers and/or measurements of
a specific categorical variable
Kappa values range between .70 and .80
MIAS - June 14, 2007
111
Outline
Corpus Annotation
Pure NLP
Lexicon development
Recognizing Contextual Polarity in Phrase-Level
Sentiment Analysis
Applications
Product review mining
MIAS - June 14, 2007
112
Who does lexicon development ?
Humans
Semi-automatic
Fully automatic
MIAS - June 14, 2007
113
What?
Find relevant words, phrases, patterns that can be used
to express subjectivity
Determine the polarity of subjective expressions
MIAS - June 14, 2007
114
Words
Adjectives (e.g. Hatzivassiloglou & McKeown 1997, Wiebe 2000, Kamps & Marx 2002, Andreevskaia &
Bergler 2006)
positive: honest important mature large patient
Ron Paul is the only honest man in Washington.
Kitchell’s writing is unbelievably mature and is only likely to
get better.
To humour me my patient father agrees yet again to my
choice of film
MIAS - June 14, 2007
115
Words
Adjectives (e.g. Hatzivassiloglou & McKeown 1997, Wiebe 2000, Kamps & Marx 2002, Andreevskaia &
Bergler 2006)
positive
negative: harmful hypocritical inefficient insecure
It was a macabre and hypocritical circus.
Why are they being so inefficient ?
subjective: curious, peculiar, odd, likely, probably
MIAS - June 14, 2007
116
Words
Adjectives (e.g. Hatzivassiloglou & McKeown 1997, Wiebe 2000, Kamps & Marx 2002, Andreevskaia &
Bergler 2006)
positive
negative
subjective: curious, peculiar, odd, likely, probable
He spoke of Sue as his probable successor.
The two species are likely to flower at different times.
MIAS - June 14, 2007
117
Other parts of speech (e.g. Turney & Littman 2003, Esuli & Sebastiani 2006)
Verbs
positive: praise, love
negative: blame, criticize
subjective: predict
Nouns
positive: pleasure, enjoyment
negative: pain, criticism
subjective: prediction
MIAS - June 14, 2007
118
Phrases
Phrases containing adjectives and adverbs (e.g. Turney 2002,
Takamura et al. 2007 )
positive: high intelligence, low cost
negative: little variation, many troubles
MIAS - June 14, 2007
119
Patterns
Lexico-syntactic patterns (Riloff & Wiebe 2003)
way with <np>: … to ever let China use force to have its
way with …
expense of <np>: at the expense of the world’s securty
and stability
underlined <dobj>: Jiang’s subdued tone … underlined
his desire to avoid disputes …
MIAS - June 14, 2007
120
How?
How do we identify subjective items?
MIAS - June 14, 2007
121
How?
How do we identify subjective items?
Assume that contexts are coherent
MIAS - June 14, 2007
122
Conjunction
MIAS - June 14, 2007
123
Statistical association
If words of the same orientation like to co-occur together,
then the presence of one makes the other more probable
Use statistical measures of association to capture this
interdependence
Mutual Information (Church & Hanks 1989)
MIAS - June 14, 2007
124
How?
How do we identify subjective items?
Assume that contexts are coherent
Assume that alternatives are similarly subjective
MIAS - June 14, 2007
125
How?
How do we identify subjective items?
Assume that contexts are coherent
Assume that alternatives are similarly subjective
MIAS - June 14, 2007
126
WordNet
MIAS - June 14, 2007
127
WordNet
MIAS - June 14, 2007
128
WordNet relations
MIAS - June 14, 2007
129
WordNet relations
MIAS - June 14, 2007
130
WordNet relations
MIAS - June 14, 2007
131
WordNet glosses
MIAS - June 14, 2007
132
WordNet examples
MIAS - June 14, 2007
133
How?
How do we identify subjective items?
Assume that contexts are coherent
Assume that alternatives are similarly subjective
Take advantage of word meanings
MIAS - June 14, 2007
134
*We cause great leaders
MIAS - June 14, 2007
135
Specific papers using
these ideas
Hatzivassiloglou & McKeown 1997
1.
Build training set: label all adjectives with frequency > 20
Test agreement with human annotators
MIAS - June 14, 2007
137
Hatzivassiloglou & McKeown 1997
1.
2.
Build training set: label all adj. with frequency > 20; test agreement
with human annotators
Extract all conjoined adjectives
nice and comfortable
nice and scenic
MIAS - June 14, 2007
138
Hatzivassiloglou & McKeown 1997
3. A supervised learning algorithm builds a graph of adjectives linked by
the same or different semantic orientation
scenic
nice
terrible
painful
handsome
fun
expensive
comfortable
MIAS - June 14, 2007
139
Hatzivassiloglou & McKeown 1997
4. A clustering algorithm partitions the adjectives into two subsets
+
slow
scenic
nice
terrible
handsome
painful
fun
expensive
comfortable
MIAS - June 14, 2007
140
Wiebe 2000
Learning Subjective Adjectives From Corpora
Learning evaluation and opinion clues
Distributional similarity process, based on
manual annotation
Refinement with lexical features
Improved results from both
MIAS - June 14, 2007
141
Lin’s (1998) Distributional Similarity
obj
mod
I
have
a
brown
dog
subj
det
Word
I
have
brown
R
W
subj have
obj dog
mod dog
...
MIAS - June 14, 2007
142
Lin’s Distributional Similarity
Word1
R W
R W
R={subj, obj, etc.}
Word2
RW
RW
RW
RW
RW
RW
RW
RW
MIAS - June 14, 2007
RW
RW
143
Bizarre
strange similar scary unusual fascinating
interesting curious tragic different
contradictory peculiar silly sad absurd
poignant crazy funny comic compelling
odd
MIAS - June 14, 2007
144
Experiments
9
10
MIAS - June 14, 2007
145
Experiments
Seeds
Distributional
similarity
Separate
corpus
Seeds +
Similar
Words
9
10
MIAS - June 14, 2007
146
Experiments
Distributional
similarity
Seeds
9
10
Separate
corpus
Seeds +
Similar
Words
p(subjective | s)
S
Words
MIAS - June 14, 2007
S > Adj > Majority
147
Turney 2002a,b
Determine the semantic orientation of each extracted
phrase based on their association with seven positive
and seven negative words
p( word1 & word 2 )
PMI ( word1 , word 2 ) log2
p
(
word
)
p
(
word
)
1
2
hits( word NEAR p _ query)hits(n _ query)
SO PMI IR( word ) log2
hits
(
word
NEAR
n
_
query
)
hits
(
p
_
query
)
MIAS - June 14, 2007
148
Turney 2002a,b
Determine the semantic orientation of each extracted
phrase based on their association with seven positive
and seven negative words
p( word1 & word 2 )
PMI ( word1 , word 2 ) log2
p
(
word
)
p
(
word
)
1
2
hits( word NEAR p _ query)hits(n _ query)
SO PMI IR( word ) log2
hits
(
word
NEAR
n
_
query
)
hits
(
p
_
query
)
MIAS - June 14, 2007
149
Riloff & Wiebe 2003
Observation: subjectivity comes in many (low-frequency) forms
better to have more data
Boot-strapping produces cheap data
High-precision classifiers label sentences as subjective or objective
Extraction pattern learner gathers patterns biased towards
subjective texts
Learned patterns are fed back into high precision classifier
MIAS - June 14, 2007
151
MIAS - June 14, 2007
152
Riloff & Wiebe 2003
Observation: subjectivity comes in many (low-frequency) forms
better to have more data
Boot-strapping produces cheap data
High-precision classifiers label sentences as subjective or objective
Extraction pattern learner gathers patterns biased towards
subjective texts
Learned patterns are fed back into high precision classifier
MIAS - June 14, 2007
153
Yu & Hatzivassiloglou 2003
Towards Answering Opinion Questions: Separating Facts from Opinions and
Identifying the Polarity of Opinion Sentences
Classifying documents: naïve bayes, words as features
Finding opinion sentences:
2 similarity approaches
Naïve bayes (n-grams, POS, counts of polar words,
counts of polar sequences, average orientation)
Multiple naïve bayes
MIAS - June 14, 2007
154
Yu & Hatzivassiloglou 2003
Towards Answering Opinion Questions: Separating Facts from Opinions and
Identifying the Polarity of Opinino Sentences
Tagging words and sentences:
modified log-likelihood ratio of collocation with pos,
neg adjectives in seed sets
Adjectives, adverbs, and verbs provide best
combination for tagging polarity of sentences
MIAS - June 14, 2007
155
Kim & Hovy 2005
Automatic Detection of Opinion Bearing Words and Sentences
WordNet-based method for collecting opinion-bearing
adjectives and verbs
manually constructed strong seed set
manually labeled reference sets (opinion-bearing or
not)
for synonyms/antonyms of seed set, calculate an
opinion strength relative to reference sets
expand further with naïve bayes classifier
MIAS - June 14, 2007
157
Kim & Hovy 2005
Automatic Detection of Opinion Bearing Words and Sentences
Corpus-based method (WSJ)
Calculate bias of words for particular text genre
(Editorials and Letter to editor)
# w in editorial docum ents
EditorialProb(w)
total words in editorial docum ents
Editorial Pr ob( w)
Score( w)
Noneditorial Pr ob( w)
MIAS - June 14, 2007
159
Esuli & Sebastiani 2005
Determining the semantic orientation of terms
through gloss analysis
use final sets as gold standard to train a classifier, which uses all or part of
the glosses in some format as features
the trained classifier can then be used to label any term that has a gloss
with sentiment
w(awful)
…
w(dire)
w(direful)
…
w(dread)
MIAS - June 14, 2007
W(dreaded)
…
…
161
Esuli & Sebastiani 2006
Determining Term Subjectivity and Term Orientation for Opinion Mining
Uses best system of 2005 paper
Additional goal of distinguishing neutral from
positive/negative
Multiple variations on learning approach, learner, training
set, feature selection
The new problem is harder! Their best accuracy is 66%
(83% in 2005 paper)
MIAS - June 14, 2007
162
Suzuki et al. 2006
Application of Semi-supervised learning to evaluative expression classification
Automatically extract and filter “evaluative expressions": The storage
capacity of this HDD is high.
Classifies these as pos, neg, or neutral
Use bootstrapping to be able to train an evaluative expression classifier
based on a larger collection of unlabeled data.
Learn contexts that contain evaluative expressions
I am really happy because [the storage capacity is high]
Unfortunately, [the laptop was too expensive].
MIAS - June 14, 2007
163
Suzuki et al. 2006
Evaluation
Attribute
Automatically extract and filter “evaluative expressions": The storage
capacity of this HDD is high.
Classifies these as pos, neg, or neutral
Use bootstrapping to be able to train an evaluative expressionSubject
classifier
based on a larger collection of unlabeled data.
Learn contexts that contain evaluative expressions
I am really happy because [the storage capacity is high]
Unfortunately, [the laptop was too expensive].
MIAS - June 14, 2007
164
Suzuki et al. 2006
Automatically extract and filter “evaluative expressions": The storage
capacity of this HDD is high.
Classifies these as pos, neg, or neutral
Use bootstrapping to be able to train an evaluative expression
classifier based on a larger collection of unlabeled data.
Learn contexts that contain evaluative expressions
I am really happy because [the storage capacity is high]
Unfortunately, [the laptop was too expensive].
MIAS - June 14, 2007
165
Suzuki et al. 2006
Comparison of semi-supervised methods:
Nigam et al.’s (2000) Naive Baiyes + EM method
Naive Bayes + EM + SVM (SVM combined with Naive
Bayes + EM using Fisher kernel)
And supervised methods:
Naive Bayes
SVM
MIAS - June 14, 2007
166
Suzuki et al. 2006
Features:
‘Phew, [the noise] of [this HDD] is annoyingly high :-(’.
Candidate evaluative expression
“Exclamation words” detected by POS tagger
Emoticons and their emotional categories
Words modifying words in the candidate evaluation expression
Words modified by words in the candidate evaluative word
MIAS - June 14, 2007
167
Both Naive Bayes + EM, and Naive Bayes + EM + SVM work better than
Naive Bayes and SVM.
Results show that Naive Bayes + EM boosted accuracy regardless of size
of labeled data
Using more unlabeled data appeared to give better results.
Qualitative analysis of the impact of the semi-supervised approaches by
looking at the top 100 features that had the highest probability P(feature—
positive) before and after EM:
more contextual features like exclamations, the happy emoticons, a
negation + ’but’, ‘therefore’ + ’interesting’, and ‘therefore‘ +
’comfortable.’
MIAS - June 14, 2007
168
Surely
… we’ve thought of everything by now?
MIAS - June 14, 2007
169
Word senses
Senses
MIAS - June 14, 2007
170
Senses
MIAS - June 14, 2007
171
Non-subjective senses of brilliant
1.
Method for identifying brilliant material in paint US Patent 7035464
2.
Halley shines in a brilliant light.
3.
In a classic pasodoble, an opening section in the
minor mode features a brilliant trumpet melody,
while the second section in the relative major begins
with the violins.
MIAS - June 14, 2007
172
Andreevskaia and Bergler 2006
Mining WordNet for Fuzzy Sentiment: Sentiment Tag Extraction from
WordNet Glosses
Using wordnet relations (synonymy, antonymy and
hyponymy) and glosses
Classify as positive, negative, or neutral
Step algorithm with known seeds:
First expand with relations
Next expand via glosses
Filter out wrong POS and multiply assigned
Evaluate against General inquirer (which contains words,
not word senses)
MIAS - June 14, 2007
173
Andreevskaia and Bergler 2006
Mining WordNet for Fuzzy Sentiment: Sentiment Tag Extraction from
WordNet Glosses
Disagreements between human labelers as a sign of
fuzzy category structure
HM and General Inquirer have 78.7% tag agreement
for shared adjectives
Find way to measure the degree of centrality of words to
the category of sentiment
Net overlap scores correlate with human agreement
MIAS - June 14, 2007
175
Outline
Corpus Annotation
Pure NLP
Lexicon development
Recognizing Contextual Polarity in Phrase-Level
Sentiment Analysis
Applications
Product review mining
MIAS - June 14, 2007
176
Recognizing Contextual Polarity in
Phrase-Level Sentiment Analysis
Wilson, Wiebe, Hoffmann HLT-EMNLP-2005
MIAS - June 14, 2007
177
Prior Polarity versus Contextual
Polarity
Most approaches use a lexicon of positive and negative words
Prior polarity: out of context, positive or negative
beautiful positive
horrid negative
A word may appear in a phrase that expresses a different polarity
in context
Contextual polarity
“Cheers to Timothy Whitfield for the wonderfully
horrid visuals.”
MIAS - June 14, 2007
178
Example
Philip Clap, President of the National Environment
Trust, sums up well the general thrust of the reaction of
environmental movements: there is no reason at all to
believe that the polluters are suddenly going to become
reasonable.
MIAS - June 14, 2007
179
Example
Philip Clap, President of the National Environment
Trust, sums up well the general thrust of the reaction of
environmental movements: there is no reason at all to
believe that the polluters are suddenly going to
become reasonable.
MIAS - June 14, 2007
180
Example
Philip Clap, President of the National
Environment Trust, sums up well the
general thrust of the reaction of
environmental movements: there is no
reason at all to believe that the polluters
are suddenly going to become
reasonable.
Contextual
prior polarity
polarity
MIAS - June 14, 2007
181
Goal of This Work
Automatically distinguish prior and contextual polarity
MIAS - June 14, 2007
182
Approach
Lexicon
All
Instances
Corpus
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances
Contextual
Polarity?
Use machine learning and variety of features
Achieve significant results for a large subset of sentiment
expressions
MIAS - June 14, 2007
183
Outline
Introduction
Manual Annotations and Corpus
Lexicon
All
Instances
Corpus
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances
Contextual
Polarity?
Prior-Polarity Subjectivity Lexicon
Experiments
Conclusions
MIAS - June 14, 2007
184
Manual Annotations
Subjective expressions of the MPQA corpus annotated
with contextual polarity
MIAS - June 14, 2007
185
Annotation Scheme
Mark polarity of subjective expressions as positive,
negative, both, or neutral
positive
African observers generally approved of his victory while
Western governments denounced it.
negative
Besides, politicians refer to good and evil …
both
Jerome says the hospital feels no different than a
hospital in the states.
neutral
MIAS - June 14, 2007
186
Annotation Scheme
Judge the contextual polarity of sentiment ultimately
being conveyed
They have not succeeded, and will never succeed, in
breaking the will of this valiant people.
MIAS - June 14, 2007
187
Annotation Scheme
Judge the contextual polarity of sentiment ultimately
being conveyed
They have not succeeded, and will never succeed,
in breaking the will of this valiant people.
MIAS - June 14, 2007
188
Annotation Scheme
Judge the contextual polarity of sentiment ultimately
being conveyed
They have not succeeded, and will never succeed,
in breaking the will of this valiant people.
MIAS - June 14, 2007
189
Annotation Scheme
Judge the contextual polarity of sentiment ultimately
being conveyed
They have not succeeded, and will never
succeed, in breaking the will of this valiant people.
MIAS - June 14, 2007
190
Corpus
425 documents from MPQA Opinion Corpus
15,991 subjective expressions in 8,984 sentences
Divided into two sets
Development set for feature development
66 docs / 2,808 subjective expressions
Experiment set
359 docs / 13,183 subjective expressions
Divided into 10 folds for cross-validation
MIAS - June 14, 2007
191
Outline
Introduction
Manual Annotations and Corpus
Prior-Polarity Subjectivity Lexicon
Lexicon
Corpus
All
Instances
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Experiments
Conclusions
MIAS - June 14, 2007
192
Prior-Polarity Subjectivity Lexicon
Over 8,000 words from a variety of sources
Both manually and automatically identified
Positive/negative words from General Inquirer and
Hatzivassiloglou and McKeown (1997)
All words in lexicon tagged with:
Prior polarity: positive, negative, both, neutral
Reliability: strongly subjective (strongsubj),
weakly subjective (weaksubj)
MIAS - June 14, 2007
193
Outline
Introduction
Manual Annotations
Corpus
Prior-Polarity Subjectivity Lexicon
Experiments
Conclusions
MIAS - June 14, 2007
194
Experiments
Lexicon
All
Instances
Corpus
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Both Steps:
BoosTexter AdaBoost.HM 5000 rounds boosting
10-fold cross validation
Give each instance its own label
MIAS - June 14, 2007
195
Definition of Gold Standard
Given an instance inst from the lexicon:
if inst not in a subjective expression:
goldclass(inst) = neutral
else if inst in at least one positive and one negative subjective
expression:
goldclass(inst) = both
else if inst in a mixture of negative and neutral:
goldclass(inst) = negative
else if inst in a mixture of positive and neutral:
goldclass(inst) = positive
else: goldclass(inst) = contextual polarity of subjective expression
MIAS - June 14, 2007
196
Features
Many inspired by Polanyi & Zaenen (2004): Contextual
Valence Shifters
Example: little threat
little truth
Others capture dependency relationships between words
Example:
wonderfully horrid
pos
mod
MIAS - June 14, 2007
197
Lexicon
All
Instances
Corpus
1.
2.
3.
4.
5.
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Word features
Modification features
Structure features
Sentence features
Document feature
MIAS - June 14, 2007
198
Lexicon
All
Instances
Corpus
1.
2.
3.
4.
5.
Word features
Modification features
Structure features
Sentence features
Document feature
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Word token
terrifies
Word part-of-speech
VB
Context
that terrifies me
Prior Polarity
negative
Reliability
strongsubj
MIAS - June 14, 2007
199
Lexicon
Step 1
All
Instances
Corpus
1.
2.
3.
4.
5.
Neutral
or
Polar?
Word features
Modification features
Structure features
Sentence features
Document feature
Step 2
Polar
Instances Contextual
Polarity?
Binary features:
Preceded by
adjective
adverb (other than not)
intensifier
strongsubj clue
weaksubj clue
poses
subj
obj
report
det
adj
challenge
mod
The human rights
Self intensifier
Modifies
det
adj
p
Modified by
strongsubj clue
weaksubj clue
Dependency
Parse Tree
a substantial …
MIAS - June 14, 2007
200
Lexicon
Step 1
All
Instances
Corpus
1.
2.
3.
4.
5.
Neutral
or
Polar?
Word features
Modification features
Structure features
Sentence features
Document feature
Step 2
Polar
Instances Contextual
Polarity?
Binary features:
In subject
[The human rights report]
poses
In copular
I am confident
poses
subj
obj
report
det
adj
In passive voice
must be regarded
challenge
mod
The human rights
det
adj
p
a substantial …
MIAS - June 14, 2007
201
Lexicon
All
Instances
Corpus
1.
2.
3.
4.
5.
Word features
Modification features
Structure features
Sentence features
Document feature
Step 1
Step 2
Polar
Instances Contextual
Neutral
or
Polar?
Polarity?
Count of strongsubj clues in
previous, current, next sentence
Count of weaksubj clues in
previous, current, next sentence
Counts of various parts of speech
MIAS - June 14, 2007
202
Lexicon
All
Instances
Corpus
1.
2.
3.
5.
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Document topic (15)
economics
health
…
4.
Word features
Modification features
Structure features
Sentence features
Document feature
Step 1
Kyoto protocol
presidential election in
Zimbabwe
Example: The disease can be contracted if a
person is bitten by a certain tick or if a person comes
into contact with the blood of a congo fever sufferer.
MIAS - June 14, 2007
203
Lexicon
Step 1
All
Instances
Corpus
Step 2
Polar
Instances Contextual
Neutral
or
Polar?
Polarity?
Results 1a
90
82.1
80
75.9
70
Word token
Word + Prior Polarity
All Features
63.4
60
50
40
Accuracy
Polar F
Neutral F
MIAS - June 14, 2007
204
Lexicon
Step 1
All
Instances
Neutral
or
Polar?
Corpus
Step 2
Polar
Instances Contextual
Polarity?
Results 1b
80
70
60
Word token
Word + Prior Polarity
All Features
50
40
30
Polar Recall
Polar Precision
MIAS - June 14, 2007
205
Step 2: Polarity Classification
Lexicon
All
Instances
Corpus
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
19,506
5,671
Classes
positive, negative, both, neutral
MIAS - June 14, 2007
206
Lexicon
Step 1
All
Instances
Corpus
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Word token
Word prior polarity
Negated
Negated subject
Modifies polarity
Modified by polarity
Conjunction polarity
General polarity shifter
Negative polarity shifter
Positive polarity shifter
MIAS - June 14, 2007
207
Lexicon
All
Instances
Corpus
Word token
Word prior polarity
Negated
Negated subject
Modifies polarity
Modified by polarity
Conjunction polarity
General polarity shifter
Negative polarity shifter
Positive polarity shifter
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Word token
terrifies
Word prior polarity
negative
MIAS - June 14, 2007
208
Lexicon
All
Instances
Corpus
Word token
Word prior polarity
Negated
Negated subject
Modifies polarity
Modified by polarity
Conjunction polarity
General polarity shifter
Negative polarity shifter
Positive polarity shifter
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Binary features:
Negated
For example:
not good
does not look very good
not only good but amazing
Negated subject
No politically prudent Israeli could
support either of them.
MIAS - June 14, 2007
209
Lexicon
All
Instances
Corpus
Word token
Word prior polarity
Negated
Negated subject
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
5 values: positive, negative,
neutral, both, not mod
Modifies polarity
Modified by polarity
Conjunction polarity
General polarity shifter
Negative polarity shifter
Positive polarity shifter
Modifies polarity
substantial: negative
Modified by polarity
5 values: positive, negative,
neutral, both, not mod
challenge: positive
substantial (pos) challenge (neg)
MIAS - June 14, 2007
210
Lexicon
All
Instances
Corpus
Word token
Word prior polarity
Negated
Negated subject
Modifies polarity
Modified by polarity
Conjunction polarity
General polarity shifter
Negative polarity shifter
Positive polarity shifter
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Conjunction polarity
5 values: positive, negative,
neutral, both, not mod
good: negative
good (pos) and evil (neg)
MIAS - June 14, 2007
211
Lexicon
All
Instances
Corpus
Word token
Word prior polarity
Negated
Negated subject
Modifies polarity
Modified by polarity
Conjunction polarity
General polarity shifter
Negative polarity shifter
Positive polarity shifter
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
General polarity shifter
have few risks/rewards
Negative polarity shifter
lack of understanding
Positive polarity shifter
abate the damage
MIAS - June 14, 2007
212
Lexicon
Step 1
All
Instances
Corpus
Neutral
or
Polar?
Polar
Instances Contextual
Polarity?
Results 2a
90
80
70
Step 2
77.2
65.7
65.1
60
50
46.2
Word token
Word + Prior Polarity
All Features
40
30
Accuracy
Pos F
Neg F
Neutral F
MIAS - June 14, 2007
213
Lexicon
Step 1
All
Instances
Corpus
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Results 2b
90
80
70
Word token
Word + Prior Polarity
All Features
60
50
40
Pos Pos Prec Neg Neg Prec
Recall
Recall
MIAS - June 14, 2007
214
Lexicon
All
Instances
Corpus
Step 1
Neutral
or
Polar?
Step 2
Polar
Instances Contextual
Polarity?
Ablation experiments removing features:
1. Negated, negated subject
2. Modifies polarity, modified by polarity
3. Conjunction polarity
4. General, negative, positive polarity shifters
Combination of features is needed to achieve
significant results over baseline
MIAS - June 14, 2007
215
Conclusions
Presented a two-step approach to phrase-level
sentiment analysis
1. Determine if an expression is neutral or polar
2. Determines contextual polarity of the ones that are
polar
Automatically identify the contextual polarity of a large
subset of sentiment expression
MIAS - June 14, 2007
217
Discussion: Lexicon <> Context
Objective
Subjective
#1
a bunch of
#2
#1
condemn
#2
headless
chicken
#1
hot
#2,4,8
#1,3,5,6,7
MIAS - June 14, 2007
218
He is not the sharpest knife in the drawer.
She is a few fries short of a Happy Meal.
Stephanie McMahon is the next Stalin.
You are no Jack Kennedy.
Whatever.
MIAS - June 14, 2007
219
That was utterly foolish.
No one would say that John is smart.
My little brother could have told you that.
Senator NN has voted with the president 95% of the
times.
MIAS - June 14, 2007
220
Outline
Corpus Annotation
Pure NLP
Lexicon development
Recognizing Contextual Polarity in Phrase-Level
Sentiment Analysis
Applications
Product review mining
MIAS - June 14, 2007
221
Product review mining
MIAS - June 14, 2007
222
Product review mining
Goal: summarize a set of reviews
Targeted opinion mining: topic is given
Two levels:
Product
Product and features
Typically done for pre-identified reviews but review
identification may be necessary
MIAS - June 14, 2007
223
Laptop review 1
A Keeper
Reviewed By: N.N. on 5/12/2007
Tech Level: average - Ownership: 1 week to 1 month
Pros: Price/Value. XP OS NOT VISTA! Screen good even in
bright daylignt. Easy to access USB, lightweight.
Cons: A bit slow - since we purchased this for vacation travel
(email & photos) speed is not a problem.
Other Thoughts: Would like to have card slots for camera/PDA
cards. Wish we could afford two so we can have a "spare".
MIAS - June 14, 2007
224
Laptop review 1
A Keeper
Reviewed By: N.N. on 5/12/2007
Tech Level: average - Ownership: 1 week to 1 month
Pros: Price/Value. XP OS NOT VISTA! Screen good even in
bright daylignt. Easy to access USB, lightweight.
Cons: A bit slow - since we purchased this for vacation travel
(email & photos) speed is not a problem.
Other Thoughts: Would like to have card slots for camera/PDA
cards. Wish we could afford two so we can have a "spare".
MIAS - June 14, 2007
225
Laptop review 2
By N.N. (New York - USA) - See all my reviews
I was looking for a laptop for long time, doing search, comparing brands,
technology, cost/benefits etc.... I should say that I am a normal user and this
laptop satisfied all my expectations, the screen size is perfect, its very light,
powerful, bright, lighter, elegant, delicate... But the only think that I regret is
the Battery life, barely 2 hours... some times less... it is too short... this
laptop for a flight trip is not good companion...
Even the short battery life I can say that I am very happy with my Laptop
VAIO and I consider that I did the best decision. I am sure that I did the best
decision buying the SONY VAIO
MIAS - June 14, 2007
226
Laptop review 2
By N.N. (New York - USA) - See all my reviews
I was looking for a laptop for long time, doing search, comparing brands,
technology, cost/benefits etc.... I should say that I am a normal user and
this laptop satisfied all my expectations, the screen size is perfect, its very
light, powerful, bright, lighter, elegant, delicate... But the only think that I
regret is the Battery life, barely 2 hours... some times less... it is too short...
this laptop for a flight trip is not good companion...
Even the short battery life I can say that I am very happy with my Laptop
VAIO and I consider that I did the best decision. I am sure that I did the best
decision buying the SONY VAIO
MIAS - June 14, 2007
227
Some challenges
Available NLP tools have harder time with review data (misspellings,
incomplete sentences)
Level of user experience (novice, …, prosumer)
Various types and formats of reviews
Additional buyer/owner narrative
What rating to assume for unmentioned features?
How to aggregate positive and negative evaluations?
How to present results?
MIAS - June 14, 2007
230
Core problems of review mining
Finding product features
Recognizing opinions
MIAS - June 14, 2007
231
Feature finding
Wide variety of linguistic expressions can evoke a
product feature
… you can't see the LCD very well in sunlight.
… it is very difficult to see the LCD.
… in the sun, the LCD screen is invisible
It is very difficult to take pictures outside in the sun
with only the LCD screen.
MIAS - June 14, 2007
232
Opinions v. Polar facts
Some statements invite emotional appraisal but do not
explicitly denote appraisal.
While such polar facts may in a particular context seem
to have an obvious value, their evaluation may be very
different in another one.
MIAS - June 14, 2007
233
A Keeper
Reviewed By: N.N. on 5/12/2007
Tech Level: average - Ownership: 1 week to 1 month
Pros: Price/Value. XP OS NOT VISTA! Screen good even in
bright daylignt. Easy to access USB, lightweight.
Cons: A bit slow - since we purchased this for vacation travel
(email & photos) speed is not a problem.
Other Thoughts: Would like to have card slots for camera/PDA
cards. Wish we could afford two so we can have a "spare".
MIAS - June 14, 2007
234
Use Coherence to resolve
orientation of polar facts
Is a sentence framed by two positive sentences likely to
also be positive?
Can context help settle the interpretation of inherently
non-evaluative attributes (e.g. hot room v. hot water in a
hotel context; Popescu & Etzioni 2005) ?
MIAS - June 14, 2007
235
Dave, Lawrence, Pennock 2003
Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of
Product Reviews
Product-level review-classification
Train Naïve Bayes classifier using a corpus of selftagged reviews available from major web sites (C|net,
amazon)
Refine the classifier using the same corpus before
evaluating it on sentences mined from broad web
searches
MIAS - June 14, 2007
236
Dave, Lawrence, Pennock 2003
Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of
Product Reviews
Feature selection
Substitution (statistical, linguistic)
I called Kodak
I called Nikon
I called Fuji
I called COMPANY
Backing off to wordnet synsets
Stemming
N-grams
arbitrary-length substrings
MIAS - June 14, 2007
237
Dave, Lawrence, Pennock 2003
Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of
Product Reviews
Feature selection
Substitution (statistical, linguistic)
Backing off to wordnet synsets
brilliant -> {brainy, brilliant, smart as a whip}
Stemming
N-grams
arbitrary-length substrings
MIAS - June 14, 2007
238
Dave, Lawrence, Pennock 2003
Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of
Product Reviews
Feature selection
Substitution (statistical, linguistic)
Backing off to wordnet synsets
Stemming
bought them
buying them
buy them
buy them
N-grams
arbitrary-length substrings
MIAS - June 14, 2007
239
Dave, Lawrence, Pennock 2003
Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of
Product Reviews
Feature selection
Substitution (statistical, linguistic)
Backing off to wordnet synsets
Stemming
N-grams
last long enough
too hard to
arbitrary-length substrings
MIAS - June 14, 2007
240
Dave, Lawrence, Pennock 2003
Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of
Product Reviews
Feature selection
Substitution (statistical, linguistic)
Backing off to wordnet synsets
Stemming
N-grams
arbitrary-length substrings
MIAS - June 14, 2007
241
Dave, Lawrence, Pennock 2003
Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of
Product Reviews
Laplace (add-one) smoothing was found to be best
2 types of test (1 balanced, 1 unbalanced)
SVM did better on Test 2 (balanced data) but not Test
1
Experiments with weighting features did not give better
results
MIAS - June 14, 2007
242
Hu & Liu 2004
Mining Opinion Features in Customer Reviews
Here: explicit product features only, expressed as nouns
or compound nouns
Use association rule mining technique rather than
symbolic or statistical approach to terminology
Extract associated items (item-sets) based on support
(>1%)
MIAS - June 14, 2007
243
Hu & Liu 2004
Mining Opinion Features in Customer Reviews
Feature pruning
compactness
“I had searched for a digital camera for 3 months.”
“This is the best digital camera on the market”
“The camera does not have a digital zoom”
Redundancy
manual ; manual mode; manual setting
MIAS - June 14, 2007
244
Hu & Liu 2004
Mining Opinion Features in Customer Reviews
For sentences with frequent feature, extract nearby
adjective as opinion
Based on opinion words, gather infrequent features (N,
NP nearest to an opinion adjective)
The salesman was easy going and let me try all the
models on display.
MIAS - June 14, 2007
245
Yi & Niblack 2005
Sentiment mining in WebFountain
MIAS - June 14, 2007
246
Yi & Niblack 2005
Sentiment mining in WebFountain
Product feature terms are extracted heuristically, with
high precision
For all definite base noun phrases,
the NN
the JJ NN
the NN NN NN
…
calculate a statistic based on likelihood ratio test
MIAS - June 14, 2007
247
Yi & Niblack 2005
Sentiment mining in WebFountain
Manually constructed
Sentiment lexicon: excellent JJ +
Pattern database: impress + PP(by; with)
Sentiment miner identifies the best fitting pattern for a
sentence based on the parse
MIAS - June 14, 2007
249
Yi & Niblack 2005
Sentiment mining in WebFountain
Manually constructed
Sentiment lexicon: excellent JJ +
Pattern database: impress + PP(by; with)
Sentiment miner identifies the best fitting pattern for a
sentence based on the parse
Sentiment is assigned to opinion target
MIAS - June 14, 2007
250
Yi & Niblack 2005
Sentiment mining in WebFountain
Discussion of hard cases:
Sentences that are ambiguous out of context
Cases that did not express a sentiment at all
Sentences that were not about the product:
Need to associate opinion and target
MIAS - June 14, 2007
251
Summary
Subjectivity is common in language
MIAS - June 14, 2007
252
Summary
Subjectivity is common in language
Recognizing it is useful in many NLP tasks
MIAS - June 14, 2007
253
Summary
Subjectivity is common in language
Recognizing it is useful in many NLP tasks
It comes in many forms and often is context-dependent
MIAS - June 14, 2007
254
Summary
Subjectivity is common in language
Recognizing it is useful in many NLP tasks
It comes in many forms and often is context-dependent
Contextual coherence and distributional similarity are
important linguistic notions in lexicon building
A wide variety of features seem to be necessary for
opinion and polarity recognition
MIAS - June 14, 2007
255
Summary
Goals for the future:
Fine resolution on opinions (expression level)
Integrating the various attributes of subjective
expressions (opinion expressions, sources, targets,
etc.)
MIAS - June 14, 2007
256
Additional material
scheme
Wilson & Wiebe 2005. Annotating Attributions and Private States
I think people are happy because Chavez has fallen.
direct subjective
span: think
source: <writer, I>
attitude:
attitude
span: think
type: positive arguing
intensity: medium
target:
target
span: people are happy because
Chavez has fallen
direct subjective
span: are happy
source: <writer, I, People>
attitude:
attitude
span: are happy
type: pos sentiment
intensity: medium
target:
target
span: Chavez has fallen
MIAS - June 14, 2007
inferred attitude
span: are happy because
Chavez has fallen
type: neg sentiment
intensity: medium
target:
target
span: Chavez
258
Work on the intensity of private states
Theresa Wilson, Janyce Wiebe and Rebecca Hwa
(2006). Recognizing strong and weak opinion clauses.
Computational Intelligence, 22 (2), pp. 73-99.
Theresa Wilson. 2007. Ph.D. Thesis. Fine-grained
Subjectivity and Sentiment Analysis: Recognizing the
Intensity, Polarity, and Attitudes of private states.
MIAS - June 14, 2007
259
James R. Martin and Peter R.R. White. 2005. The
Language of Evaluation: The Appraisal Framework.
An approach to evaluation that comes from within the
theory of systemic-functional grammar.
Website on this theory maintained by P.R. White:
http://www.grammatics.com/appraisal/index.html
MIAS - June 14, 2007
260
More work related to
lexicon building
Alina Andreevskaia and Sabine Bergler. 2006. Sentiment
Tag Extraction from WordNet Glosses.
Janyce Wiebe and Rada Mihalcea. 2006. Word Sense
and Subjectivity
Riloff, Patwardhan, Wiebe. 2006. Feature Subsumption
for Opinion Analysis.
MIAS - June 14, 2007
262
Takamura et al. 2007
Extracting Semantic Orientations of Phrases from Dictionary
Use a Potts model to categorize Adj+Noun phrases
Targets ambiguous adjectives like low, high, small, large
Connect two nouns, if one appears in gloss of other
Nodes have orientation values (pos, neg, neu) and are
connected by same or different orientation links
MIAS - June 14, 2007
263
A Sample Lexical Network
WORD
GLOSS
cost
loss or sacrifice, expenditure
loss
something lost
cost
loss
expenditure
sacrifice
lose
MIAS - June 14, 2007
264
Takamura et al 2007
Probabilistic Model on the Lexical Network (Potts
model)
index for node
set of seed words
state of node i
class label of seed word i
constants
MIAS - June 14, 2007
265
Takamura et al. 2007
The state of a seed word becomes
Neighboring nodes tend to have the same label.
“low cost” = “low expenditure”
MIAS - June 14, 2007
266
Takamura et al. 2007
Manually labeled adj+noun data provide noun seeds of
known orientation
The network assigns orientation to nouns not seen in
training data
MIAS - June 14, 2007
267
Further work on review
mining
Morinaga et. al. 2002. Mining Product Reputations on
the Web
Kobayashi et al. 2004. Collecting Evaluative Expressions
for Opinion Extraction
Hu & Liu. 2006. Opinion Feature Extraction Using Class
Sequential Rules
MIAS - June 14, 2007
269
Popescu & Etzioni 2005
Report on a product review mining system that extracts and labels
opinion expressions their attributes
They use the relaxation-labeling technique from computer vision to
perform unsupervised classification satisfying local constraints
(which they call neighborhood features)
The system tries to solve several classification problems (e.g.
opinion and target finding) at the same time rather than separately.
MIAS - June 14, 2007
270
Subjectivity and
Sentiment analysis not
discussed earlier
Question Answering
MIAS - June 14, 2007
272
Question Answering
Much work on Subjectivity & Sentiment has been motivated by QA.
Yu, H. & Hatzivassiloglou, V. (2003)
Kim, S. & Hovy, E. (AAAI-Workshop 2005)
Some QA work has also indicated that making a subjective/objective
distinction would be useful for the seemingly objective task of
definitional QA
Lita et al. (2005)
MIAS - June 14, 2007
273
Question Answering
Making subjective/objective distinction has been showed to be
useful in answering opinion based questions in news text
Stoyanov et al.(2005)
Making finer grained distinction of subjective types ( Sentiment and
Arguing) further improves the QA system ( world press and online
discussion forums)
Somasundaran et al. (2007)
MIAS - June 14, 2007
274
Information Extraction
Information Extraction has been used to learn subjective patterns
Wiebe and Riloff (2005)
Subjectivity has been shown to improve IE
Riloff et al. (2005)
MIAS - June 14, 2007
275
Summarization
Opinion Summaries from documents have been created
Stoyanov & Cardie (2006)
Carenini et al.(IUI-2006)
They combine fine grained opinions from the same source to create a source specific
summary of opinion
They summarize a large corpora of evaluative text about a single entity (product)
Different aspects of subjectivity analysis have been used to enhance summarizing
systems.
Seki et al. (2005)
Summarization based on user’s needs (benefits, positive/negative factors, commentary,
etc).
MIAS - June 14, 2007
276
Blog analysis
Analysis of sentiments on Blog posts
Chesley et al.(2006)
Sentiment has been used for blog analysis
Balog et al. (2006)
Discover irregularities in temporal mood patterns (fear, excitement, etc) appearing in a large corpus of
blogs
Kale et al. (2007)
Perform subjectivity and polarity classification on blog posts
Use link polarity information to model trust and influence in the blogosphere
Blog sentiment has been used in applications
Mishne and Glance (2006)
Analyze Blog sentiments about movies and correlate it with its sales
MIAS - June 14, 2007
277
Human Computer Interaction
Affect sensing
Liu et al. (2003)
Human Robot Interaction
Tokuhisa & Terashima (2006)
Correlate enthusiasm levels in dialogs with subjective
language for human robot interaction
MIAS - June 14, 2007
278
Visualization
Visualization of sentiments
Gregory et al. (2007)
Visualize affect distribution in social media (blogs) for a
topic.
Gamon et al. (2005)
Visualize sentiment and its orientation for a topic from large
number of customer feedback texts.
MIAS - June 14, 2007
279
Trends & Buzz
Stock market
Koppel & Shtrimberg(2004)
Correlate positive/negative news stories about publicly traded
companies and the stock price changes
Market Intelligence from message boards, forums, blogs.
Glance et al. (2005)
MIAS - June 14, 2007
280
Source and Target Finding
MIAS - June 14, 2007
281
Bethard et al. 2004
Automatic Extraction of Opinion Propositions and their Holders
Find verbs that express opinions in propositional form,
and their holders
Still, Vista officials realize they’re relatively fortunate.
Modify algorithms developed in earlier work on semantic
parsing to perform binary classification (opinion or not)
Use presence of subjectivity clues to identify opinionated
uses of verbs
MIAS - June 14, 2007
282
Choi et al.2005
Identifying sources of opinions with conditional random fields and extraction
patterns
Treats source finding as a combined sequential tagging and
information extraction task
IE patterns are high precision, lower recall
Base CRF uses information about noun phrase semantics,
morphology, syntax
IE patterns connect opinion words to sources
Conditional Random Fields given IE features perform better than
CRFs alone
MIAS - June 14, 2007
283
Kim & Hovy 2006
Extracting opinions, opinion holders, and
topics expressed in online news media text
Perform semantic role labeling (FrameNet) for a set of
adjectives and verbs (pos, neg)
Map semantic roles to holder and target
E.g. for Desiring frame: Experiencer->Holder
Train on FN data, test on FN data and on news
sentences collected and annotated by authors’
associates
Precision is higher for topics, recall for holders
MIAS - June 14, 2007
284
Choi, Breck, Cardie 2006
Joint extraction of entities and relations for opinion reocgnition
Find direct expressions of opinions and their sources
jointly
Uses sequence-tagging CRF classifiers for opinion
expressions, sources, and potential link relations
Integer linear programming combines local knowledge
and incorporates constraints
Performance better even on the individual tasks
MIAS - June 14, 2007
285
Further references on Source and
Target Finding
Breck & Cardie 2004. Playing the Telephone Game: Determining the
Hierarchical Structure of Perspective and Speech Expressions.
Bloom et al. 2007. Extracting Appraisal Expressions.
MIAS - June 14, 2007
286