Great Food, Lousy Service Topic Modeling for Sentiment Analysis in Sparse Reviews Robin Melnick Dan Preston [email protected] [email protected].
Download ReportTranscript Great Food, Lousy Service Topic Modeling for Sentiment Analysis in Sparse Reviews Robin Melnick Dan Preston [email protected] [email protected].
Great Food, Lousy Service Topic Modeling for Sentiment Analysis in Sparse Reviews Robin Melnick [email protected]
Dan Preston [email protected]
OpenTable.com
Short
Characters Words
Sparse
“An unexpected combination of Left-Bank Paris and Lower Manhattan in Omaha. Divine. Inspirational and a great value.”
• • • • Food?
Ambiance?
Service?
Noise?
Skewed
Correlations
SVM + Features, Features, Features!
tokenize punctuation id, neutralize proper nouns strip numbers contraction splitting lower casing unigram (Bag of Words) bigram trigram mixed n-grams ignore stop words stemming negation processing expanded negation processing large training set size varying dictionary size SVM scaling "white list" (only use sentiment words) remove stop words POS tagging, ADJ only POS tagging, add ADV Brill tagger sentiment "white list" (Harvard lexicon) count of sentiment words (pos/neg) balanced training set binary accuracy sub-topic classifiers, hand list WordNet topic list expansion topic-filtered n-grams topic-word proximity filtering strict entropy modeling frequency-weighted entropy modeling • • 30+ preprocessing and SVM classification features, ~50 configurations
Key Features
• Stemming • Porter 1980 via NLTK •
< fast
>, <
faster
>, <
fastest >
<
fast >
• Negation processing • • • (enhanced approach from Pang et al. 2002) “ Not a great experience.”
NOT_great
“They never disappoint !”
NOT_disappoint
• Net sentiment count • • • pos/neg lexicon (Harvard General Inquirer) running +/- count “ Incredible(+) food, but our server was rude(-) .” (0)
Results (so far)
• • Trained on 10,000 reviews Tested on ~80,000 reviews • Accuracy • • Baseline: Intermediate model: 50.0% 56.6% ( 1.13x
) • abs( average scoring delta ): 0.56
Topic Modeling
Hand-seeded topic-word list expanded via WordNet SynSets 1.
2.
3.
sub-topic classifiers topic-filtered n-grams • • • <soup FOOD
was fantastic
ADJ > <fantastic ADJ
soup
FOOD was> topic-word proximity filtering both above <fantastic ADJ/FOOD >.
Results: 1.
3.
Food
39.15% 40.05% 1.02x
Ambiance
47.26% 47.88% 1.01x
Service
53.70% 54.92% 1.02x
Noise
48.43% 50.35% 1.03x
Word-Rating Distributions
“worst” “mediocre” “decent” “solid” “exceeded”
Frequency-Weighted Entropy Model • Accuracy • • • Baseline: Intermediate model: Best (entropy) model: 50.0% 56.6% 58.6% ( 1.17x
) • abs( average scoring delta ): 0.56 0.52