Great Food, Lousy Service Topic Modeling for Sentiment Analysis in Sparse Reviews Robin Melnick [email protected]

Dan Preston [email protected]

OpenTable.com

Short

Characters Words

Sparse

“An unexpected combination of Left-Bank Paris and Lower Manhattan in Omaha. Divine. Inspirational and a great value.”

• • • • Food?

Ambiance?

Service?

Noise?

Skewed

Correlations

SVM + Features, Features, Features!

tokenize punctuation id, neutralize proper nouns strip numbers contraction splitting lower casing unigram (Bag of Words) bigram trigram mixed n-grams ignore stop words stemming negation processing expanded negation processing large training set size varying dictionary size SVM scaling "white list" (only use sentiment words) remove stop words POS tagging, ADJ only POS tagging, add ADV Brill tagger sentiment "white list" (Harvard lexicon) count of sentiment words (pos/neg) balanced training set binary accuracy sub-topic classifiers, hand list WordNet topic list expansion topic-filtered n-grams topic-word proximity filtering strict entropy modeling frequency-weighted entropy modeling • • 30+ preprocessing and SVM classification features, ~50 configurations

Key Features

• Stemming • Porter 1980 via NLTK •

< fast

>, <

faster

>, <

fastest >

 <

fast >

• Negation processing • • • (enhanced approach from Pang et al. 2002) “ Not a great experience.” 

NOT_great

“They never disappoint !” 

NOT_disappoint

• Net sentiment count • • • pos/neg lexicon (Harvard General Inquirer) running +/- count “ Incredible(+) food, but our server was rude(-) .”  (0)

Results (so far)

• • Trained on 10,000 reviews Tested on ~80,000 reviews • Accuracy • • Baseline: Intermediate model: 50.0% 56.6% ( 1.13x

) • abs( average scoring delta ): 0.56

Topic Modeling

Hand-seeded topic-word list expanded via WordNet SynSets 1.

2.

3.

sub-topic classifiers topic-filtered n-grams • • • <soup FOOD

was fantastic

ADJ > <fantastic ADJ

soup

FOOD was> topic-word proximity filtering both above  <fantastic ADJ/FOOD >.

Results: 1.

3.

Food

39.15% 40.05% 1.02x

Ambiance

47.26% 47.88% 1.01x

Service

53.70% 54.92% 1.02x

Noise

48.43% 50.35% 1.03x

Word-Rating Distributions

“worst” “mediocre” “decent” “solid” “exceeded”

Frequency-Weighted Entropy Model • Accuracy • • • Baseline: Intermediate model: Best (entropy) model: 50.0% 56.6% 58.6% ( 1.17x

) • abs( average scoring delta ): 0.56  0.52