Predictive Profiling from Massive Transactional Data Sets
Download
Report
Transcript Predictive Profiling from Massive Transactional Data Sets
Author-Topic Models
for Large Text Corpora
Padhraic Smyth
Department of Computer Science
University of California, Irvine
In collaboration with:
Mark Steyvers (UCI)
Michal Rosen-Zvi (UCI)
Tom Griffiths (Stanford)
Outline
•
Problem motivation:
•
•
Probabilistic approaches
•
•
topic models -> author-topic models
Results
•
•
•
•
Modeling large sets of documents
Author-topic results from CiteSeer, NIPS, Enron data
Applications of the model
(Demo of author-topic query tool)
Future directions
Data Sets of Interest
•
Data = set of documents
•
Large collection of documents: 10k, 100k, etc
Know authors of the documents
Know years/dates of the documents
……
•
(will typically assume bag of words representation)
•
•
•
Examples of Data Sets
•
CiteSeer:
•
•
NIPS papers
•
•
160k abstracts, 80k authors, 1986-2002
2k papers, 1k authors, 1987-1999
Reuters
•
20k newspaper articles, 114 authors
Pennsylvania Gazette
1728-1800
80,000 articles
25 million words
www.accessible.com
Enron email data
500,000 emails
5000 authors
1999-2002
Problems of Interest
•
What topics do these documents “span”?
•
Which documents are about a particular topic?
•
How have topics changed over time?
•
What does author X write about?
•
Who is likely to write about topic Y?
•
Who wrote this specific document?
•
and so on…..
A topic is represented as a
(multinomial) distribution over words
TOPIC 289
TOPIC 209
WORD
PROB.
WORD
PROB.
PROBABILISTIC
0.0778
RETRIEVAL
0.1179
BAYESIAN
0.0671
TEXT
0.0853
PROBABILITY
0.0532
DOCUMENTS
0.0527
CARLO
0.0309
INFORMATION
0.0504
MONTE
0.0308
DOCUMENT
0.0441
DISTRIBUTION
0.0257
CONTENT
0.0242
INFERENCE
0.0253
INDEXING
0.0205
PROBABILITIES
0.0253
RELEVANCE
0.0159
CONDITIONAL
0.0229
COLLECTION
0.0146
PRIOR
0.0219
RELEVANT
0.0136
....
...
...
...
P( w | z )
Cluster Models
DOCUMENT 1
DOCUMENT 2
Probabilistic
Information
Learning
Retrieval
Learning
Information
Bayesian
Retrieval
Cluster Models
DOCUMENT 1
DOCUMENT 2
Probabilistic
Information
Learning
Retrieval
Learning
Information
Bayesian
Retrieval
P(probabilistic | topic) = 0.25
P(learning
| topic) = 0.50
P(Bayesian
| topic) = 0.25
P(other words | topic) = 0.00
P(information | topic) = 0.5
P(retrieval
| topic) = 0.5
P(other words | topic) = 0.0
Graphical Model
z
Cluster Variable
w
Word
n words
Graphical Model
z
Cluster Variable
w
Word
n words
D documents
Graphical Model
Cluster
Weights
Cluster-Word
distributions
a
z
Cluster Variable
w
Word
f
n words
D documents
Cluster Models
DOCUMENT 1
DOCUMENT 2
Probabilistic
Information
Learning
Retrieval
Learning
Information
Bayesian
Retrieval
DOCUMENT 3
Probabilistic
Learning
Information
Retrieval
Topic Models
DOCUMENT 1
DOCUMENT 2
Probabilistic
Information
Learning
Retrieval
Learning
Information
Bayesian
Retrieval
Topic Models
DOCUMENT 1
DOCUMENT 2
Probabilistic
Information
Learning
Retrieval
Learning
Information
Bayesian
Retrieval
DOCUMENT 3
Probabilistic
Learning
Information
Retrieval
History of topic models
•
Latent class models in statistics (late 60’s)
•
Hoffman (1999)
•
•
Blei, Ng, and Jordan (2001, 2003)
•
•
Original application to documents
Variational methods
Griffiths and Steyvers (2003, 2004)
•
Gibbs sampling approach (very efficient)
Word/Document counts
for 16 Artificial Documents
River
Stream
Bank
Money
Loan
documents
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Can we recover the original topics and topic mixtures from this data?
Example of Gibbs Sampling
•
Assign word tokens randomly to topics:
(●=topic 1; ●=topic 2 )
River
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Stream
Bank
Money
Loan
After 1 iteration
•
Apply sampling equation to each word token
River
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Stream
Bank
Money
Loan
After 4 iterations
River
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Stream
Bank
Money
Loan
After 32 iterations
f
●
●
topic 1
stream .40
bank .35
river .25
River
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Stream
Bank
topic 2
bank .39
money .32
loan .29
Money
Loan
Topic Models
DOCUMENT 1
DOCUMENT 2
Probabilistic
Information
Learning
Retrieval
Learning
Information
Bayesian
Retrieval
DOCUMENT 3
Probabilistic
Learning
Information
Retrieval
Author-Topic Models
DOCUMENT 1
DOCUMENT 2
Probabilistic
Information
Learning
Retrieval
Learning
Information
Bayesian
Retrieval
Author-Topic Models
DOCUMENT 1
DOCUMENT 2
Probabilistic
Information
Learning
Retrieval
Learning
Information
Bayesian
Retrieval
DOCUMENT 3
Probabilistic
Learning
Information
Retrieval
Approach
•
The author-topic model
•
a probabilistic model linking authors and topics
•
•
learned from data
•
•
authors -> topics -> words
completely unsupervised, no labels
generative model
•
•
•
Different questions or queries can be answered by
appropriate probability calculus
E.g., p(author | words in document)
E.g., p(topic | author)
Graphical Model
x
Author
z
Topic
Graphical Model
x
Author
z
Topic
w
Word
Graphical Model
x
Author
z
Topic
w
Word
n
Graphical Model
a
x
Author
z
Topic
w
Word
n
D
Graphical Model
a
Author-Topic
distributions
Topic-Word
distributions
q
f
x
Author
z
Topic
w
Word
n
D
Generative Process
•
•
Let’s assume authors A1 and A2 collaborate and produce a
paper
•
A1 has multinomial topic distribution q1
•
A2 has multinomial topic distribution q2
For each word in the paper:
1. Sample an author x (uniformly) from A1, A2
2. Sample a topic z from qX
3. Sample a word w from a multinomial topic distribution fz
Graphical Model
a
Author-Topic
distributions
Topic-Word
distributions
q
f
x
Author
z
Topic
w
Word
n
D
Learning
•
Observed
•
•
Unknown
•
•
•
x, z : hidden variables
Θ, f : unknown parameters
Interested in:
•
•
•
W = observed words, A = sets of known authors
p( x, z | W, A)
p( θ , f | W, A)
But exact inference is not tractable
Step 1: Gibbs sampling of x and z
a
Marginalize
over unknown
parameters
q
f
x
Author
z
Topic
w
Word
n
D
Step 2: MAP estimates of θ and f
a
q
f
x
Author
z
Topic
w
Word
Condition on
particular
samples of
x and z
n
D
Step 2: MAP estimates of θ and f
a
q
Point estimates
of unknown
parameters
f
x
Author
z
Topic
w
Word
n
D
More Details on Learning
•
Gibbs sampling for x and z
•
•
•
Estimating θ and f
•
•
•
x and z sample -> point estimates
non-informative Dirichlet priors for θ and f
Computational Efficiency
•
•
Typically run 2000 Gibbs iterations
1 iteration = full pass through all documents
Learning is linear in the number of word tokens
Predictions on new documents
•
can average over θ and f
different runs)
(from different samples,
Gibbs Sampling
•
Need full conditional distributions for variables
•
The probability of assigning the current word i to topic j and
author k given everything else:
P( zi j , xi k | wi m, z i , x i , w i , a d )
WT
Cmj
WT
C
m' m' j V
CmjAT a
AT
C
j ' kj' Ta
number of times word w assigned to topic j
number of times topic j assigned to author k
Experiments on Real Data
•
Corpora
•
•
•
•
CiteSeer:
160K abstracts, 85K authors
NIPS:
1.7K papers, 2K authors
Enron:
115K emails, 5K authors (sender)
Pubmed:27K abstracts, 50K authors
•
Removed stop words; no stemming
•
Ignore word order, just use word counts
•
Processing time:
Nips: 2000 Gibbs iterations 8 hours
CiteSeer: 2000 Gibbs iterations 4 days
Four example topics from CiteSeer (T=300)
TOPIC 205
TOPIC 209
WORD
PROB.
WORD
DATA
0.1563
MINING
0.0674
BAYESIAN
ATTRIBUTES
0.0462
DISCOVERY
TOPIC 289
PROB.
TOPIC 10
WORD
PROB.
WORD
PROB.
RETRIEVAL
0.1179
QUERY
0.1848
0.0671
TEXT
0.0853
QUERIES
0.1367
PROBABILITY
0.0532
DOCUMENTS
0.0527
INDEX
0.0488
0.0401
CARLO
0.0309
INFORMATION
0.0504
DATA
0.0368
ASSOCIATION
0.0335
MONTE
0.0308
DOCUMENT
0.0441
JOIN
0.0260
LARGE
0.0280
CONTENT
0.0242
INDEXING
0.0180
KNOWLEDGE
0.0260
INDEXING
0.0205
PROCESSING 0.0113
DATABASES
0.0210
RELEVANCE
0.0159
AGGREGATE 0.0110
ATTRIBUTE
0.0188
CONDITIONAL
0.0229
COLLECTION
0.0146
ACCESS
0.0102
DATASETS
0.0165
PRIOR
0.0219
RELEVANT
0.0136
PRESENT
0.0095
AUTHOR
PROB.
AUTHOR
PROB.
AUTHOR
PROB.
AUTHOR
PROB.
Han_J
0.0196
Friedman_N
0.0094
Oard_D
0.0110
Suciu_D
0.0102
Rastogi_R
0.0094
Heckerman_D
0.0067
Croft_W
0.0056
Naughton_J
0.0095
Zaki_M
0.0084
Ghahramani_Z
0.0062
Jones_K
0.0053
Levy_A
0.0071
Shim_K
0.0077
Koller_D
0.0062
Schauble_P
0.0051
DeWitt_D
0.0068
Ng_R
0.0060
Jordan_M
0.0059
Voorhees_E
0.0050
Wong_L
0.0067
Liu_B
0.0058
Neal_R
0.0055
Singhal_A
0.0048
Mannila_H
0.0056
Raftery_A
0.0054
Hawking_D
0.0048
Ross_K
0.0061
Brin_S
0.0054
Lukasiewicz_T
0.0053
Merkl_D
0.0042
Hellerstein_J
0.0059
Liu_H
0.0047
Halpern_J
0.0052
Allan_J
0.0040
Lenzerini_M
0.0054
Holder_L
0.0044
Muller_P
0.0048
Doermann_D
0.0039
Moerkotte_G
0.0053
PROBABILISTIC 0.0778
DISTRIBUTION 0.0257
INFERENCE
0.0253
PROBABILITIES 0.0253
Chakrabarti_K 0.0064
More CiteSeer Topics
TOPIC 10
TOPIC 209
WORD
TOPIC 87
WORD
PROB.
SPEECH
0.1134
RECOGNITION
0.0349
BAYESIAN
0.0671
TOPIC 20
PROB.
WORD
PROB.
WORD
PROB.
PROBABILISTIC 0.0778
USER
0.2541
STARS
0.0164
INTERFACE
0.1080
OBSERVATIONS 0.0150
WORD
0.0295
PROBABILITY
0.0532
USERS
0.0788
SOLAR
0.0150
SPEAKER
0.0227
CARLO
0.0309
INTERFACES
0.0433
MAGNETIC
0.0145
ACOUSTIC
0.0205
MONTE
0.0308
GRAPHICAL
0.0392
RAY
0.0144
RATE
0.0134
DISTRIBUTION
0.0257
INTERACTIVE
0.0354
EMISSION
0.0134
SPOKEN
0.0132
INFERENCE
0.0253
INTERACTION
0.0261
GALAXIES
0.0124
SOUND
0.0127
VISUAL
0.0203
OBSERVED
0.0108
TRAINING
0.0104
CONDITIONAL
0.0229
DISPLAY
0.0128
SUBJECT
0.0101
MUSIC
0.0102
PRIOR
0.0219
MANIPULATION
0.0099
STAR
0.0087
AUTHOR
PROB.
AUTHOR
PROB.
AUTHOR
PROB.
AUTHOR
PROB.
Waibel_A
0.0156
Friedman_N
0.0094
Shneiderman_B
0.0060
Linsky_J
0.0143
Gauvain_J
0.0133
Heckerman_D
0.0067
Rauterberg_M
0.0031
Falcke_H
0.0131
Lamel_L
0.0128
Ghahramani_Z
0.0062
Lavana_H
0.0024
Mursula_K
0.0089
Woodland_P
0.0124
Koller_D
0.0062
Pentland_A
0.0021
Butler_R
0.0083
Ney_H
0.0080
Jordan_M
0.0059
Myers_B
0.0021
Bjorkman_K
0.0078
Hansen_J
0.0078
Neal_R
0.0055
Minas_M
0.0021
Knapp_G
0.0067
Renals_S
0.0072
Raftery_A
0.0054
Burnett_M
0.0021
Kundu_M
0.0063
Noth_E
0.0071
Lukasiewicz_T
0.0053
Winiwarter_W
0.0020
Christensen-J
0.0059
Boves_L
0.0070
Halpern_J
0.0052
Chang_S
0.0019
Cranmer_S
0.0055
Young_S
0.0069
Muller_P
0.0048
Korvemaker_B
0.0019
Nagar_N
0.0050
PROBABILITIES 0.0253
Some topics relate to generic word
usage
TOPIC 273
WORD
PROB.
METHOD
0.5851
METHODS
0.3321
APPLIED
0.0268
APPLYING
0.0056
ORIGINAL
0.0054
DEVELOPED
0.0051
PROPOSE
0.0046
COMBINES
0.0034
PRACTICAL
0.0031
APPLY
0.0029
AUTHOR
PROB.
Yang_T
0.0014
Zhang_J
0.0014
Loncaric_S
0.0014
Liu_Y
0.0013
Benner_P
0.0013
Faloutsos_C
0.0013
Cortadella_J
0.0012
Paige_R
0.0011
Tai_X
0.0011
Lee_J
0.0011
What can the Model be used for?
•
We can analyze our document set through the
“topic lens”
•
Applications
•
Queries
•
Who writes on this topic?
•
•
•
•
•
•
e.g., finding experts or reviewers in a particular area
What topics does this person do research on?
Discovering trends over time
Detecting unusual papers and authors
Interactive browsing of a digital library via topics
Parsing documents (and parts of documents) by topic
and more…..
Some likely topics per author (CiteSeer)
•
Author = Andrew McCallum, U Mass:
•
•
•
•
Topic 1: classification, training, generalization, decision, data,…
Topic 2: learning, machine, examples, reinforcement, inductive,…..
Topic 3: retrieval, text, document, information, content,…
Author = Hector Garcia-Molina, Stanford:
- Topic 1: query, index, data, join, processing, aggregate….
- Topic 2: transaction, concurrency, copy, permission, distributed….
- Topic 3: source, separation, paper, heterogeneous, merging…..
•
Author = Paul Cohen, USC/ISI:
- Topic 1: agent, multi, coordination, autonomous, intelligent….
- Topic 2: planning, action, goal, world, execution, situation…
- Topic 3: human, interaction, people, cognitive, social, natural….
Temporal patterns in topics: hot and cold topics
•
We have CiteSeer papers from 1986-2002
•
For each year, calculate the fraction of words
assigned to each topic
•
-> a time-series for topics
•
•
Hot topics become more prevalent
Cold topics become less prevalent
4
2
x 10
Document and Word Distribution by Year in the UCI CiteSeer Data
5
x 10
14
1.8
12
1.6
Number of Documents
1.2
8
1
6
0.8
0.6
4
0.4
2
0.2
0
1986
1988
1990
1992
1994
Year
1996
1998
2000
2002
0
Number of Words
10
1.4
0.012
CHANGING TRENDS IN COMPUTER SCIENCE
0.01
WWW
Topic Probability
0.008
0.006
INFORMATION
RETRIEVAL
0.004
0.002
0
1990
1992
1994
1996
Year
1998
2000
2002
0.012
CHANGING TRENDS IN COMPUTER SCIENCE
0.01
Topic Probability
0.008
OPERATING
SYSTEMS
WWW
PROGRAMMING
LANGUAGES
0.006
INFORMATION
RETRIEVAL
0.004
0.002
0
1990
1992
1994
1996
Year
1998
2000
2002
-3
8
x 10
HOT TOPICS: MACHINE LEARNING/DATA MINING
7
Topic Probability
6
CLASSIFICATION
5
REGRESSION
4
DATA MINING
3
2
1
1990
1992
1994
1996
Year
1998
2000
2002
-3
5.5
x 10
BAYES MARCHES ON
5
BAYESIAN
Topic Probability
4.5
PROBABILITY
4
3.5
STATISTICAL
PREDICTION
3
2.5
2
1.5
1990
1992
1994
1996
Year
1998
2000
2002
0.012
INTERESTING "TOPICS"
0.01
FRENCH WORDS:
LA, LES, UNE, NOUS, EST
Topic Probability
0.008
0.006
DARPA
0.004
0.002
0
1990
MATH SYMBOLS:
GAMMA, DELTA, OMEGA
1992
1994
1996
Year
1998
2000
2002
Four example topics from NIPS (T=100)
TOPIC 19
TOPIC 24
TOPIC 29
WORD
PROB.
WORD
PROB.
WORD
LIKELIHOOD
0.0539
RECOGNITION
0.0400
MIXTURE
0.0509
CHARACTER
0.0336
POLICY
EM
0.0470
CHARACTERS
0.0250
DENSITY
0.0398
TANGENT
GAUSSIAN
0.0349
ESTIMATION
0.0314
DIGITS
LOG
0.0263
MAXIMUM
TOPIC 87
PROB.
WORD
PROB.
KERNEL
0.0683
0.0371
SUPPORT
0.0377
ACTION
0.0332
VECTOR
0.0257
0.0241
OPTIMAL
0.0208
KERNELS
0.0217
HANDWRITTEN 0.0169
ACTIONS
0.0208
SET
0.0205
0.0159
FUNCTION
0.0178
SVM
0.0204
IMAGE
0.0157
REWARD
0.0165
SPACE
0.0188
0.0254
DISTANCE
0.0153
SUTTON
0.0164
MACHINES
0.0168
PARAMETERS
0.0209
DIGIT
0.0149
AGENT
0.0136
ESTIMATE
0.0204
HAND
0.0126
DECISION
0.0118
MARGIN
0.0151
AUTHOR
PROB.
AUTHOR
PROB.
AUTHOR
PROB.
AUTHOR
PROB.
Tresp_V
0.0333
Simard_P
0.0694
Singh_S
0.1412
Smola_A
0.1033
Singer_Y
0.0281
Martin_G
0.0394
Barto_A
0.0471
Scholkopf_B
0.0730
Jebara_T
0.0207
LeCun_Y
0.0359
Sutton_R
0.0430
Burges_C
0.0489
Ghahramani_Z
0.0196
Denker_J
0.0278
Dayan_P
0.0324
Vapnik_V
0.0431
Ueda_N
0.0170
Henderson_D
0.0256
Parr_R
0.0314
Chapelle_O
0.0210
Jordan_M
0.0150
Revow_M
0.0229
Dietterich_T
0.0231
Cristianini_N
0.0185
Roweis_S
0.0123
Platt_J
0.0226
Tsitsiklis_J
0.0194
Ratsch_G
0.0172
Schuster_M
0.0104
Keeler_J
0.0192
Randlov_J
0.0167
Laskov_P
0.0169
Xu_L
0.0098
Rashid_M
0.0182
Bradtke_S
0.0161
Tipping_M
0.0153
Saul_L
0.0094
Sackinger_E
0.0132
Schwartz_A
0.0142
Sollich_P
0.0141
REINFORCEMENT 0.0411
REGRESSION 0.0155
NIPS: support vector topic
NIPS: neural network topic
Pennsylvania Gazette Data
(courtesy of David Newman, UC Irvine)
Enron email data
500,000 emails
5000 authors
1999-2002
Enron email topics
TOPIC 36
TOPIC 72
TOPIC 23
TOPIC 54
WORD
PROB.
WORD
PROB.
WORD
PROB.
FEEDBACK
0.0781
PROJECT
0.0514
FERC
0.0554
PERFORMANCE
0.0462
PLANT
0.028
MARKET
0.0328
PROCESS
0.0455
COST
0.0182
ISO
0.0226
PEP
0.0446
MANAGEMENT
0.03
UNIT
0.0166
ORDER
COMPLETE
0.0205
FACILITY
0.0165
QUESTIONS
0.0203
SITE
0.0136
CONSTRUCTION 0.0169
WORD
PROB.
ENVIRONMENTAL 0.0291
AIR
0.0232
MTBE
0.019
EMISSIONS
0.017
0.0212
CLEAN
0.0143
FILING
0.0149
EPA
0.0133
COMMENTS
0.0116
PENDING
0.0129
COMMISSION 0.0215
SELECTED
0.0187
PROJECTS
0.0117
PRICE
0.0116
SAFETY
0.0104
COMPLETED
0.0146
CONTRACT
0.011
CALIFORNIA
0.0110
WATER
0.0092
SYSTEM
0.0146
UNITS
0.0106
FILED
0.0110
GASOLINE
0.0086
SENDER
PROB.
SENDER
PROB.
SENDER
PROB.
SENDER
PROB.
perfmgmt
0.2195
***
0.0288
***
0.0532
***
0.1339
perf eval process
0.0784
***
0.022
***
0.0454
***
0.0275
enron announcements
0.0489
***
0.0123
***
0.0384
***
0.0205
***
0.0089
***
0.0111
***
0.0334
***
0.0166
***
0.0048
***
0.0108
***
0.0317
***
0.0129
Non-work Topics…
TOPIC 66
TOPIC 182
TOPIC 113
TOPIC 109
WORD
PROB.
WORD
PROB.
WORD
PROB.
WORD
PROB.
HOLIDAY
0.0857
TEXANS
0.0145
GOD
0.0357
AMAZON
0.0312
PARTY
0.0368
WIN
0.0143
LIFE
0.0272
GIFT
0.0226
YEAR
0.0316
FOOTBALL
0.0137
MAN
0.0116
CLICK
0.0193
SEASON
0.0305
FANTASY
0.0129
PEOPLE
0.0103
SAVE
0.0147
COMPANY
0.0255
SPORTSLINE
0.0129
CHRIST
0.0092
SHOPPING
0.0140
CELEBRATION
0.0199
PLAY
0.0123
FAITH
0.0083
OFFER
0.0124
ENRON
0.0198
TEAM
0.0114
LORD
0.0079
HOLIDAY
0.0122
TIME
0.0194
GAME
0.0112
JESUS
0.0075
RECEIVE
0.0102
RECOGNIZE
0.019
SPORTS
0.011
SPIRITUAL
0.0066
SHIPPING
0.0100
MONTH
0.018
GAMES
0.0109
VISIT
0.0065
FLOWERS
0.0099
SENDER
PROB.
SENDER
PROB.
SENDER
PROB.
SENDER
PROB.
chairman & ceo
0.131
cbs sportsline com 0.0866
crosswalk com
0.2358
amazon com
0.1344
***
0.0102
houston texans 0.0267
wordsmith
0.0208
jos a bank
0.0266
***
0.0046
houstontexans 0.0203
***
0.0107
sharperimageoffers
0.0136
***
0.0022
sportsline rewards 0.0175
travelocity com
0.0094
general announcement 0.0017
pro football 0.0136
barnes & noble com
0.0089
doctor dictionary 0.0101
***
0.0061
Topical Topics
TOPIC 18
TOPIC 22
TOPIC 114
WORD
PROB.
WORD
PROB.
WORD
PROB.
TOPIC 194
WORD
PROB.
POWER
0.0915
STATE
0.0253
COMMITTEE
0.0197
LAW
0.0380
CALIFORNIA
0.0756
PLAN
0.0245
BILL
0.0189
TESTIMONY
0.0201
ELECTRICITY
0.0331
CALIFORNIA
0.0137
HOUSE
0.0169
ATTORNEY
0.0164
UTILITIES
0.0253
POLITICIAN Y
0.0137
SETTLEMENT
0.0131
PRICES
0.0249
RATE
0.0131
LEGAL
0.0100
MARKET
0.0244
EXHIBIT
0.0098
PRICE
0.0207
SOCAL
0.0119
CONGRESS
0.0112
CLE
0.0093
UTILITY
0.0140
POWER
0.0114
PRESIDENT
0.0105
SOCALGAS
0.0093
CUSTOMERS
0.0134
BONDS
0.0109
METALS
0.0091
ELECTRIC
0.0120
MOU
0.0107
DC
0.0093
PERSON Z
0.0083
SENDER
PROB.
SENDER
PROB.
SENDER
PROB.
SENDER
PROB.
***
0.1160
***
0.0395
***
0.0696
***
0.0696
***
0.0518
***
0.0337
***
0.0453
***
0.0453
***
0.0284
***
0.0295
***
0.0255
***
0.0255
***
0.0272
***
0.0251
***
0.0173
***
0.0173
***
0.0266
***
0.0202
***
0.0317
***
0.0317
BANKRUPTCY 0.0126
WASHINGTON 0.0140
SENATE
0.0135
POLITICIAN X 0.0114
LEGISLATION 0.0099
Enron email: California Energy Crisis
Message-ID: <21993848.1075843452041.JavaMail.evans@thyme>
Date: Fri, 27 Apr 2001 09:25:00 -0700 (PDT)
Subject: California Update 4/27/01
………….
FERC price cap decision reflects Bush political and economic objectives.
Politically, Bush is determined to let the crisis blame fall on Davis;
from an economic perspective, he is unwilling to create disincentives
for new power generation
The FERC decision is a holding move by the Bush administration that
looks like action, but is not. Rather, it allows the situation in California
to continue to develop virtually unabated. The political strategy
appears to allow the situation to deteriorate to the point where Davis
cannot escape shouldering the blame. Once they are politically
inoculated, the Administration can begin to look at regional solutions.
Moreover, the Administration has already made explicit (and will
certainly restate in the forthcoming Cheney commission report) its
opposition to stronger price caps …..
Enron email: US Senate Bill
Message-ID: <23926374.1075846156491.JavaMail.evans@thyme>
Date: Thu, 15 Jun 2000 08:59:00 -0700 (PDT)
From: ***************
To: ***************
Subject: Senate Commerce Committee Pipeline Safety Markup
The Senate Commerce Committee held a markup today where Senator John McCain's
(R-AZ) pipeline safety legislation, S. 2438, was approved. The overall
outcome was not unexpected -- the final legislation contained several
provisions that went a little bit further than Enron and INGAA would have
liked, ……………
2) McCain amendment to Section 13 (b) (on operator assistance investigations)
-- Approved by voice vote. …….
3) Sen. John Kerry (D-MA) Amendment on Enforcement -- Approved by voice
vote. Another confusing vote, in which many members did not understand the
changes being made, but agreed to it on the condition that clarifications be
made before Senate floor action. Late last night, Enron led a group
including companies from INGAA and AGA in providing comments to Senator Kerry
which caused him to make substantial changes to his amendment before it was
voted on at markup, including dropping provisions allowing citizen suits and
other troubling issues. In the end, the amendment that passed was
acceptable to industry.
Enron email: political donations
10/16/2000 04:41 PM
Subject: Ashcroft Senate Campaign Request
We have received a request from the Ashcroft Senate campaign for $10,000 in
soft money. This is the race where Governor Carnahan is the challenger. Enron
PAC has contributed $10,000 and Enron has also contributed $15,000 soft money
in this campaign to Senator Ashcroft. Ken Lay has been personally interested
in the Ashcroft campaign. Our polling information is that Ashcroft is
currently leading 43 to 38 with an undecided of 19 percent.
…………………………………………………………………………………………………………………………………
Message-ID: <2546687.1075846182883.JavaMail.evans@thyme>
Date: Mon, 16 Oct 2000 14:13:00 -0700 (PDT)
From: *****
To: *****
Subject: Re: Ashcroft Senate Campaign Request
If you can cover it I would say yes. It's a key race and we have been close
to Ashcroft for years. Let's make sure he knows we gave it.... we need to
follow up with him. Last time I talked to him he basically recited the
utilities' position on electric restructuring. Let's make it clear that we
want to talk right after the election.
PubMed-Query Topics
TOPIC 188
WORD
BIOLOGICAL
TOPIC 63
PROB.
WORD
TOPIC 32
TOPIC 85
PROB.
WORD
PROB.
0.1002
PLAGUE
0.0296
BOTULISM
0.1014
AGENTS
0.0889
MEDICAL
0.0287
BOTULINUM
0.0888
THREAT
0.0396
CENTURY 0.0280
BIOTERRORISM 0.0348
MEDICINE
0.0266
0.0669
INHIBITORS
0.0366
0.0340
INHIBITOR
0.0220
0.0245
PLASMA
0.0204
0.0203
POTENTIAL
0.0305
EPIDEMIC
0.0106
ATTACK
0.0290
GREAT
0.0091
CHEMICAL
0.0288
WARFARE
0.0219
CHINESE
0.0083
ANTHRAX
0.0146
FRENCH
0.0082
PARALYSIS
0.0124
PROB.
AUTHOR
PROB.
AUTHOR
0.0563
TYPE
HISTORY
PROB.
PROTEASE
0.0916
0.0877
0.0328
AUTHOR
HIV
PROB.
TOXIN
WEAPONS
EPIDEMICS 0.0090
WORD
CLOSTRIDIUM
INFANT
NEUROTOXIN
AMPRENAVIR 0.0527
0.0184
APV
0.0169
BONT
0.0167
DRUG
0.0169
FOOD
0.0134
RITONAVIR
0.0164
IMMUNODEFICIENCY0.0150
AUTHOR
PROB.
Atlas_RM
0.0044
Károly_L
0.0089
Hatheway_CL
0.0254
Sadler_BM
0.0129
Tegnell_A
0.0036
Jian-ping_Z
0.0085
Schiavo_G
0.0141
Tisdale_M
0.0118
Aas_P
0.0036
Sabbatani_S
0.0080
Sugiyama_H
0.0111
Lou_Y
0.0069
Arnon_SS
0.0108
Stein_DS
0.0069
Simpson_LL
0.0093
Haubrich_R
0.0061
Greenfield_RA
Bricaire_F
0.0032
0.0032
Theodorides_J 0.0045
Bowers_JZ
0.0045
PubMed-Query Topics
TOPIC 40
WORD
TOPIC 89
PROB.
WORD
ANTHRACIS
0.1627
CHEMICAL 0.0578
ANTHRAX
0.1402
SARIN
0.0454
BACILLUS
0.1219
AGENT
0.0332
SPORES
0.0614
GAS
0.0312
CEREUS
0.0382
SPORE
0.0274
THURINGIENSIS 0.0177
AGENTS
VX
PROB.
ENZYME
0.0938
MUSTARD
0.0639
ACTIVE
0.0429
EXPOSURE
0.0444
SM
0.0361
0.0264
SKIN
0.0208
REACTION
0.0225
0.0232
EXPOSED
0.0185
AGENT
0.0140
0.0124
TOXIC
0.0197
PRODUCTS 0.0170
PROB.
EPIDERMAL
DAMAGE
AUTHOR
Mock_M
0.0203
Minami_M
0.0093
Phillips_AP
0.0125
Hoskin_FC
0.0092
Smith_WJ
Welkos_SL
0.0083
Benschop_HP 0.0090
Turnbull_PC
0.0071
Raushel_FM 0.0084
Wild_JR
SITE
0.0399
0.0308
STERNE
0.0067
0.0353
SUBSTRATE
ENZYMES
0.0220
Fouet_A
PROB.
0.0657
NERVE
AUTHOR
WORD
0.0343
ACID
PROB.
HD
PROB.
SULFUR
0.0152
AUTHOR
WORD
0.0268
SUBTILIS
INHALATIONAL 0.0104
TOPIC 178
TOPIC 104
0.0075
0.0129
0.0116
PROB.
Monteiro-Riviere_NA 0.0284
SUBSTRATES 0.0201
FOLD
CATALYTIC
RATE
AUTHOR
0.0176
0.0154
0.0148
PROB.
Masson_P
0.0166
0.0219
Kovach_IM
0.0137
Lindsay_CD
0.0214
Schramm_VL
0.0094
Sawyer_TW
0.0146
Meier_HL
0.0139
Barak_D
Broomfield_CA
0.0076
0.0072
PubMed-Query Author Model
•
P. M. Lindeque, South Africa
TOPICS
• Topic
• Topic
• Topic
• Topic
• Topic
1:
2:
3:
4:
5:
water, natural, foci, environmental, source
anthracis, anthrax, bacillus, spores, cereus
species, sp, isolated, populations, tested
epidemic, occurred, outbreak, persons
positive, samples, negative, tested
prob=0.33
prob=0.13
prob=0.06
prob=0.06
prob=0.05
PAPERS
•
•
•
•
Vaccine-induced protections against anthrax in cheetah
Airborne movement of anthrax spores from carcass sites in the Etosha
National Park
Ecology and epidemiology of anthrax in the Etosha National Park
Serology and anthrax in humans, livestock, and wildlife
PubMed-Query: Topics by Country
ISRAEL, n=196 authors
TOPIC 188
p=0.049
BIOLOGICAL
AGENTS
THREAT
BIOTERRORISM
WEAPONS
POTENTIAL
ATTACK
CHEMICAL
WARFARE
ANTHRAX
TOPIC 6
p=0.045
INJURY
INJURIES
WAR
TERRORIST
MILITARY
MEDICAL
VICTIMS
TRAUMA
BLAST
VETERANS
TOPIC 133
p=0.043
HEALTH
PUBLIC
CARE
SERVICES
EDUCATION
NATIONAL
COMMUNITY
INFORMATION
PREVENTION
LOCAL
TOPIC 104
p=0.027
HD
TOPIC 7
p=0.026
RENAL
HFRS
VIRUS
SYNDROME
FEVER
TOPIC 79
p=0.024
FINDINGS
CHEST
CT
LUNG
CLINICAL
HEMORRHAGIC
PULMONARY
CONCLUSIONS
BACKGROUND
HANTAVIRUS
HANTAAN
ABNORMAL
STUDY
TOPIC 197
p=0.023
PATIENTS
HOSPITAL
PATIENT
ADMITTED
TWENTY
HOSPITALIZED
CONSECUTIVE
INVOLVEMENT
OBJECTIVES
PROSPECTIVELY
MUSTARD
EXPOSURE
SM
SULFUR
SKIN
EXPOSED
AGENT
EPIDERMAL
DAMAGE
TOPIC 159
p=0.025
EMERGENCY
RESPONSE
MEDICAL
PREPAREDNESS
DISASTER
MANAGEMENT
TRAINING
EVENTS
BIOTERRORISM
LOCAL
CHINA, n=1775 authors
TOPIC 177
p=0.045
SARS
RESPIRATORY
SEVERE
COV
SYNDROME
ACUTE
CORONAVIRUS
CHINA
TOPIC 49
p=0.024
METHODS
RESULTS
CONCLUSION
OBJECTIVE
POTENTIAL
ATTACK
CHEMICAL
WARFARE
ANTHRAX
MEDICAL
VICTIMS
TRAUMA
BLAST
VETERANS
NATIONAL
COMMUNITY
INFORMATION
PREVENTION
LOCAL
SKIN
TOPIC 7
p=0.026
RENAL
HFRS
VIRUS
SYNDROME
FEVER
TOPIC 79
p=0.024
FINDINGS
CHEST
CT
LUNG
CLINICAL
HEMORRHAGIC
PULMONARY
CONCLUSIONS
BACKGROUND
HANTAVIRUS
HANTAAN
PUUMALA
ABNORMAL
STUDY
INVOLVEMENT
COMMON
OBJECTIVES
INVESTIGATE
HANTAVIRUSES
RADIOGRAPHIC
DESIGN
EXPOSED
MANAGEMENT
TRAINING
EVENTS
BIOTERRORISM
LOCAL
PubMed-Query: Topics by
Country
EPIDERMAL
AGENT
DAMAGE
CHINA, n=1775 authors
TOPIC 177
p=0.045
SARS
RESPIRATORY
SEVERE
COV
SYNDROME
ACUTE
CORONAVIRUS
CHINA
KONG
PROBABLE
TOPIC 49
p=0.024
METHODS
RESULTS
CONCLUSION
OBJECTIVE
TOPIC 197
p=0.023
PATIENTS
HOSPITAL
PATIENT
ADMITTED
TWENTY
HOSPITALIZED
CONSECUTIVE
PROSPECTIVELY
DIAGNOSED
PROGNOSIS
3 of 300 example topics (TASA)
TOPIC 82
WORD
TOPIC 77
TOPIC 166
PROB.
WORD
PROB.
WORD
PROB.
PLAY
0.0601
MUSIC
0.0903
PLAY
0.1358
PLAYS
0.0362
DANCE
0.0345
BALL
0.1288
STAGE
0.0305
SONG
0.0329
GAME
0.0654
MOVIE
0.0288
PLAY
0.0301
PLAYING
0.0418
SCENE
0.0253
SING
0.0265
HIT
0.0324
ROLE
0.0245
SINGING
0.0264
PLAYED
0.0312
AUDIENCE
0.0197
BAND
0.0260
BASEBALL
0.0274
THEATER
0.0186
PLAYED
0.0229
GAMES
0.0250
PART
0.0178
SANG
0.0224
BAT
0.0193
FILM
0.0148
SONGS
0.0208
RUN
0.0186
ACTORS
0.0145
DANCING
0.0198
THROW
0.0158
DRAMA
0.0136
PIANO
0.0169
BALLS
0.0154
REAL
0.0128
PLAYING
0.0159
TENNIS
0.0107
CHARACTER 0.0122
RHYTHM
0.0145
HOME
0.0099
ACTOR
0.0116
ALBERT
0.0134
CATCH
0.0098
ACT
0.0114
MUSICAL
0.0134
FIELD
0.0097
MOVIES
0.0114
DRUM
0.0129
PLAYER
0.0096
ACTION
0.0101
GUITAR
0.0098
FUN
0.0092
0.0097
BEAT
0.0097
THROWING
0.0083
0.0094
BALLET
0.0096
PITCHER
0.0080
SET
SCENES
Word sense disambiguation
(numbers & colors topic assignments)
A Play082 is written082 to be performed082 on a stage082 before a live093
audience082 or before motion270 picture004 or television004 cameras004 ( for
later054 viewing004 by large202 audiences082). A Play082 is written082
because playwrights082 have something ...
He was listening077 to music077 coming009 from a passing043 riverboat. The
music077 had already captured006 his heart157 as well as his ear119. It was
jazz077. Bix beiderbecke had already had music077 lessons077. He
wanted268 to play077 the cornet. And he wanted268 to play077 jazz077...
Jim296 plays166 the game166. Jim296 likes081 the game166 for one. The
game166 book254 helps081 jim296. Don180 comes040 into the house038. Don180
and jim296 read254 the game166 book254. The boys020 see a game166 for
two. The two boys020 play166 the game166....
Finding unusual papers for an author
Perplexity = exp [entropy (words | model) ]
= measure of surprise for model on data
Can calculate perplexity of unseen documents,
conditioned on the model for a particular author
Papers and Perplexities: M_Jordan
Factorial Hidden Markov Models
687
Learning from Incomplete Data
702
Papers and Perplexities: M_Jordan
Factorial Hidden Markov Models
687
Learning from Incomplete Data
702
MEDIAN PERPLEXITY
2567
Papers and Perplexities: M_Jordan
Factorial Hidden Markov Models
687
Learning from Incomplete Data
702
MEDIAN PERPLEXITY
2567
Defining and Handling Transient
Fields in Pjama
14555
An Orthogonally Persistent JAVA
16021
Papers and Perplexities: T_Mitchell
Explanation-based Learning for
Mobile Robot Perception
1093
Learning to Extract Symbolic
Knowledge from the Web
1196
Papers and Perplexities: T_Mitchell
Explanation-based Learning for
Mobile Robot Perception
1093
Learning to Extract Symbolic
Knowledge from the Web
1196
MEDIAN PERPLEXITY
2837
Papers and Perplexities: T_Mitchell
Explanation-based Learning for
Mobile Robot Perception
1093
Learning to Extract Symbolic
Knowledge from the Web
1196
MEDIAN PERPLEXITY
2837
Text Classification from Labeled and
Unlabeled Documents using EM
3802
A Method for Estimating
Occupational Radiation Dose…
8814
Author prediction with CiteSeer
•
Task: predict (single) author of new CiteSeer
abstracts
•
Results:
•
•
For 33% of documents, author guessed correctly
Median rank of true author = 26 (out of 85,000)
Who wrote what?
Test of model:
1) artificially combine abstracts from different authors
2) check whether assignment is to correct original
author
A method1 is described which like the kernel1 trick1 in support1 vector1 machines1 SVMs1 lets us
generalize distance1 based2 algorithms to operate in feature1 spaces usually nonlinearly related
to the input1 space This is done by identifying a class of kernels1 which can be represented as
norm1 based2 distances1 in Hilbert spaces It turns1 out that common kernel1 algorithms such as
SVMs1 and kernel1 PCA1 are actually really distance1 based2 algorithms and can be run2 with that
class of kernels1 too As well as providing1 a useful new insight1 into how these algorithms work
the present2 work can form the basis1 for conceiving new algorithms
This paper presents2 a comprehensive approach for model2 based2 diagnosis2 which includes
proposals for characterizing and computing2 preferred2 diagnoses2 assuming that the system2
description2 is augmented with a system2 structure2 a directed2 graph2 explicating the
interconnections between system2 components2 Specifically we first introduce the notion of a
consequence2 which is a syntactically2 unconstrained propositional2 sentence2 that characterizes
all consistency2 based2 diagnoses2 and show2 that standard2 characterizations of diagnoses2 such
as minimal conflicts1 correspond to syntactic2 variations1 on a consequence2 Second we propose
a new syntactic2 variation on the consequence2 known as negation2 normal form NNF and discuss
its merits compared to standard variations Third we introduce a basic algorithm2 for computing
consequences in NNF given a structured system2 description We show that if the system2
structure2 does not contain cycles2 then there is always a linear size2 consequence2 in NNF which
can be computed in linear time2 For arbitrary1 system2 structures2 we show a precise connection
between the complexity2 of computing2 consequences and the topology of the underlying system2
structure2 Finally we present2 an algorithm2 that enumerates2 the preferred2 diagnoses2
characterized by a consequence2 The algorithm2 is shown1 to take linear time2 in the size2 of the
consequence2 if the preference criterion1 satisfies some general conditions
Written by
(1) Scholkopf_B
Written by
(2) Darwiche_A
The Author-Topic
Browser
Querying on
author
Pazzani_M
Querying on topic
relevant to author
Querying on
document written
by author
Stability of Topics
•
Content of topics is arbitrary across runs of model
(e.g., topic #1 is not the same across runs)
•
However,
•
•
•
Majority of topics are stable over processing time
Majority of topics can be aligned across runs
Topics appear to represent genuine structure in
data
Comparing NIPS topics from the same Markov chain
10
16
20
14
30
12
40
10
50
t1
ANALOG
CIRCUIT
CHIP
CURRENT
VOLTAGE
VLSI
INPUT
OUTPUT
CIRCUITS
FIGURE
PULSE
SYNAPSE
SILICON
CMOS
MEAD
.043
.040
.034
.025
.023
.022
.018
.018
.015
.014
.012
.011
.011
.009
.008
t2
ANALOG
CIRCUIT
CHIP
VOLTAGE
CURRENT
VLSI
OUTPUT
INPUT
CIRCUITS
PULSE
SYNAPSE
SILICON
FIGURE
CMOS
GATE
.044
.040
.037
.024
.023
.023
.022
.019
.015
.012
.012
.011
.010
.009
.009
8
60
KL distance
Re-ordered topics at t2=2000
BEST KL = 0.54
70
80
90
100
20
40
60
topics at t1=1000
80
100
6
4
2
WORST KL = 4.78
t1
FEEDBACK
ADAPTATION
CORTEX
REGION
FIGURE
FUNCTION
BRAIN
COMPUTATIONAL
FIBER
FIBERS
ELECTRIC
BOWER
FISH
SIMULATIONS
CEREBELLAR
.040
.034
.025
.016
.015
.014
.013
.013
.012
.011
.011
.010
.010
.009
.009
t2
ADAPTATION
FIGURE
SIMULATION
GAIN
EFFECTS
FIBERS
COMPUTATIONAL
EXPERIMENT
FIBER
SITES
RESULTS
EXPERIMENTS
ELECTRIC
SITE
NEURO
.051
.033
.026
.025
.016
.014
.014
.014
.013
.012
.012
.012
.011
.009
.009
Gibbs Sampler Stability (NIPS data)
New Applications/ Future Work
•
Reviewer Recommendation
•
•
Change Detection/Monitoring
•
•
•
Which authors are on the leading edge of new topics?
Characterize the “topic trajectory” of this author over time
Author Identification
•
•
“Find reviewers for this set of grant proposals who are active
in relevant topics and have no conflicts of interest”
Who wrote this document? Incorporation of stylistic
information (stylometry)
Additions to the model
•
•
•
Modeling citations
Modeling topic persistence in a document
…..
Summary
•
Topic models are a versatile probabilistic model for text data
•
Author-topic models are a very useful generalization
•
•
Equivalent to topics model with 1 different author per document
Learning has linear time complexity
•
•
Gibbs sampling is practical on very large data sets
Experimental results
•
•
On multiple large complex data sets, the resulting topic-word and
author-topic models are quite interpretable
Results appear stable relative to sampling
•
Numerous possible applications….
•
Current model is quite simple….many extensions possible
Further Information
•
www.datalab.uci.edu
•
•
•
Steyvers et al, ACM SIGKDD 2004
Rosen-Zvi et al, UAI 2004
www.datalab.uci.edu/author-topic
•
•
JAVA demo of online browser
additional tables and results
BACKUP SLIDES
Author-Topics Model
a
Author-Topic
distributions
Topic-Word
distributions
q
f
x
Author
z
Topic
w
Word
n
D
Topics Model: Topics, no Authors
q
Topic-Word
distributions
f
x
Author
z
Topic
w
Word
Document-Topic
Distributions
n
D
Author Model: Authors, no Topics
a
Author-Word
Distributions
a
Author
w
Word
f
n
D
Comparison Results
14000
•
Train models on part
of a new document
and predict remaining
words
•
Without having seen
any words from new
document, authortopic information
helps in predicting
words from that
document
•
Topics model is more
flexible in adapting to
new document after
observing a number of
words
Author model
10000
8000
6000
4000
24
6
10
25
64
16
4
Author-Topics
1
2000
Topics model
0
Perplexity (new words)
12000
# Observed words in document
Latent Semantic Analysis
(Landauer & Dumais, 1997)
word/document counts
high dimensional space
Doc1
Doc2
Doc3 …
RIVER
34
0
0
STREAM
12
0
0
BANK
5
19
6
MONEY
…
0
…
16
…
1
…
SVD
STREAM
RIVER
BANK
Words with similar co-occurence patterns across documents
end up with similar vector representations
MONEY
Topics
LSA
•
•
•
•
Geometric
•
Probabilistic
•
Fully generative
•
Topic dimensions are
often interpretable
•
Modular language of
bayes nets/ graphical
models
Partially generative
Dimensions are
not interpretable
Little flexibility to expand
model (e.g., syntax)
Modeling syntax and semantics
(Steyvers, Griffiths, Blei, and Tenenbaum)
semantics: probabilistic topics
q
long-range, document specific,
dependencies
short-range dependencies constant
across all documents
z
z
z
w
w
w
x
x
x
syntax: 3rd order HMM