Transcript CS 294-5: Statistical Natural Language Processing

CS 5331: Applied Machine Learning
Spring 2011
Bayesian Networks (+MDP)
Feb 11 2011
Mohan Sridharan
1
Bayes’ Nets
 A Bayes’ net is an efficient
encoding of a probabilistic
model of a domain.
 Questions we can ask:
 Inference: given a fixed BN, what is P(X | e)?
 Representation: given a BN graph, what kinds of distributions
can it encode?
 Modeling: what BN is most appropriate for a given domain?
2
Bayes’ Net Semantics
 Formal semantics of a Bayes’ net.
 A set of nodes, one per variable X.
 A directed acyclic graph (DAG).
 A conditional distribution for each node:
 A collection of distributions over X, one for
each combination of parents’ values:
A1
An
X
 CPT: conditional probability table.
 Description of a noisy “causal” process.
A Bayes net = Topology (graph) + Local Conditional Probabilities.
3
Building the (Entire) Joint
 Can take a Bayes’ net and build any entry from the full
joint distribution it encodes
 Typically, there’s no reason to build ALL of it.
 Build what we need on the fly.
 Every BN implicitly defines a joint distribution over a
domain, specified by local probabilities and graph
structure.
4
Example: Coins
 Extra arcs do not prevent independence, just
allow non-independence.
X1
X2
X1
X2
h
0.5
h
0.5
h
0.5
h|h
0.5
t
0.5
t
0.5
t
0.5
t|h
0.5
h|t
0.5
t|t
0.5
 Adding unneeded arcs not
wrong, just inefficient!
5
Topology Limits Distributions
Y
 Given graph topology G, only certain
joint distributions can be encoded.
X
Z
Y
X
 The graph structure guarantees
certain (conditional) independences.
Z
 There might be more independence!
 Adding arcs increases the set of
distributions, but has several costs.
 Full conditioning can encode any
distribution.
Y
X
Z
6
Independence in a BN
 Important question about a BN:




Are two nodes independent given certain evidence?
If yes, can prove using algebra (tedious in general).
If no, can prove with a counter example.
Example:
X
Y
Z
 Question: are X and Z necessarily independent?
 Answer: no. Example: low pressure causes rain, which
causes traffic.
 X can influence Z and Z can influence X (via Y).
 Addendum: they could be independent: how?
7
Causal Chains
 This configuration is a causal chain:
X
Y
X: Low pressure
Z
Y: Rain
Z: Traffic
 Is X independent of Z given Y?
Yes!
 Evidence along the chain “blocks” the influence.
8
Common Cause
Y
 Another basic configuration: two effects
of the same cause
 Are X and Z independent?
 Are X and Z independent given Y?
X
Z
Y: Project due
X: Newsgroup
busy
Yes!
Z: Lab full
 Observing the cause blocks the influence
between effects!
9
Common Effect
 Two causes of one effect (v-structures).
 Are X and Z independent?
 Yes: the ballgame and the rain cause traffic, but
they are not correlated.
 Still need to prove they must be -- try it!
X
Z
Y
 Are X and Z independent given Y?
 No: seeing traffic puts the rain and the ballgame
in competition as explanation!
 This is backwards from the other cases!
X: Raining
Z: Ballgame
Y: Traffic
 Observing an effect activates influence between
possible causes.
10
The General Case
 Any complex example can be analyzed using
these three canonical cases.
 General question: in a given BN, are two
variables independent (given evidence)?
 Solution: analyze the graph 
 Recipe: shade evidence nodes.
11
Reachability (D-Separation)
 Are X, Y conditionally independent
given evidence {Z}?
Active Triples
Inactive Triples
 Yes, if X and Y “separated” by Z
 Look for active paths from X to Y
 No active paths = independence!
 Path is active if each triple is active:
 Causal chain A  B  C where B
is unobserved (either direction).
 Common cause A  B  C where
B is unobserved.
 Common effect (aka v-structure)
A  B  C where B or one of its
descendents is observed.
 All it takes to block a path is a
single inactive segment!
12
Example
Yes
R
B
T
T’
13
Example
L
Yes
Yes
R
D
B
T
Yes
T’
14
Example
 Variables:
 R: Raining
 T: Traffic
 D: Roof drips
 S: I’m sad
 Questions:
R
T
D
S
Yes
15
Causality?
 Bayes’ nets that reflect the true causal patterns:
 Often simpler (nodes have fewer parents).
 Often easier to think about.
 Often easier to elicit from experts.
 BNs need not actually be causal 
 Sometimes no causal net exists over the domain.
 E.g. consider the variables Traffic and Drips.
 Arrows reflect correlation not causation.
 What do the arrows really mean?
 Topology may happen to encode causal structure.
 Topology only guaranteed to encode conditional independence!
16
Changing Bayes’ Net Structure
 The same joint distribution can be encoded in
many different Bayes’ nets:
 Causal structure tends to be the simplest.
 Analysis question: given some edges, what
other edges do you need to add?
 One answer: fully connect the graph (inefficient).
 Better answer: don’t make any false conditional
independence assumptions 
17
Example: Alternate Alarm
Earthq
uake
Burglary
If we reverse the edges, we make
different conditional independence
assumptions
John
calls
Alarm
Mary
calls
Alarm
John
calls
Mary
calls
To capture the same joint
distribution, we have to add more
edges to the graph
Burglary
Earthqu
ake
18
Summary
 Bayes nets compactly encode joint distributions.
 Guaranteed independencies of distributions can be
deduced from BN graph structure.
 D-separation gives precise conditional independence
guarantees from graph alone.
 A Bayes’ net’s joint distribution may have further
(conditional) independence that is not detectable until
you inspect its specific distribution.
19
Inference
 Inference: calculating some useful
quantity from a joint probability
distribution.
 Examples:
 Posterior probability:
 Most likely explanation:
B
E
A
J
M
20
Inference by Enumeration
 Given unlimited time, inference in BNs is easy.
 Recipe:
 State the marginal probabilities you need.
 Figure out ALL the atomic probabilities you need.
 Calculate and combine them.
B
E
 Example:
A
J
M
21
Example: Enumeration
 This simple method only needs the BN to synthesize the
joint entries.
22
Inference by Enumeration?
23
Variable Elimination
 Why is inference by enumeration so slow?
 Need to join the whole joint distribution before you sum out the
hidden variables.
 End up repeating a lot of work!
 Idea: interleave joining and marginalizing!
 Called “Variable Elimination”.
 Still NP-hard, but usually much faster than inference by
enumeration.
 We will need some new notation to define VE.
24
Variable Elimination
 What you need to know:
 Should be able to run it on small examples, understand the factor
creation / reduction flow.
 Better than enumeration: saves time by marginalizing variables as soon
as possible rather than at the end.
 Special cases of VE:
 On tree-structured graphs, variable elimination runs in polynomial time,
like tree-structured CSPs (Constraint Satisfaction Problems).
 Pseudocode: Figure 14.11 (AIMA).
 Complexity of exact inference? (Section 14.4.3, AIMA).
 Clustering algorithms (Section 14.4.4, AIMA).
25
Approximate Inference

Simulation has a name: sampling.

Sampling is a hot topic in machine learning, and it is really simple (later!) 

Basic idea:
 Draw N samples from a sampling distribution S.
 Compute an approximate posterior probability.
 Show this converges to the true probability P.

Why sample?
 Learning: get samples from a distribution you do not know.
 Inference: getting a sample is faster than computing the right answer
(e.g. with variable elimination).
26
Other Stuff
 More Bayesian Nets: Chapter 14, AIMA.
 Coming up: MDPs and RL.
27
Reinforcement Learning
 Basic idea:
 Receive feedback in the form of rewards.
 Agent’s utility is defined by the reward function.
 Must learn to act so as to maximize expected rewards.
28
Reinforcement Learning
 Basic idea:




Receive feedback in the form of rewards.
Agent’s utility is defined by the reward function.
Must learn to act so as to maximize expected rewards.
Change the rewards, change the learned behavior!
 Examples:
 Playing a game, reward at the end for winning / losing
 Vacuuming a house, reward for each piece of dirt picked up
 Automated taxi, reward for each passenger delivered
 First: Need to master MDPs.
29
Grid World
 The agent lives in a grid.
 Walls block the agent’s path.
 The agent’s actions do not always
go as planned:



80% of the time, the action North
takes the agent North (if there is
no wall there).
10% of the time, North takes the
agent West; 10% East.
If there is a wall in the direction
the agent would have been taken,
the agent stays put.
 Big rewards come at the end.
30
Markov Decision Processes
 An MDP is defined by:
 A set of states s  S.
 A set of actions a  A.
 A transition function T(s, a, s’):
 Probability that a from s leads to s’.
 P(s’ | s, a) – also called the model.
 A reward function R(s, a, s’):
 Sometimes just R(s) or R(s’).
 A start state (or distribution).
 Maybe a terminal state..
 MDPs are a family of non-deterministic
search problems:
 Reinforcement learning: MDPs where
the T and R are unknown.
31
What is Markov about MDPs?
 Andrey Markov (1856-1922)
 “Markov” generally means that given the
present state, the future and the past are
independent.
 For MDPs, “Markov” means:
32
Solving MDPs
 In deterministic single-agent search problem, want an
optimal plan, or sequence of actions, from start to goal.
 In an MDP, we want an optimal policy *: S → A.
 A policy  gives an action for each state.
 An optimal policy maximizes expected utility if followed.
 Defines a reflex agent.
Optimal policy when
R(s, a, s’) = -0.03 for all
non-terminals s.
33
Coming Up…
 More on MDPs.
 Intro to RL.
 Presenter: Muneem.
34