Transcript CCMs - Dan Roth

Constrained Conditional Models
for
Natural Language Processing
Ming-Wei Chang, Lev Ratinov, Dan Roth
Department of Computer Science
University of Illinois at Urbana-Champaign
March 2009
EACL
Page 1
Nice to Meet You
Page 2
Constraints Conditional Models (CCMs)
 Informally:
 Everything that has to do with constraints (and learning models)
 Formally:
 We typically make decisions based on models such as:
arg maxy wT f ( y, x)
 With CCMs we make decisions based on models such as:
arg max y wT f ( y, x)   c d ( y,1C )
 We do not define the learning method cC
 but we’ll discuss it and make suggestions
 CCMs make predictions in the presence/guided by constraints
Page 3
Constraints Driven Learning and Decision Making

Why Constraints?


The Goal: Building a good NLP systems easily
We have prior knowledge at our hand


How can we use it?
We suggest that often knowledge can be injected directly
Can use it to guide learning
 Can use it to improve decision making
 Can use it to simplify the models we need to learn


How useful are constraints?



Useful for supervised learning
Useful for semi-supervised learning
Sometimes more efficient than labeling data directly
Inference
Page 5
A process that maintains and
updates a collection of propositions
about the state of affairs.
Comprehension
(ENGLAND, June, 1989) - Christopher Robin is alive and well. He lives in
England. He is the same person that you read about in the book, Winnie the
Pooh. As a boy, Chris lived in a pretty home called Cotchfield Farm. When
Chris was three years old, his father wrote a poem about him. The poem was
printed in a magazine for others to read. Mr. Robin then wrote a book. He
made up a fairy tale land where Chris lived. His friends were animals. There
was a bear called Winnie the Pooh. There was also an owl and a young pig,
called a piglet. All the animals were stuffed toys that Chris owned. Mr. Robin
made them come to life with his words. The places in the story were all near
Cotchfield Farm. Winnie the Pooh was written in 1925. Children still love to
read about Christopher Robin and his animal friends. Most people don't know
he is a real person who is grown now. He has written two books of his own.
They tell what it is like to be famous.
1. Christopher Robin was born in England.
3. Christopher Robin’s dad was a magician.
2. Winnie the Pooh is a title of a book.
4. Christopher Robin must be at least 65 now.
This is an Inference Problem
Page 6
This Tutorial: Constrained Conditional Models

Part 1: Introduction to CCMs

Examples:




First summary: why are CCM important
Problem Setting


Features and Constraints; Some hints about training issues
Part 2: Introduction to Integer Linear Programming


NE + Relations
Information extraction – correcting models with CCMS
What is ILP; use of ILP in NLP
Part 3: Detailed examples of using CCMs



Semantic Role Labeling in Details
Coreference Resolution
Sentence Compression
BREAK
Page 7
This Tutorial: Constrained Conditional Models (2nd part)

Part 4: More on Inference –

Other inference algorithms



Using hard and soft constraints
Part 5: Training issues when working with CCMS


Formalism (again)
Choices of training paradigms -- Tradeoffs



Search (SRL); Dynamic Programming (Transliteration); Cutting Planes
Examples in Supervised learning
Examples in Semi-Supervised learning
Part 6: Conclusion

Building CCMs
Features and Constraints; Objective functions; Different Learners.

Mixed models vs. Joint models; where is Knowledge coming from

THE END
Page 8
Learning and Inference
 Global decisions in which several local decisions play a role but
there are mutual dependencies on their outcome.
 E.g. Structured Output Problems – multiple dependent output variables
 (Learned) models/classifiers for different sub-problems
 In some cases, not all local models can be learned simultaneously
 Key examples in NLP are Textual Entailment and QA
 In these cases, constraints may appear only at evaluation time
 Incorporate models’ information, along with prior
knowledge/constraints, in making coherent decisions
 decisions that respect the local models as well as domain & context
specific knowledge/constraints.
Page 9
This Tutorial: Constrained Conditional Models

Part 1: Introduction to CCMs

Examples:




First summary: why are CCM important
Problem Setting


Features and Constraints; Some hints about training issues
Part 2: Introduction to Integer Linear Programming



NE + Relations
Information extraction – correcting models with CCMS
What is ILP; use of ILP in NLP
Semantic Role Labeling in Details
Part 3: Detailed examples of using CCMs


Coreference Resolution
Sentence Compression
BREAK
Page 10
This Tutorial: Constrained Conditional Models (2nd part)

Part 4: More on Inference –

Other inference algorithms



Using hard and soft constraints
Part 5: Training issues when working with CCMS


Formalism (again)
Choices of training paradigms -- Tradeoffs



Search (SRL); Dynamic Programming (Transliteration); Cutting Planes
Examples in Supervised learning
Examples in Semi-Supervised learning
Part 6: Conclusion

Building CCMs
Features and Constraints; Objective functions; Different Learners.

Mixed models vs. Joint models; where is Knowledge coming from

THE END
Page 11
Inference with General Constraint Structure
Improvement over
[Roth&Yih’04]
no inference: 2-5%
Recognizing Entities and Relations
other
0.05
other
0.10
other
0.05
per
0.85
per
0.60
per
0.50
loc
0.10
loc
0.30
loc
0.45
Dole ’s wife, Elizabeth , is a native ofSome
N.C.Questions:
E1
E2
R12
R23
How to guide the global
E3
inference?
Why not learn Jointly?
irrelevant
irrelevant
0.05
irrelevant
0.10
spouse_of
spouse_of
0.45
0.45
spouse_of
0.05
born_in
born_in
0.50
born_in
0.85
Note:
Non Sequential
Model
Models could be learned separately; constraints may come up only at decision time.
Page 12
Task of Interests: Structured Output

For each instance, assign values to a set of variables
Output variables depend on each other

Common tasks in


Natural language processing


Information extraction


Entities, Relations,…
Many pure machine learning approaches exist



Parsing; Semantic Parsing; Summarization; Transliteration; Co-reference
resolution,…
Hidden Markov Models (HMMs); CRFs
Structured Perceptrons ad SVMs…
However, …
Page 13
Information Extraction via Hidden Markov Models
Lars Ole Andersen . Program analysis and specialization for the
C Programming language. PhD thesis. DIKU ,
University of Copenhagen, May 1994 .
Prediction result of a trained HMM
[AUTHOR]
[TITLE]
[EDITOR]
[BOOKTITLE]
[TECH-REPORT]
[INSTITUTION]
[DATE]
Lars Ole Andersen . Program analysis and
specialization for the
C
Programming language
. PhD thesis .
DIKU , University of Copenhagen , May
1994 .
Unsatisfactory results !
Page 14
Strategies for Improving the Results

(Pure) Machine Learning Approaches



Higher Order HMM/CRF?
Increasing the window size?
Adding a lot of new features



Requires a lot of labeled examples
What if we only have a few labeled examples?
Any other options?


Increasing the model complexity
Can we keep the learned model
simple and still make expressive
decisions?
Humans can immediately tell bad outputs
The output does not make sense
Page 15
Information extraction without Prior Knowledge
Lars Ole Andersen . Program analysis and specialization for the
C Programming language. PhD thesis. DIKU ,
University of Copenhagen, May 1994 .
Prediction result of a trained HMM
[AUTHOR]
[TITLE]
[EDITOR]
[BOOKTITLE]
[TECH-REPORT]
[INSTITUTION]
[DATE]
Lars Ole Andersen . Program analysis and
specialization for the
C
Programming language
. PhD thesis .
DIKU , University of Copenhagen , May
1994 .
Violates lots of natural
constraints!
Page 16
Examples of Constraints

Each field must be a consecutive list of words and can appear
at most once in a citation.

State transitions must occur on punctuation marks.

The citation can only start with AUTHOR or EDITOR.

The words pp., pages correspond to PAGE.
Four digits starting with 20xx and 19xx are DATE.
Quotations can appear only in TITLE
Easy to express pieces of “knowledge”
…….



Non Propositional; May use Quantifiers
Page 17
Information Extraction with Constraints


Adding constraints, we get correct results!
 Without changing the model
[AUTHOR]
[TITLE]
[TECH-REPORT]
[INSTITUTION]
[DATE]
Lars Ole Andersen .
Program analysis and specialization for the
C Programming language .
PhD thesis .
DIKU , University of Copenhagen ,
May, 1994 .
Page 18
Problem Setting

Random Variables Y:
y4
y1
C(y1,y4)
y5
y2
C(y2,y3,y6,y7,y8)
y6
y7
y3
y8
observations




Conditional Distributions P (learned by models/classifiers)
Constraints C– any Boolean function
defined over partial assignments (possibly: + weights W )
Goal: Find the “best” assignment
 The assignment that achieves the highest global performance.
This is an Integer Programming Problem
Y*=argmaxY PY (+ WC) subject to constraints C
Page 19
Formal Model
Penalty for violating
the constraint.
Subject to constraints
Weight Vector for
“local” models
A collection of Classifiers;
Log-linear models (HMM,
CRF) or a combination
(Soft) constraints
component
How far y is from
a “legal” assignment
How to solve?
How to train?
This is an Integer Linear Program
How to decompose the global
objective function?
Solving using ILP packages gives an
exact solution.
Search techniques are also possible
Should we incorporate constraints
in the learning process?
Page 20
Features Versus Constraints
Ái :
X £ Y ! R;
Ci : X £ Y ! {0,1};
d: X £ Y ! R;
In principle, constraints and features can encode the same propeties
 In practice, they are very different


Features
Local , short distance properties – to allow tractable inference
 Propositional (grounded):
 E.g. True if:
“the” followed by a Noun occurs in the sentence”


Constraints
Global properties
Indeed, used differently
 Quantified, first order logic expressions
 E.g.True if:
“all yis in the sequence y are assigned different values.”

Page 21
Encoding Prior Knowledge

Consider encoding the knowledge that:


The “Feature” Way





Entities of type A and B cannot occur simultaneously in a sentence
Results in higher order HMM, CRF
May require designing a model tailored to knowledge/constraints
Large number of new features: might require more labeled data
Wastes parameters to learn indirectly knowledge we have.
The Constraints Way



Need more training data
A form of supervision
Keeps the model simple; add expressive constraints directly
A small set of constraints
Allows for decision time incorporation of constraints
Page 22
Constrained Conditional Models – 1st Summary


Everything that has to do with Constraints and Learning models
In both examples, we started with learning models



We then included constraints on the output


As a way to “correct” the output of the model
In both cases this allows us to


Either for components of the problem
 Classifiers for Relations and Entities
Or the whole problem
 Citations
Learn simpler models than we would otherwise
As presented, global constraints did not take part in training

Global constraints were used only at the output.

We will later call this training paradigm L+I
Page 23
This Tutorial: Constrained Conditional Models

Part 1: Introduction to CCMs

Examples:




First summary: why are CCM important
Problem Setting


Features and Constraints; Some hints about training issues
Part 2: Introduction to Integer Linear Programming



NE + Relations
Information extraction – correcting models with CCMS
What is ILP; use of ILP in NLP
Semantic Role Labeling in Details
Part 3: Detailed examples of using CCMs


Coreference Resolution
Sentence Compression
BREAK
Page 24
This Tutorial: Constrained Conditional Models (2nd part)

Part 4: More on Inference –

Other inference algorithms



Using hard and soft constraints
Part 5: Training issues when working with CCMS


Formalism (again)
Choices of training paradigms -- Tradeoffs



Search (SRL); Dynamic Programming (Transliteration); Cutting Planes
Examples in Supervised learning
Examples in Semi-Supervised learning
Part 6: Conclusion

Building CCMs
Features and Constraints; Objective functions; Different Learners.

Mixed models vs. Joint models; where is Knowledge coming from

THE END
Page 25
Formal Model
Penalty for violating
the constraint.
(Soft) constraints
component
Weight Vector for
“local” models
A collection of Classifiers;
Log-linear models (HMM,
CRF) or a combination
How far y is from
a “legal” assignment
How to solve?
How to train?
This is an Integer Linear Program
How to decompose the global objective
function?
Solving using ILP packages gives an
exact solution.
Search techniques are also possible
Should we incorporate constraints in
the learning process?
Page 26
Inference with constraints.


We start with adding constraints to existing models.
It’s a good place to start, because conceptually all you do is
to add constraints to what you were doing before and the
performance improves.
Page 27
Constraints and Integer Linear Programming (ILP)



ILP is powerful (NP-complete)
ILP is popular – inference for many models, such as Viterbi
for CFR, have already been implemented.
Powerful off-the shelf solvers exist. All we need is to write
down the objective function and the constraints, there is no
need to write code.
Page 28
Linear Programming



Key contributors: Leonid Kantorovich, George B. Dantzig,
John von Neumann.
Optimization technique with linear objective function and
linear constraints.
Note that the word “Integer” is absent.
Page 29
Example (Thanks James Clarke)

Telfa Co. produces tables and chairs


Each table makes 8$ profit, each chair makes 5$ profit.
We want to maximize the profit.
Page 30
Example (Thanks James Clarke)

Telfa Co. produces tables and chairs





Each table makes 8$ profit, each chair makes 5$ profit.
A table requires 1 hour of labor and 9 sq. feet of wood.
A chair requires 1 hour of labor and 5 sq. feet of wood.
We have only 6 hours of work and 45sq. Feet of wood.
We want to maximize the profit.
Page 31
Solving LP problems.
Page 32
Solving LP problems.
Page 33
Solving LP problems.
Page 34
Solving LP problems.
Page 35
Solving LP problems
Page 36
Solving LP Problems.
Page 37
Integer Linear Programming- integer solutions.
Page 38
Integer Linear Programming.

In NLP, we are dealing with discrete outputs, therefore we’re
almost always interested in integer solutions.

ILP is NP-complete, but often efficient for large NLP
problems.

In some cases, the solutions to LP are integral (e.g totally
unimodular constraint matrix.).

Next, we show an example for using ILP with constraints. The matrix
is not totally unimodular, but LP gives integral solutions.
Page 39
Back to Example: Recognizing Entities and Relations
other
0.05
other
0.10
other
0.05
per
0.85
per
0.60
per
0.50
loc
0.10
loc
0.30
loc
0.45
Dole ’s wife, Elizabeth , is a native of N.C.
E1
E2
R12
E3
R23
irrelevant
irrelevant
0.05
irrelevant
0.10
spouse_of
spouse_of
0.45
0.45
spouse_of
0.05
born_in
born_in
0.50
born_in
0.85
Page 40
Back to Example: Recognizing Entities and Relations
other
0.05
other
0.10
other
0.05
per
0.85
per
0.60
per
0.50
loc
0.10
loc
0.30
loc
0.45
NLP
with ,ILPkey issues:
Dole ’s wife,
Elizabeth
is a native
of N.C.
E1)1 Write down
E2 the objective function.
E3
2) Write down the constraints as linear
R23
R12
inequalities
irrelevant
irrelevant
0.05
irrelevant
0.10
spouse_of
spouse_of
0.45
0.45
spouse_of
0.05
born_in
born_in
0.50
born_in
0.85
Page 41
Back to Example: Recognizing Entities and Relations
other
0.05
other
0.10
other
0.05
per
0.85
per
0.60
per
0.50
loc
0.10
loc
0.30
loc
0.45
Dole ’s wife, Elizabeth , is a native of N.C.
E1
E2
R12
E3
R23
irrelevant
irrelevant
0.05
irrelevant
0.10
spouse_of
spouse_of
0.45
0.45
spouse_of
0.05
born_in
born_in
0.50
born_in
0.85
Page 42
Back to Example: cost function
other
0.05
other
0.10
other
0.05
per
0.85
per
0.60
per
0.50
loc
0.10
loc
0.30
loc
0.45
Dole
E1
R12
R21
Elizabeth
E2
R23
N.C.
E3
R32
R13
R31
x{E1 = per}, x{E1 = loc}, …,
x{R12 = spouse_of}, x{R12 = born_in}, …, x{R12 = } , …  {0,1}
irrelevant
irrelevant
0.05
irrelevant
0.10
spouse_of
spouse_of
0.45
0.45
spouse_of
0.05
born_in
born_in
0.50
born_in
0.85
Page 43
Back to Example: cost function
other
0.05
other
0.10
other
0.05
per
0.85
per
0.60
per
0.50
loc
0.10
loc
0.30
loc
0.45
Dole
E1
R12
Cost function:
R21
Elizabeth
E2
R23
N.C.
E3
R32
R13
R31
c{E1 = per}· x{E1 = per} + c{E1 = loc}· x{E1 = loc} + … +
c{R12 = spouse_of}· x{R12 = spouse_of} + … + c{R12 = }· x{R12 = } + …
irrelevant
irrelevant
0.05
irrelevant
0.10
spouse_of
spouse_of
0.45
0.45
spouse_of
0.05
born_in
born_in
0.50
born_in
0.85
Page 44
Adding Constraints

Each entity is either a person, organization or location:


(R12 = spouse_of)  (E1 = person)  (E2 = person)




x{E1 = per}+ x{E1 = loc}+ x{E1 = org} + x{E1 = }=1
x{R12 = spouse_of}  x{E1 = per}
x{R12 = spouse_of}  x{E2 = per}
We need more consistency constraints.
Any Boolean constraint can be written with a set of linear
inequalities, and an efficient algorithm exist
[Rizzollo2007].
Page 45
CCM for NE-Relations


We showed how a CCM formulation for the NE-Relation
problem.
In this case, the cost of each variable was learned
independently, using trained classifiers.
c{E1 = per}· x{E1 = per} + c{E1 = loc}· x{E1 = loc} + … +
c{R12 = spouse_of}· x{R12 = spouse_of} + … + c{R12 = }· x{R12 = } + …

Other expressive problems can be formulated as Integer
Linear Programs.

For example, HMM/CRF inference, Viterbi algorithms
Page 46
TSP as an Integer Linear Program
 Dantzig et al. were the first to
suggest a practical solution to the
problem using ILP.
Page 47
Reduction from Traveling Salesman to ILP
1
x12c12
3
Every node has
ONE outgoing edge
x32c32
2
Page 48
Reduction from Traveling Salesman to ILP
1
x12c12
3
x32c32
Every node has
ONE incoming edge
2
Page 49
Reduction from Traveling Salesman to ILP
1
x12c12
3
x32c32
2
The solutions
are binary (int)
Page 50
Reduction from Traveling Salesman to ILP
1
x12c12
3
x32c32
2
No sub graph
contains a cycle
Page 51
Viterbi as an Integer Linear Program
y0
y1
y2
yN
x0
x1
x2
xN
Page 52
Viterbi with ILP
y0
y1
y2
yN
x0
x1
x2
xN
y01
y02
-Log{P(x0|y0=1) P(y0=1)}
S
y0M
Page 53
Viterbi with ILP
y0
y1
y2
yN
x0
x1
x2
xN
y01
y02
S
-Log{P(x0|y0=2)P(y0=2)}
y0M
Page 54
Viterbi with ILP
y0
y1
y2
yN
x0
x1
x2
xN
y01
y02
S
-Log{P(x0|y0=M) P(y0=M)}
y0M
Page 55
Viterbi with ILP
y0
y1
y2
yN
x0
x1
x2
xN
y01
y11
y02
y12
-Log{P(x2|y1=1)P(y1=1|y0=1)}
S
y0M
y1M
Page 56
Viterbi with ILP
y0
y1
y2
yN
x0
x1
x2
xN
y01
y11
y02
y12
-Log{P(x2|y1=2)P(y1=1|y0=1)}
S
y0M
y1M
Page 57
Viterbi with ILP
S
y0
y1
y2
yN
x0
x1
x2
xN
y01
y11
y21
yN1
y02
y12
y22
yN2
y0M
y1M
y2M
yNM
T
Page 58
Viterbi with ILP
y0
y1
y2
yN
x0
x1
x2
xN
Viterbi = shortest path (ST ) with dynamic programming.
We saw TSP  ILP.
Now show: Shortest Path  ILP.
S
y01
y11
y21
yN1
y02
y12
y22
yN2
y0M
y1M
y2M
yNM
T
Page 59
Shortest path with ILP.

For each edge (u,v),


xuv =1 if (u,v) is on the shortest path, 0 otherwise.
cuv = the weight of the edge.
Page 60
Shortest path with ILP.

For each edge (u,v),


xuv =1 if (u,v) is on the shortest path, 0 otherwise.
cuv = the weight of the edge.
Page 61
Shortest path with ILP.

For each edge (u,v),


xuv =1 if (u,v) is on the shortest path, 0 otherwise.
cuv = the weight of the edge.
All nodes except the S and the T have eq. number of ingoing and outgoing edges
Page 62
Shortest path with ILP.

For each edge (u,v),


xuv =1 if (u,v) is on the shortest path, 0 otherwise.
cuv = the weight of the edge.
All nodes except the S and the T have eq. number of ingoing and outgoing edges
The source node has one outgoing edge more than ingoing edges.
Page 63
Shortest path with ILP.

For each edge (u,v),


xuv =1 if (u,v) is on the shortest path, 0 otherwise.
cuv = the weight of the edge.
All nodes except the S and the T have eq. number of ingoing and outgoing edges
The source node has one outgoing edge more than ingoing edges.
The target node has one ingoing edge more than outgoing edges.
Page 64
Shortest path with ILP.
 For each edge (u,v),
Interestingly, in this formulation, the solution to the general LP problem will have
xuv =1 if Thus,
(u,v) we
is on
the
shortest path,
0 otherwise.
integersolutions.
have
a reasonably
fast alternative
to Viterbi with ILP.

cuv = the weight of the edge.
Page 65
Who cares that Viterbi is an ILP?


Assume you want to learn an HMM/CRF model (e.g.,
Extracting fields from citations (IE))
But, you also want to add expressive constraints:




Do:




No field can appear twice in the data.
The fields <journal> and <tech-report> are mutually exclusive.
The field <author> must appear at least once.
Learn HMM/CRF
Convert to an LP
Modify the LP canonical constraint matrix
A concrete example for adding constraints over CRF will be
shown
This Tutorial: Constrained Conditional Models

Part 1: Introduction to CCMs

Examples:




First summary: why are CCM important
Problem Setting


Features and Constraints; Some hints about training issues
Part 2: Introduction to Integer Linear Programming


NE + Relations
Information extraction – correcting models with CCMS
What is ILP; use of ILP in NLP
Part 3: Detailed examples of using CCMs



Semantic Role Labeling in Details
Coreference Resolution
Sentence Compression
BREAK
Page 67
This Tutorial: Constrained Conditional Models (2nd part)

Part 4: More on Inference –

Other inference algorithms



Using hard and soft constraints
Part 5: Training issues when working with CCMS


Formalism (again)
Choices of training paradigms -- Tradeoffs



Search (SRL); Dynamic Programming (Transliteration); Cutting Planes
Examples in Supervised learning
Examples in Semi-Supervised learning
Part 6: Conclusion

Building CCMs
Features and Constraints; Objective functions; Different Learners.

Mixed models vs. Joint models; where is Knowledge coming from

THE END
Page 68
CCM Examples

Many works in NLP make use of constrained conditional
models, implicitly or explicitly.
Next we describe in details three examples.

Example 2: Semantic Role Labeling.



Example 3 (Co-ref):


The use of inference with constraints to improve semantic parsing
combining classifiers through objective function outperforms
pipeline.
Example 4 (Sentence Compression):

Simple language model with constraints outperforms complex
models.
Page 69
Example 2: Semantic Role Labeling
Who did what to whom, when, where, why,…
Demo:http://L2R.cs.uiuc.edu/~cogcomp
Approach :
1) Reveals several relations.
2) Produces a very good
semantic parser. F1~90%
3) Easy and fast: ~7 Sent/Sec
(using Xpress-MP)
Top ranked system in CoNLL’05
shared task
Key difference is the Inference
Simple sentence:
I left my pearls to my daughter in my will .
[I]A0 left [my pearls]A1 [to my daughter]A2 [in my will]AM-LOC .




A0
A1
A2
AM-LOC
Leaver
Things left
Benefactor
Location
I left my pearls to my daughter in my will .
SRL Dataset

PropBank [Palmer et. al. 05]

Core arguments: A0-A5 and AA



13 types of adjuncts labeled as AM-arg


different semantics for each verb
specified in the PropBank Frame files
where arg specifies the adjunct type
In this problem, all the information is given, but
conceptually, we could train different components on
different resources.
Page 72
Identify Vocabulary
Algorithmic Approach

I left my nice pearls to her
candidate arguments
Identify argument candidates


I left my nice pearls to her
[ [
[
[
[
]
] ]
]
]
Pruning [Xue&Palmer, EMNLP’04]
Argument Identifier
Binary classification (SNoW)
Classify argument candidates



Argument Classifier
Multi-class classification (SNoW)
Inference





Use the estimated probability distribution
given by the argument classifier
Use structural and linguistic constraints
Infer the optimal global output
Inference over (old and
I new)
left my Vocabulary
nice pearls to her
[ [
[
[
[
]I left
] my
] nice pearls] to her
]
I left my nice pearls to her
Learning

Both argument identifier and argument classifier are trained
using SNoW.



Sparse network of linear functions
Multiclass classifier that produces a probability distribution over
output values.
Features (some examples)


Voice, phrase type, head word, parse tree path from predicate, chunk
sequence, syntactic frame, …
Conjunction of features
Page 74
Inference

The output of the argument classifier often violates some
constraints, especially when the sentence is long.

Finding the best legitimate output is formalized as an
optimization problem and solved via Integer Linear
Programming (ILP).

Input:



The scores from the argument classifier
Structural and linguistic constraints
ILP allows incorporating expressive (non-sequential)
constraints on the variables (the arguments types).
Page 75
Semantic Role Labeling (SRL)
I left my pearls to my daughter in my will .
0.5
0.05
0.15
0.1
0.15
0.15
0.1
0.6
0.1
0.05
0.05
0.05
0.05
0.05
0.7
0.2
0.6
0.05
0.05
0.3
0.15
0.2
0.2
0.1
0.2
Page 76
Semantic Role Labeling (SRL)
I left my pearls to my daughter in my will .
0.5
0.05
0.15
0.1
0.15
0.15
0.1
0.6
0.1
0.05
0.05
0.05
0.05
0.05
0.7
0.2
0.6
0.05
0.05
0.3
0.15
0.2
0.2
0.1
0.2
Page 77
Semantic Role Labeling (SRL)
I left my pearls to my daughter in my will .
0.5
0.05
0.15
0.1
0.15
0.15
0.1
0.6
0.1
0.05
0.05
0.05
One inference
problem for each
verb predicate.
0.05
0.05
0.7
0.2
0.6
0.05
0.05
0.3
0.15
0.2
0.2
0.1
0.2
Page 78
Integer Linear Programming Inference

For each argument ai


Goal is to maximize



Set up a Boolean variable: ai,t indicating whether ai is classified as t
 i score(ai = t ) ai,t
Subject to the (linear) constraints
If score(ai = t ) = P(ai = t ), the objective is to find the
assignment that maximizes the expected number of
arguments that are correct and satisfies the constraints.
Page 79
Constraints

No duplicate argument classes
a  POTARG x{a = A0}  1

Any Boolean rule can be encoded as
a set of linear inequalities.
R-ARG
If there is an R-ARG phrase, there is an ARG
Phrase
 a2  POTARG , a  POTARG x{a = A0}  x{a2 = R-A0}

C-ARG

a2  POTARG ,
If there is an C-ARG phrase, there is an ARG before it
 (a  POTARG)  (a is before a2 ) x{a = A0}  x{a2 = C-A0}
Universally quantified
rulesto encode
LBJ: allows a developer
constraints in FOL; these are
compiled into linear inequalities
automatically.
Many other possible constraints:
 Unique labels
 No overlapping or embedding
 Relations between number of arguments; order constraints
 If verb is of type A, no argument of type B
Joint inference can be used also to combine different SRL Systems.
Page 80
Summary: Semantic Role Labeling
Who did what to whom, when, where, why,…
Demo:
http://L2R.cs.uiuc.edu/~cogcomp
Page 81
Example 3: co-ref (thanks Pascal Denis)
Page 82
Example 1: co-ref (thanks Pascal Denis)
Traditional approach: train a classifier which predicts
the probability that two named entities co-refer:
Pco-ref(Clinton, NPR) = 0.2 (order sensitive!)
Pco-ref(Lewinsky, her) = 0.6
Decision Process: E.g., If Pco-ref(s1, s2) > 0.5, label two entities as co-referent
or, clustering
Page 83
Example 1: co-ref (thanks Pascal Denis)
Evaluation:
Cluster all the entities based on co-ref classifier predictions.
The evaluation is based on cluster overlap.
Traditional approach: train a classifier which predicts
the probability that two named entities co-refer:
Pco-ref(Clinton, NPR) = 0.2
Pco-ref(Lewinsky, her) = 0.6
Decision Process: E.g., If Pco-ref(s1, s2) > 0.5, label two entities as co-referent
or, clustering
Page 84
Example 1: co-ref (thanks Pascal Denis)
NPR
Two types of entities:
“Base entities”
“Anaphors” (pointers)
Page 85
Example 1: co-ref (thanks Pascal Denis)
1)
2)
Error analysis:
“Base entities” that “point” to anaphors.
Anaphors that don’t “point” to anything.
Page 86
Proposed solution.
NPR
Pipelined approach:
Identify anaphors.
Forbid links of certain types.
Page 87
Proposed solution.
NPR
Doesn’t work, despite 80% accuracy of the
anaphorisity classifier.
The reason: error propagation in the pipeline.
Page 88
Joint Solution with CCMs
Page 89
Joint solution using CCMs
Page 90
Joint solution using CCMs
This performs considerably better than the pipelined
approach, and actually improves over the baseline
without anaphorisity classifier (though marginally)
Page 91
Joint solution using CCMs
α
β
Note:
If we have reasons to believe one of the classifiers significantly more
than the other, we can scale with tuned parameters
Page 92
Example 4 - Sentence Compression
(thanks James Clarke).
Page 93
Example
Example:
0
1
2
3
4
5
6
7
8
Big
fish
eat
small
fish
in
a
small
pond
Big
fish
in
a
pond
 0  1   5   6   8  1
 015   156   568  1
Page 94
Language model-based compression
Page 95
Example 2 : summarization (thanks James Clarke)
This formulation requires some additional constraints
Big fish eat small fish in a small pond
No selection of decision variables can make these trigrams
appear consecutively in output.
We skip these constraints here.
Page 96
Trigram model in action
Page 97
Modifier Constraints
Page 98
Example
Page 99
Example
Page 100
Sentential Constraints
Page 101
Example
Page 102
Example
Page 103
More constraints
Page 104
Other CCM Examples: Opinion Recognition

Y. Choi, E. Breck, and C. Cardie. Joint Extraction of Entities
and Relations for Opinion Recognition EMNLP-2006

Semantic parsing variation:



Agent=entity
Relation=opinion
Constraints:



A an agent can have at most two opinions.
An opinion should be linked to only one agent.
The usual non-overlap constraints.
Page 105
Other CCM Examples: Temporal Ordering

N. Chambers and D. Jurafsky. Jointly Combining Implicit
Constraints Improves Temporal Ordering. EMNLP-2008.
Page 106
Other CCM Examples: Temporal Ordering

N. Chambers and D. Jurafsky. Jointly Combining Implicit
Constraints Improves Temporal Ordering. EMNLP-2008.
Three types of edges:
1) Annotation relations before/after
2) Transitive closure constraints
3) Time normalization constraints
Page 107
Related Work: Language generation.

Regina Barzilay and Mirella Lapata. Aggregation via Set
Partitioning for Natural Language Generation.HLTNAACL-2006.

Constraints:


Transitivity: if (ei,ej)were aggregated, and (ei,ejk) were too, then
(ei,ek) get aggregated.
Max number of facts aggregated, max sentence length.
Page 108
MT & Alignment


Ulrich Germann, Mike Jahr, Kevin Knight, Daniel Marcu,
and Kenji Yamada. Fast decoding and optimal decoding for
machine translation. ACL 2001.
John DeNero and Dan Klein. The Complexity of Phrase
Alignment Problems. ACL-HLT-2008.
Page 109
Summary of Examples

We have shown several different NLP solution that make
use of CCMs.

Examples vary in the way models are learned.

In all cases, constraints can be expressed in a high level
language, and then transformed into linear inequalities.

Learning based Java (LBJ) [Rizollo&Roth’07] describe an
automatic way to compile high level description of
constraint into linear inequalities.
Page 110
Solvers
 All applications presented so far used ILP for inference.
 People used different solvers
 Xpress-MP
 GLPK
 lpsolve
R
 Mosek
 CPLEX
 Next we discuss other inference approaches to CCMs
Page 111
This Tutorial: Constrained Conditional Models (2nd part)

Part 4: More on Inference –

Other inference algorithms



Using hard and soft constraints
Part 5: Training issues when working with CCMS


Formalism (again)
Choices of training paradigms -- Tradeoffs



Search (SRL); Dynamic Programming (Transliteration); Cutting Planes
Examples in Supervised learning
Examples in Semi-Supervised learning
Part 6: Conclusion

Building CCMs
Features and Constraints; Objective functions; Different Learners.

Mixed models vs. Joint models; where is Knowledge coming from

THE END
112
Where Are We ?

We hope we have already convinced you that
 Using constraints is a good idea for addressing NLP problems


We were talking about using expressive constraints




Constrained conditional models provide a good platform
To improve existing models
Learning + Inference
The problem: inference
A powerful inference tool: Integer Linear Programming


SRL, co-ref, summarization, entity-and-relation…
Easy to inject domain knowledge
113
Constrained Conditional Model : Inference
Constraint violation penalty
(Soft) constraints
component
Weight Vector for
“local” models
A collection of Classifiers;
Log-linear models (HMM,
CRF) or a combination
How far y is from
a “legal” assignment
How to solve?
How to train?
This is an Integer Linear Program
How to decompose the global
objective function?
Solving using ILP packages gives an
exact solution.
Search techniques are also possible
Should we incorporate constraints
in the learning process?
114
Advantages of ILP Solvers: Review

ILP is Expressive: We can solve many inference problems


ILP is Easy to Use: Many available packages




Converting inference problems into ILP is easy
(Open Source Packages): LPSolve, GLPK, …
(Commercial Packages): XPressMP, Cplex
No need to write optimization code!
Why should we consider other inference options?
115
ILP: Speed Can Be an Issue

Inference problems in NLP

Sometimes large problems are actually easy for ILP




When ILP isn’t fast enough, and one needs to
resort to approximate solutions.
The Problem: General Solvers vs. Specific Solvers



E.g. Entities-Relations
Many of them are not “difficult”
ILP is a very, very general solver
But, sometimes the structure of the problem allows for simpler
inference algorithms.
Next we give examples for both cases.
116
Example 1: Search based Inference for SRL

The objective function
max cij  xij
i, j
Classification confidence

Indicator variable
assigns the j-th class for the i-th token
Constraints





Maximize summation of the scores
subject to linguistic constraints
Unique labels
No overlapping or embedding
If verb is of type A, no argument of type B
…
Intuition: check constraints’ violations on partial
assignments
117
Inference using Beam Search
Shape: argument
Color: label
Beam size = 2,
Constraint:
Only one Red
Rank them
according to
classification
confidence!
Rank them according to
classification confidence!

For each step, discard partial assignments that violate
constraints!
118
Heuristic Inference

Problems of heuristic inference


Problem 1: Possibly, sub-optimal solution
Problem 2: May not find a feasible solution


Drop some constraints, solve it again
Using search on SRL gives comparable results to using ILP, but is
much faster.
119
Example 2: Exploiting Structure in Inference: Transliteration



How to get a score for the pair?
Previous approaches:
 Extract features for each source and target entity pair
The CCM approach:
 Introduce an internal structure (characters)
 Constrain character mappings to “make sense”.
120
Transliteration Discovery with CCM
Assume the weights
are given.
More on this latter.
Score
= sum
of the
Score
= sum
ofmappings’
the
weight
mappings’ weight
s. t. mapping satisfies constraints
• Natural constraints
• Pronunciation constraints
• One-to-One
• Non-crossing
•…

A weight is assigned to each edge.
Include it or not? A binary decision.
The problem now: inference

How to find the best mapping that satisfies the constraints?
121
Finding The Best Character Mappings

An Integer Linear
Programming Problem
c x
max
iS , jT
ij ij
Maximize the mapping score
0  xij  1, xij  Z
Pronounciation constraint
(i, j )  B, xij  0,
One-to-one constraint
i,  xij  1,
j
Non-crossing constraint

Is this the best inference
algorithm?
i, j , k , m, i  k , m  j ,
xij  xkm  1
...
122
Finding The Best Character Mappings

A Dynamic Programming
Algorithm
We can
decompose
the inference
problem into
two parts
Maximize the mapping score
Restricted mapping constraints
One-to-one constraint
Non-crossing constraint

Exact and fast!
Take Home Message:
Although ILP can solve
most problems, the
fastest inference
algorithm depends on
the constraints and can
be simpler
123
Other Inference Options

Constraint Relaxation Strategies

Try Linear Programming


Cutting plane algorithms  do not use all constraints at first



[Roth and Yih, ICML 2005]
Dependency Parsing: Exponential number of constraints
[Riedel and Clarke, EMNLP 2006]
Other search algorithms


A-star, Hill Climbing…
Gibbs Sampling Inference [Finkel et. al, ACL 2005]
 Named Entity Recognition: enforce long distance constraints
 Can be considered as : Learning + Inference
 One type of constraints only
124
Inference Methods – Summary

Why ILP? A powerful way to formalize the problems


Heuristic inference algorithms are useful sometimes!



However, not necessarily the best algorithmic solution
Beam search
Other approaches: annealing …
Sometimes, a specific inference algorithm can be designed

According to your constraints
125
This Tutorial: Constrained Conditional Models (2nd part)

Part 4: More on Inference –

Other inference algorithms



Using hard and soft constraints
Part 5: Training issues when working with CCMS


Formalism (again)
Choices of training paradigms -- Tradeoffs



Search (SRL); Dynamic Programming (Transliteration); Cutting Planes
Examples in Supervised learning
Examples in Semi-Supervised learning
Part 6: Conclusion

Building CCMs
Features and Constraints; Objective functions; Different Learners.

Mixed models vs. Joint models; where is Knowledge coming from

THE END
126
Constrained Conditional Model: Training
Constraint violation penalty
(Soft) constraints
component
Weight Vector for
“local” models
A collection of Classifiers;
Log-linear models (HMM,
CRF) or a combination
How far y is from
a “legal” assignment
How to solve?
How to train?
This is an Integer Linear Program
How to decompose the global
objective function?
Solving using ILP packages gives an
exact solution.
Search techniques are also possible
Should we incorporate constraints
in the learning process?
127
Where are we?

Algorithmic Approach: Incorporating general constraints



Showed that CCMs allow for formalizing many problems
Showed several algorithmic ways to incorporate global constraints in
the decision.
Training: Coupling vs. Decoupling Training and Inference.




Incorporating global constraints is important but
Should it be done only at evaluation time or also at training time?
How to decompose the objective function and train in parts?
Issues related to:


Modularity, efficiency and performance, availability of training data
Problem specific considerations
128
Training in the presence of Constraints
Decompose Model (SRL case)

Decompose Model from constraints
General Training Paradigm:



First Term: Learning from data (could be further decomposed)
Second Term: Guiding the model by constraints
Can choose if constraints’ weights trained, when and how, or taken
into account only in evaluation.
129
Comparing Training Methods

Option 1: Learning + Inference (with Constraints)


Ignore constraints during training
Option 2: Inference (with Constraints) Based Training

Consider constraints during training

In both cases: Decision Making with Constraints

Question: Isn’t Option 2 always better?

Not so simple…

Next, the “Local model story”
130
Training Methods
Learning + Inference (L+I)
Learn models independently
y1
y3
f1 ( x )
Inference Based Training (IBT)
Learn all models together!
Intuition
Learning with
constraints may make
learning more difficult
y2
y4
f2(x)
f3(x)
x3
x1
x4
y5
Y
f4(x)
f5(x)
x5
x2
x6
x7 X
131
Training with Constraints
Example: Perceptron-based Global Learning
True Global Labeling
Local Predictions
Apply
Constraints:
Y
-1
1
-1
-1
1
Y’
-1
1
1
-1
1
1
f1 ( x )
X
f2(x)
x3
x1
f3(x)
x4
x5
x2
x6
x7
f 4( x )
Y
f5(x)
Which one is better?
When and Why?
132
Claims [Punyakanok et. al , IJCAI 2005]

Theory applies to the case of local model (no Y in the features)

When the local modes are “easy” to learn, L+I outperforms IBT.


In many applications, the components are identifiable and easy to learn
(e.g., argument, open-close, PER).
Only when the local problems become difficult to solve in isolation, IBT
outperforms L+I, but needs a larger number of training examples.
L+I: cheaper computationally; modular
IBT is better in the limit, and other extreme cases.


Other training paradigms are possible
Pipeline-like Sequential Models: [Roth, Small, Titov: AI&Stat’09]


Identify a preferred ordering among components
Learn k-th model jointly with previously learned models
133
Bound Prediction
L+I vs. IBT: the more identifiable
individual problems are, the better
overall performance is with L+I

Local
 ≤ opt + ( ( d log m + log 1/ ) / m )1/2

Global
 ≤ 0 + ( ( cd log m + c2d + log 1/ ) / m )1/2
Indication for
hardness of
problem
Bounds
Simulated Data
opt
=0
opt=0.1
opt=0.2
134
Relative Merits: SRL
In some cases problems are hard due
to lack of training data.
Semi-supervised learning
L+I is better.
When the problem
is artificially made
harder, the tradeoff
is clearer.
hard
Difficulty of the learning problem
(# features)
easy
135
L+I & IBT: General View – Structured Perceptron

The theory applies when F(x,y) = F(x)
For each iteration
For each (X, YGOLD ) in the training data
YPRED=
If YPRED != YGOLD
λ = λ + F(X, YGOLD ) - F(X, YPRED)
endif
The difference between
endfor
L+I and IBT
136
Comparing Training Methods (Cont.)
Decompose Model (SRL case)

Local Models (train independently) vs. Structured Models



Decompose Model from constraints
In many cases, structured models might be better due to expressivity
But, what if we use constraints?
Local Models+Constraints vs. Structured Models +Constraints



Hard to tell: Constraints are expressive
For tractability reasons, structured models have less expressivity than
the use of constraints.
Local can be better, because local models are easier to learn
137
Example: Semantic Role Labeling Revisited
s
A
A
A
A
A
B
B
B
B
B
C
C
C
C
C
Sequential Models
t
Local Models
Conditional Random Field
Logistic Regression
Global perceptron
Local Avg. Perceptron
Training: sentence based
Training: token based.
Testing: find the shortest path
Testing: find the best assignment locally
with constraints
with constraints
138
Which Model is Better? Semantic Role Labeling

Experiments on SRL: [Roth and Yih, ICML 2005]

Story: Inject constraints into conditional random field mdoels
Sequential Models
L+I
IBT
Model
Baseline
CRF
66.46
CRF-D
CRF-IBT
69.14
Local
L+I
Avg. P
58.15
+ Constraints
71.94
73.91
69.82
74.49
Training
38 than
145Models
Local
Models48
are noware
better
Sequential
Models!
Sequential
Models
better
than
Local
! 0.8
Time
(With
constraints)
(No constraints)
139
Summary: Training Methods

Many choices for training a CCM
 Learning + Inference (Training without constraints)
 Inference based Learning (Training with constraints)
 Based

on this, what kind of models you want to use?
Advantages of L+I
Require fewer training examples
 More efficient; most of the time, better performance
 Modularity; easier to incorporate already learned models.

140
Constrained Conditional Model: Soft Constraints
Constraint violation penalty
Subject to constraints
(1) Why use soft constraint?
(Soft) constraints
component
How far y is from
a “legal” assignment
(2) How to model “degree of violations”
(3) How to solve?
(4) How to train?
This is an Integer Linear Program
How to decompose the global
objective function?
Solving using ILP packages gives an
exact solution.
Search techniques are also possible
Should we incorporate constraints
in the learning process?
141
(1) Why Are Soft Constraints Important?

Some constraints may be violated by the data.

Even when the gold data violates no constraints, the model
may prefer illegal solutions.



We want a way to rank solutions based on the level of constraints’
violation.
Important when beam search is used
Working with soft constraints


Need to define the degree of violation
Need to assign penalties for constraints
142
Example: Information extraction
Lars Ole Andersen . Program analysis and specialization for the
C Programming language. PhD thesis. DIKU ,
University of Copenhagen, May 1994 .
Prediction result of a trained HMM
[AUTHOR]
[TITLE]
[EDITOR]
[BOOKTITLE]
[TECH-REPORT]
[INSTITUTION]
[DATE]
Lars Ole Andersen . Program analysis and
specialization for the
C
Programming language
. PhD thesis .
DIKU , University of Copenhagen , May
1994 .
Violates lots of natural
constraints!
143
Examples of Constraints

Each field must be a consecutive list of words and can appear
at most once in a citation.

State transitions must occur on punctuation marks.

The citation can only start with AUTHOR or EDITOR.

The words pp., pages correspond to PAGE.
Four digits starting with 20xx and 19xx are DATE.
Quotations can appear only in TITLE
…….



144
(2) Modeling Constraints’ Degree of Violations
c ( y N ) 
1 - if assigning yN to xN violates the constraint C
with respect to assignment (x1,..,xN-1;y1,…,yN-1)
0 - otherwise
•Count how many times it violated the constraints:
d(y,1c(X))= ∑Φc(yi)
State transition must
occur on punctuations.
i, yi 1  yi  xi 1 is a punctuation
Lars
Lars
AUTH
AUTH
Ole
Ole
BOOK
AUTH
Andersen
Andersen
EDITOR
EDITOR
.
.
EDITOR
EDITOR
Φ
Φcc(y
(y11)=0
)=0
Φ
Φcc(y
(y22)=1
)=0
Φ
Φcc(y
(y33)=1
)=1
Φ
Φcc(y
(y44)=0
)=0
∑Φc(yi) =2
=1
145
Example: Degree of Violation?

It might be the case that all “good” assignments violate
some constraints


We lost the ability to judge which assignment is better
Better options: Choose the one with fewer violations!
Lars
Ole
Andersen
.
AUTH
AUTH
EDITOR
EDITOR
Φc(y1)=0
Φc(y2)=0
Φc(y3)=1
Φc(y4)=0
Lars
Ole
Andersen
.
AUTH
BOOK
EDITOR
EDITOR
Φc(y1)=0
Φc(y2)=1
Φc(y3)=1
Φc(y4)=0
The first one
is better
because of
d(y,1c(X))!
146
Hard Constraints vs. Weighted Constraints
Constraints are close to
perfect
Labeled data might not
follow the constraints
147
(3) Constrained Conditional Model with Soft Constraints
Constraint violation penalty
(Soft) constraints
component
How far y is from
a “legal” assignment
How to solve?
How to train?
This is an Integer Linear Program
How to decompose the global
objective function?
Solving using ILP packages gives an
exact solution.
Search techniques are also possible
Should we incorporate constraints
in the learning process?
148
Inference with Beam Search (& Soft Constraints)
Run beam search inference as
before. Rather than eliminating
illegal assignments, re-rank them.
Au
L. Shari et.
C.
al.
Fellbaum,
ILP is another option!
1998. Building semantic
ed.
WordNet:
and
Concordances. In
its
applications.
149
Assume that We Have Made a Mistake…
Assumption: the best assignment
according to the objective function
Start from here
Au Au
L. Shari
C.
Au Au Date Title
Title
et. al. 1998. Building semantic
Fellbaum,
ed.
WordNet:
and
Journal
Concordances. In
its
applications.
150
Top Choice from Learned Models Is Not Good
: ∑Φc1(yi) =1 ; ∑Φc2(yi) =1+1
Each field must be a
consecutive list of words
and can appear at most
once in a citation.
Au Au
Au Au Date
L.
et.
Shari
Au
C.
al.
Fellbaum,
State transitions must
occur on punctuation
marks.
Two violations.
Title
Title
1998. Building semantic
ed.
WordNet:
and
Journal
Book
Concordances.
In
its
applications.
151
The Second Choice from Learned Models Is Not Good
∑Φc2(yi) =1+1
State transitions must
occur on punctuation
marks.
Two violations.
Au Au
Au Au Date
L.
et.
Shari
al.
Title
Title
1998. Building semantic
Journal
Book
Concordances.
In
Editor
C.
Fellbaum,
ed.
WordNet:
and
its
applications.
152
Use Constraints to Find the Best Assignment
∑Φc2(yi) =1
State transitions must
occur on punctuation
marks.
one violations.
Au Au
L. Shari
Au Au Date Title
Title
et. al. 1998. Building semantic
Editor
C.
Fellbaum,
ed.
WordNet:
and
Journal
Concordances.
its
Editor
In
applications.
153
Soft Constraints Help Us Find a Better Assignment
We can do this
because we use soft
constraints. If we use hard
constraints, the score of
all assignment given the
prefix is negative infinity.
Au Au
Au Au Date
L.
et.
Shari
Editor
C.
al.
Editor
Fellbaum,
Title
Title
Journal
Editor
1998. Building semantic
Concordances.
In
Editor
ed.
Book
its
Book
WordNet:
Book
and
Book
applications.
154
(4) Constrained Conditional Model with Soft Constraints
Constraint violation penalty
(Soft) constraints
component
How far y is from
a “legal” assignment
How to solve?
How to train?
This is an Integer Linear Program
How to decompose the global
objective function?
Solving using ILP packages gives an
exact solution.
Search techniques are also possible
Should we incorporate constraints
in the learning process?
155
Compare Training Methods (with Soft Constraints)

Need to figure out the penalty as well…

Option 1: Learning + Inference (with Constraints)
 Learn the weights and penalties separately

Option 2: Inference (with Constraints) Based Training
 Learn the weights and penalties together

The tradeoff between L+I and IBT is similar to what
we have seen earlier.
156
Inference Based Training With Soft Constraints
• Example: Perceptron
• Update penalties as well !
For each iteration
For each (X, YGOLD ) in the training data
YPRED=
If YPRED != YGOLD
λ = λ + F(X, YGOLD ) - F(X, YPRED)
ρI = ρI+ d(YGOLD,1C(X)) - d(YPRED,1C(X)), I = 1,..
endif
endfor
157
L+I v.s IBT for Soft Constraints

Test on citation recognition:




L+I: HMM + weighted constraints
IBT: Perceptron + weighted constraints
Same feature set
With constraints
Without
constraints


Factored
Model
is better
Few
labeled
examples,
HMM > perceptron
More significant
with a small
# of examples
Many
labeled examples,
perceptron
> HMM
Agrees with earlier results
in the supervised setting
ICML’05, IJCAI’5
158
Summary – Soft Constraints

Using soft constraints is important sometimes



Some constraints might be violated in the gold data
We want to have the notion of degree of violation
Degree of violation

One approximation: count how many time the constraints was
violated

How to solve? Beam search, ILP, …

How to learn?

L+I v.s. IBT
159
Constrained Conditional Model: Injecting Knowledge
Constraint violation penalty
(Soft) constraints
component
Weight Vector for
“local” models
A collection of Classifiers;
Log-linear models (HMM,
CRF) or a combination
How far y is from
a “legal” assignment
via to
Constraints
How to solve?Inject Prior KnowledgeHow
train?
This is an Integer Linear Program
How to decompose the global
objective function?
Solving using ILP packages gives an
A.K.A.Should we incorporate constraints
exact solution.
in the learning
process?
Semi/unsupervised
with Constrained
Conditional
Model
Search techniques areLearning
also possible
160
Constraints As a Way To Encode Prior Knowledge

Consider encoding the knowledge that:


The “Feature” Way


Entities of type A and B cannot occur simultaneously in a sentence
Requires larger models
The Constraints Way



Need more training data
A effective way to inject
knowledge
Keeps the model simple; add expressive constraints directly
A small set of constraints
Allows for decision time incorporation of constraints
We can use constraints as a way to replace training data
161
Constraint Driven Semi/Un Supervised Learning
CODL
Use constraints
to generate better
training
In traditional
semi/unsupervised
Learning,
samples
in semi/unsupervised
models
can drift
away from correctleaning.
model
Resource 
Seed Examples 
Better
ModelModel
BetterFeedback
Feedback
Prediction + Constraints
Learn from
labeled
Learn
from data
labeled data
Label unlabeled data
Unlabeled Data
162
162
Semi-Supervised Learning with Soft Constraints (L+I)

Learning model weights
 Example:

HMM
Constraints Penalties
 Hard
Constraints : infinity
 Weighted Constraints:

½i = -log P{Constraint Ci is violated in training data}
Learn the weights and
penalty separately

Only 10 constraints
163
Semi-supervised Learning with Constraints
Supervised
learning algorithm
parameterized by 
[Chang, Ratinov, Roth, ACL’07]
Inference based
augmentation of the
training set (feedback)
(inference with constraints).
=learn(T)
For N iterations do
T=
For each x in unlabeled dataset
{y1,…,yK} InferenceWithConstraints(x,C, )
T=T  {(x, yi)}i=1…k
=  +(1- )learn(T)
Learn from new training data.
Weigh supervised and
unsupervised model.
164
Value of Constraints in Semi-Supervised Learning
Objective function:
Learning w/o Constraints: 300 examples.
Constraints are used to
Bootstrap a semisupervised learner
Poor model + constraints
used to annotate
unlabeled data, which in
turn is used to keep
training the model.
Learning w 10 Constraints
# of available labeled examples
300
165
Unsupervised Constraint Driven Learning

Semi-supervised Constraint Driven Learning
 In [Chang et.al. ACL 2007], they


Reason: We need a good starting point!
Unsupervised Constraint Driven Learning
 Sometimes,



use a small labeled dataset
good resources exist
We can use the resource to initialize the model
We do not need labeled instances at all!
Application: transliteration discovery
166
Why Adding Constraints?

Before talking about transliteration discovery
 Let’s think about why adding constraints again
 Reason:
We want to capture the dependencies
between different outputs

A new question: What happen if you are trying
to solve a single output classification problem
 Can
you still inject prior knowledge?
167
Why Add Constraints?
Use constraints
to capture the
dependencies
Structure Output Problem:
Dependencies between
different outputs
y1
y2
y3
y4
x3
x1
y5
x4
Y
x5
x2
x6
x7
X
168
Why Add Constraints?
Hard to find
constraints!?
Single Output Problem:
Only one output
y1
x3
x1
Y
x4
x5
x2
x6
Intuition: introduce
structural hidden
variables
x7
X
169
Adding Constraints Through Hidden Variables
Single Output Problem
y1
with hidden variables
f1
Y
f2
f3
f4
f5
x3
x1
x4
x5
x2
x6
Use constraints
to capture the
dependencies
x7
X
170
Example: Transliteration Discovery

Hidden variables
 Character mappings

Character mappings is not our final goal
 But, it provides a intermediate representation that
satisfies natural constraints.
The score for each pair depends on
the character mappings.
171
Transliteration Discovery with Hidden Variables


For each source NE, find the best target candidate
For each NE pair, the score is equal to


the score of the best hidden set of character mapping (edges)
Add constraints when predicting hidden variables
g ( H , N s , Nt )
score( N s , Nt )  arg maxH
| Nt |
g ( H , N s , Nt )   F ( H , N s , Nt )   c d ( H ,1c )
T
A CCM Formulation, use constraints
to bias the prediction!
Alternative View:
Finding the best feature representation
cC
Violation
172
Natural Constraints

A Dynamic Programming
Algorithm
We can
decompose
the problem
into two parts
Maximize the mapping score
Pronouncition constraints
One-to-one constraint
Non-crossing constraint

Exact and fast!
Take Home Message:
Although ILP can solve
most problems, the
fastest inference
algorithm depends on
the constraints and can
be simpler
173
Algorithm: High Level View


Model  Resource (Romanization Table)
Until Convergence
 Use model + Constraints to get assignments for both
hidden variables (F) and labels (Y)
D  D  {, F ( N s , N t* | W )}
D  D  {, F ( N s , N t | W )}, N t  N t*
 Update
the model with newly-labeled F and y
W  train(D)

Get feedback from both hidden variables and labels
174
Results - Russian
Higher,
better
No NER on Russian
Unsupervised Constraint Driven Learning
No labeled data
No temporal Information
No constraints
+ 20 labeled example + temporal
No constraints
+ 20 labeled example
175
Results - Hebrew
Unsupervised Constraint Driven Learning
No labeled data
No temporal Information
No constraints
+ 250 labeled example
176
Summary

Adding constraints is an effective way to inject knowledge


Injecting constraints sometimes are more effective than
annotating data


300 labeled data v.s. 20 labeled data + constraints
On the cases of single output problems


Constraints can correct our predictions on unlabeled data
Use hidden variables to capture the constraints
Other approaches are possible

[Duame, EMNLP 2008]

Only get feedback from the examples where constraints are satisfied
177
This Tutorial: Constrained Conditional Models (2nd part)

Part 4: More on Inference –

Other inference algorithms



Using hard and soft constraints
Part 5: Training issues when working with CCMS


Formalism (again)
Choices of training paradigms -- Tradeoffs



Search (SRL); Dynamic Programming (Transliteration); Cutting Planes
Examples in Supervised learning
Examples in Semi-Supervised learning
Part 6: Conclusion

Building CCMs
Features and Constraints; Objective functions; Different Learners.

Mixed models vs. Joint models; where is Knowledge coming from

THE END
Page178
178
Conclusion

Constrained Conditional Models combine



Learning conditional models with using declarative expressive constraints
Within a constrained optimization framework
Our goal was to:



Introduce a clean way of incorporating constraints to bias and improve
decisions of learned models
A clean way to use (declarative) prior knowledge to guide semisupervised learning
Provide examples for the diverse usage CCMs have already found in NLP

Significant success on several NLP and IE tasks (often, with ILP)
179
Technical Conclusions

Presented and discussed inference algorithms



Present and discussed learning models to be used along with
constraints




How to use constraints to make global decisions
The formulation is an Integer Linear Programming formulation, but
algorithmic solutions can employ a variety of algorithms
Training protocol matters
We distinguished two extreme cases – training with/without constraints.
Issues include performance, as well as modularity of the solution and the
ability to use previously learned models.
We did not attend to the question of “how to find constraints”


In order to emphasize the idea that we think background knowledge is
important, it exists, and to focus on using it.
But, we talked about learning weights for constraints, and it’s clearly
possible to learn constraints.
180
Summary: Constrained Conditional Models
Conditional Markov Random Field
y1
y4
y2
y5
y6
Constraints Network
y3
y7
y1
y8
y4

Linear objective functions
Typically Á(x,y) will be local
functions, or Á(x,y) = Á(x)


y5
y6
y3
y7
y8
 i ½i dC(x,y)
y* = argmaxy  wi Á(x; y)

y2



Expressive constraints over output
variables
Soft, weighted constraints
Specified declaratively as FOL formulae
Clearly, there is a joint probability distribution that represents
this mixed model. Key difference from MLNs which provide a concise
We would like to: definition of a model, but the whole joint one.


Learn a simple model or several simple models
Make decisions with respect to a complex model
181
Designing CCMs
y1
y4
y2
y5
y6
y3
y7
y1
y8
y4

Linear objective functions
Typically Á(x,y) will be local
functions, or Á(x,y) = Á(x)
y5
y6
y3
y7
y8
 i ½i dC(x,y)
y* = argmaxy  wi Á(x; y)

y2



Expressive constraints over output
variables
Soft, weighted constraints
Specified declaratively as FOL formulae
DecideLBJ
what
variables
are of
interest
– learn model (s)
(Learning
Based
Java):
http://L2R.cs.uiuc.edu/~cogcomp
A modeling
for Constrained
Supports
 Think
aboutlanguage
constraints
among theConditional
variables Models.
of interest
programming along with building learned models, high level specification of
 Design your objective function

constraints and inference with constraints
182
Questions?

Thank you!
183