Transcript Duality in QPs - Carnegie Mellon University

+
Quadratic Programming
and Duality
Sivaraman Balakrishnan
+
Outline

Quadratic Programs

General Lagrangian Duality

Lagrangian Duality in QPs
+
Norm approximation

Problem

Interpretation

Geometric – try to find projection of b into ran(A)

Statistical – try to find solution to b = Ax + v


v is a measurement noise (choose norm so that v is small in that
norm)
Several others
+
Examples

-- Least Squares Regression

-- Chebyshev

-- Least Median Regression

More generally can use *any* convex penalty function
+ Picture from BV
+
Least norm

Perfect measurements 

Not enough of them 

Heart of something known as compressed sensing

Related to regularized regression in the noisy case
+
Smooth signal reconstruction

S(x) is a smoothness penalty

Least squares penalty


Smooths out noise and sharp transitions
Total variation (peak to valley intuition)

Smooths out noise but preserves sharp transitions
+
Euclidean Projection

Very fundamental idea in constrained minimization

Efficient algorithms to project onto many many convex sets
(norm balls, special polyhedra etc)

More generally finding minimum distance between
polyhedra is a QP
+
Quadratic
Programming
Duality
+
General recipe

Form Lagrangian

How to figure out signs?
+
Primal & Dual Functions

Primal

Dual
+
Primal & Dual Programs

Primal Programs

Constraints are now implicit in the primal

Dual Program
+
Lagrangian Properties

Can extract primal and dual problem

Dual problem is always concave


Dual problem is always a lower bound on primal


Proof
Proof
Strong duality gives complementary slackness

Proof
+
Some examples of QP duality

Consider the example from class

Lets try to derive dual using Lagrangian
+
General PSD QP

Primal

Dual
+
SVM – Lagrange Dual

Primal SVM

Dual SVM

Recovering Primal Variables and Complementary Slackness