Transcript Document

Mixture Density Networks
Qiang Lou
Outline

Simple Example

Introduction of MDN

Analysis of MDN
Weights Optimization
Prediction
Simple Example
The inverse problem of x=t+0.3*sin(2лt)
Input: x, output: t
Learn form the example
Obviously, we find the problem of the
conventional neural networks.
multi-valued mapping
Reason:
f(x,w*)= E[Y|X], the average of the
correct target values, sometimes is not
correct solution.
Solution:
mixture density networks
MDN: overcome the limits mentioned above
---- using a linear combination of kernel function:
Three parameters:
coefficients:
means:
variances:
How to model the parameters?
---- using the outputs of the conventional NN
a)
Coefficients:
b)
Variances:
c)
Means can be directly represented by output of
NN:
Basic structure of MDN
Weights Optimization
Similar to the conventional NN:
maximum likelihood (minimize the negative
logarithm of the likelihood).
We try to minimize E(w), which is equivalent to
maximize the likelihood.
Weights Optimization
Using chain rule and back propagation:
start off the algorithm:
Prediction

General Way
take the conditional average of the target data:

Accurate Way
take the solution of the most probable components
μk , where k = arg maxk(
)
Results of example
Problems
1)
2)
The number of the outputs of the MDN
Assume: L models in the mixture model
K outputs in conventional NN
Outputs of MDN: (K+2) L
Thank you !