> For the complete documentation index, see [llms.txt](https://ykkim.gitbook.io/wiki/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ykkim.gitbook.io/wiki/machine-learning/lecture-notes/density-estimation.md).

# Parameter, Density Estimation

## Lecture Note

[Read class PPT for detail](https://people.engr.tamu.edu/rgutier/web_courses/csce666_s20/l6.pdf)

[Click here for Lecture PPT](https://people.engr.tamu.edu/rgutier/web_courses/csce666_s20/l6.pdf): TAMU

## Introduction

For Bayesian Classifier, it becomes the problem of Probability Density Function(pdf) Estimation for continuous distribution of Likelihood.

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-4eb01f39252349bc1320521f14961745a0a9fc88%2Fimage.png?alt=media)

How to estimate $$p(\bf{x}|w\_i)$$ with samples of x?

> X 중에 w\_*i* 에 속하는 샘플 집합 Xi를 가지고 p(x|w\_i)를 추정하라

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-4a6785763d7987edae54996fafaa497dac8f0a5f%2Fimage.png?alt=media)

It is modeling the probability density function (pdf) of the unknown probability distribution from which the dataset has been drawn.

The methods are classifed as

### Parametric model

* [MAP, MLE](/wiki/machine-learning/lecture-notes/density-estimation/map-mle.md#reference)
* Gaussian Mixture Model
* multivariate normal distribution (MND)

### Nonparametric model

* Kernel Density Estimation, Histogram, Parzen Window
* k-Nearest neighbor

##

## Kernel Density Estimation Method

It has the parameter of b, the bandwidth

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-fa524a117f1c830453f4be7ef7fe696c8797ccf5%2Fimage.png?alt=media)

* Example: Gaussian kernel (1D) for k

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-e7a292616de5f0271528e5013ba974af10ea90c8%2Fimage-20210729152458677.png?alt=media)

**Find the optimum b:** We look for such a value of b that minimizes the difference between the real shape of f(x) and the shape of our model f\_b(x).

### Example

Let {xi}N i=1 be a one-dimensional dataset (a multi-dimensional case is similar) whose examples were drawn from a distribution with an unknown pdf f with xi 2 R for all i = 1, . . . ,N.

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-55178e9b53d67936a6c4f0fb94dd87fed79b4b1b%2Fimage.png?alt=media)

**How to measure the goodness of estimation?**

A reasonable choice of measure of this difference is called the mean integrated squared error (MISE):

we square the difference between the real pdf f and our model of it f^hat\_b.

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-e7a292616de5f0271528e5013ba974af10ea90c8%2Fimage-20210729152458677.png?alt=media)

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-0703faebc04056b04e8afc02ae33117267611646%2Fimage-20210729152520277.png?alt=media)

Now, to find the optimal value b\* for b, we minimize the cost defined as,

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-d89db2322276efa8fdd38f4c1e6ae5437c05c797%2Fimage-20210729153933940.png?alt=media)

####

#### Limitations

* Memory -based method, that need to store all the samples of training.
* For additional sample, need to re-calculate the whole process.
* Curse-of Dimension exists: Use only for low-dimension problems

## Gaussian Mixture Model

[Read GMM](/wiki/machine-learning/lecture-notes/density-estimation/gaussian-mixture-model.md#gaussian-mixture-model)

## Reference

The Hundred-Page Machine Learning Book <http://themlbook.com/wiki/doku.php>

머신러닝/패턴인식, 오일석

{% embed url="<https://towardsdatascience.com/gaussian-mixture-models-explained-6986aaf5a95>" %}