> For the complete documentation index, see [llms.txt](https://ykkim.gitbook.io/wiki/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ykkim.gitbook.io/wiki/machine-learning/lecture-notes/bayesian-classifier.md).

# Bayesian Classifier

## Review on Probability

[See Terminology Review before reading further](/wiki/machine-learning/lecture-notes/bayesian-classifier/terminology-review.md#probability)

## Concept

Assume we are to classify an object based on the evidence provided by feature vector x, as class w1 or class w2.

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-2745d65bc4e65b64b252adcc153ddae9c1c5b544%2Fimage.png?alt=media)

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-809a40516c88327f74741d533b9fd7d6a0e46544%2Fimage.png?alt=media)

## Bayesian Classification

### Decision Rule (binary class problem)

* Let w1: class 1, w2: class 2.

If P(w1|X)> P(w2|X), then X belongs to w1. Else w2

Applying Bayesian rule, it becomes minimum error Bayesian Classifier

### Minimum Error Bayesian Classifier

If p(X|w1)P(w1)> p(X|w2)P(w2), then X belongs to w1. Else w2

> Since X can be either discrete,continuous, use small p for p(X|w1). For discrete lable w, use P(w), P(w|x)

### Likelihood Ratio Test (LRT)

Since p(x) does not affect decision rule, rearrange using the term $$\Delta$$.

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-a08b248a7a96dfed196e91224ce4e1e30a73ef44%2Fimage.png?alt=media)

### Probability of Error

For binary classification,

P(e)=P(e|w1)P(w1)+P(e|w2)P(w2)

> If P(w1)=P(w2)=0.5, then P(e)=0.5(e1+e2)

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-7e70ce585b5765e32b7880a3721d4c220fae4856%2Fimage.png?alt=media)

>

#### How good is the LRT decision rule?

Optimal decision rule will minimize P(e|x) at every value of x so the integral is minimized

P(e)=Integral\_INF { P(e|x)p(x)dx}

> For any given problem, the minimum probability error is achieved by LRT decision. The best classifier

### Example: LRT

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-6ab5c57742e2b7e1a5df08eb9b781d6f0e14608b%2Fimage.png?alt=media)

## Bayes Risk

Penalty of misclassifying can have different weight for each class.

* For example, misclassifying a cancer sufferer as a healthy patient is a much more serious problem than the other way around

### Minimum risk Bayesian Classifier

Let C\_ij is the cost of choosing class w\_i when w\_j is the true class.

e.g. C21 is wrong classification as w2 when the true class is w1.

#### Bayes Risk R

Expected value of the cost:

R= E\[C] = { c11 p(x|w1)P(w1)+ c12p(x|w2)P(w2)} + c21 p(x|w1)P(w1)+ c22p(x|w2)P(w2)}

After some rearrangement, it becomes a form of Likelihood Ratio

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-785415d0a4658afdd3192ff638ec1a13ec128a63%2Fimage.png?alt=media)

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-d8c0a1261116a65e4831ed03bf423d13eac01c27%2Fimage.png?alt=media)

### Example

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-7359436df0875885063d05e39315524dec978b49%2Fimage.png?alt=media)