# Bayesian Classifier

## Review on Probability

[See Terminology Review before reading further](/wiki/machine-learning/lecture-notes/bayesian-classifier/terminology-review.md#probability)

## Concept

Assume we are to classify an object based on the evidence provided by feature vector x, as class w1 or class w2.

![](/files/-Mg5sMvnvt1R-a1b6p7x)

![](/files/-Mg5smQiNr-X1tZctxnD)

## Bayesian Classification

### Decision Rule (binary class problem)

* Let w1: class 1, w2: class 2.

If P(w1|X)> P(w2|X), then X belongs to w1. Else w2

Applying Bayesian rule, it becomes minimum error Bayesian Classifier

### Minimum Error Bayesian Classifier

If p(X|w1)P(w1)> p(X|w2)P(w2), then X belongs to w1. Else w2

> Since X can be either discrete,continuous, use small p for p(X|w1). For discrete lable w, use P(w), P(w|x)

### Likelihood Ratio Test (LRT)

Since p(x) does not affect decision rule, rearrange using the term $$\Delta$$.

![](/files/-Mg5kuPjkNAtoTaoAPP0)

### Probability of Error

For binary classification,

P(e)=P(e|w1)P(w1)+P(e|w2)P(w2)

> If P(w1)=P(w2)=0.5, then P(e)=0.5(e1+e2)

![](/files/-Mg5mjyZaI5_rO3rdatv)

>

#### How good is the LRT decision rule?

Optimal decision rule will minimize P(e|x) at every value of x so the integral is minimized

P(e)=Integral\_INF { P(e|x)p(x)dx}

> For any given problem, the minimum probability error is achieved by LRT decision. The best classifier

### Example: LRT

![](/files/-Mg5nPgkw6zcz1CxUi1Q)

## Bayes Risk

Penalty of misclassifying can have different weight for each class.

* For example, misclassifying a cancer sufferer as a healthy patient is a much more serious problem than the other way around

### Minimum risk Bayesian Classifier

Let C\_ij is the cost of choosing class w\_i when w\_j is the true class.

e.g. C21 is wrong classification as w2 when the true class is w1.

#### Bayes Risk R

Expected value of the cost:

R= E\[C] = { c11 p(x|w1)P(w1)+ c12p(x|w2)P(w2)} + c21 p(x|w1)P(w1)+ c22p(x|w2)P(w2)}

After some rearrangement, it becomes a form of Likelihood Ratio

![](/files/-Mg5nzpQHGyC3BOz0hg8)

![](/files/-Mg5nuDMcHUbLI5Rrvy2)

### Example

![](/files/-Mg5nctd33_ZKsf5YLQq)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://ykkim.gitbook.io/wiki/machine-learning/lecture-notes/bayesian-classifier.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.