> For the complete documentation index, see [llms.txt](https://ykkim.gitbook.io/wiki/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ykkim.gitbook.io/wiki/machine-learning/probability-and-statistics-for-machine-learning/bayesian-updating-with-continuous-priors.md).

# Bayesian Updating with Continuous Priors

> Ref) [MIT OCW Lecture Note](https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading13a.pdf)

Up to now, we have only done Bayesian updating when we had a finite number of hypothesis, e.g. Have disease(1) or not(0). Now we will study Bayesian updating when there is a **continuous range of hypotheses**.

### Examples with \*\*\*\* continuous ranges of hypotheses

**Ex1)** Suppose you have a system that can succeed or fail with probability p. Then we can hypothesize that p is anywhere in the range \[0, 1]. That is, we have a continuous range of hypotheses. We will often model this example with a ‘bent’ coin with unknown probability p of heads.

**Ex2)** We model gestational length for single births by a normal distributions. The parameters $$\mu$$ and $$\sigma$$ of a normal distribution can be any real number in $$(-\infty,\infty)$$ and $$(0,\infty)$$.

{% hint style="info" %}
We model the random process giving rise to the data by a distribution with parameters called a **parameterized distribution**. Every possible choice of the parameter(s) is a hypothesis.
{% endhint %}

###

### The law of total probability

The law of total probability for continuous probability distributions is essentially the same as for discrete distributions. **Prior predictive probability** can be calculated as follows.

#### Discrete Hypothesis

$$
P(D) = \sum\limits\_{i = 1}^n {p(x|{\theta \_i})p({\theta \_i})}
$$

**Continuous Hypothesis**

$$
p(x) = \int\_a^b {p(x|\theta )f(\theta )d\theta }
$$

### Bayes' theorem for continuous probability densities

* $$\theta$$ is a continuous parameter with pdf $$f(\theta)$$and range $$\[a, b]$$.
* $$x$$ is random discrete data.
* Together they have likelihood $$p(x|\theta)$$

$$
f(\theta |x)d\theta = {{p(x|\theta )f(\theta )d\theta } \over {p(x)}} = {{p(x|\theta )f(\theta )d\theta } \over {\int\_a^b {p(x|\theta )f(\theta )d\theta } }}
$$

**Proof** can be done using Bayes' theorem in discrete priors.

$$
f(\theta |x)d\theta = P(H|D) = {{P(D|H)P(H)} \over {P(D)}} = {{p(x|\theta )f(\theta )d\theta } \over {p(x)}}
$$

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-a3c0388d47cb3ecd5f20f4622915a2766ad889e5%2Fimage%20\(328\)%20\(1\).png?alt=media)

### Bayesian updating with continuous priors

**Ex)** We have a bent coin with unknown probability $$\theta$$ of heads. Suppose we toss it once and get tails. Assume a flat prior( $$\theta$$ has range \[0,1] then $$f(\theta)=1$$ ) and find the posterior probability for $$\theta$$.

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-d7ff0139886f33b4d0e28507c351d4ce516aae0d%2Fimage.png?alt=media)

### From discrete to continuous Bayesian updating

1. Approximate the continuous range of hypotheses by a finite number.
2. Create the discrete updating table for the finite number of hypotheses.
3. Consider how the table changes as the number of hypotheses goes to infinity.

**Ex)** To keep things concrete, we will work with the ‘bent’ coin with a flat prior f(θ) = 1 from above example. Our goal is to go from discrete to continuous by increasing the number of hypotheses.

**4 hypotheses.** We slice \[0, 1] into 4 equal intervals: \[0, 1/4], \[1/4, 1/2], \[1/2, 3/4], \[3/4, 1]. Each slice has width Δθ = 1/4. We put our 4 hypotheses θi at the centers of the four slices:\
θ1: ‘θ = 1/8’, θ2: ‘θ = 3/8’, θ3: ‘θ = 5/8’, θ4: ‘θ = 7/8’.\
The flat prior gives each hypothesis a probability of 1/4 = 1 · Δθ. We have the table:

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-5dd44520984d85ed6157a240982998ec9bdc7348%2Fimage.png?alt=media)

Here are the density histograms of the prior and posterior pmf. The prior and posterior pdfs from above example are superimposed on the histograms in red.

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-5c99b027969df8b9ef95a5b466e627f63443ab80%2Fimage.png?alt=media)

**8 hypotheses.** Next we slice \[0,1] into 8 intervals each of width Δθ = 1/8 and use the center of each slice for our 8 hypotheses. The flat prior gives each hypothesis the probability 1/8 = 1 · Δθ. Here are the table and density histograms.

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-6dfd650da59e784783d0a675a493dc52f11da314%2Fimage.png?alt=media)

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-bb1e9a97a96dbff5a85b55a6ed02f4f7d8ad81c5%2Fimage.png?alt=media)

**20 hypotheses.** Finally we slice \[0,1] into 20 pieces. This is essentially identical to the previous two cases. Let’s skip right to the density histograms.

![](https://3698175758-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MAwtzMy_pbrChIExFtN%2Fuploads%2Fgit-blob-3f19177bde9c74ce49c8e4566f76d218fb02895b%2Fimage.png?alt=media)

Looking at the sequence of plots we see how the prior and posterior density histograms converge to the prior and posterior probability density functions.