🖍️
gitbook_docs
  • Introduction
  • Machine Learning
    • Recommended Courses
      • For Undergrad Research
      • Math for Machine Learning
    • ML Notes
      • Covariance Correlation
      • Feature Selection
      • Linear Regression
      • Entropy, Cross-Entropy, KL Divergence
      • Bayesian Classifier
        • Terminology Review
        • Bayesian Classifier for Normally Distributed classes
      • Linear Discriminant Analysis
      • Logistic Regression
        • Logistic Regression Math
      • Logistic Regression-MaximumLikelihood
      • SVM
        • SVM concept
        • SVM math
      • Cross Validation
      • Parameter, Density Estimation
        • MAP, MLE
        • Gaussian Mixture Model
      • E-M
      • Density Estimation(non-parametric)
      • Unsupervised Learning
      • Clustering
      • kNN
      • WaveletTransform
      • Decision Tree
    • Probability and Statistics for Machine Learning
      • Introduction
      • Basics of Data Analysis
      • Probability for Discrete Random Variable
      • Poisson Distribution
      • Chi-Square Distribution
      • P-value and Statistical Hypothesis
      • Power and Sample Size
      • Hypothesis Test Old
      • Hypothesis Test
      • Multi Armed Bandit
      • Bayesian Inference
      • Bayesian Updating with Continuous Priors
      • Discrete Distribution
      • Comparison of Bayesian and frequentist inference
      • Confidence Intervals for Normal Data
      • Frequenist Methods
      • Null Hypothesis Significance Testing
      • Confidence Intervals: Three Views
      • Confidence Intervals for the Mean of Non-normal Data
      • Probabilistic Prediction
  • Industrial AI
    • PHM Dataset
    • BearingFault_Journal
      • Support Vector Machine based
      • Autoregressive(AR) model based
      • Envelope Extraction based
      • Wavelet Decomposition based
      • Prediction of RUL with Deep Convolution Nueral Network
      • Prediction of RUL with Information Entropy
      • Feature Model and Feature Selection
    • TempCore Journal
      • Machine learning of mechanical properties of steels
      • Online prediction of mechanical properties of hot rolled steel plate using machine learning
      • Prediction and Analysis of Tensile Properties of Austenitic Stainless Steel Using Artificial Neural
      • Tempcore, new process for the production of high quality reinforcing
      • TEMPCORE, the most convenient process to produce low cost high strength rebars from 8 to 75 mm
      • Experimental investigation and simulation of structure and tensile properties of Tempcore treated re
    • Notes
  • LiDAR
    • Processing of Point Cloud
    • Intro. 3D Object Detection
    • PointNet
    • PointNet++
    • Frustrum-PointNet
    • VoxelNet
    • Point RCNN
    • PointPillars
    • LaserNet
  • Simulator
    • Simulator List
    • CARLA
    • Airsim
      • Setup
      • Tutorial
        • T#1
        • T#2
        • T#3: Opencv CPP
        • T#4: Opencv Py
        • Untitled
        • T#5: End2End Driving
  • Resources
    • Useful Resources
    • Github
    • Jekyll
  • Reinforcement Learning
    • RL Overview
      • RL Bootcamp
      • MIT Deep RL
    • Textbook
    • Basics
    • Continuous Space RL
  • Unsupervised Learning
    • Introduction
  • Unclassified
    • Ethics
    • Conference Guideline
  • FPGA
    • Untitled
  • Numerical Method
    • NM API reference
Powered by GitBook
On this page
  • Lecture Note
  • Introduction
  • Parametric model
  • Nonparametric model
  • Kernel Density Estimation Method
  • Example
  • Gaussian Mixture Model
  • Reference

Was this helpful?

  1. Machine Learning
  2. ML Notes

Parameter, Density Estimation

PreviousCross ValidationNextMAP, MLE

Last updated 3 years ago

Was this helpful?

Lecture Note

: TAMU

Introduction

For Bayesian Classifier, it becomes the problem of Probability Density Function(pdf) Estimation for continuous distribution of Likelihood.

How to estimate p(x∣wi)p(\bf{x}|w_i)p(x∣wi​) with samples of x?

X 중에 w_i 에 속하는 샘플 집합 Xi를 가지고 p(x|w_i)를 추정하라

It is modeling the probability density function (pdf) of the unknown probability distribution from which the dataset has been drawn.

The methods are classifed as

Parametric model

  • Gaussian Mixture Model

  • multivariate normal distribution (MND)

Nonparametric model

  • Kernel Density Estimation, Histogram, Parzen Window

  • k-Nearest neighbor

Kernel Density Estimation Method

It has the parameter of b, the bandwidth

  • Example: Gaussian kernel (1D) for k

Find the optimum b: We look for such a value of b that minimizes the difference between the real shape of f(x) and the shape of our model f_b(x).

Example

Let {xi}N i=1 be a one-dimensional dataset (a multi-dimensional case is similar) whose examples were drawn from a distribution with an unknown pdf f with xi 2 R for all i = 1, . . . ,N.

How to measure the goodness of estimation?

A reasonable choice of measure of this difference is called the mean integrated squared error (MISE):

we square the difference between the real pdf f and our model of it f^hat_b.

Now, to find the optimal value b* for b, we minimize the cost defined as,

Limitations

  • Memory -based method, that need to store all the samples of training.

  • For additional sample, need to re-calculate the whole process.

  • Curse-of Dimension exists: Use only for low-dimension problems

Gaussian Mixture Model

Reference

머신러닝/패턴인식, 오일석

The Hundred-Page Machine Learning Book

http://themlbook.com/wiki/doku.php
Read class PPT for detail
Click here for Lecture PPT
https://towardsdatascience.com/gaussian-mixture-models-explained-6986aaf5a95towardsdatascience.com
Read GMM
MAP, MLE