Revision

Back to Machine Learning

Ordinary Least Squares

See Wikipedia page of Ordinary Least Squares.

Link with Least Squares

Ordinary Least Squares is a special case of Least Squares. OLS is the most commonly used estimator of the LS family. To have more insights on the difference see:

• Wikipedia page of Least Squares,
• Wikipedia page of Ordinary Least Squares,
• This StackOverflow page,
• This StackExchange page,
• This document by JJ Jacquelin (in french).

Maximum Likelihood

Given a sample of data \((X, Y)\) (or simply \(Y\) if we are not in a prediction framework using features), the goal of maximum likelihood estimation is to make inferences about the population that is most likely to have generated the sample, specifically the joint probability distribution of the random variables.

The goal of MLE is to find the best set of parameters \(\theta\) of a probability distribution function to match the empirical distribution of the value \(Y\) given the features \(X\). Hence the goal is to find the \(\theta\) that maximise the probability density function for the values we get.

Resources

See:

• Wikipedia page of Maximum Likelihood Estimation.

Maximum a Posteriori

See:

• Wikipedia page of Maximum a Posteriori.

Method of moments

See:

• Wikipedia page of Method of moments.

Bayes estimator

See:

• Wikipedia page of Bayes estimator.

Other estimators

See:

• This list of estimators on Wikipedia.