Revision

Back to Probability distribution

Moments

Expected value

Definition

\[\mathbb{E}(X)=\int_{-\infty}^{\infty}x \cdot f(x)dx\]

Where:

\(f\) is the probability density function of \(X\).

Estimator

The estimator of the expected value or mean is:

\[\mathbb{E}(X)=\frac{1}{n}\sum_{i=1}^{n}x_i\]

Resources

See : Wikipedia page for expected value.

Variance and standard deviation

Definition

\[Var(X) =\sigma^2 =\mathbb{E}\left[\left(X-\mathbb{E}(X)\right)^2\right] =\mathbb{E}\left[X^2-2X\mathbb{E}(X)+\mathbb{E}(X)^2\right] =\mathbb{E}\left(X^2\right)-2\mathbb{E}(X)\mathbb{E}(X)+\mathbb{E}(X)^2 =\mathbb{E}\left(X^2\right)-\mathbb{E}(X)^2\] \[Var(X)=\sigma^2=\int_{-\infty}^{\infty}(x-\mu)^2f(x)dx=\int_{-\infty}^{\infty}x^2f(x)dx-\mu^2\]

Where:

\(f\) is the probability density function of \(X\).

The standard deviation is the square root of the variance it is \(\sigma\).

Estimator

Unbiased variance (with Bessel’s correction ie n-1 in place of n in the denominator):

\[Var(X)=\sigma^2=\frac{1}{n-1}\sum_{i=1}^{n}\left(x_i-\bar{x}\right)^2=\left(\frac{1}{n}\sum_{i=1}^{n}x_i^2\right)-\bar{x}^2\]

Where:

\(\bar{x}\) is the estimator of the mean

The estimator of the standard deviation is just \(\sigma\), the square root of the estimated variance.

Properties

For X and Y two random variables and a and b two determistic values:

\(Var(X) \geq 0\),
\(Var(a) = 0\),
\(Var(X+a)=Var(X)\),
\(Var(aX)=a^2Var(X)\),
\(Var(aX+bY) = a^2 Var(X) + b^2 Var(Y) + 2 ab Cov(X,Y)\).

Resources

See : Wikipedia page for variance.

Covariance

Definition

\(Cov(X, Y)=\mathbb{E}\left[\left(X-\mathbb{E}(X)\right)\left(Y-\mathbb{E}(Y)\right)\right]=\mathbb{E}(XY)-\mathbb{E}(X)\mathbb{E}(Y)\)

Estimator

Unbiased covariance estimator (with Bessel’s correction ie n-1 in place of n in the denominator):

\[Cov(X, Y)=\frac{1}{n-1}\sum_{i=1}^{n}\left(x_i-\bar{x}\right)\left(y_i-\bar{y}\right)\]

Properties

For X and Y two random variables and a and b two determistic values:

\(X\text{, }Y\text{ independents } \implies Cov(X,Y)=0\),
\(Cov(X,Y)=0 \not\implies X\text{, }Y\text{ independents}\),
\(Cov(X,X)=Var(X)\),
\(Cov(X,a)=0\),
\(Cov(X,Y)=Cov(Y,X)\),
\(Cov(aX,bY) = ab Cov(X,Y)\),
\(Cov(a+X,b+Y) = Cov(X,Y)\),
\(Cov(aX+bY,cW+dV) = ac\;Cov(X,W) + ad\;Cov(X,V) + bc\;Cov(Y,W) + bd\;Cov(Y,V)\) (for V, W random variables and c, d deterministic).

For \(Cov(X,Y)=0 \not\implies X\text{, }Y\text{ independents}\), an example is to take \(X\) centered on 0 and \(Y=X^2\). Covariance is null but \(Y\) is totally determined by \(X\).

Resources

See : Wikipedia page for covariance.

Correlation

\[\rho(X, Y)=\frac{Cov(X,Y)}{\sigma_X \sigma_Y}\]

Where:

\(\sigma_X\) is the standard deviation of X
\(\sigma_Y\) is the standard deviation of Y

Estimator

The estimator of the correlation is the estimator of the covariance of X and Y divided by the product of the estimated standard deviation of X and Y.

Properties

For X and Y two random variables and a and b two determistic values:

\(X\text{, }Y\text{ independents } \implies \rho(X,Y)=0\),
\(\rho(X,Y)=0 \not\implies X\text{, }Y\text{ independents}\),

For \(rho(X,Y)=0 \not\implies X\text{, }Y\text{ independents}\), an example is to take \(X\) centered on 0 and \(Y=X^2\). Correlation is null but \(Y\) is totally determined by \(X\).

Resources

See : Wikipedia page for correlation.

Skewness

Skewness is the third moment of a distribution. It measures the asymmetry of a probability distribution.

Kurtosis

Kurtosis is the forth moment of a distribution. It describes the shape of a probability distribution and in particular the probability of extreme values (fat tails).

Characteristic function

The characteristic function of a random variable X is:

\[\phi_X(t)=\mathbb{E}\left[e^{itX}\right]=\mathbb{E}\left[\cos(tX)\right]+i\mathbb{E}\left[\sin(tX)\right]\]

If the distribution of this random variable has a probability density then its characteristic function is the inverse Fourier transform of this probability density function:

\[\phi_X(t)=\int_{\mathbb{R}}f_X(x)e^{itx}dx\]

The characteristic function is very useful to compute the moments of a distribution. Its successive derivatives taken at 0 give the moment of equivalent order.

\[\frac{d\phi_X(t)}{dt}=\mathbb{E}\left[iXe^{itX}\right]\] \[\frac{d\phi_X(0)}{dt}=i\mathbb{E}[X]\]

So the first derivative gives the mean.