Definition

The covariance of two random variables XX and YY is defined by:

Cov[X,Y]≜𝔼[(X−𝔼[X])(Y−𝔼[Y])]\mathrm{Cov}[X,Y]\triangleq \mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])]

or equivalently,

Cov[X,Y]=𝔼[XY]−𝔼[X]𝔼[Y]\mathrm{Cov}[X,Y] = \mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]

Note that variance is a special case:

Var[X]=Cov[X,X]\mathrm{Var}[X]=\mathrm{Cov}[X,X]

General rules

Cov⁡(∑i=1naiXi,∑j=1majXj)=∑i=1n∑j=1mCov⁡(Xi,Yj)\operatorname{Cov}\left(\sum_{i=1}^n a_i X_i, \sum_{j=1}^m a_j X_j\right) = \sum_{i=1}^n \sum_{j=1}^m \operatorname{Cov}(X_i, Y_j) Var⁡(∑i=1nXi)=∑i=1nVar⁡(Xi)+∑∑i<j⁡Cov⁡(Xi,Xj)\operatorname{Var}\left( \sum_{i=1}^n X_i \right) = \sum_{i=1}^n \operatorname{Var}(X_i) + \mathop{\sum\sum}_{i < j} \operatorname{Cov}(X_i, X_j)

Sample covariance matrix

for a sample of data on random variables we may define the sample covariance matrix as a KK-by-KK matrix 𝐐=[qjk]\mathbf{Q} = [q_{jk}] where

qjk=1N−1∑i=1N(xij−x‾j)(xik−x‾k)q_{jk} = \frac{1}{N-1}\sum_{i=1}^N (x_{ij}-\bar{x}_j)(x_{ik}-\bar{x}_k)

with qjkq_{jk} as estimate of covariance between jj-th and kk-th variables underlying the data; we may also write

𝐐=1N−1∑i=1N(𝐱i−𝐱‾)(𝐱i−𝐱‾)T\mathbf{Q} = \frac{1}{N-1}\sum_{i=1}^N (\mathbf{x}_i-\bar{\mathbf{x}})(\mathbf{x}_i-\bar{\mathbf{x}})^T

Notes

See also

References:

  1. https://cs229.stanford.edu/section/cs229-prob.pdf
  2. https://www.randomservices.org/random/expect/Covariance.html
  3. V. M. Panaretos, Statistics for Mathematicians. in Compact Textbooks in Mathematics. Cham: Springer International Publishing, 2016. doi: 10.1007/978-3-319-28341-8. p. 158.
  4. https://en.wikipedia.org/wiki/Covariance
  5. https://en.wikipedia.org/wiki/Sample_mean_and_covariance