1. Expectations
• $E[X] = \int xf(x)dx$
• $E[c] = c$
• $E[X + c] = E[X] + c$
• $E[cX] = cE[X]$
• $E[X+Y] = E[X] + E[Y]$
• $E[XY] = E[X]E[Y]$ if $X$ and $Y$ are independent
• $E[\vec{X}] = (E[X_1] \ E[X_2] \ ... \ E[X_d])^T$
2. Variance
• $Var(X) = E[(X - E[X])^2]$
• $Var(X) = E[X^2] - (E[X])^2$
• $Var(c) = 0$
• $Var(X + c) = Var(X)$
• $Var(cX) = c^2Var(X)$
• $Var(X+Y) = Var(X)Var(Y)$ if $X$ and $Y$ are independent
3. Covariance
• $Cov(X,Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X][Y]$
• $Var(X) = Cov(X,X)$
• $Cov(X,c) = 0$
• $Cov(aX, bY) = abCov(X,Y)$
• $Cov(\vec{X}) = E[(\vec{X} - E[\vec{X}])(\vec{X} - E[\vec{X}])^T] = E[\vec{X}\vec{X}^T] - E[\vec{X}](E[\vec{X}])^T$
• $Cov(\vec{X}|\vec{Y}) = E[(\vec{X} - E[\vec{X}|\vec{Y}])(\vec{X} - E[\vec{X}|\vec{Y}])^T|\vec{Y}]$
4. Mean Square Error: $MSE(\hat{\theta}) = E[(\hat{\theta} - \theta)^2] = E[(\hat{\theta}-E[\hat{\theta}])^2] + (E[\hat{\theta}]-\theta)^2 = Var(\hat{\theta}) + Bias(\hat{\theta})^2$, more detail explanation can be found on wikipedia.
5. Law of Total Expectation $E[X] = E[E[X|Y]]$