According to Wikipedia, “a loss function or cost function is a function that maps an event or values of one or more variables onto a real number intuitively representing some ‘cost’ associated with the event.”(link) The definition works well in Machine Learning.

Classification:

0/1 loss: $1\{f(x_i)\not=y_i\}$

However, 0/1 loss is not convex, thus hard to optimize. To work around this problem, we use surrogate loss functions. These are convex functions that simulate the 0/1 loss and will always be the upper bound of 0/1 loss:

Hinge loss: $max(0, 1-f(x_i)y_i)$

Logistic loss: $\frac{1}{\ln2}\ln(1+e^{-f(x_i)y_i})$

entropy

cross-entropy:

Exponential loss:

Squared loss:

Regression:

1. L1 loss / absolute loss: $|y_i-f(x_i)|$
2. L2 loss / squared loss: $(y_i-f(x_i))^2$
• Gini index