According to Wikipedia, “a loss function or cost function is a function that maps an event or values of one or more variables onto a real number intuitively representing some ‘cost’ associated with the event.”(link) The definition works well in Machine Learning.


0/1 loss: 1\{f(x_i)\not=y_i\}

However, 0/1 loss is not convex, thus hard to optimize. To work around this problem, we use surrogate loss functions. These are convex functions that simulate the 0/1 loss and will always be the upper bound of 0/1 loss:

Hinge loss: max(0, 1-f(x_i)y_i)

Logistic loss: \frac{1}{\ln2}\ln(1+e^{-f(x_i)y_i})



Exponential loss:

Squared loss:


  1. L1 loss / absolute loss: |y_i-f(x_i)|
  2. L2 loss / squared loss: (y_i-f(x_i))^2
    • Gini index

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s