What are options to calibrate probabilities produced from the output of a classifier that does not produce natural probabilities?

The two most common calibration approaches are:

(a) Platt scaling

(b) Isotonic regression

At a high level, Platt scaling fits a logistic regression on the original predictions, where the target is the original class labels and the input is the array of raw predicted probabilities. Isotonic regression follows a similar approach but instead fits a piecewise non-decreasing function to the original predictions rather than a logistic regression. Platt scaling tends to work better when the raw probabilities are not concentrated around 0 or 1, which tends to occur from Ensemble based methods like Random Forest and GBM. On the other hand, Isotonic regression provides better calibration for algorithms like Naive Bayes that produce many probabilities at the extremes. 

Author

Help us improve this post by suggesting in comments below:

– modifications to the text, and infographics
– video resources that offer clear explanations for this question
– code snippets and case studies relevant to this concept
– online blogs, and research publications that are a “must read” on this topic

Leave the first comment

Partner Ad
Find out all the ways that you can
Contribute