H2o.ai Calibration Calibration Platt

Question

I noticed a relatively new addition to the h2o.ai suite, the ability to perform additional Platt scaling to improve calibration of output probabilities. (See calibrate_modelthe h2o manual .) However, several guidelines are available in the online reference documents. In particular, I wonder if Platt scaling is enabled:

• How does this affect the model leaderboard? That is, platt scaling is calculated after the ranking metric or earlier?
• How does this affect computing performance?
• Could it calibration_framebe the same as validation_frameit should or should not (both in the calculation and in the theoretical point of view)?

Thanks in advance

Answer 1

Calibration is a step after processing after completion of the model. Therefore, it does not affect the leaderboard and does not affect learning indicators. It adds 2 more columns to the scored frame (with calibrated forecasts).

This article provides guidance on creating a calibration frame:

• Divide the data set into test and train.
• Divide the train set into simulation and model calibration.

It also says: The most important step is to create a separate dataset to perform calibration with.

, , , . , - (, - H2O GLM), " ", . , , . :

" , , . , , ( ). 10% , , , 50 ".