Follow up on Calibrator estimators

[sfilipi]

There are a few follow ups to the work to create calibrator estimators:

1- The public classes in Microsoft.ML.Core/Prediction/Calibrator.cs need to have descriptive XML documentation.
2- The Calibrators need to be added as property to the BinaryClassificationContext
3- Some of the Calibrators have a public constructor, that allows initializing the parameters. Does it make sense to expose and wire those parameters to the CalibratorEstimators, and set them from there?
So they will all look like FixedPalttCalibratorEstimator

cc @Zruty0

[Zruty0]

There are 2 cases of calibrators: one where we set the parameters (slope, offset) manually, and one where we learn them.

Both cases will need an estimator: first one will just be a regular trainer, and the second will be a trivial estimator. I world prefer these 2 cases to be invoked via 2 different methods, instead of a single overload.

[TomFinley]

It seems to me that these when fit, given that they're parameterized with model parameters, should be producing a model with CalibratedModelParametersBase, with the sub-model being the input model. I do not see that the estimators currently are doing this.

I am at least confident we should not have two mechanisms for doing this. Right now we have two, and I'm not sure why:

1. CalibratorTransformer<TCalibrator>, and

2. BinaryPredictionTransformer<CalibratedModelParametersBase<TModel, TCalibrator>>

[Ivanidzo4ka]

I can be wrong, but for me it looks like done issue.
#2601 does 1) and 3) and I believe @wschin is done with making sure we return calibrated or not calibrated model.

[TomFinley]

I can be wrong, but for me it looks like done issue.
#2601 does 1) and 3)

Hi @Ivanidzo4ka , you are omitting 2 I think. The estimator for calibration is still not accessible through MLContext, the IHostEnvironment constructor is still public at the present time, as we see below.

machinelearning/src/Microsoft.ML.Data/Prediction/CalibratorCatalog.cs

Lines 279 to 282 in eb60021

	public PlattCalibratorEstimator(IHostEnvironment env,
	string labelColumn = DefaultColumnNames.Label,
	string scoreColumn = DefaultColumnNames.Score,
	string weightColumn = null) : base(env, new PlattCalibratorTrainer(env), labelColumn, scoreColumn, weightColumn)

There is still not Calibrator's property that is some sort of CalibratorsCatalog accessible through BinaryClassification. Calibrated models are great, but we also just want to be able to calibrate scores as these estimators do (per @sfilipi), and we haven't finished writing that catalog yet.

[Ivanidzo4ka]

Ok, let's at least add it to Project 13, otherwise it has chance to be missed.

[Ivanidzo4ka]

Just to make clear.
We want something like this:

            var pipeline = mlContext.BinaryClassification.Trainers.StochasticDualCoordinateAscentNonCalibrated(
                labelColumn: "Label", featureColumn: "Features", loss: new HingeLoss(), l2Const: 0.001f)
                .Append(mlContext.BinaryClassification.Calibrators.PairAdjacentViolators("Label"));

            // Step 3: Train the pipeline created.
            var model = pipeline.Fit(data);
           // Get submodel and calibrator.
            var subModel = model.LastTransformer.Model.SubModel;
            var calibrator = model.LastTransformer.Model.Calibrator;

Or something else?

Right now CalibratorEstimator returns

machinelearning/src/Microsoft.ML.Data/Prediction/CalibratorCatalog.cs

Line 150 in 412e1f9

public abstract class CalibratorTransformer<TICalibrator> : RowToRowTransformerBase, ISingleFeaturePredictionTransformer<TICalibrator>

which is quite blank.