Cannot set the threshold on a binary predictor

[rogancarr]

It is no longer possible to set a custom Threshold and ThresholdColumn on a binary classifier.

Previously, we had been using BinaryPredictionTransformer. Recently, BinaryPredictionTransformer was marked as internal and is no longer available for usage outside of the library.

Related to question #403

[rogancarr]

One suggestion is to make the Threshold and ThresholdColumn properties of the prediction transform settable. They are currently only gettable.

Something like this?

var model = mlContext.BinaryClassification.Trainers.LogisticRegression().Fit(train);
var predictor = model.LastTransformer;
predictor.Threshold = 0.01;
// or
predictor.SetThreshold(0.01);
predictor.SetThresholdColumn("Bananas");

[Ivanidzo4ka]

https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Tests/Scenarios/Api/Estimators/ReconfigurablePrediction.cs

[rogancarr]

@Ivanidzo4ka I am pulling these out of Tests/ and into a new Functional.Tests that is does not have InternalsVisibleTo available in accordance with #2306 .

This test relies on internal APIs, so it doesn't work from and end-user perspective.

[eerhardt]

YAY! The first value of these tests is now realized. 😄

[rogancarr]

For anyone looking for this functionality, there is a workaround discussed in #2645.

[eerhardt]

Note that we are doing this exact scenario in our samples:

https://github.com/dotnet/machinelearning-samples/blob/73803ce861e16b48edefaad97771bcd721e268d6/samples/csharp/getting-started/BinaryClassification_SpamDetection/SpamDetectionConsoleApp/Program.cs#L61-L66

            // The dataset we have is skewed, as there are many more non-spam messages than spam messages.
            // While our model is relatively good at detecting the difference, this skewness leads it to always
            // say the message is not spam. We deal with this by lowering the threshold of the predictor. In reality,
            // it is useful to look at the precision-recall curve to identify the best possible threshold.
            var inPipe = new TransformerChain<ITransformer>(model.Take(model.Count() - 1).ToArray());
            var lastTransformer = new BinaryPredictionTransformer<IPredictorProducing<float>>(mlContext, model.LastTransformer.Model, inPipe.GetOutputSchema(data.Schema), model.LastTransformer.FeatureColumn, threshold: 0.15f, thresholdColumn: DefaultColumnNames.Probability);

@CESARDELATORRE

[rogancarr]

@eerhardt This is broken as of 0.11.

[eerhardt]

I think we need to fix it. This is a “take back” in functionality that we said we needed for v1.

[rogancarr]

I've added these back to Project 13. Let's discuss these in scrum today and update the issues once we have made a group decision.