Cannot produce a Precision-Recall Curve from our Binary Classification Evaluator

[rogancarr]

In the Supported Scenarios for V1 (see #2498) we want to be able to easily plot a PR curve from output of the evaluator.

This is not currently possible in the current set of APIs*. As detailed in #2465, we cannot set a threshold on a binary classification prediction, so we cannot loop over the various thresholds. Furthermore, the Evaluator, while it has enough information to return lists of precisions and recalls at the various cutoffs, does not have an avenue to return this information.

* Well, actually this is possible using custom mappers like so:

1. Get a list of all unique probability scores
   (e.g. by reading the IDataView as an enumerable and creating a dictionary of observed values)
2. For each unique value of the probability:
   a. Write a custom mapper to produce PredictedLabel at that probability threshold.
   b. Calculate Precision and Recall with these labels.

However, it doesn't seem like this workaround qualifies as satisfying the V1 Scenario — it's a more advanced use of the APIs, and it requires an individual to understand the ins-and-outs of calculating PR curves

[shauheen]

Thanks for filing the issue @rogancarr, this is clearly a lack of functionality but we can surely do this post March.

[rogancarr]

@shauheen Sounds good! The scenario is technically blocked, but there is a workaround (documented above) because our APIs are so flexible.

[CESARDELATORRE]

That workaround looks pretty complex for a getting started sample.
We're hitting this issue in one of the samples (Spam detection):

https://github.com/dotnet/machinelearning-samples/blob/73803ce861e16b48edefaad97771bcd721e268d6/samples/csharp/getting-started/BinaryClassification_SpamDetection/SpamDetectionConsoleApp/Program.cs#L61-L66

We cannot use the new BinaryPredictionTransformer() any longer..

I hope we can fix/simplify this for 0.12 or by v1.0 in any case?

[rogancarr]

@CESARDELATORRE Current plan is for post-v1 @shauheen

[CESARDELATORRE]

@asthana86 @shauheen @eerhardt - So, shall we remove the "Spam Detection" sample from the repo?
Or @rogancarr can you help on implementing the mentioned workaround above in the "Spam Detection" sample? - But I'm not sure if the workaround is going to be too complex for a getting started sample...
Thoughts?

[rogancarr]

I'd be happy to help, but let's first resolve if this is in or out for V1.

[shauheen]

This along with #2465 would be stretch goals. At the moment we should not take these.

[ganik]

We do have functionality that calculates PR curve, its just we dont expose it yet (see internal):

internal sealed class BinaryClassifierEvaluator : RowToRowEvaluatorBase<BinaryClassifierEvaluator.Aggregator>
    {
        public sealed class Arguments
        {
            [Argument(ArgumentType.AtMostOnce, HelpText = "Probability value for classification thresholding")]
            public Single Threshold;

            [Argument(ArgumentType.AtMostOnce, HelpText = "Use raw score value instead of probability for classification thresholding", ShortName = "useRawScore")]
            public bool UseRawScoreThreshold = true;

            [Argument(ArgumentType.AtMostOnce, HelpText = "The number of samples to use for p/r curve generation. Specify 0 for no p/r curve generation", ShortName = "numpr")]
            public int NumRocExamples;

            [Argument(ArgumentType.AtMostOnce, HelpText = "The number of samples to use for AUC calculation. If 0, AUC is not computed. If -1, the whole dataset is used", ShortName = "numauc")]
            public int MaxAucExamples = -1;

            [Argument(ArgumentType.AtMostOnce, HelpText = "The number of samples to use for AUPRC calculation. Specify 0 for no AUPRC calculation", ShortName = "numauprc")]
            public int NumAuPrcExamples = 100000;
        }

So Arguments need to be public and our BinaryClassifier Evaluator should accept them

       /// <summary>
        /// Evaluates scored binary classification data, if the predictions are not calibrated.
        /// </summary>
        /// <typeparam name="T">The shape type for the input data.</typeparam>
        /// <param name="catalog">The binary classification catalog.</param>
        /// <param name="data">The data to evaluate.</param>
        /// <param name="label">The index delegate for the label column.</param>
        /// <param name="pred">The index delegate for columns from uncalibrated prediction of a binary classifier.
        /// Under typical scenarios, this will just be the same tuple of results returned from the trainer.</param>
        /// <returns>The evaluation results for these uncalibrated outputs.</returns>
        public static BinaryClassificationMetrics Evaluate<T>(
            this BinaryClassificationCatalog catalog,
            DataView<T> data,
            Func<T, Scalar<bool>> label,
            Func<T, (Scalar<float> score, Scalar<bool> predictedLabel)> pred)