Reconcile the check for Probability in the BinaryClassifierEvaluator with the logic in the calibrators

[sfilipi]

Fitting an IDataView that contains a score column(produced by a binary classifier scorer), currently will append a column with the name probability to the IDataView .

If one column with the name "Probability" exists, another one will be added. (Maybe if the user wants to try out different calibrators?). The calibrators don't complain about its existence.

Currently the BinaryClassifierEvaluator checks that there are no more than one probability columns.

Reconcile the behavior/expectations, or is this ok, and we can leave the user cleanup if more than one Probability columns.

cc @yaeldekel @TomFinley

[TomFinley]

Hi @sfilipi , just a clarifying question, in the time since you wrote this issue our public API has changed to the point where the API via MLContext.BinaryClassification.Evaluate has explicit names for the columns, and also after you wrote this issue the class BinaryClassifierEvaluator, which is no longer public after #2453. (Obviously written after this issue!) Does that obviate the need for this issue, since there is now only one public surface area? Or is this issue still relevant?

It might still be relevant if the public surface area were behaving strangely in the case where multiple models were being evaluated, but if that is true that is not clear to me. If it is still relevant, a worked example to demonstrate the problem would help enormously with at least my ability to appreciate it.

cc @shauheen