Strange error message

[CBrauer]

Hello,

I just upgraded my project to the new pre-release versions of ML.NET and I got the following error message when I ran my program:

I have added a Zip file of my program and dataset. I hope you guys can help me find out why I'm getting this error,

Charles

bug.zip

[kere-nel]

@justinormont This appears to be an AutoML issue, could you take a look at it?

[justinormont]

This is an error in the AutoML API code.

It appears due to the handling of NaN values in the AutoML code, which can occur for the F1Score metric that you're optimizing towards. Along with F1, log-loss reduction can also return a NaN.

F1 can return a NaN value when both the precision and recall are zero, as F1 = 2 * precision * recall / (precision + recall), which turns into 0 / 0. Precision/recall being zero occurs for a trivial model which always guesses the majority class zero.

Two things to fix in ML.NET/AutoML:

1. Check why AutoML isn't handling the NaN for the metric values
2. Consider returning F1=0.0 instead of NaN -- this is done in sklearn (code), and others (note)
   
   Our F1 calculation code:
   

   machinelearning/src/Microsoft.ML.Data/Evaluators/BinaryClassifierEvaluator.cs

   Line 483 in 610ffcb

   public Double F1 { get { return 2 * PrecisionPos * RecallPos / (PrecisionPos + RecallPos); } }

[CBrauer]

Thanks Justin. I'll change your code as your suggest.