NaN metric value handling in AutoML

[justinormont]

AutoML API code is not handing NaN values for metrics. During the sweep, when a model returns a NaN value for the metric being optimized, AutoML crashes.

See background: #4648 (comment)

[harishsk]

@justinormont You suggested fixing two things in #4648 . The second one is being fixed in ML.NET. Does the first work item still need to be done as part of this bug fix? If so, can you please add some repro steps for this bug?

[justinormont]

@CBrauer has a repro in bug.zip from his original bug report: #4648 (comment)

Hello,

I just upgraded my project to the new pre-release versions of ML.NET and I got the following error message when I ran my program:

I have added a Zip file of my program and dataset. I hope you guys can help me find out why I'm getting this error,

Charles

bug.zip

In this example, he is optimizing towards the F1 metric, which can currently be NaN. The AutoML code crashes when it receives the NaN within the metric its optimizing towards.

... Does the first work item still need to be done as part of this bug fix?

The AutoML code does need to be robust to NaN values for its optimization metric. NaN values are the expected values at times.

Another way to reproduce is in a debugger and replacing the model's returned metric w/ NaN.

[harishsk]

@CBrauer The attached zip file did not contain the csv files for validation and test. I reduced the training file by 40% and created two new files for validation and test. With that, I have not been able to reproduce the issue you are seeing.

Can you please update the zip file with the necessary files that reproduce the issue?

[najeeb-kazmi]

We don't need to split the file into train, validation, and test. AutoML does the split internally. In this case, the training set is used as the validation set just to evaluate metrics from the best AutoML run. The choice of dataset is unrelated to AutoML training, which is the relevant part of the code for this bug.

I can reproduce the error with the data and code provided. It uses 1.5.0-preview and 0.17.0-preview, which do not have the fix for F1 score returning 0 instead of Nan from #4674. The fix for F1 is there in preview2.

I'll look at how NaN metrics can be handled in AutoML. F1 no longer returns NaN, but LogLossReduction can still return NaN #4648 (comment). I'll look at whether NaN in LogLossReduction can be handled, or if AutoML should generally handle NaNs, or both.