Debugging hanging AutoFitImageClassificationTrainTest

[mstfbl]

Will be using this draft PR for general debugging purposes on CI

Notes:
Windows builds have 7,168 MBs of RAM

[mstfbl]

Testing of AutoFitImageClassificationTrainTest is taking too long per test, so tesitng it with 1000 iterations isn't feasible.

[mstfbl]

The tests AutoFitRecommendationTest and AutoFitRegressionTest are passing. AutoFitImageClassificationTrainTest is displaying errors every now and then.

[mstfbl]

The reason why AutoFitImageClassificationTrainTest is crashing is after running var result = context.Auto().CreateMulticlassClassificationExperiment(0).Execute(trainDataset, testDataset, columnInference.ColumnInformation), result.Best run might not always be updated. I saw this as I caught result.BestRun having a null value right when it is being called. Exact location where error is thrown.

[mstfbl]

The original bug with AutoFitImageClassificationTrainTest is occuring due to the null returned value here:

machinelearning/src/Microsoft.ML.AutoML/Utils/BestResultUtil.cs

Line 61 in f0a8a76

if (!results.Any()) { return null; }

For some reason, sometimes the validationMetrics of an IEnumerable<(RunDetail) is null.

[mstfbl]

There's an issue with this Evaluation function:

machinelearning/src/Microsoft.ML.Data/Evaluators/MulticlassClassificationEvaluator.cs

Lines 506 to 535 in f0a8a76

	public MulticlassClassificationMetrics Evaluate(IDataView data, string label, string score, string predictedLabel)
	{
	Host.CheckValue(data, nameof(data));
	Host.CheckNonEmpty(label, nameof(label));
	Host.CheckNonEmpty(score, nameof(score));
	Host.CheckNonEmpty(predictedLabel, nameof(predictedLabel));
	var roles = new RoleMappedData(data, opt: false,
	RoleMappedSchema.ColumnRole.Label.Bind(label),
	RoleMappedSchema.CreatePair(AnnotationUtils.Const.ScoreValueKind.Score, score),
	RoleMappedSchema.CreatePair(AnnotationUtils.Const.ScoreValueKind.PredictedLabel, predictedLabel));
	var resultDict = ((IEvaluator)this).Evaluate(roles);
	Host.Assert(resultDict.ContainsKey(MetricKinds.OverallMetrics));
	var overall = resultDict[MetricKinds.OverallMetrics];
	var confusionMatrix = resultDict[MetricKinds.ConfusionMatrix];
	MulticlassClassificationMetrics result;
	using (var cursor = overall.GetRowCursorForAllColumns())
	{
	var moved = cursor.MoveNext();
	Host.Assert(moved);
	result = new MulticlassClassificationMetrics(Host, cursor, _outputTopKAcc ?? 0, confusionMatrix);
	moved = cursor.MoveNext();
	Host.Assert(!moved);
	}
	return result;
	}
	}

The returned result value can sometimes be null, which is what is causing AutoFitImageClassificationTrainTest to sometimes fail.

[mstfbl]

I figured out the cause of the occasional crash of AutoFitImageClassificationTrainTest. When any exception occurs in RunnerUtil.TrainAndScorePipeline, instead of throwing the error, it is instead caught and ignored while a null metrics value (in line 49) is sent up through the call stack instead.

machinelearning/src/Microsoft.ML.AutoML/Experiment/Runners/RunnerUtil.cs

Lines 25 to 49 in f0a8a76

	try
	{
	var estimator = pipeline.ToEstimator(trainData, validData);
	var model = estimator.Fit(trainData);
	var scoredData = model.Transform(validData);
	var metrics = metricsAgent.EvaluateMetrics(scoredData, labelColumn);
	var score = metricsAgent.GetScore(metrics);
	if (preprocessorTransform != null)
	{
	model = preprocessorTransform.Append(model);
	}
	// Build container for model
	var modelContainer = modelFileInfo == null ?
	new ModelContainer(context, model) :
	new ModelContainer(context, modelFileInfo, model, modelInputSchema);
	return (modelContainer, metrics, null, score);
	}
	catch (Exception ex)
	{
	logger.Error($"Pipeline crashed: {pipeline.ToString()} . Exception: {ex}");
	return (null, null, ex, double.NaN);

This is the exception being caught:

System.ArgumentException : PIPELINE CRASHES - Line 55 - RunnerUtil.cs - Pipeline crash string: xf=ValueToKeyMapping{ col=Label:Label} xf=RawByteImageLoading{ col=ImagePath_featurized:ImagePath imageFolder=} xf=ColumnCopying{ col=Features:ImagePath_featurized} tr=ImageClassification{} xf=KeyToValueMapping{ col=PredictedLabel:PredictedLabel} cache=- - Exception String: System.FormatException: Tensorflow exception triggered while loading model. ---> System.Runtime.InteropServices.SEHException: External component has thrown an exception.

When reproduced locally, the exception string is:

"Could not find a part of the path 'C:\Users\mubal\AppData\Local\Temp\Microsoft.ML.AutoML\experiment_y1gbdyum.xdu\Model1.zip'."

[mstfbl]

Update: AutoFitImageClassificationTrainTest with 100 iterations fail on Windows x64 builds with:

System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception

but also I get:
System.FormatException: Tensorflow exception triggered while loading model. ---> System.OutOfMemoryException
and:
System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

in Tensorflow.c_api.TF_SessionRun, which is the C++ implementation of TensorFlow's training code. This seems related to Issue SciSharp/TensorFlow.NET#485

As mentioned in this PR #4755, we still cannot see details about the crash in Tensorflow.c_api.TF_SessionRun.

[mstfbl]

Will be using this PR to debug AutoFitImageClassificationTrainTest hanging occasionally on Windows builds.

[mstfbl]

AutoFitImageClassificationTrainTest is still occasionally hanging, mostly due to indisposed Tensorflow objects after the test is complete. I found this comment in Microsoft.ML.Vision/ImageClassificationTrainer.TrainModelCore to be of interest:

machinelearning/src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

Lines 714 to 721 in 5d531d3

	// Leave the ownership of _session so that it is not disposed/closed when this object goes out of scope
	// since it will be used by ImageClassificationModelParameters class (new owner that will take care of
	// disposing).
	var session = _session;
	_session = null;
	return new ImageClassificationModelParameters(Host, session, _classCount, _jpegDataTensorName,
	_resizedImageTensorName, _inputTensorName, _softmaxTensorName);

[mstfbl]

Adding the fix (model as IDisposable)?.Dispose(); for freeing Tensor objects worked! This fix is necessary, as these Tensor objects made in the C TensorFlow libraries are not automatically cleaned up by C#'s Garbage Collector.

Edit: While this fix works, it is not safe to assume that this model can be disposed in RunnerUtil.cs. The user might be accessing this model during disposal, which would result in use-after-free and/or null reference errors.

[mstfbl]

Freeing Tensor objects in model in a finally statement in TrainAndScorePipeline works in fixing memory bug, and is safe to do when model is never saved in memory and written to disk always.