Evaluating model that loads images from disk throws an exception in ASP.NET core

[adriansd27]

System information

• OS version/distro: Windows 10
• .NET Version (eg., dotnet --info): ASP.NET Core 3.1

Issue

I'm running ASP.NET Core 3.1. Training the model works fine. PredictionEnginePool predicts fine. However if I try to get the metrics of the trained model I'm getting the below exception. Is it that ASP.NET Core doesn't support System.Drawing.Bitmap? If so, is there any alternative option to evaluate a model in ASP.NET Core?

Source code / logs

 var trainingData = _mlContext.Data.LoadFromEnumerable(imageDataTags);
 var model = pipeline.Fit(trainingData);
 var testingData = _mlContext.Data.LoadFromEnumerable(testImageDataTags);
 var predictions = model.Transform(testingData);
 var metrics = _mlContext.MulticlassClassification.Evaluate(predictions,"LabelKey");

 ---> System.ArgumentException: Parameter is not valid.
   at System.Drawing.Bitmap..ctor(String filename, Boolean useIcm)
   at System.Drawing.Bitmap..ctor(String filename)
   at Microsoft.ML.Data.ImageLoadingTransformer.Mapper.<>c__DisplayClass4_0.<MakeGetterImageDataViewType>b__0(Bitmap& dst)
   at Microsoft.ML.Transforms.Image.ImageResizingTransformer.Mapper.<>c__DisplayClass3_0.<MakeGetter>b__1(Bitmap& dst)
   at Microsoft.ML.Transforms.Image.ImagePixelExtractingTransformer.Mapper.<>c__DisplayClass5_0`1.<GetGetterCore>b__1(VBuffer`1& dst)
   at Microsoft.ML.Transforms.TensorFlowTransformer.TensorValueGetterVec`1.GetTensor()
   at Microsoft.ML.Transforms.TensorFlowTransformer.Mapper.UpdateCacheIfNeeded(Int64 position, ITensorValueGetter[] srcTensorGetters, String[] activeOutputColNames, OutputCache outputCache)
   at Microsoft.ML.Transforms.TensorFlowTransformer.Mapper.<>c__DisplayClass9_0`1.<MakeGetter>b__4(VBuffer`1& dst)
   at Microsoft.ML.Data.SchemaBindablePredictorWrapperBase.<>c__DisplayClass18_0`2.<GetValueGetter>b__0(TDst& dst)
   at Microsoft.ML.Data.DataViewUtils.Splitter.InPipe.Impl`1.Fill()
   at Microsoft.ML.Data.DataViewUtils.Splitter.<>c__DisplayClass5_1.<ConsolidateCore>b__2()
   --- End of inner exception stack trace ---
   at Microsoft.ML.Data.DataViewUtils.Splitter.Batch.SetAll(OutPipe[] pipes)
   at Microsoft.ML.Data.DataViewUtils.Splitter.Cursor.MoveNextCore()
   at Microsoft.ML.Data.RootCursorBase.MoveNext()
   at Microsoft.ML.Data.EvaluatorBase`1.ProcessData(IDataView data, RoleMappedSchema schema, Func`2 activeColsIndices, TAgg aggregator, AggregatorDictionaryBase[] dictionaries)
   at Microsoft.ML.Data.EvaluatorBase`1.Microsoft.ML.Data.IEvaluator.Evaluate(RoleMappedData data)
   at Microsoft.ML.Data.MulticlassClassificationEvaluator.Evaluate(IDataView data, String label, String score, String predictedLabel)
   at Microsoft.ML.MulticlassClassificationCatalog.Evaluate(IDataView data, String labelColumnName, String scoreColumnName, String predictedLabelColumnName, Int32 topKPredictionCount)
   at PDCSBE.Services.Implementation.PredictionService.TrainModel() in D:\...\Services\Implementation\PredictionService.cs:line 91
   at PDCSBE.Api.Controllers.PredictController.TrainModel(ModelTrainerDataInputDto input) in D:\...\Controllers\PredictController.cs:line 26
   at lambda_method(Closure , Object , Object[] )
   at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.<Invoke>g__AwaitMatcher|8_0(EndpointRoutingMiddleware middleware, HttpContext httpContext, Task`1 matcherTask)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

HEADERS
=======
Accept: */*
Accept-Encoding: gzip, deflate, br
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 214
Content-Type: application/json
Host: localhost:44370
User-Agent: PostmanRuntime/7.24.1
Postman-Token: ced1eca6-9e0b-4292-9887-f46a6d1c57d5

[ddobric]

System.Drawing is not a part of .net core, because it has a dependency to win-system32.
This can help:
https://developers.de/2018/01/22/how-to-use-system-drawing-in-net-core/

[antoniovs1029]

Hi, @adriansd27 .

Has following @ddobric 's suggestion helped you to fix this issue?

If not, can you please share a .zip containing your project and some sample data so that I can try to reproduce your problem? I'll need to know what your pipeline looks like, and how you're loading the images, so having a full repro would be very helpful. It would also help if you try to run your same code not in an ASP.Net Core app but actually on a Console App, and let us know the results.

I believe that if you're able to use PredictionEnginePool without having any problem, then this might actually be a bug, because PredictionEnginePool would also be using System.Drawing.Bitmap and it would be doing so without problems... so there's no reason for Evaluate to run into problems if PredictionEnginePool isn't.

I also wonder if this is related to #4126 . I've recently fixed that issue, but my fix will be available only on the next release. There, the user was also getting a similar stack trace to the one you're getting here... but there, the user was using in-memory images (i.e. they weren't using ImageLoadingTransformer) and the bug was that the ImageResizingTransformer was incorrectly disposing the last image in the dataset, making it unusable for later uses. Here, by looking at your stack trace, it seems you're not using in-memory images and you're actually using ImageLoadingTransformer so issue #4126 shouldn't be related to your issue as far as I know, so this bit is confusing to me, as your case is somewhat similar to that other case.

Another recommendation is to check if all your image files are in good state, have you actually tried to run PredictionEnginePool over all of them? perhaps there's some problem with only particular files or some particular filenames.

[adriansd27]

Hey @antoniovs1029 ,

Unfortunately I'm not able to share the solution zip. I can assure you that all the images are in good state, I've tried this with different datasets.

Cheers,
Adrian

[antoniovs1029]

I see. @adriansd27 could you please share a simplified solution where your error is reproduced using some mock image dataset?

Or at least share what your pipeline looks like, as well as to how are you creating the testImageDataTags enumerable. I'd imagine there's some problem in here.

Also I'd like to know if your pipeline ends with a tensorflow estimator, or not? Have you tried also changing your pipeline to see if it works with another kind of pipeline?

Thanks.

[adriansd27]

I've made my own metrics implementation, so I'm not able to test this anymore. I've also switched to in memory loading of images. However, at some point, I've encountered this problem again when evaluating the model with my own metrics methods, at this line of code:

var imagePrediction = _predictionEnginePool.Predict("ImageClassificationModel", image);

I've applied your fix for yielding list over the parameter of that method and the exception started to occur in the yield list function.

I've been able to fix it by using pure functions for every method that used bitmap. Since ASP.NET Core apps are parallel, using impure functions with bitmap type may cause errors.

So the root of the cause might be using impure functions with bitmap images while running in an asynchronous app.

[frank-dong-ms-zz]

Close this issue as seems we are lack of information to repro this issue so no further action from our end. Feel free to reopen if necessary, thanks.