ImageLoadingTransformer hides exceptions in Mapper.MakeGetter

[mareklinka]

System information

• OS version/distro: Windows 10 1809
• .NET Version (eg., dotnet --info): 2.2.202

Issue

I recently tried to use ML.NET in a Xamarin-based UWP app. I targeted the earliest version of UWP that support .NET Standard 2.0 and everything installed correctly. My intention was to make use a separately-trained TensorFlow model to predict some data from an image.

While my code worked without issues in a .NET Core 2.2 console app, in the UWP it was failing. I was constantly getting an exception when attempting to load the target image - Image [whatever] was not found. No additional details, no inner exception.

Only after a fair amount of headscratching and trying various things I managed to find the culprit (which was obvious in hindsight): UWP does not support bitmaps. After I tried to replicate a little bit of the ImageLoader code, I got an unsupported platform exception.

This is all fair but then looking into the code some more, I found this:

machinelearning/src/Microsoft.ML.ImageAnalytics/ImageLoader.cs

Line 215 in b5a8d99

throw Host.Except($"Image {src.ToString()} was not found.");

The code is catching all exceptions and throwing a custom exception in their place, without providing the base exception. I think this could be improved as part of making the API surface friendlier to use - spending an hour on this, I started to think I was doing something insanely wrong. The inner exception would have told me the root cause in 10 seconds flat.

So there are basically two issues:

1. Hiding the base exceptions
2. Image analytics won't work in UWP at all because of bitmaps

I think a fix for 1. could be relatively simple. 2. will be much more difficult and might not be desirable but I wanted to throw it out there. Maybe using a platform-agnostic implementation of image handling could be useful.

Opinions? Thoughts?

[Ivanidzo4ka]

@eerhardt @danmosemsft Can you provide information regarding System.Drawing.Common and it's UWP support?
If it doesn't support UWP what is best alternative for System.Drawing?

[eerhardt]

@safern @JeremyKuhne @nattress - do you have any recommendations here on using System.Drawing on UWP?

[safern]

@safern @JeremyKuhne @nattress - do you have any recommendations here on using System.Drawing on UWP?

Not really, Drawing is just a platform not supported assembly in UWP, we basically just throw there. I don't know if there are any plans on supporting it in the near future (even if we can because of all the P/Invokes to GDI+).

[Ivanidzo4ka]

So #1, yes it's a bug, and it need to be fixed
#2 We will come up with plan how to enable image support in UWP. Our whole UWP story isn't great and I guess we would tackle this issue during work on UWP support.

[labChariot]

I was working with the exact same setup - Xamarin-based UWP utilising a previously trained TensorFlow model - and hit the PlatformNotSupportedException for System.Drawing when calling into the PredictionEngine:

System.PlatformNotSupportedException: 'System.Drawing is not supported on this platform.'

This exception was originally thrown at this call stack:
    System.Drawing.Bitmap.Bitmap(string)
    Microsoft.ML.Data.ImageLoadingTransformer.Mapper.MakeGetterImageDataViewType.AnonymousMethod__0(ref System.Drawing.Bitmap)
    Microsoft.ML.Transforms.Image.ImageResizingTransformer.Mapper.MakeGetter.AnonymousMethod__1(ref System.Drawing.Bitmap)
    Microsoft.ML.Transforms.Image.ImagePixelExtractingTransformer.Mapper.GetGetterCore.AnonymousMethod__1(ref Microsoft.ML.Data.VBuffer<TValue>)
    Microsoft.ML.Transforms.TensorFlowTransformer.TensorValueGetterVec<T>.GetTensor()
    Microsoft.ML.Transforms.TensorFlowTransformer.Mapper.UpdateCacheIfNeeded(long, Microsoft.ML.Transforms.TensorFlowTransformer.ITensorValueGetter[], string[], Microsoft.ML.Transforms.TensorFlowTransformer.Mapper.OutputCache)
    Microsoft.ML.Transforms.TensorFlowTransformer.Mapper.MakeGetter.AnonymousMethod__4(ref Microsoft.ML.Data.VBuffer<T>)
    Microsoft.ML.Data.TypedCursorable<TRow>.TypedRowBase.CreateDirectVBufferSetter.AnonymousMethod__0(TRow)
    Microsoft.ML.Data.TypedCursorable<TRow>.TypedRowBase.FillValues(TRow)
    Microsoft.ML.PredictionEngineBase<TSrc, TDst>.Predict(TSrc)
    ...
    [Call Stack Truncated]

It's my first time doing anything with ML.NET. Has there been any plan formulated to bring this sort of image support to ML.NET on UWP?

[antoniovs1029]

Hi, @labChariot . As of now, ML.NET doesn't have any plan to provide image support on UWP.

[labChariot]

Thanks @antoniovs1029

[brianfeucht]

ImageSharp just released RC2. I would love an implementation of the ImagePixelExtractingTransformer which is based of ImageSharp instead of GDI

I started to take a look at implementing something locally, but access modifiers on the base classes make that difficult. I haven't had time to look it is beyond that.

[XorZy]

Is there any progress on migrating from System.Drawing to ImageSharp?
I saw it was mentioned on the Roadmap but it's not really clear if it's still being worked on.
It's a very desirable change since NET7 will not support System.Drawing at all (even with a configuration switch) on any platform other than Windows.

[dakersnar]

cc @luisquintanilla

[mapo80]

Same issue here. Any update?

[luisquintanilla]

There is a PR #6363 that should address this issue.