Pretrained DNN Image Featurization

[vaeksare]

Support for a DNN Image Featurizer Transform is to be added to ML.NET. This will allow users to use 1 of 4 pretrained DNN models (ResNet18, ResNet50, ResNet101, and AlexNet) trained on ImageNet in order to featurize an input image.

This transform will use the ONNX Transform as the backbone of doing input preprocessing and applying the pretrained DNN model.

[vaeksare]

@markusweimer

[wschin]

Do we want to have those pre-trained model as a separated nuget? I'd expect their sizes are not small.

[mdabros]

@wschin From a users point of view, I would prefer to have each pre-trained model in separate nuget package. Both to minimize the size of the packages, but also to have a separate "assembly" version for each model, so you can easily track if a pre-trained model have been updated.
Changes/updates to the pre-trained models can have significant impact on the ML products build on top of those.

[justinormont]

I would recommend placing the pre-trained models in the CDN and downloading them on first use. This is the method used elsewhere in ML.NET.

We should consider how/where to store the original models, ONNX conversion script, and instructions, so that a user can re-create the ONNX files. Perhaps we can start another repo (eg: https://github.com/dotnet/machinelearning-resources/) to store these larger files in a more accessible form.

[markusweimer]

@justinormont, I'd like to understand the CDN idea a bit better. When would the models be downloaded from the CDN? I can see a couple distinct times during the creation of a pipeline:

• When installing the NuGet into the app.
• When referencing a specific model the first time in code.
• When training a model referencing a specific model the first time.

I don't see a huge win of a CDN over the NuGet approach. Either way, the model should become part of the app after compilation and training. Downloading resources in an app from a CDN is problematic, as the app developer looses control over that part of their app that way.

[Zruty0]

@vaeksare , are you currently working on it? If yes, please assign to yourself.

[vaeksare]

At this point, it seems like the NuGet approach is the best way, with an individual NuGet being created for each model (and the baseline Transform code residing in the OnnxTransform NuGet since that is the minimum required to run it, and the actual transform code is so small that it doesn't make sense to put it into its own separate distribution). @Zruty0 you mentioned extension methods earlier, do we believe those are the best way to achieve functionality for each additional model?

[Zruty0]

Yes :)

[vaeksare]

Actually a slight concern regarding the extension method approach. Currently, if we are to stick to how other similar transforms work (and how this transform worked in the past), the model to use would be passed in to the constructor of the transform as an argument. In this case, If we use this approach again, I don't believe the implementation can be done using extension methods. Rather, the user would be told to/need to download the correct NuGet containing that transform in order for model argument that is passed in to work.

[Zruty0]

As far as I understand, currently the 'model to use' is passed as an enum to the constructor of the transform.

Since we cannot have 'extension enums' in the language, but we do have extension methods, the suggestion was to change the creation interface.
For example, before it was:

new DnnImageFeaturizer(model: DnnImageModel.ResNet) // DnnImageModel is an enum

and now it could be

new DnnImageFeaturizer(m => m.ResNet()) // here, ResNet() is an extension to some 'model selector' subclass
// or
new DnnImageFeaturizer().ResNet() // here, ResNet() is an extension to DnnImageFeaturizer itself