Pre-trained detection models for PyTorch Hub

[chenliu0831]

🚀 Feature

First of all, thanks for the amazing work.

I noticed currently there's no pre-trained detection models exposed via hub_conf.py. It looks like the building blocks are already there. Is there plan to support it?

Thanks

Motivation

Increase coverage of supported vision tasks via Torch Hub.

Pitch

• entrypoints for detection models are added to hub_conf.py
• some helpers e.g. for bbox parsing could also be added there

[fmassa]

Hi,

Adding the detection models in hub_conf.py wouldn't be enough in general, because those models require custom C++ / CUDA ops which are only present in torchvision.
Given that TorchHub doesn't compile the repos it clones, we wouldn't have a full-fleshed detection model with the current setup for TorchHub.

ccing @ailzhang to see if there are other options.

[chenliu0831]

Gotcha. Related, I think Fairseq actually is doing some Cython compilation inside their hubconf.py. Might be an option if the custom op compilation is fast.

[fmassa]

@chenliu0831 understood, that could be a solution for it, although I'm not sure if we should be doing any compilation in the torchhub setup, as it would take a long time to run.

[ailzhang]

@fmassa IMHO it's worth adding to make the detection models more accessible. Users will build for the first time, but as we cache the build, they should be good until they force a rebuild.

[fmassa]

We could try adding support for it, although it will require users to have a C++ compiler + nvcc (if using GPUs), which might not always be the case, so we would need to give good error messages in those cases.

[chenliu0831]

Awesome, I'll take a shot.

[AdeelH]

@chenliu0831, any update on this?

[AdeelH]

@fmassa, I tried adding the detection models to hubconf.py (AdeelH@2b35026) and importing them via torch hub. I found that it mostly works.

Note that I did NOT build either pytorch or torchvision from source. I installed them using

pip3 install torch==1.10.2+cu113 torchvision==0.11.3+cu113 torchaudio==0.10.2+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

This is the code I used to test the imports:

import torch
import torchvision

model = torch.hub.load( 
	'/home/adeel/vision',
	'ssd300_vgg16',
	source='local',
	num_classes=2,
	force_reload=True,
	pretrained=False,
	pretrained_backbone=False).eval()

model(torch.zeros(1, 3, 300, 300))

Observations

• The snippet above fails if I make the hubconf.py changes over the main branch.
• The snippet above succeeds if I make the hubconf.py changes over the 0.11 release, presumably because the version matches my torchvision installation's version.
• The snippet above fails if I remove import torchvision (line 2).

In each failure case, I get this error:

RuntimeError: Couldn't load custom C++ ops. This can happen if your PyTorch and torchvision versions are incompatible, or if you had errors while compiling torchvision from source. For further information on the compatible versions, check https://github.com/pytorch/vision#installation for the compatibility matrix. Please check your PyTorch version with torch.__version__ and your torchvision version with torchvision.__version__ and verify if they are compatible, and if not please reinstall torchvision so that it matches your PyTorch install.

---

What do you make of this? Is it worth adding the detection models to hubconf.py?

[datumbox]

@AdeelH The detection models require custom C++ ops that need compilation. TorchHub can't import models with such dependencies. Right now there is no way to add support. We might look on the future for options but this is quite a sizeable work and it's not going to be done just by adding them on the hubconf.py.

[AdeelH]

Thanks for the reply. I get that, but what I've reported above is that importing via torch hub is in fact working in some cases without having to do any additional compilation. If that is known/expected behavior, then I apologize.

[datumbox]

I believe what you describe is expected. The specific version that you have installed will work because the binaries contain the compiled C++/CUDA ops.

I'm not an expert on the internals of TorchHub. @NicolasHug do you know if the point 2 on the Observation section above is expected?

[NicolasHug]

As @datumbox mentioned in general one should not expect models with compiled extensions to work with torchhub.

However, since you already have torchvision 0.11 installed, these extensions have been compiled already and are present somewhere on your machine.

import torch and import torchvision do quite a bit of magic under the hood, resolving dynamic lib symbols, and all sorts of shenanigans that I don't understand enough. But I would bet that when you're on the 0.11 branch, those symbols get resolved properly, which is why you're able to load the models in this specific case (and why it still fails if import torchvision is removed)