development plan of "functional_tensor"

[Nic-Ma]

❓ Questions and Help

Hi torchvision team,

This is Nic from NVIDIA, thanks for sharing your great work on data processing solutions!

1. I saw you developed "functional_tensor.py" to support Tensor type data but didn't find where it is used in transforms, may I know the reason?
2. And what's your future plan of transforms for Numpy and Tensor data type?
   Actually, I found 2 Tensor only transforms, others are for PIL or numpy.
3. If you want to support both Tensor and Numpy for all transforms, explicitly ask users to select transform for Numpy or for Tensor?
   Or implicitly detect data type in transforms and use "function.py" or "function_tensor.py"?

Thanks in advance.

[fmassa]

Hi,

Great questions! The some of the ideas of the existence of functional_tensor.py are exposed in #1375 , and we should consider it as work-in-progress right now.

Once the transforms work in functional_tensor.py is finished (i.e., parity with functional.py), we will adapt transforms.py to support both PIL images and PyTorch tensors, in a way that is torchscriptable.
This might mean converting the transforms to inherit from nn.Module instead of object.
So the transforms would look like

# in Rotate
if _is_pil_image(input):
    return pil_rotate(input, ...)
return tensor_rotate(input, ...)

# define `pil_rotate` with `torch.jit.unused` so that
# it works in torchscript

The plan is that all transforms should support tensors as arguments. This means that it will work on both CPU and CUDA, and be torchscriptable).
There are currently no plans to support numpy ndarray in the transforms, except in conversion functions such as ToTensor.

The current missing functions that we need for parity mostly require changes in PyTorch, in order to add uint8 support for interpolate, as well as (if possible) efficient support for channels_last format.

Plus, we are also working on adding image decoding operators in torchvision, which will mean that the entire pipeline can be torchscripted and work on torch Tensors.

In the long term, the idea would be that with the help of NestedTensor, we would be able to do the preprocessing of batches on the CPU / GPU in parallel in an efficient way, while using the same abstractions. But that is still far in the future.

Let me know if you have questions, or if there are aspects of this that interests you and you would like to collaborate on some points.

[Nic-Ma]

Hi @fmassa ,

Thanks very much for your detailed explanation!
And glad to see that we are in a similar direction for Tensor transforms.
Actually, we are developing a medical domain-specific AI toolkit MONAI which contains many numpy based transforms for medical images(especially higher dimensions, medical image use cases, etc.): transforms for array and transforms for dict.

And we are evaluating whether it's the right time to re-implement Tensor based transforms to be compatible with existing numpy transforms or replace them directly. We just completed the alpha version and don't have back-compatible problems, do you have any suggestion here?

We are facing exactly the same Pros(GPU, tracing, autodiff, etc) and Cons(NCHW, cases not support, etc.) as you shared in #1375

May I know the schedule of your development plan? For example, when do you guys plan to release the compatible version torchvision?

Thanks.

[fmassa]

Hi @Nic-Ma

We are planning on allowing all transforms in torchvision to have tensor support for next release, which should be in the next 3-4 months. This won't include nested tensor support, but only normal Tensors for now.

We just completed the alpha version and don't have back-compatible problems, do you have any suggestion here?

I would say that in the future, the tensor APIs might converge and that interoperability will be possible. It's already the case between some operations in PyTorch and numpy, where you can pass torch tensors to numpy functions and get back torch tensors (for example, np.sin(torch.rand(2)) works fine).
For now, I think if you want to support both numpy and PyTorch, there might be a need to duplicate parts of the API, which might not be desirable. But if you are ok keeping only the PyTorch implementation, it might in the future also support numpy arrays (via implicit convertion).

Let me know if you have further questions.

[Nic-Ma]

Hi @fmassa ,

Thanks very much for your sharing!
We totally understand your strategy and schedule now.
Will evaluate the solution effort and risk for our MONAI project again.
And looking forward to the next version torchvision!
Thanks.

[fmassa]

This work has been finished with the help from @vfdev-5 , and will be present in the next release of torchvision.