[NOMERGE] Feedback Transforms API implementation

[datumbox]

No description provided.

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit dc7d0d2 (more details on the Dr. CI page):

---

• 2/2 failures introduced in this PR

---

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

unittest_prototype (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

�[31m============================== �[31m�[1m5 ...0.22s�[0m�[31m ===============================�[0m

    raise TypeError(f"{cls} is not a generic class")
TypeError: <class 'torch.utils.data.datapipes.iter.grouping.ShardingFilterIterDataPipe'> is not a generic class
------ generated xml file: /home/circleci/project/test-results/junit.xml -------
=========================== short test summary info ============================
ERROR test/test_prototype_builtin_datasets.py - TypeError: <class 'torch.util...
ERROR test/test_prototype_datasets_api.py - TypeError: <class 'torch.utils.da...
ERROR test/test_prototype_datasets_utils.py - TypeError: <class 'torch.utils....
ERROR test/test_prototype_models.py - TypeError: <class 'torch.utils.data.dat...
ERROR test/test_prototype_transforms_functional.py - TypeError: <class 'torch...
!!!!!!!!!!!!!!!!!!! Interrupted: 5 errors during collection !!!!!!!!!!!!!!!!!!!!
�[31m============================== �[31m�[1m5 errors�[0m�[31m in 0.22s�[0m�[31m ===============================�[0m


Exited with code exit status 2

---

1 failure not recognized by patterns:

Job	Step	Action
lint_python_and_config	Lint Python code and config files	🔁 rerun

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[pmeier]

Right now there is no custom feature type for videos and the tentatively plan is to treat videos as Image's with an extra batch dimensions cc @bjuncek. Since the kernel should be agnostic to batch dimensions, they are automatically supported by all image transforms.

[datumbox]

We should finalize this plan soon. We got outstanding work with Videos so we need to be able to have an idea of how this works for them. Once the plan is finalized, we will be able to parallelize the work of porting all transforms concurrently.

I agree about the underlying kernels being agnostic, sounds good.

[pmeier]

For reference, the initial support for videos happens in #5335. @bjuncek is checking with other researchers working on videos if the API is sufficient or if we are missing something important.

[bjuncek]

Annoyingly, I still haven't gotten a reply from any of them.
@datumbox , maybe you could help me offline in order to make this discussion happen?

tentatively plan is to treat videos as Image's with an extra batch dimensions

Is there a downside to creating a video feature class that for the time being implement just this, but would be possible to extend with stuff like audio or cc-data or encoded bytes if support for this will be needed?

[datumbox]

@bjuncek ping me offline with the names of the people you try to talk to to see if I can help.

When we say "extra branch dimensions", could you clarify the proposal?

[pmeier]

I don't think there is real downside other than being more verbose in our transforms implementations. For example, we would need to have resize_video that simply defers to resize_image. This has to happen for every single image / video kernel.

[datumbox]

Spoke offline with @pmeier and I'm going to close this. The comments will be addressed in a follow up PR.