(WIP) Initial implementation of the new videoReader API (#2683)

* adding base files

* setup modification to actually build the thing

* video api constructor registration

* FAIL metadata

* FAIL update for QS

* revert

* debugging with Victor

* adding base files

* setup modification to actually build the thing

* video api constructor registration

* FAIL metadata

* FAIL update for QS

* revert

* debugging with Victor

* metadata registration works

* API build next

* test

* Merge change

* formatting parameters to avoid the segfault

* next now works on a video

* make size of the output tensor format dependent

* Make next work on audio stream only as well

* refactoring the _setCurrentStream param

* Fixing the last frame return and sensor

* todo docs

* Formatting

* cleanup and comments

* introducing new tests for the API

* cleanup

* Comment out unnecesary format (will add following FFMPEG fix)

* Reformat parsing function

* removing the seek bug `get_decoder_params`

* Removing unnecessary code/variables

* enforce RGB24 as a reading format (will crash before ffmpeg fix)

* permute the dimensions to return (RGB x H x W)

* Changing the return type to std::tuple<torch::Tensor, double> as opposed to tensor list

* Adjusting tests for the new return type

* remove unnecessary jitter

* clangangangang

* Metadata return changes (#1)

* remove implicit calls to set a current stream (#2)

* Adding new tests to check the accuracy of the seek

* cleanup debugging statements

* adding base files

* setup modification to actually build the thing

* video api constructor registration

* FAIL metadata

* FAIL update for QS

* revert

* debugging with Victor

* adding base files

* video api constructor registration

* FAIL metadata

* FAIL update for QS

* revert

* debugging with Victor

* metadata registration works

* API build next

* test

* Merge change

* formatting parameters to avoid the segfault

* next now works on a video

* make size of the output tensor format dependent

* Make next work on audio stream only as well

* refactoring the _setCurrentStream param

* Fixing the last frame return and sensor

* todo docs

* Formatting

* cleanup and comments

* introducing new tests for the API

* cleanup

* Comment out unnecesary format (will add following FFMPEG fix)

* Reformat parsing function

* removing the seek bug `get_decoder_params`

* Removing unnecessary code/variables

* enforce RGB24 as a reading format (will crash before ffmpeg fix)

* permute the dimensions to return (RGB x H x W)

* Changing the return type to std::tuple<torch::Tensor, double> as opposed to tensor list

* Adjusting tests for the new return type

* remove unnecessary jitter

* clangangangang

* Metadata return changes (#1)

* remove implicit calls to set a current stream (#2)

* Adding new tests to check the accuracy of the seek

* cleanup debugging statements

* Addressing PR comments

* addressing Francisco's comments

* CLANG build formatting

* Updated testing to test against pyav for the video tensor reads

* Formatting

* remove pyav from pip deps and add it to conda build

* add pyav and ffmeped to conda builds

* Formatting?

* Setting up linter once and for all hopefully

* Testing pyav

* Fix to 8.0.0

* Try 6.2.0

* See what happens with av from pip

* Remove FFMPEG blocker

* What is going on?

* More tests

* Forgot something

* unblocker

* Check if cache is messing up with things

* Now try with different ffmpeg

* Now try with different ffmpeg

* Testing pyav

* Fix to 8.0.0

* Try 6.2.0

* See what happens with av from pip

* What is going on?

* More tests

* Forgot something

* Check if cache is messing up with things

* Now try with different ffmpeg

* Now try with different ffmpeg

* Do not install av

* Test with ffmpeg 4.2

* clean up video tests

* cleaning up the tests a bit to better test partial reading

* arrgh linter

* Forgot the av test

* forgot av test

* checkout build files from master

* revert circleci

* addressing Franciscos comments

* addressing Franciscos comments

* Ignore ffmpeg in travis

Co-authored-by: Francisco Massa <[email protected]>
Co-authored-by: Edgar Andrés Margffoy Tuay <[email protected]>

Author: 3 people

.travis.yml

@@ -38,7 +38,6 @@ before_install:
     fi
   - conda install av -c conda-forge
 
-
 install:
   # Using pip instead of setup.py ensures we install a non-compressed version of the package
   # (as opposed to an egg), which is necessary to collect coverage.
@@ -55,7 +54,7 @@ install:
     cd -
 
 script:
-  - pytest --cov-config .coveragerc --cov torchvision --cov $TV_INSTALL_PATH -k 'not TestVideoReader and not TestVideoTransforms and not TestIO' test --ignore=test/test_datasets_download.py
+  - pytest --cov-config .coveragerc --cov torchvision --cov $TV_INSTALL_PATH -k 'not TestVideo and not TestVideoReader and not TestVideoTransforms and not TestIO' test --ignore=test/test_datasets_download.py
   - pytest test/test_hub.py
 
 after_success:

setup.py

@@ -347,10 +347,13 @@ def get_extensions():
         base_decoder_src_dir = os.path.join(this_dir, 'torchvision', 'csrc', 'cpu', 'decoder')
         base_decoder_src = glob.glob(
             os.path.join(base_decoder_src_dir, "*.cpp"))
+        # Torchvision video API
+        videoapi_src_dir = os.path.join(this_dir, 'torchvision', 'csrc', 'cpu', 'video')
+        videoapi_src = glob.glob(os.path.join(videoapi_src_dir, "*.cpp"))
         # exclude tests
         base_decoder_src = [x for x in base_decoder_src if '_test.cpp' not in x]
 
-        combined_src = video_reader_src + base_decoder_src
+        combined_src = video_reader_src + base_decoder_src + videoapi_src
 
         ext_modules.append(
             CppExtension(
@@ -359,6 +362,7 @@ def get_extensions():
                 include_dirs=[
                     base_decoder_src_dir,
                     video_reader_src_dir,
+                    videoapi_src_dir,
                     ffmpeg_include_dir,
                     extensions_dir,
                 ],