Using VideoClips for loading a video dataset

[ekosman]

🐛 Describe the bug

Basically, I'm trying to load Kintetics400 using the VideoClips module. However, I encounter this issue. It is important to note that this error does not occur when I try to load a subset of the dataset (e.g. only 100 classes).

This is my code:

self.root = dataset_path
        self.show_errors = False
        classes = list(sorted(list_dir(dataset_path)))
        class_to_idx = {classes[i]: i for i in range(len(classes))}
        self.samples = make_dataset(self.root, class_to_idx, extensions, is_valid_file=None)
        self.classes = classes
        self.steps_between_frames = steps_between_frames
        self.total_clip_duration_in_frames = frames_per_clip * steps_between_frames
        video_list = [x[0] for x in self.samples]
        precomputed_metadata_path = path.join(dataset_path, 'metadata.pt')
        flag_metadata_exists = False
        if path.exists(precomputed_metadata_path):
            _precomputed_metadata = torch.load(precomputed_metadata_path)
            flag_metadata_exists = True

        logging.info("Preparing VideoClips...")
        self.video_clips = VideoClips(
            video_list,
            self.total_clip_duration_in_frames,
            step_between_clips,
            None,
            _precomputed_metadata,
            num_workers=num_workers,
            _video_width=_video_width,
            _video_height=_video_height,
            _video_min_dimension=_video_min_dimension,
            _audio_samples=_audio_samples,
            _audio_channels=_audio_channels,
        )

Output:

  6%|▌         | 764/12927 [26:36<5:28:32,  1.62s/it]2021-09-02 18:14:07,979 moov atom not found
  6%|▌         | 765/12927 [26:49<17:16:11,  5.11s/it]2021-09-02 18:14:21,003 moov atom not found
 19%|█▉        | 2478/12927 [1:13:44<3:29:40,  1.20s/it]2021-09-02 19:01:22,169 error while decoding MB 22 42, bytestream -14
2021-09-02 19:01:22,169 Invalid NAL unit size (0 > 5459).
2021-09-02 19:01:22,170 Error splitting the input into NAL units.
 23%|██▎       | 2945/12927 [1:25:48<9:58:30,  3.60s/it] 2021-09-02 19:20:02,882 moov atom not found
 49%|████▊     | 6282/12927 [2:54:43<10:47:10,  5.84s/it]2021-09-02 20:42:24,446 Invalid NAL unit size (-2025917314 > 14478).
2021-09-02 20:42:24,447 missing picture in access unit with size 14482
2021-09-02 20:42:24,447 Invalid NAL unit size (-2025917314 > 14478).
2021-09-02 20:42:24,447 Error splitting the input into NAL units.
 98%|█████████▊| 12630/12927 [5:45:26<09:35,  1.94s/it]2021-09-02 23:33:00,092 moov atom not found
 98%|█████████▊| 12642/12927 [5:45:35<05:53,  1.24s/it]2021-09-02 23:33:17,606 moov atom not found
 98%|█████████▊| 12643/12927 [5:45:46<07:46,  1.64s/it]
Traceback (most recent call last):
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torchvision/datasets/video_utils.py", line 146, in _compute_frame_pts
    for batch in dl:
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
    return self._process_data(data)
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
    data.reraise()
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 1.
Original Traceback (most recent call last):
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torchvision/datasets/video_utils.py", line 65, in __getitem__
    return read_video_timestamps(self.video_paths[idx])
  File "/home/eitank/anaconda3/envs/torch/lib/python3.7/site-packages/torchvision/io/video.py", line 354, in read_video_timestamps
    video_fps = float(video_stream.average_rate)
TypeError: float() argument must be a string or a number, not 'NoneType'

python-BaseException
Backend Qt5Agg is interactive backend. Turning interactive mode on.
2021-09-02 23:33:26,149 Invalid NAL unit size (1031162844 > 19556).
2021-09-02 23:33:26,149 missing picture in access unit with size 19560
2021-09-02 23:33:26,149 Invalid NAL unit size (1031162844 > 19556).
2021-09-02 23:33:26,149 Error splitting the input into NAL units.
2021-09-02 23:33:30,536 moov atom not found

Process finished with exit code 1

Versions

Collecting environment information...
PyTorch version: 1.7.1+cu110
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.18.0
Libc version: glibc-2.9

Python version: 3.7.0 (default, Oct  9 2018, 10:31:47)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-80-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: GeForce RTX 3090
GPU 1: GeForce RTX 3090

Nvidia driver version: 460.91.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.0
[pip3] pytorch-lightning==1.3.3
[pip3] torch==1.7.1+cu110
[pip3] torch-cluster==1.5.9
[pip3] torch-geometric==1.7.2
[pip3] torch-scatter==2.0.7
[pip3] torch-sparse==0.6.9
[pip3] torch-spline-conv==1.2.1
[pip3] torchaudio==0.7.2
[pip3] torchmetrics==0.3.2
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.8.2+cu110
[conda] _tflow_select             2.3.0                       mkl    defaults
[conda] blas                      1.0                         mkl    conda-forge
[conda] mkl                       2020.2                      256    defaults
[conda] mkl-service               2.3.0            py37he904b0f_0    defaults
[conda] mkl_fft                   1.2.0            py37h23d657b_0    defaults
[conda] mkl_random                1.1.1            py37h0573a6f_0    defaults
[conda] numpy                     1.21.0                   pypi_0    pypi
[conda] numpy-base                1.19.1           py37hfa32c7d_0    defaults
[conda] pytorch-lightning         1.3.3                    pypi_0    pypi
[conda] tensorflow                2.0.0           mkl_py37h66b46cc_0    defaults
[conda] tensorflow-base           2.0.0           mkl_py37h9204916_0    defaults
[conda] torch                     1.7.1+cu110              pypi_0    pypi
[conda] torch-cluster             1.5.9                    pypi_0    pypi
[conda] torch-geometric           1.7.2                    pypi_0    pypi
[conda] torch-scatter             2.0.7                    pypi_0    pypi
[conda] torch-sparse              0.6.9                    pypi_0    pypi
[conda] torch-spline-conv         1.2.1                    pypi_0    pypi
[conda] torchaudio                0.7.2                    pypi_0    pypi
[conda] torchmetrics              0.3.2                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.8.2+cu110              pypi_0    pypi

[bjuncek]

Hmmm - it looks to me as if one of the videos are not properly encoded and due to the failure, either on reading or on video side the videoclip generation fails.

Just to clarify, are you using the videos downloaded from the official repo?

[ekosman]

Sure @bjuncek. After debugging through VideoClips, I was able to identify the corrupted video file. However, I was wondering if there's any possibility to identify corrupted files automatically during this loading. Right now, the exception is not very informative and it is hard to identify the source of the problem without manually debugging the internals of VideoClips by placing a breakpoint in the relevant method.

As per your question, I use the list of video ids provided in DeepMind's website and you youtube-dl for downloading the files. Specifically, I use this code: https://github.com/Showmax/kinetics-downloader

[bjuncek]

Yeah, I think that makes sense. Could you attach the video in question? I'll make sure there is some workaround for this :)

Also, there is an official video list provided by Joao at CVDF https://github.com/cvdfoundation/kinetics-dataset which claims to fix some of the corrupted videos. Just FYI

[ekosman]

Many thanks @bjuncek ! I wasn't aware of that repo.

Regarding the corrupted video, I already replaced it and don't have it anymore :(

[bjuncek]

I've just tried this on several videos with several python versions and it seems resolved.
I'll close it, but feel free to re-open if you run into any issues?