DALI
DALI copied to clipboard
NVIDIA
Adds interleaving to videos to read them as specified in #2621
Why we need this PR?
- This PR adds the feature mentioned in #2621. It allows to merge multiple videos into one batch by using a specified interleave size and type. To generate the *_CONTINUOUS behavoir of the PR it was previously necessary to construct multiple VideoReaders. Further, it was previously impossible to rearange video sequences such that they are read simultaneously while still respecting the boundary between different video batches.
What happened in this PR?
-
What solution was applied: The PR changes the order of the frames in VideoLoader by interleaving them. This process is done by picking the frames from different videos in a round robin order. Further, the interleave size specifies the size of the circular buffer that is used to interleave the frames. This parameter is specified when creating a new VideoReader object. For this change a new type was introduced that is used to determine how samples are selected. The options are:
-
SHORTEN_CONTINUOUS Merges all sequences while disregarding boundary inbetween different videos except for the last ones. In this case the last sequences are all shortened to the same size.
Example in between videos: see SHORTEN_CONTINUOUS labels frames [0 1 2 3] [11340 11340 11340 11340] [0 1 6 3] [11370 11370 0 11370] [0 1 6 3] [11400 11400 30 11400] [0 5 6 3] [11430 0 60 11430] [0 5 6 3] [11460 30 90 11460] [0 5 6 3] [11490 60 120 11490] [0 5 6 7] [11520 90 150 0] [4 5 6 7] [ 0 120 180 30] Example at the end of all videos: labels frames [ 8 9 10 11] [10290 10890 10980 10830] [ 8 9 10 11] [10320 10920 11010 10860] [ 0 1 2 3] [ 0 0 0 0] [ 0 1 2 3] [ 30 30 30 30] with video 8 (11100 frames); video 9 (11130 frames); video 10 (11010 frames) and video 11 (11310 frames). -
REPEAT_CONTINUOUS Merges all sequences while disregarding boundary inbetween different videos except for the last ones. In this case the last sequences of the last video are repeated until all video sequences have been read.
Example in between videos: see SHORTEN_CONTINUOUS Example at the end of all videos: labels frames [ 8 9 10 11] [10770 210 420 11310] [ 8 9 10 11] [10800 240 450 0] ... [ 8 9 10 11] [11100 540 750 300] [ 0 1 2 3] [ 0 0 0 0] with video 8 (11100 frames); video 9 (11130 frames); video 10 (11010 frames) and video 11 (11310 frames). -
CLAMP_CONTINUOUS Merges all sequences while disregarding boundary inbetween different videos except for the last ones. In this case the last sequence of a video is repeated until all sequences have been read.
Example in between videos: see SHORTEN_CONTINUOUS Example at the end of all videos: labels frames [ 8 9 10 11] [11070 11130 11010 11310] [ 8 9 10 11] [11100 11130 11010 11310] [ 0 1 2 3] [ 0 0 0 0] [ 0 1 2 3] [ 30 30 30 0] with video 8 (11100 frames); video 9 (11130 frames); video 10 (11010 frames) and video 11 (11310 frames). -
SHORTEN Merges all sequences while keeping the boundary inbetween different videos. In this case all videos are shortened to the shortest video in the current video batch.
Example inbetween videos: labels frames [0 1 2 3] [11310 11310 11310 11310] [0 1 2 3] [11340 11340 11340 11340] [4 5 6 7] [ 0 0 0 0] [4 5 6 7] [ 30 30 30 30] with video 0 (11520 frames), video 1 (11400 frames), video 2 (11340 frames) and video 3 (11490 frames). -
REPEAT Merges all sequences while keeping the boundary inbetween different videos. In this case videos are repeated until the whole video batch has been read.
Example inbetween videos: labels frames [0 1 2 3] [11340 11340 11340 11340] [0 1 2 3] [11370 11370 0 11370] ... [0 1 2 3] [11520 90 150 0] [4 5 6 7] [ 0 0 0 0] with video 0 (11520 frames), video 1 (11400 frames), video 2 (11340 frames) and video 3 (11490 frames). -
CLAMP Merges all sequences while keeping the boundary inbetween different videos. In this case last sequences of the videos are repeated until the whole video batch has been read.
Example inbetween videos: labels frames [0 1 2 3] [11310 11310 11310 11310] [0 1 2 3] [11340 11340 11340 11340] ... [0 1 2 3] [11490 11400 11340 11490] [0 1 2 3] [11520 11400 11340 11490] [4 5 6 7] [ 0 0 0 0] [4 5 6 7] [ 30 30 30 30] with video 0 (11520 frames), video 1 (11400 frames), video 2 (11340 frames) and video 3 (11490 frames).
-
-
Affected modules and functionalities: Changes to: - dali/operators/reader/loader/loader.h - dali/operators/reader/loader/loader.cc - dali/operators/reader/loader/video_loader.h - dali/pipeline/data/types.h - dali/python/backend_impl.cc - dali/python/nvidia/dali/types.py - include/dali/core/common.h
-
Key points relevant for the review: Implementation.
-
Validation and testing: Run tests not sure if more tests are required.
-
Documentation (including examples): Added docstrings for the newly introduced parameters.
JIRA TASK: NA
Hi @TheTimmy, thank you for the contribution.
We will assign the reviewers tomorrow and will get back to you.
In the meantime, can you think about adding a test to this PR? You can find some example tests for video reader (using Python) here: https://github.com/NVIDIA/DALI/blob/master/dali/test/python/test_video_pipeline.py
Some video tests failed in our internal CI:
FAIL: test_video_pipeline.test_shorten_interleave_mode
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/opt/dali/dali/test/python/test_video_pipeline.py", line 635, in test_shorten_interleave_mode
_test_interleave_mode(_test_shorten_interleave_mode)
File "/opt/dali/dali/test/python/test_video_pipeline.py", line 355, in _test_interleave_mode
test_func(size, 40, LONG_ITER)
File "/opt/dali/dali/test/python/test_video_pipeline.py", line 388, in _test_shorten_interleave_mode
assert(time[k] == last_time[k] + sequence_length)
AssertionError:
-------------------- >> begin captured stdout << ---------------------
[0] [0]
[0] [1]
[0] [2]
[0] [3]
[0] [4]
[0] [5]
[0] [0]
[0] [2]
[0] [4]
[0] [6]
[0] [8]
[0] [10]
[0 1 2 3 4] [0 0 0 0 0]
[0 1 2 3 4] [1 1 1 1 1]
[0 1 2 3 4] [2 2 2 2 2]
[0 1 2 3 4] [3 3 3 3 3]
[0 1 2 3 4] [4 4 4 4 4]
[0 1 2 3 4] [5 5 5 5 5]
[0 1 2 3 4] [0 0 0 0 0]
[0 1 2 3 4] [2 2 2 2 2]
[0 1 2 3 4] [4 4 4 4 4]
[0 1 2 3 4] [6 6 6 6 6]
[0 1 2 3 4] [8 8 8 8 8]
[0 1 2 3 4] [10 10 10 10 10]
[0 1 2 3 4] [0 0 0 0 0]
[0 1 2 3 4] [40 40 40 40 40]
[0 1 2 3 4] [80 80 80 80 80]
[0 1 2 3 4] [120 120 120 120 120]
[0 1 2 3 4] [160 160 160 160 160]
[0 1 2 3 4] [200 200 200 200 200]
[0 1 2 3 4] [0 0 0 0 0]
--------------------- >> end captured stdout << ----------------------
======================================================================
FAIL: test_video_pipeline.test_repeat_interleave_mode
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/opt/dali/dali/test/python/test_video_pipeline.py", line 638, in test_repeat_interleave_mode
_test_interleave_mode(_test_repeat_interleave_mode)
File "/opt/dali/dali/test/python/test_video_pipeline.py", line 355, in _test_interleave_mode
test_func(size, 40, LONG_ITER)
File "/opt/dali/dali/test/python/test_video_pipeline.py", line 428, in _test_repeat_interleave_mode
assert(any(time[j] != 0 for j in range(batch_size)))
AssertionError:
-------------------- >> begin captured stdout << ---------------------
[0] [0]
[0] [1]
[0] [2]
[0] [3]
[0] [4]
[0] [5]
[0] [0]
[0] [2]
[0] [4]
[0] [6]
[0] [8]
[0] [10]
[0 1 2 3 4] [0 0 0 0 0]
[0 1 2 3 4] [1 1 1 1 1]
[0 1 2 3 4] [2 2 2 2 2]
[0 1 2 3 4] [3 3 3 3 3]
[0 1 2 3 4] [4 4 4 4 4]
[0 1 2 3 4] [5 5 5 5 5]
[0 1 2 3 4] [0 0 0 0 0]
[0 1 2 3 4] [2 2 2 2 2]
[0 1 2 3 4] [4 4 4 4 4]
[0 1 2 3 4] [6 6 6 6 6]
[0 1 2 3 4] [8 8 8 8 8]
[0 1 2 3 4] [10 10 10 10 10]
[0 1 2 3 4] [0 0 0 0 0]
[0 1 2 3 4] [40 40 40 40 40]
[0 1 2 3 4] [80 80 80 80 80]
[0 1 2 3 4] [120 120 120 120 120]
[0 1 2 3 4] [160 160 160 160 160]
[0 1 2 3 4] [200 200 200 200 200]
[0 1 2 3 4] [0 0 0 0 0]
--------------------- >> end captured stdout << ----------------------
======================================================================
FAIL: test_video_pipeline.test_clamp_interleave_mode
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/opt/dali/dali/test/python/test_video_pipeline.py", line 641, in test_clamp_interleave_mode
_test_interleave_mode(_test_clamp_interleave_mode)
File "/opt/dali/dali/test/python/test_video_pipeline.py", line 355, in _test_interleave_mode
test_func(size, 40, LONG_ITER)
File "/opt/dali/dali/test/python/test_video_pipeline.py", line 468, in _test_clamp_interleave_mode
assert(time[k] == last_time[k] + sequence_length or time[k] == max_time[k])
AssertionError:
-------------------- >> begin captured stdout << ---------------------
[0] [0]
[0] [1]
[0] [2]
[0] [3]
[0] [4]
[0] [5]
[0] [0]
[0] [2]
[0] [4]
[0] [6]
[0] [8]
[0] [10]
[0 1 2 3 4] [0 0 0 0 0]
[0 1 2 3 4] [1 1 1 1 1]
[0 1 2 3 4] [2 2 2 2 2]
[0 1 2 3 4] [3 3 3 3 3]
[0 1 2 3 4] [4 4 4 4 4]
[0 1 2 3 4] [5 5 5 5 5]
[0 1 2 3 4] [0 0 0 0 0]
[0 1 2 3 4] [2 2 2 2 2]
[0 1 2 3 4] [4 4 4 4 4]
[0 1 2 3 4] [6 6 6 6 6]
[0 1 2 3 4] [8 8 8 8 8]
[0 1 2 3 4] [10 10 10 10 10]
[0 1 2 3 4] [0 0 0 0 0]
[0 1 2 3 4] [40 40 40 40 40]
[0 1 2 3 4] [80 80 80 80 80]
[0 1 2 3 4] [120 120 120 120 120]
[0 1 2 3 4] [160 160 160 160 160]
[0 1 2 3 4] [200 200 200 200 200]
[0 1 2 3 4] [0 0 0 0 0]
--------------------- >> end captured stdout << ----------------------
@awolant Thanks for letting me know, I will look into this. This might be a bug in the test code as the output seems to be correct. Would you mind sharing the videos that result in the failure?
@awolant Thanks for letting me know, I will look into this. This might be a bug in the test code as the output seems to be correct. Would you mind sharing the videos that result in the failure?
Please use https://github.com/NVIDIA/DALI_extra (you need LFS) and check how this test is invoked in qa/TL0_videoreader_test/test.sh.
Hi @TheTimmy, Have you got a chance to see what is the issue bending failing tests? After an internal discussion about this PR we think:
- providing a set of reliable tests is a key to allow us to maintain the work you contributed
- extending documentation to provide a compact but complete description of functionality is more than welcome. We can assist in forging the language in a way that is consistent with the rest of the DALI documentation
Hi @TheTimmy, Have you got a chance to see what is the issue bending failing tests? After an internal discussion about this PR we think:
* providing a set of reliable tests is a key to allow us to maintain the work you contributed * extending documentation to provide a compact but complete description of functionality is more than welcome. We can assist in forging the language in a way that is consistent with the rest of the DALI documentation
Hi @JanuszL, Sorry, I'm currently quite busy with other work so I haven't found the time lately. However, as soon as I get some free minutes I will definitly look at it again and check the code/tests.
Hi @JanuszL, Sorry, I'm currently quite busy with other work so I haven't found the time lately. However, as soon as I get some free minutes I will definitly look at it again and check the code/tests.
Sure. Just wanted to check. Just let me know if you need any guidance.