MONAI icon indicating copy to clipboard operation
MONAI copied to clipboard

Published 20 hours ago • verified publisher icon Project-MONAI

Video dataset

Open rijobro opened this issue 2 years ago • 35 comments

Addresses: https://github.com/Project-MONAI/MONAI/issues/4746.

Description

Adds video datasets. Videos can be from file or capture device (e.g., webcam).

Requires a corresponding tutorial.

Status

Hold

Types of changes

  • [x] Non-breaking change (fix or new feature that would not break existing functionality).
  • [x] New tests added to cover the changes.

rijobro avatar Jul 26 '22 15:07 rijobro

Hi @binliunls ,

I think this great PR almost achieved all the necessary features of the video loading part. Please take a look and maybe test it locally in your use case.

Thanks in advance.

Nic-Ma avatar Jul 27 '22 08:07 Nic-Ma

Hi @rijobro, I had a few tests on this videodataset this week. However my program stuck when I try to use dataloader whose num_workers equals to 1 or above to load data with the videodataset. I am currently finding the reason why this happened. And I think maybe this would be helpful for you too.

binliunls avatar Aug 04 '22 06:08 binliunls

Hi @binliunls ,

Do you mean DataLoader or ThreadDataLoader with use_thread_workers=True? I think this line in this PR is not thread safe:

self.cap.set(cv2.CAP_PROP_POS_FRAMES, index)

And GPU transforms can't work in multi-processing in your use case.

So if num_workers > 0, both multi-thread and multi-processing workers should not work in theory here. @rijobro right?

Thanks.

Nic-Ma avatar Aug 04 '22 07:08 Nic-Ma

Hi @binliunls ,

Do you mean DataLoader or ThreadDataLoader with use_thread_workers=True? I think this line in this PR is not thread safe:

self.cap.set(cv2.CAP_PROP_POS_FRAMES, index)

And GPU transforms can't work in multi-processing in your use case.

So if num_workers > 0, both multi-thread and multi-processing workers should not work in theory here. @rijobro right?

Thanks.

Hi @Nic-Ma : I tried DataLoader and ThreadDataLoader. They both stuck when set the number workers >= 1. @rijobro Could you please double check this problem, in case I did something wrong with the environment or the code.

binliunls avatar Aug 04 '22 07:08 binliunls

Hi @binliunls ,

Are you execute transforms on GPU? It should not work with num_workers > 0 in DataLoader.

Thanks.

Nic-Ma avatar Aug 04 '22 07:08 Nic-Ma

Hi @binliunls ,

Are you execute transforms on GPU? It should not work with num_workers > 0 in DataLoader.

Thanks.

I tried to move the transform from GPU to CPU and the program still stuck.

binliunls avatar Aug 04 '22 07:08 binliunls

Hi @binliunls ,

I think your multi-processing issue may be similar to: https://www.pythonfixing.com/2022/05/fixed-python-opencv-multiprocessing.html Let's keep using num_workers=0 with ThreadDataLoader and GPU transforms so far. @rijobro Could you please help confirm the issue? If True, maybe we need to highlight it in the doc-string of VideoDataset?

Thanks in advance.

Nic-Ma avatar Aug 04 '22 07:08 Nic-Ma

Hi @binliunls ,

I think your multi-processing issue may be similar to: https://www.pythonfixing.com/2022/05/fixed-python-opencv-multiprocessing.html Let's keep using num_workers=0 with ThreadDataLoader and GPU transforms so far. @rijobro Could you please help confirm the issue? If True, maybe we need to highlight it in the doc-string of VideoDataset?

Thanks in advance.

Hi @Nic-Ma: The link below solved my problem. It may cause problems when the main process and children process share the same obeject, which is the video "cap" object in our case. I moved the "open_video" function call from the init function to functions like "get_frame", where the "cap" is needed . The new program can run. https://stackoverflow.com/questions/62126858/python-multiprocessing-and-opencv-videocapture-read-error

binliunls avatar Aug 04 '22 08:08 binliunls

Hi @binliunls ,

If you move the "open_video" call to every get frame, will it be slower as it frequently opens and closes?

Thanks.

Nic-Ma avatar Aug 04 '22 08:08 Nic-Ma

Hi @binliunls ,

If you move the "open_video" call to every get frame, will it be slower as it frequently opens and closes?

Thanks. Hi @Nic-Ma Yes it will be 1.5x slower than @rijobro one if num_worker=0&batch_size=4 for both videodatasets. However @rijobro old videodataset will be 1.5x slower if I set num_worker=4 and keep the batch_size=4 for the new one. Thanks.

binliunls avatar Aug 04 '22 09:08 binliunls

Ok, so I will put a multiprocessing argument into the constructor, which is False by default. This will do open_video at construction and only perform once. This will not be threadsafe and the user should use num_workers==0. If True, we do open_video each time a new frame is requested. This will be threadsafe, useful if num_workers>0 but slower if num_workers==0. Does that sound reasonable?

rijobro avatar Aug 05 '22 10:08 rijobro

Hi @rijobro ,

I think it's OK to add this multiprocessing, the solution looks good to me. But I think it's "process safe" not "thread safe", the VideoDataset still can't work with ThreadDataLoader if set use_thread_workers=True. It's not a problem so far.

Thanks.

Nic-Ma avatar Aug 05 '22 11:08 Nic-Ma

@binliunls I've made the various changes (including the RGB option you requested here). Could you give it a go and let me know if it all works as expected?

rijobro avatar Aug 05 '22 12:08 rijobro

@Nic-Ma @wyli I'm seeing circular import errors. Is this specific to this PR or are we seeing this elsewhere? I can't see changes in this code affecting the Convolution class it's mentioning.

rijobro avatar Aug 05 '22 14:08 rijobro

I think it's in this PR only, and I couldn't find out the reason...most likely from the unit tests?

wyli avatar Aug 05 '22 14:08 wyli

@binliunls I've made the various changes (including the RGB option you requested here). Could you give it a go and let me know if it all works as expected?

Hi @rijobro , I will run some tests on this new dataset. Meanwhile I think maybe the channel to front operation in the link below is not necessary, since users can do this by adding "AsChannelFirst" to their transforms. And this will be a block to users who just want to see what the frames look like, because they are going to get a channel first image and have to move the axis again. https://github.com/rijobro/MONAI/blob/2d52bfe27e7dd2f238103cc32f8f1837c1fa8214/monai/data/video_dataset.py#L108

binliunls avatar Aug 05 '22 15:08 binliunls

Hm, I think for pytorch based applications, we always want the channel in front of spatial dimensions. Granted, matplotlib and opencv expect them at the end. I'll put this as an option, too. It's true that we could leave it for the transforms, but I think that it's going to be needed frequently enough to have it done by default.

rijobro avatar Aug 05 '22 15:08 rijobro

Hi @rijobro , just for your information. There is a similar ensure_channel_first arg in the LoadImaged transform: https://github.com/Project-MONAI/MONAI/blob/dev/monai/transforms/io/dictionary.py#L76 Thanks.

Nic-Ma avatar Aug 05 '22 15:08 Nic-Ma

@wyli Any idea why I'm seeing this? The import is nested in an optional_import.

rijobro avatar Aug 05 '22 15:08 rijobro

/black

@wyli Any idea why I'm seeing this? The import is nested in an optional_import.

I see, that's an issue from type annotations and you've addressed that.

wyli avatar Aug 05 '22 16:08 wyli

/build

Hi @Nic-Ma, I can make the minor doc updates in https://github.com/Project-MONAI/MONAI/pull/4838

wyli avatar Aug 08 '22 09:08 wyli

/build

wyli avatar Aug 08 '22 12:08 wyli

/build

wyli avatar Aug 08 '22 12:08 wyli

error from the tests, I suspect the docker image lacks FFmpeg which is under GPL licence @rijobro @binliunls any idea?

[2022-08-08T13:17:56.960Z] ======================================================================
[2022-08-08T13:17:56.960Z] ERROR: test_dataset (tests.test_video_datasets.TestVideoFileDataset)
[2022-08-08T13:17:56.960Z] ----------------------------------------------------------------------
[2022-08-08T13:17:56.960Z] Traceback (most recent call last):
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_video_datasets.py", line 117, in test_dataset
[2022-08-08T13:17:56.960Z]     super().test_dataset(known_num_frames, known_fps)
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_video_datasets.py", line 70, in test_dataset
[2022-08-08T13:17:56.960Z]     ds = self.get_ds(max_num_frames=max_num_frames)
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_video_datasets.py", line 40, in get_ds
[2022-08-08T13:17:56.960Z]     return self.ds(video_source=self.video_source, transform=TRANSFORMS, *args, **kwargs)  # type: ignore
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 127, in __init__
[2022-08-08T13:17:56.960Z]     VideoDataset.__init__(self, *args, **kwargs)
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 74, in __init__
[2022-08-08T13:17:56.960Z]     self.cap = self.open_video(video_source)
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 94, in open_video
[2022-08-08T13:17:56.960Z]     raise RuntimeError(f"Failed to open video: {video_source}")
[2022-08-08T13:17:56.960Z] RuntimeError: Failed to open video: /home/jenkins/agent/workspace/MONAI-premerge/monai/tests/testing_data/endo.mp4
[2022-08-08T13:17:56.960Z] 
[2022-08-08T13:17:56.960Z] ======================================================================
[2022-08-08T13:17:56.960Z] ERROR: test_multiple_sources (tests.test_video_datasets.TestVideoFileDataset)
[2022-08-08T13:17:56.960Z] ----------------------------------------------------------------------
[2022-08-08T13:17:56.960Z] Traceback (most recent call last):
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_video_datasets.py", line 61, in test_multiple_sources
[2022-08-08T13:17:56.960Z]     ds1 = self.get_ds()
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_video_datasets.py", line 40, in get_ds
[2022-08-08T13:17:56.960Z]     return self.ds(video_source=self.video_source, transform=TRANSFORMS, *args, **kwargs)  # type: ignore
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 127, in __init__
[2022-08-08T13:17:56.960Z]     VideoDataset.__init__(self, *args, **kwargs)
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 74, in __init__
[2022-08-08T13:17:56.960Z]     self.cap = self.open_video(video_source)
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 94, in open_video
[2022-08-08T13:17:56.960Z]     raise RuntimeError(f"Failed to open video: {video_source}")
[2022-08-08T13:17:56.960Z] RuntimeError: Failed to open video: /home/jenkins/agent/workspace/MONAI-premerge/monai/tests/testing_data/endo.mp4
[2022-08-08T13:17:56.960Z] 
[2022-08-08T13:17:56.960Z] ======================================================================
[2022-08-08T13:17:56.960Z] ERROR: test_multiprocessing (tests.test_video_datasets.TestVideoFileDataset)
[2022-08-08T13:17:56.960Z] ----------------------------------------------------------------------
[2022-08-08T13:17:56.960Z] Traceback (most recent call last):
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_video_datasets.py", line 54, in test_multiprocessing
[2022-08-08T13:17:56.960Z]     ds = self.get_ds(max_num_frames=100, multiprocessing=multiprocessing)
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_video_datasets.py", line 40, in get_ds
[2022-08-08T13:17:56.960Z]     return self.ds(video_source=self.video_source, transform=TRANSFORMS, *args, **kwargs)  # type: ignore
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 128, in __init__
[2022-08-08T13:17:56.960Z]     num_frames = self.get_num_frames()
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 139, in get_num_frames
[2022-08-08T13:17:56.960Z]     num_frames = int(self._get_cap().get(cv2.CAP_PROP_FRAME_COUNT))
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 99, in _get_cap
[2022-08-08T13:17:56.960Z]     return self.open_video(self.video_source) if self.multiprocessing else self.cap
[2022-08-08T13:17:56.960Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/data/video_dataset.py", line 94, in open_video
[2022-08-08T13:17:56.960Z]     raise RuntimeError(f"Failed to open video: {video_source}")
[2022-08-08T13:17:56.960Z] RuntimeError: Failed to open video: /home/jenkins/agent/workspace/MONAI-premerge/monai/tests/testing_data/endo.mp4
[2022-08-08T13:17:56.960Z] 
[2022-08-08T13:17:56.960Z] ----------------------------------------------------------------------

wyli avatar Aug 08 '22 13:08 wyli

error from the tests, I suspect the docker image lacks FFmpeg which is under GPL licence @rijobro @binliunls any idea?

Yes, I have encountered this problem in docker environment several times. Not only lack the FFMPEG, but also the opencv-python in the docker has some problems to load the ffmpeg if installed. I usually delete the opencv-python in the docker manually and reinstall the ffmpeg and opencv-python to make it work for loading videos. However I haven't tested this using the latest docker, so I cannot say whether will this problem appear in it.

binliunls avatar Aug 08 '22 13:08 binliunls

ok, the VideoDataset module is generic and doesn't strictly require ffmpeg, the test cases need it. do you have simpler test cases?

wyli avatar Aug 08 '22 13:08 wyli

ok, the VideoDataset module is generic and doesn't strictly require ffmpeg, the test cases need it. do you have simpler test cases?

Hi @wyli ,

I think the VideoDataset depends on OpenCV, the get_frame and open_video, etc. How about making the opencv tests as skipped in CI? Like cupy tests? https://github.com/Project-MONAI/MONAI/blob/dev/tests/test_cucim_transform.py#L65

Thanks.

Nic-Ma avatar Aug 09 '22 03:08 Nic-Ma

there is a version of opencv in our docker image release, it's probably from the base image and compiled without ffmpeg. In order to use the video dataset, what would be the installation steps from a docker image user's perspective?

wyli avatar Aug 09 '22 06:08 wyli

Hi @rijobro ,

Would you like to fix the CI issue and let's merge this PR then?

Thanks in advance.

Nic-Ma avatar Aug 10 '22 15:08 Nic-Ma

Will wait to see if this passes CI, added .avi example video in case .mp4 codec missing. If both missing, skip. Also silence stderr error from OpenCV C++.

rijobro avatar Aug 12 '22 12:08 rijobro