MONAI icon indicating copy to clipboard operation
MONAI copied to clipboard

SharedCacheDataset

Open myron opened this issue 2 years ago • 0 comments

This introduces a new SharedCacheDataset(Dataset) class to address https://github.com/Project-MONAI/MONAI/issues/3843

It has several improvements / changes compared to the CacheDataset https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/dataset.py#L690

  • It caches into a shared memory in RAM, so all processes in multigpu run with DistributedDataParallel can read/write to the same shared memory (globally accessible)
  • it works on the fly (there is non need to wait to pre-cache the data), it will cache it on the first data access, and then use it from the cache

If tested it and used for many tasks already in DDP / multigpu I'll add unit tests too if it looks good

Types of changes

  • [x] Non-breaking change (fix or new feature that would not break existing functionality).
  • [ ] Breaking change (fix or new feature that would cause existing functionality to change).
  • [ ] New tests added to cover the changes.
  • [ ] Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • [ ] Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • [ ] In-line docstrings updated.
  • [ ] Documentation updated, tested make html command in the docs/ folder.

myron avatar Oct 11 '22 08:10 myron