MONAI
MONAI copied to clipboard
SharedCacheDataset
This introduces a new SharedCacheDataset(Dataset) class to address https://github.com/Project-MONAI/MONAI/issues/3843
It has several improvements / changes compared to the CacheDataset https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/dataset.py#L690
- It caches into a shared memory in RAM, so all processes in multigpu run with DistributedDataParallel can read/write to the same shared memory (globally accessible)
- it works on the fly (there is non need to wait to pre-cache the data), it will cache it on the first data access, and then use it from the cache
If tested it and used for many tasks already in DDP / multigpu I'll add unit tests too if it looks good
Types of changes
- [x] Non-breaking change (fix or new feature that would not break existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running
./runtests.sh -f -u --net --coverage
. - [ ] Quick tests passed locally by running
./runtests.sh --quick --unittests --disttests
. - [ ] In-line docstrings updated.
- [ ] Documentation updated, tested
make html
command in thedocs/
folder.