streaming
streaming copied to clipboard
mosaicml
A Data Streaming Library for Efficient Neural Network Training
*In which we blow away 1) torch dist, 2) shared memory, and 3) filelock #YOLO* ## Nuke torch dist Can we better contain or even eliminate Streaming's dependencies on PyTorch,...
- [x] replace prefix registration/lookup aka local dir collision detection dist - [ ] replace shared memory, shared array, shared scalar, and shared barrier - [x] replace streaming dataset init...
Is it possible to use streaming dataset as a distributed key value store? i have a set of keys (strings like "xyz_123") each that correspond to an numpy array ideally...
In `streaming/base/format/base/writer.py`: ```py @classmethod def _get_timer(cls) -> Timer: """Get a timer tree for the process of writing a dataset. Returns: Timer: The tree of timers. """ return Timer([ ('write', Timer([...
## 🚀 Feature Request Hey MosaicML team! Thank you so much for this awesome project! I was wondering if there are any plans to make this framework agnostic: Remove the...
Thanks for your great work! Could this saving cache path be optional instead of always writes into `/tmp/streaming` ? https://github.com/mosaicml/streaming/blob/cb8e872359643fa84782c4e95c496dc66e495c44/streaming/base/dataset.py#L515
it'd be great if the files could be hosted on https e.g. for experimentation that the dataset could be hosted on localhost or a local network with a command line...
## Description of changes: Add a ingestion helper utility for Huggingface datasets downloading. Building on snapshot_download, some improvements include - Enable resume = True. retry when bad network happens -...
## Environment AWS Deep Learning Machine with 8xA100 and CUDA 11.8 ## To reproduce Steps to reproduce the behavior: 1. Use StreamingDataset to load ImageNet from a local SSD using...
## Description of changes: - Moved local directory creation and existence check from CloudUploader to Writer class ## Issue #, if available: ## Merge Checklist: _Put an `x` without space...
