armory
armory copied to clipboard
new dataset builder integration
By @davidslater in Slack
- [ ] Move in new dataset building and loading machinery, with tests, and a few representative datasets (that are not currently being modified, i.e., not CARLA). Update documentation of how building and loading work. Demonstrate how dataset loading would change with the new setup, but no actual changes to existing codebase.
- [ ] Migrate over all current datasets and replace data loading calls in scenarios. Add tests and ensure functional parity. Ensure all S3 datasets are updated to match.
- [ ] Remove data subdirectory of armory and any other cleanup?
For new datasets, I was thinking that we put them into a new part of the armory-public-data
bucket, like
armory-public-data/tfdsv4/<dataset>/<tar gz file>
so that can keep them nicely separated from the current batch.