armory icon indicating copy to clipboard operation
armory copied to clipboard

new dataset builder integration

Open mwartell opened this issue 2 years ago • 1 comments

By @davidslater in Slack

  • [ ] Move in new dataset building and loading machinery, with tests, and a few representative datasets (that are not currently being modified, i.e., not CARLA). Update documentation of how building and loading work. Demonstrate how dataset loading would change with the new setup, but no actual changes to existing codebase.
  • [ ] Migrate over all current datasets and replace data loading calls in scenarios. Add tests and ensure functional parity. Ensure all S3 datasets are updated to match.
  • [ ] Remove data subdirectory of armory and any other cleanup?

mwartell avatar May 16 '22 20:05 mwartell

For new datasets, I was thinking that we put them into a new part of the armory-public-data bucket, like armory-public-data/tfdsv4/<dataset>/<tar gz file> so that can keep them nicely separated from the current batch.

davidslater avatar May 16 '22 21:05 davidslater