flyte
flyte copied to clipboard
Plugin: petastorm datatype for structured dataset
Motivation: Why do you think this is important?
Working with large parquet files can be cumbersome and hard, especially in tight training loops. Petastorm can simplify this. This can be a simple type plugin for structureddataset in Flyte https://github.com/uber/petastorm
Goal: What should the final outcome look like, ideally?
Work with large datasets
Describe alternatives you've considered
Self converting it
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
- [X] Yes
Have you read the Code of Conduct?
- [X] Yes