Plugin: petastorm datatype for structured dataset

[kumare3]

Motivation: Why do you think this is important?

Working with large parquet files can be cumbersome and hard, especially in tight training loops. Petastorm can simplify this. This can be a simple type plugin for structureddataset in Flyte https://github.com/uber/petastorm

Goal: What should the final outcome look like, ideally?

Work with large datasets

Describe alternatives you've considered

Self converting it

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes