flyte
flyte copied to clipboard
[Core feature] LiteralBlob and StructuredDataset metadata
Motivation: Why do you think this is important?
At runtime, Flyte knows about various metadata elements for a file / underlying dataset. It would be amazing to catalog them and show them in the UI. This metadata should be associated with the runtime objects and should be optional
Goal: What should the final outcome look like, ideally?
Example:
- For FlyteFile - Blob.Single
- Size of the file,
- zipped boolean
- (we already store the format in the type)
- For Directory / Blob.Multi
- Size of the total dataset
- zipped boolean
- an Index for the directory with all the files stored
- (we already store the format in the type)
- StructureDataset
- Number of paritions
- Zipped
- Format
- Index of all the files / partitions
Note We could also use this metadata to document lifecycle - for example if an expiration is set. This can be useful in determining cache hits based on data TTLs. For example, each File / dataset etc can have the max-age set and this can be valid until date.
Describe alternatives you've considered
Store is separately, but then every File will become a directory and the format is only python. UI will have to implement this again.
Propose: Link/Inline OR Additional context
Add Blob metadata to https://github.com/flyteorg/flyte/blob/cd37d1b5b61789d651e3aa1686910593ccbeaea0/flyteidl/protos/flyteidl/core/literals.proto#L39
And Structured Dataset metadata to https://github.com/flyteorg/flyte/blob/cd37d1b5b61789d651e3aa1686910593ccbeaea0/flyteidl/protos/flyteidl/core/literals.proto#L69
Also ideally the FileAccessProvider interface returns this metadata after each upload? https://github.com/flyteorg/flytekit/blob/070027d8781d9b9f830723a9a4cae5daa6359a33/flytekit/core/data_persistence.py#L311
Are you sure this issue hasn't been raised already?
- [X] Yes
Have you read the Code of Conduct?
- [X] Yes