[Core feature] LiteralBlob and StructuredDataset metadata

[kumare3]

Motivation: Why do you think this is important?

At runtime, Flyte knows about various metadata elements for a file / underlying dataset. It would be amazing to catalog them and show them in the UI. This metadata should be associated with the runtime objects and should be optional

Goal: What should the final outcome look like, ideally?

Example:

1. For FlyteFile - Blob.Single

• Size of the file,
• zipped boolean
• (we already store the format in the type)

2. For Directory / Blob.Multi

• Size of the total dataset
• zipped boolean
• an Index for the directory with all the files stored
• (we already store the format in the type)

3. StructureDataset

• Number of paritions
• Zipped
• Format
• Index of all the files / partitions

Note We could also use this metadata to document lifecycle - for example if an expiration is set. This can be useful in determining cache hits based on data TTLs. For example, each File / dataset etc can have the max-age set and this can be valid until date.

Describe alternatives you've considered

Store is separately, but then every File will become a directory and the format is only python. UI will have to implement this again.

Propose: Link/Inline OR Additional context

Add Blob metadata to https://github.com/flyteorg/flyte/blob/cd37d1b5b61789d651e3aa1686910593ccbeaea0/flyteidl/protos/flyteidl/core/literals.proto#L39

And Structured Dataset metadata to https://github.com/flyteorg/flyte/blob/cd37d1b5b61789d651e3aa1686910593ccbeaea0/flyteidl/protos/flyteidl/core/literals.proto#L69

Also ideally the FileAccessProvider interface returns this metadata after each upload? https://github.com/flyteorg/flytekit/blob/070027d8781d9b9f830723a9a4cae5daa6359a33/flytekit/core/data_persistence.py#L311

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes