SR DataSet Conversions

[Spartee]

Description

In order to make SmartRedis DataSet objects more amenable to activies like online analysis and training, we should investigate data formats that make sense to be able to convert to and from.

Additionally, this epic will also track the effort necessary to do aggregations for when global domains are needed before converting datasets.

To investigate

• xarray
• Zarr
• conduit
• feather
• Parquet
• HDF5

the goal is to have a to_<file format> and from_<file_format> method for each of the types we end up choosing.

If you would like to see others, please let us know.

[ashao]

https://zarr.readthedocs.io/en/stable/index.html should definitely be on there a well.

[ashao]

Woops you ninja-edited that in there.

[mellis13]

Closing this epic as the scope and strategy is likely much different than this was created in Feb 2022.