ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

get_description exposes column names as internal protos in a fragile and hard to consume way

Open poodlewars opened this issue 1 year ago • 0 comments

Describe the bug

No response

Steps/Code to Reproduce

from arcticdb import Arctic
import pandas as pd

ac = Arctic("lmdb://tst")
ac.create_library("blah")

lib = ac["blah"]
df = pd.DataFrame(data={"a": [pd.Timestamp(2023,8,22),pd.Timestamp(2023,8,22),pd.Timestamp(2023,8,22)], "b": ["4", "5", "6"]})
lib.write("sym", df)
lib.get_description("sym").columns[0].dtype

This gives:

value_type: NANOSECONDS_UTC
size_bits: S64

Which is:

  • A hard to consume string
  • Refers to our internal protobuf type NANOSECONDS_UTC, meaning proto renames can be breaking API changes currently

Expected Results

We should maintain enums for column types as part of the external versioned API and expose them. We should expose column and index type information in an easier-to-consume way than these strings we currently expose.

OS, Python Version and ArcticDB Version

Any

Backend storage used

Any

Additional Context

No response

poodlewars avatar Aug 22 '23 09:08 poodlewars