databroker icon indicating copy to clipboard operation
databroker copied to clipboard

Fix dtype deserialization

Open danielballan opened this issue 1 year ago • 1 comments

Closes #820

danielballan avatar Nov 07 '24 12:11 danielballan

Okay so not only the incoming dtype needs to be deserialized correctly, but in some cases also the structured arrays. So the changes needed are a little more extensive.

  1. Shape Validation (mongo): https://github.com/danielballan/databroker/blob/6c7c26a8752709ee102e2551c2a12bd4d671a9f2/databroker/mongo_normalized.py#L932-L946

numpy.asarray(item): the conversion of item to an ndarray has to be done with the appropriate numpy dtype, from the data_key['dtype_descr'] if it exists, and each structured array element has to be turned into a tuple

  1. Config xarray: https://github.com/danielballan/databroker/blob/6c7c26a8752709ee102e2551c2a12bd4d671a9f2/databroker/mongo_normalized.py#L1081

columns[key] = numpy.array(column, dtype=numpy_dtype)

if numpy_dtype is a structured array, the data in column needs to be turned into a list of lists of tuples.

How would you like to proceed, sould I open my own PR or do you want to implement these changes?

Bilchreis avatar Nov 12 '24 11:11 Bilchreis