rerun icon indicating copy to clipboard operation
rerun copied to clipboard

Implement python arrow serialization for all our types

Open emilk opened this issue 1 year ago • 2 comments

Incredibly, we don't 😭

In particular, we don't implement it for batches of a lot of things.

rerun_py/rerun_sdk/rerun/blueprint/datatypes/visible_time_range.py
126:        raise NotImplementedError  # You need to implement native_to_pa_array_override in visible_time_range_ext.py

rerun_py/rerun_sdk/rerun/blueprint/datatypes/visible_time_range_boundary.py
100:        raise NotImplementedError  # You need to implement native_to_pa_array_override in visible_time_range_boundary_ext.py

rerun_py/rerun_sdk/rerun/blueprint/components/entity_properties_component.py
67:        raise NotImplementedError  # You need to implement native_to_pa_array_override in entity_properties_component_ext.py

rerun_py/rerun_sdk/rerun/components/visualizer_overrides.py
58:        raise NotImplementedError  # You need to implement native_to_pa_array_override in visualizer_overrides_ext.py

rerun_py/rerun_sdk/rerun/datatypes/uvec4d.py
64:        raise NotImplementedError  # You need to implement native_to_pa_array_override in uvec4d_ext.py

rerun_py/rerun_sdk/rerun/datatypes/uvec2d.py
64:        raise NotImplementedError  # You need to implement native_to_pa_array_override in uvec2d_ext.py

rerun_py/rerun_sdk/rerun/datatypes/uvec3d.py
64:        raise NotImplementedError  # You need to implement native_to_pa_array_override in uvec3d_ext.py

rerun_py/rerun_sdk/rerun/datatypes/translation_and_mat3x3.py
118:        raise NotImplementedError  # You need to implement native_to_pa_array_override in translation_and_mat3x3_ext.py

rerun_py/rerun_sdk/rerun/datatypes/float32.py
59:        raise NotImplementedError  # You need to implement native_to_pa_array_override in float32_ext.py

rerun_py/rerun_sdk/rerun/datatypes/translation_rotation_scale3d.py
168:        raise NotImplementedError  # You need to implement native_to_pa_array_override in translation_rotation_scale3d_ext.py

rerun_py/rerun_sdk/rerun/datatypes/tensor_buffer.py
287:        raise NotImplementedError  # You need to implement native_to_pa_array_override in tensor_buffer_ext.py

rerun_py/rerun_sdk/rerun/datatypes/tensor_dimension.py
84:        raise NotImplementedError  # You need to implement native_to_pa_array_override in tensor_dimension_ext.py

rerun_py/rerun_sdk/rerun/datatypes/time_int.py
59:        raise NotImplementedError  # You need to implement native_to_pa_array_override in time_int_ext.py

rerun_py/rerun_sdk/rerun/datatypes/scale3d.py
92:        raise NotImplementedError  # You need to implement native_to_pa_array_override in scale3d_ext.py
``

emilk avatar Apr 29 '24 12:04 emilk

The majority of those appear to be datatypes used as sub-fields of other datatypes or components. Not to say they shouldn't implement native_to_pa_array, but the reason they don't is that the recursive dispatch story for python serializers is annoying.

The right direction here for many of these will be to move the serialization code from the parent types to the currently unimplemented children, and likely to build out some supporting dispatch helpers to make it less painful to do all the required arrow plumbing.

jleibs avatar Apr 29 '24 14:04 jleibs

We should soon be able to do this via codegen, once all arrow unions are gone:

  • #6388

emilk avatar Aug 05 '24 11:08 emilk