[Bug]: Cannot write empty custom column of DynamicTable

Open rly opened this issue 2 years ago • 0 comments

What happened?

If a user adds a custom ~indexed~ column to a DynamicTable as part of their pipeline, but the column happens to be empty, HDMF cannot determine the dtype of the column when writing to HDF5 and errors out. A user should be able to specify the dtype of a column when creating the custom column to help the object mapper determine which dtype to use when creating the (empty) dataset for that column.

Separately, it is also useful for the user to specify the dtype of a column on creation to aid in validation and so that the object mapper does not have to infer the dtype from the first element on write.

Steps to Reproduce

from hdmf.common import DynamicTable, HDF5IO, get_manager

table = DynamicTable(
    name="test",
    description="test"
)
table.add_column(
    name="test",
    description="test",
    data=[],
    index=True
)

with HDF5IO("test.h5", "w", manager=get_manager()) as io:
    io.write(table)

Traceback

Traceback (most recent call last):
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 755, in build
    bldr_data, dtype = self.convert_dtype(spec, container.data, spec_dtype=spec_dtype)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 201, in convert_dtype
    ret, ret_dtype = cls.__check_edgecases(spec, value, spec_dtype)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 314, in __check_edgecases
    raise ValueError(msg)
ValueError: Cannot infer dtype of empty list or tuple. Please use numpy array with specified dtype.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/rly/Documents/NWB/linked-arrays/test.py", line 15, in <module>
    io.write(table)
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/backends/hdf5/h5tools.py", line 375, in write
    super().write(**kwargs)
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/backends/io.py", line 98, in write
    f_builder = self.__manager.build(container, source=self.__source, root=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/manager.py", line 171, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext, export=export)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/manager.py", line 770, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext, export=export)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 716, in build
    self.__add_datasets(builder, self.__spec.datasets, container, manager, source, export)
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1020, in __add_datasets
    self.__add_containers(builder, spec, attr_value, build_manager, source, container, export)
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1108, in __add_containers
    self.__add_containers(builder, spec, container, build_manager, source, parent_container, export)
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1070, in __add_containers
    new_builder = build_manager.build(value, source=source, spec_ext=spec, export=export)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/manager.py", line 171, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext, export=export)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/manager.py", line 770, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext, export=export)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rly/mambaforge/envs/dev/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 758, in build
    raise Exception(msg) from ex
Exception: could not resolve dtype for VectorData 'test'

Operating System

macOS

Python Executable

Conda

Python Version

3.11

Package Versions

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] Have you checked the Contributing document?
[X] Have you ensured this bug was not already reported?

Nov 15 '23 20:11 rly