exdir icon indicating copy to clipboard operation
exdir copied to clipboard

Should object arrays be allowed

Open simetenn opened this issue 6 years ago • 3 comments

We should decide if we should allow object arrays or not. There are several advantages and disadvantages to this. Currently we allow object arrays, but we have not made a conscious decision to support them. They just happen to work.

simetenn avatar May 11 '18 08:05 simetenn

They should not be allowed and no longer seem to work because memmap fails in this case. I think we should catch this before memmap does and just check dtype.hasobject. If true, we raise an exception saying that this is not supported.

dragly avatar Nov 08 '18 13:11 dragly

Also not supported w. h5py:

In [8]: f = h5py.File('objecttest.h5', 'w')

In [9]: f['o'] = np.array([object()])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-ff5a86282ebb> in <module>()
----> 1 f['o'] = np.array([object()])

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_hl/group.pyc in __setitem__(self, name, obj)
    289 
    290         else:
--> 291             ds = self.create_dataset(None, data=obj, dtype=base.guess_dtype(obj))
    292             h5o.link(ds.id, self.id, name, lcpl=lcpl)
    293 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_hl/group.pyc in create_dataset(self, name, shape, dtype, data, **kwds)
    106         """
    107         with phil:
--> 108             dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
    109             dset = dataset.Dataset(dsid)
    110             if name is not None:

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_hl/dataset.pyc in make_new_dset(parent, shape, dtype, data, chunks, compression, shuffle, fletcher32, maxshape, compression_opts, fillvalue, scaleoffset, track_times)
     95         else:
     96             dtype = numpy.dtype(dtype)
---> 97         tid = h5t.py_create(dtype, logical=1)
     98 
     99     # Legacy

h5py/h5t.pyx in h5py.h5t.py_create()

h5py/h5t.pyx in h5py.h5t.py_create()

h5py/h5t.pyx in h5py.h5t.py_create()

TypeError: Object dtype dtype('O') has no native HDF5 equivalent

espenhgn avatar Nov 08 '18 15:11 espenhgn

Good point. I think we should also raise a TypeError to have as close to the same behavior as h5py as possible. That eases porting from h5py to Exdir.

dragly avatar Nov 09 '18 13:11 dragly