pynwb
pynwb copied to clipboard
Published
20 hours ago •
NeurodataWithoutBorders
NeurodataWithoutBorders
problem reading older icephys file
Description
PyNWB no longer is able to open the file in dandiset 25, which uses the old style of icephys.
Steps to Reproduce
$ dandi download https://dandiarchive.org/dandiset/000025/draft
$ cd 000025/
$ ipython
from pynwb import NWBHDF5IO
In [2]: io = NWBHDF5IO('001_140709EXP_A1_ndx_v3.nwb','r', load_namespaces=True)
/Users/bendichter/opt/miniconda3/lib/python3.9/site-packages/hdmf/spec/namespace.py:532: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.1.3 because version 1.5.0 is already loaded.
warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
/Users/bendichter/opt/miniconda3/lib/python3.9/site-packages/hdmf/spec/namespace.py:532: UserWarning: Ignoring cached namespace 'core' version 2.1.0 because version 2.4.0 is already loaded.
warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
In [3]: io.read()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-3-e070f3e77a80> in <module>
----> 1 io.read()
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/backends/hdf5/h5tools.py in read(self, **kwargs)
496 % (self.source, self.__mode))
497 try:
--> 498 return call_docval_func(super().read, kwargs)
499 except UnsupportedOperation as e:
500 if str(e) == 'Cannot build data. There are no values.': # pragma: no cover
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py in call_docval_func(func, kwargs)
422 def call_docval_func(func, kwargs):
423 fargs, fkwargs = fmt_docval_args(func, kwargs)
--> 424 return func(*fargs, **fkwargs)
425
426
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
581 def func_call(*args, **kwargs):
582 pargs = _check_args(args, kwargs)
--> 583 return func(args[0], **pargs)
584 else:
585 def func_call(*args, **kwargs):
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/backends/io.py in read(self, **kwargs)
39 # TODO also check that the keys are appropriate. print a better error message
40 raise UnsupportedOperation('Cannot build data. There are no values.')
---> 41 container = self.__manager.construct(f_builder)
42 return container
43
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
581 def func_call(*args, **kwargs):
582 pargs = _check_args(args, kwargs)
--> 583 return func(args[0], **pargs)
584 else:
585 def func_call(*args, **kwargs):
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py in construct(self, **kwargs)
278 # we are at the top of the hierarchy,
279 # so it must be time to resolve parents
--> 280 result = self.__type_map.construct(builder, self, None)
281 self.__resolve_parents(result)
282 self.prebuilt(result, builder)
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
581 def func_call(*args, **kwargs):
582 pargs = _check_args(args, kwargs)
--> 583 return func(args[0], **pargs)
584 else:
585 def func_call(*args, **kwargs):
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py in construct(self, **kwargs)
783 if build_manager is None:
784 build_manager = BuildManager(self)
--> 785 obj_mapper = self.get_map(builder)
786 if obj_mapper is None:
787 dt = builder.attributes[self.namespace_catalog.group_spec_cls.type_key()]
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
581 def func_call(*args, **kwargs):
582 pargs = _check_args(args, kwargs)
--> 583 return func(args[0], **pargs)
584 else:
585 def func_call(*args, **kwargs):
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py in get_map(self, **kwargs)
713 break
714 spec = self.__ns_catalog.get_spec(namespace, data_type)
--> 715 mapper = mapper_cls(spec)
716 self.__mappers[container_cls] = mapper
717 return mapper
~/opt/miniconda3/lib/python3.9/site-packages/pynwb/io/file.py in __init__(self, spec)
53 self.map_spec('icephys_electrodes', icephys_spec.get_neurodata_type('IntracellularElectrode'))
54 self.map_spec('sweep_table', icephys_spec.get_neurodata_type('SweepTable'))
---> 55 self.map_spec('intracellular_recordings', icephys_spec.get_neurodata_type('IntracellularRecordingsTable'))
56 self.map_spec('icephys_simultaneous_recordings', icephys_spec.get_neurodata_type('SimultaneousRecordingsTable'))
57 self.map_spec('icephys_sequential_recordings', icephys_spec.get_neurodata_type('SequentialRecordingsTable'))
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
580 if is_method:
581 def func_call(*args, **kwargs):
--> 582 pargs = _check_args(args, kwargs)
583 return func(args[0], **pargs)
584 else:
~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py in _check_args(args, kwargs)
573 if parse_err:
574 msg = '%s: %s' % (func.__qualname__, ', '.join(parse_err))
--> 575 raise ExceptionType(msg)
576
577 return parsed['args']
TypeError: ObjectMapper.map_spec: None is not allowed for 'spec' (expected 'Spec', not None)
Environment
Python Executable: Conda
Python Version: Python 3.9
Operating System: macOS
HDMF Version: 3.1.1
PyNWB Version: 2.0.0
Checklist
- [x] Have you ensured the bug was not already reported?
- [x] Have you included a brief and descriptive title?
- [x] Have you included a clear description of the problem you are trying to solve?
- [x] Have you included a minimal code snippet that reproduces the issue you are encountering?
- [x] Have you checked our Contributing document?
Ok, that should not happen. I'll take a look. I opened an icephys files from dandiset 20 yesterday without an issue. I'm wondering whether something strange is happening where things might be colliding with an extension in the file.
Validation output:
$python -m pynwb.validate 001_140709EXP_A1_ndx_v3.nwb
Traceback (most recent call last):
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pynwb/validate.py", line 136, in <module>
main()
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pynwb/validate.py", line 72, in main
ns_deps = NWBHDF5IO.load_namespaces(catalog, path)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/utils.py", line 583, in func_call
return func(args[0], **pargs)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/backends/hdf5/h5tools.py", line 147, in load_namespaces
return cls.__load_namespaces(namespace_catalog, namespaces, open_file_obj)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/backends/hdf5/h5tools.py", line 181, in __load_namespaces
d.update(namespace_catalog.load_namespaces(cls.__ns_spec_path, reader=reader))
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/utils.py", line 583, in func_call
return func(args[0], **pargs)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/spec/namespace.py", line 538, in load_namespaces
ret[ns['name']] = self.__load_namespace(ns, reader, resolve=resolve)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/spec/namespace.py", line 448, in __load_namespace
self.__load_spec_file(reader, s['source'], catalog, types_to_load=types_to_load, resolve=resolve)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/spec/namespace.py", line 402, in __load_spec_file
temp_dict = {k: None for k in __reg_spec(self.__group_spec_cls, spec_dict)}
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/spec/namespace.py", line 389, in __reg_spec
return catalog.auto_register(spec_obj, spec_source)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/utils.py", line 583, in func_call
return func(args[0], **pargs)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/spec/catalog.py", line 102, in auto_register
ret.extend(self.auto_register(group_spec, source_file))
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/utils.py", line 583, in func_call
return func(args[0], **pargs)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/spec/catalog.py", line 100, in auto_register
self.register_spec(dataset_spec, source_file)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/utils.py", line 583, in func_call
return func(args[0], **pargs)
File "/home/firma/.pyenv/versions/3.8.6/lib/python3.8/site-packages/hdmf/spec/catalog.py", line 46, in register_spec
raise ValueError("'%s' - cannot overwrite existing specification" % type_name)
ValueError: 'ScratchData' - cannot overwrite existing specification
@t-b thanks for the additional case. Could you provide a link to the file. As far as I can tell the issue is somehow related to the use of extensions in the file. Loading existing (i.e., v2.3 and earlier) ICEphys files appears to work if the file doesn't use an extension, i.e., it seems that loading the extension from file triggers an edge-case we missed in the tests.
@t-b thanks for the clarification. Interesting that the validator fails differently than the normal read here. In particular, it seems that in the log output you provided for the validator, the error occurs already in the load_namespaces function during read. In contrast in the log output that Ben reported for reading the file, the error appears to happen much later in the ObjectMapper, even though both have load_namespaces enabled. This may mean that we are actually with two separate issues. We'll take a look.