Cannot use cached namespaces from NWB file instead of default

[gouwens]

Description

At the Allen Institute, we have some NWB2 files that can't be read by the latest versions of pynwb because of a namespace problem. These NWB files use the "Index" data type which was removed in hdmf-common version 1.2.0.

When I try to load the file, it ignores the hdmf-common namespace version 1.1.3 cached in the file (which still has the "Index" type) because it has already loaded version 1.5.0 (I think just automatically because of the global namespace catalog?). There doesn't seem to be any way of overriding that default, so loading fails because "Index" is not specified in version 1.5.0.

Steps to Reproduce

import pynwb
nwb_path = "Ctgf-T2A-dgCre;Ai14-533670.04.02.02.nwb"
dataset = pynwb.NWBHDF5IO(nwb_path, mode='r', load_namespaces=True)

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/spec/namespace.py:533: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.1.3 because version 1.5.0 is already loaded.
  % (ns['name'], ns['version'], self.__namespaces.get(ns['name'])['version']))
/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/spec/namespace.py:533: UserWarning: Ignoring cached namespace 'core' version 2.2.4 because version 2.4.0 is already loaded.
  % (ns['name'], ns['version'], self.__namespaces.get(ns['name'])['version']))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_9995/732225968.py in <module>
----> 1 dataset = pynwb.NWBHDF5IO(nwb_path, mode='r', load_namespaces=True)

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
    581             def func_call(*args, **kwargs):
    582                 pargs = _check_args(args, kwargs)
--> 583                 return func(args[0], **pargs)
    584         else:
    585             def func_call(*args, **kwargs):

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/pynwb/__init__.py in __init__(self, **kwargs)
    228 
    229             tm = get_type_map()
--> 230             super(NWBHDF5IO, self).load_namespaces(tm, path, file=file_obj, driver=driver)
    231             manager = BuildManager(tm)
    232 

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
    581             def func_call(*args, **kwargs):
    582                 pargs = _check_args(args, kwargs)
--> 583                 return func(args[0], **pargs)
    584         else:
    585             def func_call(*args, **kwargs):

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py in load_namespaces(cls, **kwargs)
    145         if file_obj is None:  # need to close the file object that we just opened
    146             with open_file_obj:
--> 147                 return cls.__load_namespaces(namespace_catalog, namespaces, open_file_obj)
    148         return cls.__load_namespaces(namespace_catalog, namespaces, open_file_obj)
    149 

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py in __load_namespaces(cls, namespace_catalog, namespaces, file_obj)
    179         for ns in order:
    180             reader = readers[ns]
--> 181             d.update(namespace_catalog.load_namespaces(cls.__ns_spec_path, reader=reader))
    182 
    183         return d

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
    581             def func_call(*args, **kwargs):
    582                 pargs = _check_args(args, kwargs)
--> 583                 return func(args[0], **pargs)
    584         else:
    585             def func_call(*args, **kwargs):

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/build/manager.py in load_namespaces(self, **kwargs)
    472         load_namespaces here has the advantage of being able to keep track of type dependencies across namespaces.
    473         '''
--> 474         deps = call_docval_func(self.__ns_catalog.load_namespaces, kwargs)
    475         for new_ns, ns_deps in deps.items():
    476             for src_ns, types in ns_deps.items():

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/utils.py in call_docval_func(func, kwargs)
    422 def call_docval_func(func, kwargs):
    423     fargs, fkwargs = fmt_docval_args(func, kwargs)
--> 424     return func(*fargs, **fkwargs)
    425 
    426 

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
    581             def func_call(*args, **kwargs):
    582                 pargs = _check_args(args, kwargs)
--> 583                 return func(args[0], **pargs)
    584         else:
    585             def func_call(*args, **kwargs):

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/spec/namespace.py in load_namespaces(self, **kwargs)
    536         # now load specs into namespace
    537         for ns in to_load:
--> 538             ret[ns['name']] = self.__load_namespace(ns, reader, resolve=resolve)
    539         self.__included_specs[ns_path_key] = ret
    540         return ret

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/spec/namespace.py in __load_namespace(self, namespace, reader, resolve)
    458                 registered_types = set()
    459                 for ndt in types_to_load:
--> 460                     self.__register_type(ndt, inc_ns, catalog, registered_types)
    461                 included_types[s['namespace']] = tuple(sorted(registered_types))
    462             else:

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/spec/namespace.py in __register_type(self, ndt, inc_ns, catalog, registered_types)
    468 
    469     def __register_type(self, ndt, inc_ns, catalog, registered_types):
--> 470         spec = inc_ns.get_spec(ndt)
    471         spec_file = inc_ns.catalog.get_spec_source_file(ndt)
    472         self.__register_dependent_types(spec, inc_ns, catalog, registered_types)

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
    581             def func_call(*args, **kwargs):
    582                 pargs = _check_args(args, kwargs)
--> 583                 return func(args[0], **pargs)
    584         else:
    585             def func_call(*args, **kwargs):

/local1/anaconda3/envs/py3/lib/python3.7/site-packages/hdmf/spec/namespace.py in get_spec(self, **kwargs)
    148         spec = self.__catalog.get_spec(data_type)
    149         if spec is None:
--> 150             raise ValueError("No specification for '%s' in namespace '%s'" % (data_type, self.name))
    151         return spec
    152 

ValueError: No specification for 'Index' in namespace 'core'

Environment

Python Executable: Conda
Python Version: Python 3.7.7
Operating System: Linux
HDMF Version: 3.1.1
PyNWB Version: 2.0.0

Checklist

• [x] Have you ensured the bug was not already reported?
• [x] Have you included a brief and descriptive title?
• [x] Have you included a clear description of the problem you are trying to solve?
• [x] Have you included a minimal code snippet that reproduces the issue you are encountering?
• [x] Have you checked our Contributing document?

[rly]

@gouwens Thanks for the bug report. As we look into potential solutions, could you share one of these files? Thanks.

[gouwens]

Sure thing. I've attached an example file in a ZIP archive.

Ctgf-T2A-dgCre;Ai14-533670.04.02.02.nwb.zip

[tmchartrand]

I'm running into this too, now affecting validating files for upload to DANDI. seems like it's maybe also related to https://github.com/NeurodataWithoutBorders/pynwb/issues/1289 ?