intake-esm icon indicating copy to clipboard operation
intake-esm copied to clipboard

Passing Path objects to `esm_datastore`

Open aulemahal opened this issue 3 years ago • 0 comments

This is very minor, but I feel we should be able to pass pathlib.Path objects to intake_esm.esm_datastore.

Ex:

from pathlib import Path
from intake_esm import esm_datastore
p = Path('/path/to/catalog.json')

esm_datastore(p)

fails with

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 esm_datastore(p)

File .conda/espoprops/lib/python3.10/site-packages/intake_esm/core.py:94, in esm_datastore.__init__(self, obj, progressbar, sep, registry, read_csv_kwargs, storage_options, intake_kwargs)
     92     self.esmcat = ESMCatalogModel.from_dict(obj)
     93 else:
---> 94     self.esmcat = ESMCatalogModel.load(
     95         obj, storage_options=self.storage_options, read_csv_kwargs=read_csv_kwargs
     96     )
     98 self.derivedcat = registry or default_registry
     99 self._entries = {}

File .conda/espoprops/lib/python3.10/site-packages/intake_esm/cat.py:220, in ESMCatalogModel.load(cls, json_file, storage_options, read_csv_kwargs)
    218 storage_options = storage_options if storage_options is not None else {}
    219 read_csv_kwargs = read_csv_kwargs or {}
--> 220 _mapper = fsspec.get_mapper(json_file, **storage_options)
    222 with fsspec.open(json_file, **storage_options) as fobj:
    223     data = json.loads(fobj.read())

File .conda/espoprops/lib/python3.10/site-packages/fsspec/mapping.py:230, in get_mapper(url, check, create, missing_exceptions, alternate_root, **kwargs)
    199 """Create key-value interface for given URL and options
    200 
    201 The URL will be of the form "protocol://location" and point to the root
   (...)
    227 ``FSMap`` instance, the dict-like key-value store.
    228 """
    229 # Removing protocol here - could defer to each open() on the backend
--> 230 fs, urlpath = url_to_fs(url, **kwargs)
    231 root = alternate_root if alternate_root is not None else urlpath
    232 return FSMap(root, fs, check, create, missing_exceptions=missing_exceptions)

File .conda/espoprops/lib/python3.10/site-packages/fsspec/core.py:392, in url_to_fs(url, **kwargs)
    372 def url_to_fs(url, **kwargs):
    373     """
    374     Turn fully-qualified and potentially chained URL into filesystem instance
    375 
   (...)
    390         The file-systems-specific URL for ``url``.
    391     """
--> 392     chain = _un_chain(url, kwargs)
    393     if len(chain) > 1:
    394         inkwargs = {}

File .conda/espoprops/lib/python3.10/site-packages/fsspec/core.py:346, in _un_chain(path, kwargs)
    342     return out
    343 x = re.compile(".*[^a-z]+.*")  # test for non protocol-like single word
    344 bits = (
    345     [p if "://" in p or x.match(p) else p + "://" for p in path.split("::")]
--> 346     if "::" in path
    347     else [path]
    348 )
    349 if len(bits) < 2:
    350     return []

TypeError: argument of type 'PosixPath' is not iterable

So in fact, the error comes from fsspec. I nevertheless feel that intake_esm would be the best place to cast the path to string.

EDIT: The annotations of the functions actually mention pathlib.Path so this was supposed to work but was broken by the switch to fsspec I believe.

aulemahal avatar Oct 19 '22 15:10 aulemahal