satpy icon indicating copy to clipboard operation
satpy copied to clipboard

Automatic conversion of remote files to FSFile objects breaks pathlib.Path support

Open BENR0 opened this issue 2 years ago • 2 comments

Describe the bug Automatic conversion of remote files to FSFile objects introduced in #2096 breaks pathlib.Path support.

To Reproduce For example

from satpy import Scene
from pathlib import Path

filenames = Path("/path/to/seviri/netcdf_files").glob("*.nc")
filenames = [list(filenames)[0]]]

msgscn= Scene(reader="seviri_l1b_nc", filenames=filenames)

Expected behavior The Scene should accept pathlib.Path object without error.

Actual results

/dev/pytroll/satpy/satpy/scene.py in __init__(self, filenames, reader, filter_parameters, reader_kwargs)
    129 
    130         if filenames:
--> 131             filenames = convert_remote_files_to_fsspec(filenames, storage_options)
    132 
    133         self._readers = self._create_reader_instances(filenames=filenames,

/dev/pytroll/satpy/satpy/utils.py in convert_remote_files_to_fsspec(filenames, storage_options)
    549     if isinstance(filenames, dict):
    550         return _check_file_protocols_for_dicts(filenames, storage_options)
--> 551     return _check_file_protocols(filenames, storage_options)
    552 
    553 

/dev/pytroll/satpy/satpy/utils.py in _check_file_protocols(filenames, storage_options)
    561 
    562 def _check_file_protocols(filenames, storage_options):
--> 563     local_files, remote_files, fs_files = _sort_files_to_local_remote_and_fsfiles(filenames)
    564 
    565     if remote_files:

/dev/pytroll/satpy/satpy/utils.py in _sort_files_to_local_remote_and_fsfiles(filenames)
    578         if isinstance(f, FSFile):
    579             fs_files.append(f)
--> 580         elif urlparse(f).scheme in ('', 'file') or "\\" in f:
    581             local_files.append(f)
    582         else:

/miniconda3/envs/dev/lib/python3.8/urllib/parse.py in urlparse(url, scheme, allow_fragments)
    370     Note that we don't break the components up in smaller bits
    371     (e.g. netloc is a single string) and we don't expand % escapes."""
--> 372     url, scheme, _coerce_result = _coerce_args(url, scheme)
    373     splitresult = urlsplit(url, scheme, allow_fragments)
    374     scheme, netloc, url, query, fragment = splitresult

/miniconda3/envs/dev/lib/python3.8/urllib/parse.py in _coerce_args(*args)
    122     if str_input:
    123         return args + (_noop,)
--> 124     return _decode_args(args) + (_encode_result,)
    125 
    126 # Result objects are more helpful than simple tuples

/miniconda3/envs/dev/lib/python3.8/urllib/parse.py in _decode_args(args, encoding, errors)
    106 def _decode_args(args, encoding=_implicit_encoding,
    107                        errors=_implicit_errors):
--> 108     return tuple(x.decode(encoding, errors) if x else '' for x in args)
    109 
    110 def _coerce_args(*args):

/miniconda3/envs/dev/lib/python3.8/urllib/parse.py in <genexpr>(.0)
    106 def _decode_args(args, encoding=_implicit_encoding,
    107                        errors=_implicit_errors):
--> 108     return tuple(x.decode(encoding, errors) if x else '' for x in args)
    109 
    110 def _coerce_args(*args):

AttributeError: 'PosixPath' object has no attribute 'decode'

Environment Info:

  • OS: Linux
  • Satpy Version: 0.36.0
  • PyResample Version:

Additional context While #2096 completely breaks pathlib.Path support, I think pathlib.Path support also depends on the reader. Even before some readers did not accept pathlib.Path objects (e.g. modis_l1b) depending on which library is used to read the files. Maybe this is something that should be documented?

BENR0 avatar May 25 '22 10:05 BENR0

There are other aspects where pathlib.Path support in Satpy is incomplete. For example, save_dataset does not take pathlib.Path.

gerritholl avatar May 25 '22 12:05 gerritholl

@gerritholl yes I saw your issue (#1122) regarding that

BENR0 avatar May 25 '22 13:05 BENR0