epath behaves weird using fsspec on gcs

[hmeyer]

Via GCloud Console I clicked "CREATE FOLDER" to create empty_folder. Then I did:

$ touch /tmp/test.txt
$ gsutil cp /tmp/test.txt gs://henning-test/folder/test.txt

$ python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from etils import epath
>>> epath.gpath._is_tf_installed()
False
>>> e = epath.Path('gs://henning-test/empty_folder')
>>> f'{e.exists()=} {e.is_dir()=}'
'e.exists()=True e.is_dir()=False'
>>> list(e.iterdir())
[PosixGPath('gs://henning-test/empty_folder')]

>>> f = epath.Path('gs://henning-test/folder')
>>> f'{f.exists()=} {f.is_dir()=}'
'f.exists()=True f.is_dir()=False'

>>> t = epath.Path('gs://henning-test/folder/test.txt')
>>> f'{t.exists()=} {t.is_dir()=}'
't.exists()=True t.is_dir()=False'
>>> list(t.iterdir())
[PosixGPath('gs://henning-test/folder/test.txt/test.txt')]

[Conchylicultor]

p.is_dir() directly call fs.isdir(): https://github.com/google/etils/blob/5b96e70ca0a2d332ffa435b98bf37fe29e2f2300/etils/epath/backend.py#L394

So I think this is a fsspec bug. You can check with:

import fsspec  # pylint: disable=g-import-not-at-top

fs = fsspec.filesystem('gcs')

print(fs.isdir('gs://henning-test/folder'))

Might be related to: https://github.com/fsspec/gcsfs/issues/574

[hmeyer]

Good point. Added https://github.com/fsspec/gcsfs/issues/574#issuecomment-1854123740