adlfs icon indicating copy to clipboard operation
adlfs copied to clipboard

LS seems not to be working

Open GabrielAzevedoFerreiraQB opened this issue 3 years ago • 2 comments

LS is not showing the actual directories. it does not behave as expected for reading directories, but works as expected if we pass it the path to a file. I have the following blob: image

and am running the following

st_op = {'connection_string': connect_str}
fs = fsspec.filesystem('abfs', **st_op)
print(fs.ls('abfs://', detail=True))
print(fs.ls('abfs://demo/', detail=True))
print(fs.open('abfs://demo/crash.csv').read())
print(fs.ls('abfs://demo/dir/', detail=True))
<class 'adlfs.spec.AzureDatalakeFileSystem'>
<adlfs.spec.AzureBlobFileSystem object at 0x7fb04593f350>
containers:(OK!) [{'metadata': {}, 'deleted': None, 'last_modified': datetime.datetime(2021, 4, 21, 7, 16, 44, tzinfo=datetime.timezone.utc), 'etag': '"0x8D904956B0F24A6"', 'name': '9c6d625f-ccad-4f75-b9bb-24659ea5da2a', 'size': None, 'type': 'directory'}, {'metadata': {}, 'deleted': None, 'last_modified': datetime.datetime(2021, 4, 22, 5, 36, 35, tzinfo=datetime.timezone.utc), 'etag': '"0x8D905509818F626"', 'name': 'another', 'size': None, 'type': 'directory'}, {'metadata': {}, 'deleted': None, 'last_modified': datetime.datetime(2021, 4, 21, 7, 12, 57, tzinfo=datetime.timezone.utc), 'etag': '"0x8D90494E42673D3"', 'name': 'demo', 'size': None, 'type': 'directory'}]
single container is empty (Not OK!!!)  []
reading a file works (OK) b'driver_id,time,crash,source\nadkavy,1213037028000,0,San_Francisco\nadkavy,1213036968000,0,San_Francisc'
reading a file works (OK) []
ls on an (existing) file works []
ls on an (existing) folder does not work (Not OK!!)

Process finished with exit code 0

What happened:

What you expected to happen:

Minimal Complete Verifiable Example:

# Put your MCVE code here

Anything else we need to know?:

Environment:

  • Dask version:
  • Python version:
  • Operating System:
  • Install method (conda, pip, source):

pip show fsspec Name: fsspec
Version: 2021.4.0
Summary: File-system specification
Home-page: http://github.com/intake/filesystem_spec
Author: None
Author-email: None
License: BSD
Location: /opt/anaconda3/envs/lineage_viz37/lib/python3.7/site-packages
Requires: 
Required-by: s3fs, kedro, adlfs

pip show adlfs 
Name: adlfs
Version: 0.7.3
Summary: Access Azure Datalake Gen1 with fsspec and dask
Home-page: https://github.com/dask/adlfs/
Author: None
Author-email: None
License: BSD
Location: /opt/anaconda3/envs/lineage_viz37/lib/python3.7/site-packages
Requires: azure-identity, azure-datalake-store, azure-core, aiohttp, msrestazure, requests, azure-storage-blob, fsspec
Required-by: 

What do you get with print(fs.ls('abfs://demo/dir', detail=True))?

Please also see this discussion on fsspec.

hayesgb avatar May 13 '21 13:05 hayesgb