adlfs
adlfs copied to clipboard
LS seems not to be working
LS is not showing the actual directories. it does not behave as expected for reading directories, but works as expected if we pass it the path to a file.
I have the following blob:
and am running the following
st_op = {'connection_string': connect_str}
fs = fsspec.filesystem('abfs', **st_op)
print(fs.ls('abfs://', detail=True))
print(fs.ls('abfs://demo/', detail=True))
print(fs.open('abfs://demo/crash.csv').read())
print(fs.ls('abfs://demo/dir/', detail=True))
<class 'adlfs.spec.AzureDatalakeFileSystem'>
<adlfs.spec.AzureBlobFileSystem object at 0x7fb04593f350>
containers:(OK!) [{'metadata': {}, 'deleted': None, 'last_modified': datetime.datetime(2021, 4, 21, 7, 16, 44, tzinfo=datetime.timezone.utc), 'etag': '"0x8D904956B0F24A6"', 'name': '9c6d625f-ccad-4f75-b9bb-24659ea5da2a', 'size': None, 'type': 'directory'}, {'metadata': {}, 'deleted': None, 'last_modified': datetime.datetime(2021, 4, 22, 5, 36, 35, tzinfo=datetime.timezone.utc), 'etag': '"0x8D905509818F626"', 'name': 'another', 'size': None, 'type': 'directory'}, {'metadata': {}, 'deleted': None, 'last_modified': datetime.datetime(2021, 4, 21, 7, 12, 57, tzinfo=datetime.timezone.utc), 'etag': '"0x8D90494E42673D3"', 'name': 'demo', 'size': None, 'type': 'directory'}]
single container is empty (Not OK!!!) []
reading a file works (OK) b'driver_id,time,crash,source\nadkavy,1213037028000,0,San_Francisco\nadkavy,1213036968000,0,San_Francisc'
reading a file works (OK) []
ls on an (existing) file works []
ls on an (existing) folder does not work (Not OK!!)
Process finished with exit code 0
What happened:
What you expected to happen:
Minimal Complete Verifiable Example:
# Put your MCVE code here
Anything else we need to know?:
Environment:
- Dask version:
- Python version:
- Operating System:
- Install method (conda, pip, source):
pip show fsspec Name: fsspec
Version: 2021.4.0
Summary: File-system specification
Home-page: http://github.com/intake/filesystem_spec
Author: None
Author-email: None
License: BSD
Location: /opt/anaconda3/envs/lineage_viz37/lib/python3.7/site-packages
Requires:
Required-by: s3fs, kedro, adlfs
pip show adlfs
Name: adlfs
Version: 0.7.3
Summary: Access Azure Datalake Gen1 with fsspec and dask
Home-page: https://github.com/dask/adlfs/
Author: None
Author-email: None
License: BSD
Location: /opt/anaconda3/envs/lineage_viz37/lib/python3.7/site-packages
Requires: azure-identity, azure-datalake-store, azure-core, aiohttp, msrestazure, requests, azure-storage-blob, fsspec
Required-by:
What do you get with print(fs.ls('abfs://demo/dir', detail=True))
?
Please also see this discussion on fsspec.