astro-sdk
astro-sdk copied to clipboard
Paths property from Location does not return files correctly
Describe the bug When I'm passing a bucket folder uri it is returning all files + the folder
Version
- Astro: [e.g. 0.6.0]
- OS: [eg. Debian]
To Reproduce Steps to reproduce the behavior:
- Write the DAG '...'
- Create connection '....'
- Run using '....'
- See error
Expected behavior it should only return files
Screenshots If applicable, add screenshots to help explain your problem.
Additional context Add any other context about the problem here.
discussion: https://github.com/astronomer/astro-sdk/pull/596#discussion_r944035899
Look like in google python SDK in list_blobs
if I'm passing pattern like *
it considering it as string not pattern.
>>> from google.cloud import storage
>>> storage_client = storage.Client()
>>> bucket = storage_client.bucket("pankaj_test_dtt")
>>> blobs = bucket.list_blobs(max_results=3)
>>> [blob.name for blob in blobs]
['homes.csv', 'homes2.csv']
>>>
>>> blobs = bucket.list_blobs(prefix="*")
>>> [blob.name for blob in blobs]
[]
>>> blobs = bucket.list_blobs(prefix="*.csv")
>>> [blob.name for blob in blobs]
[]
>>> blobs = bucket.list_blobs(prefix="home")
>>> [blob.name for blob in blobs]
['homes.csv', 'homes2.csv']
@utkarsharma2 I remember we were discussing about supporting patterns in better way in past, WDYT about this ticket?
We can close this ticket. I think only the second point in this ticket is not done, but we can do it if it comes up from some user.