astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Paths property from Location does not return files correctly

Open pankajastro opened this issue 1 year ago • 1 comments

Describe the bug When I'm passing a bucket folder uri it is returning all files + the folder

Version

  • Astro: [e.g. 0.6.0]
  • OS: [eg. Debian]

To Reproduce Steps to reproduce the behavior:

  1. Write the DAG '...'
  2. Create connection '....'
  3. Run using '....'
  4. See error

Expected behavior it should only return files

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

pankajastro avatar Aug 12 '22 12:08 pankajastro

discussion: https://github.com/astronomer/astro-sdk/pull/596#discussion_r944035899

pankajastro avatar Aug 12 '22 19:08 pankajastro

Look like in google python SDK in list_blobs if I'm passing pattern like * it considering it as string not pattern.

>>> from google.cloud import storage
>>> storage_client = storage.Client()
>>> bucket = storage_client.bucket("pankaj_test_dtt")
>>> blobs = bucket.list_blobs(max_results=3)
>>> [blob.name for blob in blobs]
['homes.csv', 'homes2.csv']
>>> 
>>> blobs = bucket.list_blobs(prefix="*")
>>> [blob.name for blob in blobs]
[]
>>> blobs = bucket.list_blobs(prefix="*.csv")
>>> [blob.name for blob in blobs]
[]
>>> blobs = bucket.list_blobs(prefix="home")
>>> [blob.name for blob in blobs]
['homes.csv', 'homes2.csv']

pankajastro avatar Aug 16 '22 11:08 pankajastro

@utkarsharma2 I remember we were discussing about supporting patterns in better way in past, WDYT about this ticket?

pankajastro avatar Jan 16 '23 17:01 pankajastro

We can close this ticket. I think only the second point in this ticket is not done, but we can do it if it comes up from some user.

sunank200 avatar Mar 21 '23 13:03 sunank200