airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Fix lazy loading of ObjectStoragePath fs backends

Open bolkedebruin opened this issue 1 year ago • 3 comments

The new structure introduces eager loading, which we don't want. This re-instates lazy loading at the expense of that storage_options might not be fully parsed.

@potiuk @ap-- @Taragolis


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

bolkedebruin avatar Feb 20 '24 12:02 bolkedebruin

@ap-- could you take a look if this makes sense? particularly the __str__ function as its a bit of a hack to workaround https://github.com/fsspec/universal_pathlib/issues/198

bolkedebruin avatar Feb 20 '24 13:02 bolkedebruin

This fails du to an import when the path needs to be parsed. I will open a new issue in universal_pathlib to see how long this could be delayed.

ap-- avatar Feb 20 '24 20:02 ap--

I see a few tests are failing. Closing and reopening to return them.

dirrao avatar Feb 21 '24 03:02 dirrao

closing as this will be solved differently

bolkedebruin avatar Mar 10 '24 11:03 bolkedebruin