Universal-pathlib 0.2.3 seems to break compatibility with 0.2.2 (at least breaks mypy checks).

[potiuk]

Body

Manual MyPy Airflow:

 airflow/io/path.py:201: error: Unexpected keyword argument "overwrite" for
"rename" of "UPath"  [call-arg]
            return self.rename(target, overwrite=True)
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Manual MyPy Providers:

airflow/providers/common/io/xcom/backend.py:145: error: Argument 1 to
"joinpath" of "UPath" has incompatible type "Optional[str]"; expected
"Union[str, PathLike[str]]"  [arg-type]
                p = base_path.joinpath(dag_id, run_id, task_id, f"{uuid.uu...
                                       ^~~~~~
airflow/providers/common/io/xcom/backend.py:145: error: Argument 2 to
"joinpath" of "UPath" has incompatible type "Optional[str]"; expected
"Union[str, PathLike[str]]"  [arg-type]
                p = base_path.joinpath(dag_id, run_id, task_id, f"{uuid.uu...
                                               ^~~~~~
airflow/providers/common/io/xcom/backend.py:145: error: Argument 3 to
"joinpath" of "UPath" has incompatible type "Optional[str]"; expected
"Union[str, PathLike[str]]"  [arg-type]
    ...          p = base_path.joinpath(dag_id, run_id, task_id, f"{uuid.uuid...
                                                        ^~~~~~~

Should be reviewed and fixed.

cc: @jscheffl @bolkedebruin

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

Seems that this one might be fixed by https://github.com/fsspec/universal_pathlib/issues/257

[jscheffl]

Seems that this one might be fixed by fsspec/universal_pathlib#257

That fixes Problem #1, the second problem that we call the join with Optional[str] because we allow None values passed for dag_id, run_id and task_id is not fixed. I checked this but the interface is inherited from XCom and I assume that is something we need to fix on our side.

[ap--]

Ping me once you fix the Optional[str] issue on your side. I can then release a new intermediate version of universal-pathlib with the #257 fix.

[potiuk]

Ping me once you fix the Optional[str] issue on your side. I can then release a new intermediate version of universal-pathlib with the #257 fix.

I don't think it was "serious" but I protected against it in https://github.com/apache/airflow/pull/41921 @ap-- - I guess it will pass this time, feel free to relase a new version/