prefect
prefect copied to clipboard
Value Error Thrown When Deploying Flows Outside of Current Directory
First check
- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.
Bug summary
For our use case, we dynamically load modules based on the modules name. These modules can sometimes be installed as system packages and are not relative to the current directory for creating a deployment.
Introduced in 2.3.0, when attempting to use a module that is not relative to the current directory, Prefect fails with a value error.
Reproduction
import importlib
module, method = module_name.rsplit('.', 1)
mod = importlib.import_module(module)
workflow_flow = getattr(mod, method)
deployment = Deployment.build_from_flow(
flow=workflow_flow,
name=schedule.id,
version=1,
work_queue_name="default",
schedule=deployment_schedule,
)
deployment.apply()
A basic repro:
>>> Path("/home/user").absolute().relative_to(Path(".").absolute())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/user/.pyenv/versions/3.8.8/lib/python3.8/pathlib.py", line 907, in relative_to
raise ValueError("{!r} does not start with {!r}"
ValueError: '/home/user' does not start with '/Users/user/test_dir'
Error
deployment = Deployment.build_from_flow(
--
File "/opt/venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 212, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 141, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/opt/venv/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/opt/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/opt/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/opt/venv/lib/python3.8/site-packages/prefect/deployments.py", line 570, in build_from_flow
entry_path = Path(flow_file).absolute().relative_to(Path(".").absolute())
File "/usr/local/lib/python3.8/pathlib.py", line 908, in relative_to
raise ValueError("{!r} does not start with {!r}"
ValueError: '/opt/venv/lib/python3.8/site-packages/workflow_etl/flows/flow.py' does not start with '/opt/app'
https://github.com/PrefectHQ/prefect/blob/main/src/prefect/deployments.py#L570
Versions
2.3.0
2.3.1
2.3.2
Additional context
This change appears introduced here: https://github.com/PrefectHQ/prefect/pull/6554 https://github.com/PrefectHQ/prefect/pull/6695