prefect icon indicating copy to clipboard operation
prefect copied to clipboard

Value Error Thrown When Deploying Flows Outside of Current Directory

Open ColeMurray opened this issue 1 year ago • 0 comments

First check

  • [X] I added a descriptive title to this issue.
  • [X] I used the GitHub search to find a similar issue and didn't find it.
  • [X] I searched the Prefect documentation for this issue.
  • [X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

For our use case, we dynamically load modules based on the modules name. These modules can sometimes be installed as system packages and are not relative to the current directory for creating a deployment.

Introduced in 2.3.0, when attempting to use a module that is not relative to the current directory, Prefect fails with a value error.

Reproduction

import importlib

module, method = module_name.rsplit('.', 1)
mod = importlib.import_module(module)
workflow_flow = getattr(mod, method)

deployment = Deployment.build_from_flow(
    flow=workflow_flow,
    name=schedule.id,
    version=1,
    work_queue_name="default",
    schedule=deployment_schedule,
)
deployment.apply()

A basic repro:

>>> Path("/home/user").absolute().relative_to(Path(".").absolute())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/user/.pyenv/versions/3.8.8/lib/python3.8/pathlib.py", line 907, in relative_to
    raise ValueError("{!r} does not start with {!r}"
ValueError: '/home/user' does not start with '/Users/user/test_dir'

Error

deployment = Deployment.build_from_flow(
--
File "/opt/venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 212, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 141, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/opt/venv/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/opt/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/opt/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/opt/venv/lib/python3.8/site-packages/prefect/deployments.py", line 570, in build_from_flow
entry_path = Path(flow_file).absolute().relative_to(Path(".").absolute())
File "/usr/local/lib/python3.8/pathlib.py", line 908, in relative_to
raise ValueError("{!r} does not start with {!r}"
ValueError: '/opt/venv/lib/python3.8/site-packages/workflow_etl/flows/flow.py' does not start with '/opt/app'

https://github.com/PrefectHQ/prefect/blob/main/src/prefect/deployments.py#L570

Versions

2.3.0
2.3.1
2.3.2

Additional context

This change appears introduced here: https://github.com/PrefectHQ/prefect/pull/6554 https://github.com/PrefectHQ/prefect/pull/6695

ColeMurray avatar Sep 10 '22 04:09 ColeMurray